Server Admin Log

2017-09-28

22:18 MaxSem: Deployed fix for T176247
21:30 ejegg: updated CiviCRM from 3f48cb0 to 2138869
21:23 XenoRyet: Updated SmashPig from dc62203 to d056a13
19:36 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.1
19:34 legoktm: mysql:wikiadmin@db1052 [enwiki]> delete from user_groups where ug_user =17629530 limit 1; (T176798)
19:31 bblack: cp4026 - depool->reboot for bios updates, isolcpus fix
19:28 legoktm: mysql:wikiadmin@db1062 [centralauth]> UPDATE localuser SET lu_attached_method="new" WHERE lu_attached_method is NULL AND lu_name="Rua";
19:14 ejegg: updated fundraising tools 5708cf1 Add 'first_donation_date' column to Silverpop export
19:14 robh: cp4024 initial tests done (takes about 2 hours or so) then prompted for more in depth testing (hit yes, no failures so far) T174891
19:00 bblack: cp4025 - depool->reboot for bios updates, isolcpus fix
18:54 thcipriani@tin: Synchronized php-1.31.0-wmf.1/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: Revision::newFromTitle may return null T176882 (duration: 00m 50s)
18:42 bblack: cp4023 - depool->reboot for bios updates, isolcpus fix
18:37 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create rollbacker group at viwiktionary T176979 (duration: 00m 50s)
18:26 thcipriani@tin: Synchronized docroot/noc: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging PART II T104148 (duration: 00m 49s)
18:25 thcipriani@tin: Synchronized wmf-config: SWAT: copy squid.php->reverse-proxy.php, squid-labs->reverse-proxy-staging T104148 (duration: 00m 51s)
17:58 arlolra: Updated Parsoid to 2f4b9a8c (T176363, T176151, T170832)
17:52 mutante: librenms - renewed LE TLS cert ..via puppet, after Icinga alerted and accidentally still had "do_acme: false" in Hiera
17:46 arlolra@tin: Finished deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c (duration: 15m 25s)
17:43 bblack: cp4028 - depool->reboot for bios updates, isolcpus fix
17:40 moritzm: installing apache updates on cobalt
17:32 bblack: cp4022 - depool->reboot for bios updates, isolcpus fix
17:30 arlolra@tin: Started deploy [parsoid/deploy@84342fe]: Updating Parsoid to 2f4b9a8c
17:25 marostegui: Run fixStuckGlobalRename.php --wiki=enwiktionary --logwiki=metawiki "CodeCat" "Rua" on terbium - T176985
17:20 bblack: cp4027 - depool->reboot for bios updates, isolcpus fix
17:08 bblack: cp4021 - depool->reboot for bios update
17:05 robh: bios firmware uploaded and awaiting reboot for application on cp402[1235678]
16:48 bd808: Created centralauth.{analytics,web}.db.svc.eqiad.wmflabs Designate CNAME (T176978)
16:26 robh: cp4024 has had ilom and bios firmware updaed per epsa error checking, now re-running epsa utility T174891
16:09 robh: cp4024 bios update in progress (has been for last 5 minutes), letting it alone for a solid 30 to let it finish
16:01 marostegui: Global rename of CodeCat → Rua - T176985
15:22 hoo: Ran scap pull on mwdebug1001 after experiments with T176844
14:54 dcausse: restarting elastic on relforge servers to reinstall prod ltr plugin
14:51 marostegui: Stop MySQL on db1035 as server is going to be decommissioned - T176931
14:45 moritzm: installing apache updates on puppet mastes (will trigger a few failing puppet runs, but ircecho disabled temporarily)
14:42 marostegui: Remove db1035 from tendril - T176931
14:31 moritzm: upgrading mw1276 (API canary) to HHVM 3.18.5
14:28 moritzm: upgrading mwdebug* to HHVM 3.18.5
13:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 49s)
13:55 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1035 from config as it will be decommissioned - T176931 (duration: 00m 50s)
13:40 zeljkof: EU SWAT finished
13:38 zfilipin@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: SWAT: ve.init.mw.DesktopArticleTarget: Remove hack for reversed tabs in RTL in Vector (T50017) (duration: 00m 50s)
13:37 moritzm: upgrading mw1262 to mw1265 (canary app servers) to HHVM 3.18.5
13:35 moritzm: uploaded HHVM 3.18.5 for jessie-wikimedia to apt.wikimedia.org
13:26 marostegui: Optimize table on s4 codfw master (db2051), with replication (lag might be generated) - T174509
13:13 ema: bounce pybal on ulsfo primaries to pick up bgp-local-ips config change T103882
13:12 marostegui: Drop redundant indexes from pagelinks and templatelinks on s4 - T174509
13:10 moritzm: upgrading mw1261 to HHVM 3.18.5
13:08 ema: bounce pybal on ulsfo secondaries to pick up bgp-local-ips config change T103882
13:02 marostegui: Optimize template links and pagelinks on db2053 and db2046 - T174509
13:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2053 db2046 - T174509 (duration: 00m 49s)
12:48 dcausse: restarting elastic on relforge servers to test a new version of the ltr plugin
12:27 ema: reboot cp402[2-6] to disable numa isolation
10:30 gehel: restart elasticsearch on relforge to validate new logging config - T175242
10:09 elukey: incrementally add mw1312-17 api-appservers to serve live traffic (weights will be raised incrementally)
10:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1308.eqiad.wmnet
09:59 elukey: added new mediawiki jobrunner - mw1308
09:52 ema: upgrade pybal to 1.14.0 on lvs1009-10
09:49 aaron@tin: Synchronized wmf-config/CommonSettings.php: Enable logging of post-send DB updates (duration: 00m 55s)
09:42 ema: upgrade pybal to 1.14.0 on lvs400[13]
08:00 elukey@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1288.eqiad.wmnet
07:27 moritzm: upgrading app servers in deployment-prep to HHVM 3.18.5
07:14 moritzm: installing apache updates on silver/wikitech.wikimedia.org
06:59 moritzm: installing apache updates on graphite* hosts
06:28 elukey: restart varnish backend on cp3032
05:54 marostegui: Optimize tables pagelinks and templatelinks on dbstore2001 (s2) - T174509
05:53 marostegui: Optimize tables pagelinks and templatelinks on dbstore2002 (s2) - T174509
05:43 marostegui: Run maintain-views on labsdb1001, labsdb1003, labsdb1009, labsdb1010, labsdb1011 for hiwikivoyage - T173027
05:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2049 and db2041 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 47s)
05:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2064 db2063 - T174509 (duration: 00m 48s)
05:21 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2076 db2067 db2060 - T174509 (duration: 00m 48s)
05:19 tstarling@tin: Finished scap: updating Vector skin for gerrit 381153 (duration: 20m 39s)
04:59 tstarling@tin: Started scap: updating Vector skin for gerrit 381153
03:57 bblack: ulsfo dns-depooled at ~03:51
03:26 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 28 03:26:18 UTC 2017 (duration 6m 10s)
03:20 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 44s)
02:42 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 24s)
00:36 twentyafterfour: upgrading phabricator, expect minimal downtime

2017-09-27

23:39 robh: cp4024 is running hardware testing, leave it alone T174891
23:17 catrope@tin: Synchronized wmf-config/Wikibase.php: Make CirrusSearch default for wbsearchentities on testwikidatawiki T175741 (duration: 00m 48s)
22:34 maxsem@tin: Synchronized php-1.31.0-wmf.1/includes/EditPage.php: https://gerrit.wikimedia.org/r/#/c/381139/1 (duration: 00m 49s)
19:14 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 -> wmf.1
19:13 demon@tin: Synchronized php: symlink bump (duration: 00m 47s)
19:10 demon@tin: Synchronized php-1.31.0-wmf.1/maintenance/backup.inc: unbreak dumps (duration: 00m 49s)
18:46 catrope@tin: Finished scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter (duration: 22m 55s)
18:39 bblack: re-pooling upload@ulsfo (text@ulsfo remains pooled)
18:23 catrope@tin: Started scap: SWAT: T176762, T175358, T172023, T174719, and add html5-misnesting to Linter
18:21 ejegg: updated payments-wiki from 6104e3d to c2a3d43
17:40 ejegg: updated SmashPig Adyen settings
17:29 bblack: uploaded nginx-1.13.5-1+wmf1 to stretch-wikimedia, and matching nginx-1.13.5-1+wmf1~jessie1 to jessie-wikimedia
16:27 XioNoX: setting local-as to selected transit BGP sessions - T167840
16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.ulsfo.wmnet
16:00 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.ulsfo.wmnet
15:59 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.ulsfo.wmnet
15:58 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.ulsfo.wmnet
15:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1103 with low weight - T172679 (duration: 00m 47s)
15:51 ema: lvs4002: upgrade pybal to 1.14.0
15:49 ema: lvs4004: upgrade pybal to 1.14.0
15:28 _joe_: uploaded prometheus-statsd-exporter to stretch-wikimedia T175539
15:24 bblack: rebooting cp402[2356] (bnx2x oops)
15:15 moritzm: revoking salt certs for WMCS
15:12 mlitn@tin: Synchronized wmf-config/CommonSettings-labs.php: Remove 3D beta config (duration: 00m 47s)
15:11 mlitn@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Remove 3D beta config (duration: 00m 48s)
15:10 mlitn@tin: Synchronized wmf-config/CommonSettings.php: Load 3D extension (duration: 00m 49s)
15:09 mlitn@tin: Synchronized wmf-config/InitialiseSettings.php: Config 3D to be loaded on test, test2 (duration: 00m 48s)
15:00 mlitn@tin: Finished scap: Enable 3D extension (duration: 37m 09s)
14:58 ema: lvs1007: upgrade pybal to 1.14.0
14:45 elukey: rolling restart of all the Yarn nodemanager daemons on analytics1028-1068
14:43 bblack: cp1008/pinkunicorn upgraded to test build of nginx-1.13.5-1+wmf1
14:28 volans: uploaded cumin_1.2.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
14:24 akosiaris: change wtp1001 to wtp1024 weights to 5 from 15 in preparation for deprecation
14:24 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1024.eqiad.wmnet
14:24 mlitn@tin: Started scap: Enable 3D extension
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1023.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1022.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1021.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1020.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1019.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1018.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1017.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1016.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1015.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1014.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1013.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1012.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1011.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1010.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1009.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1008.eqiad.wmnet
14:23 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1007.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1006.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1005.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1004.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1003.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1002.eqiad.wmnet
14:22 akosiaris@puppetmaster1001: conftool action : set/weight=5; selector: name=wtp1001.eqiad.wmnet
14:17 akosiaris: T175242 restart eventstreams across the fleet to pick up the change in a rolling restart manner with a batch size of 2
14:16 akosiaris: T175242 restart parsoid across the fleet to pick up the change in a rolling restart manner with a batch size of 5
14:03 akosiaris: T175242 restart tilerator, tileratorui, restbase across the fleet to pick up the change in a rolling restart manner with a batch size of 2
13:43 akosiaris: T175242 re-enable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/. Run puppet as well in a batched execution
13:42 anomie@tin: Synchronized php-1.31.0-wmf.1/includes/specials/SpecialWatchlist.php: SWAT: {{gerrit|380968}} Fix watchlist "in the last X hours" display (duration: 00m 48s)
13:42 elukey: raised Hadoop HDFS namenode master daemon max heap size to 6G (prev 4G) on analytics100[12]
13:31 akosiaris: T175242 eventstreams requires manual restart
13:25 akosiaris: T175242 parsoid requires manual restart
13:15 akosiaris: T175242 restbase requires manual restart
13:11 addshore: SWAT done
13:11 addshore@tin: Synchronized php-1.31.0-wmf.1/extensions/TwoColConflict/modules/ext.TwoColConflict.BaseVersionSelector.css: SWAT: Address changes in the label style in OOUI (duration: 00m 46s)
13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add autopatrolled user group to dty.wikipedia (T176709) (duration: 00m 47s)
13:01 akosiaris: T175242 tilerator and tileratorui need manually restart
12:54 akosiaris: T175242 enabled puppet in aqs kafka maps maps-test selected hosts and ran puppet manually.
12:47 akosiaris: T175242 disable puppet across aqs kafka maps maps-test ores restbase restbase-dev sca scb wtp clusters for merging https://gerrit.wikimedia.org/r/#/c/376500/
12:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1103 to the array of hosts (duration: 00m 48s)
12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1103 to the array of hosts (duration: 00m 49s)
12:14 moritzm: installing apache updates on einsteinium/tegmen
12:06 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 00m 49s)
12:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on bnwiki (T176771) (duration: 01m 00s)
09:45 marostegui: Stop mysql on db1035 to copy its data to db1103 - T172679
09:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1035 to transfer its data to db1103 - T172679 (duration: 00m 48s)
08:55 mobrovac@tin: Finished deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100 (duration: 01m 26s)
08:53 mobrovac@tin: Started deploy [changeprop/deploy@7a3dc66]: Use the /page/title/{title}/{revision} end point for revision-visibility-change - T158100
08:43 mobrovac@tin: Finished deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100 (duration: 10m 01s)
08:34 elukey: raise traffic weights to 30 for mw13[19-28] incrementally - T165519
08:33 mobrovac@tin: Started deploy [restbase/deploy@1cc530b]: Introduce the /page/title/{title}/{revision} end point - T158100
08:16 marostegui: Optimize tables pagelinks templatelinks on db2064 and db2063 - T174509
08:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 db2063 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:55 marostegui: Optimize pagelinks and template links tables on db2076 db2067 db2060 - T174509
07:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2076 db2067 db2060 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:37 marostegui: Optimize pagelinks and template links tables on db2070 db2069 db2042 db2016 - T174509
07:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2070, db2069 and db2042 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 48s)
07:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2071 and db2072 - T174509 (duration: 00m 47s)
06:57 marostegui: Deploy alter table on s6 codfw - T174509
06:55 marostegui: Deploy alter table on db1052 (enwiki master) - T174509
06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add task number to db2044 line (duration: 00m 48s)
06:37 ema: restart varnish-be on cp3040
06:36 moritzm: installing git security updates (our config is not affected, but still fixing the underlying issue)
06:33 ema: restart varnish-be on cp3041
06:29 marosteg1i: Reboot db2044 after storage failure - T174764
06:15 elukey: restart varnish backend on cp3043
05:41 madhuvishy: service nova-fullstack restart on labnet1001
03:17 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 27 03:17:22 UTC 2017 (duration 6m 22s)
03:11 l10nupdate@tin: scap sync-l10n completed (1.31.0-wmf.1) (duration: 15m 55s)
02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 08m 42s)

2017-09-26

23:25 thcipriani@tin: Synchronized php-1.31.0-wmf.1/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWSignatureTool.js: SWAT: Revert MWSignatureTool case (duration: 00m 48s)
23:20 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 48s)
23:19 thcipriani@tin: Synchronized php-1.31.0-wmf.1/resources/src/mediawiki.rcfilters/mw.rcfilters.init.js: SWAT: RCFilters: Log performance data T176652 (duration: 00m 51s)
22:03 no_justification: gerrit master restart, back momentarily
20:32 mutante: phab1001 - restarted apache after upgrade
20:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T174764 (duration: 00m 50s)
20:20 mutante: phab1001 (prod phab) - upgrade Apache package
20:15 mutante: phab2001 - install Apache and MariaDB package upgrades
19:16 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.31.0-wmf.1
18:25 mepps: Updated crm from 89b8007 to 3f48cb0
18:00 demon@tin: Finished scap: 1.31.0-wmf.1 bootstrap + testwiki (duration: 47m 56s)
17:12 demon@tin: Started scap: 1.31.0-wmf.1 bootstrap + testwiki
15:50 volans: powecycled cp2012, no ping or ssh console stuck
15:18 XioNoX: starting eqiad frack switch to new infra - T174218
15:06 gehel: restarting blazegraph on all wdqs nodes for heap resize - T175919
14:47 mobrovac@tin: Finished deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940 (duration: 20m 35s)
14:36 marostegui: Optimize tables pagelinks and templatelinks on db2071 and db2072 from s1 - T174509
14:27 mobrovac@tin: Started deploy [restbase/deploy@f989dd9]: Remove the old mobileapps module completely - T169940
14:26 hoo: mwscript refreshLinks.php --wiki elwiki --namespace 0 on terbium has finished (T151717)
14:12 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2071 and db2072 to optimize templatelinks and pagelinks tables - T174509 (duration: 00m 44s)
13:39 marostegui: Deploy alter table on s6 master (with replication enable, so lag will be generated) db2028 - T163979
13:25 hashar@tin: Synchronized wmf-config/filebackend.php: Expose Thumbor swift username - T144479 (duration: 00m 44s)
13:24 hashar@tin: Synchronized wmf-config/CommonSettings.php: Enable asia-specific Navigation Timing metric - T169522 (duration: 00m 47s)
13:22 godog: upgrade grafana to 4.5.2 on labmon1001 - T175980
13:15 hashar@tin: Synchronized wmf-config/CommonSettings.php: Disable RelatedArticles instrumentation on all wikis - T174944 (duration: 00m 45s)
13:02 marostegui: Drop redundant indexes from s6 - T174509
13:00 elukey: add mw132[7,8] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
12:44 gehel: restarting blazegraph on wdqs1004 / wdqs2001 for heap resive - T175919
12:27 marostegui: Drop redundant indexes from s2 - T174509
12:12 marostegui: Drop redundant indexes from enwiki on eqiad - T174509
12:03 kartik@tin: Finished deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction (duration: 03m 03s)
12:00 kartik@tin: Started deploy [cxserver/deploy@aa232ee]: Update cxserver, registry reconstruction
11:58 moritzm: uploaded prometheus-apache-exporter 0.3-1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
11:57 marostegui: Drop redundant indexes from enwiki on codfw - T174509
11:49 moritzm: uploaded prometheus-hhvm-exporter 0.3.1+deb9u1 to apt.wikimedia.org/stretch-wikimedia
11:30 kartik@tin: (no justification provided)
11:29 kartik@tin: Finished deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction (duration: 00m 52s)
11:28 kartik@tin: Started deploy [cxserver/deploy@f1d4851]: Update cxserver, registry reconstruction
10:52 volans: uploaded cumin_1.1.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
10:34 joal@tin: Finished deploy [analytics/refinery@5d806c6]: Regular analytics deploy (duration: 08m 00s)
10:26 joal@tin: Started deploy [analytics/refinery@5d806c6]: Regular analytics deploy
10:20 _joe_: rebooting copper (again)
09:12 elukey: add mw132[4,5,6] to live traffic (new appservers) - weights will be increased incrementally from 5 to 20
09:10 _joe_: rebooting copper to verify docker setup is correct
08:46 moritzm: installing apache updates on mw* app servers
08:44 godog: run puppet on ms-fe* to reduce conntrack generated by swift internal clients towards backends - T173731
08:24 godog: roll-restart thumbor to upgrade to 1.5 - T173804
08:09 godog: roll-restart swift-proxy to apply https://gerrit.wikimedia.org/r/#/c/380483/ - T161535
08:08 marostegui: Drop old temporary tar.gz files from dbstore1001 to get back some disk space
08:07 marostegui: Drop table trackbacks from s3 - T175051
07:48 mattflaschen@tin: Synchronized wmf-config/CommonSettings-labs.php: RCFilters: Beta Cluster only (duration: 00m 46s)
07:46 marostegui: Drop trackbacks table from s7 - T175051
07:28 marostegui: Drop trackbacks table from s6 - T175051
07:16 moritzm: installing perl security updates
06:45 _joe_: restarting varnish backend on cp3033, throwing 503s
06:29 mobrovac@tin: Finished deploy [cpjobqueue/deploy@17c3833]: (no justification provided) (duration: 00m 28s)
06:28 mobrovac@tin: Started deploy [cpjobqueue/deploy@17c3833]: (no justification provided)
06:18 marostegui: Drop table trackbacks from s4 - T175051
06:06 marostegui: Drop table trackbacks from s5 (only exists on dewiki) - T175051
05:48 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 - T176573 (duration: 00m 44s)
05:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove old db1055 comments (duration: 00m 46s)
02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 26 02:29:31 UTC 2017 (duration 5m 52s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 26s)

2017-09-25

23:58 ejegg: updated payments-wiki from d2ce697 to 6104e3d
23:15 mutante: gerrit2001 reinstalled with stretch, revoked old puppet cert, accepted new puppet cert, initial run that will do base and all the gerrit things at once.. (T168562)
22:53 mutante: gerrit2001 - PXE booting
22:34 mutante: gerrit2001 - schedule downtime for reinstall with strech, reboot imminent
20:57 hoo: Manually pruned the sites and site_identifiers table on hiwikivoyage, than ran populateSitesTable.php. (T173030)
20:37 arlolra: Updated Parsoid to 4230c27b (T176425, T173029)
20:30 arlolra@tin: Finished deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b (duration: 16m 52s)
20:13 arlolra@tin: Started deploy [parsoid/deploy@376faad]: Updating Parsoid to 4230c27b
19:58 XioNoX: renumbering ams BGP communities - T167840
19:25 robh@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4021.ulsfo.wmnet
19:02 robh: cp4021 coming down for memory swap
19:02 robh@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4021.ulsfo.wmnet
18:54 MaxSem: started populating ip_changes on group2 wikis
18:38 demon@tin: Synchronized scap/plugins/clean.py: no-op/consistency (duration: 00m 45s)
18:22 ejegg: updated CiviCRM from 45c56cf to 89b8007
17:58 demon@tin: Pruned MediaWiki: 1.30.0-wmf.15 (duration: 02m 46s)
17:07 gehel@tin: Finished deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update (duration: 01m 51s)
17:05 gehel@tin: Started deploy [wdqs/wdqs@3eec185]: Deploy new Blazegraph binary & GUI update
17:02 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Disable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 46s)
16:37 demon@tin: Synchronized php-1.30.0-wmf.19/maintenance/getConfiguration.php: getConfiguration: Don't bail when a valid variable is set null (duration: 00m 47s)
15:57 urandom: T169940: Converting restbase-ng data tables to size-tiered compaction
14:39 gehel: dropping badly named apifeature index from cirrus elasticsearch codfw/eqiad - T176430
14:31 gehel: rolling restart of logstash for config change - T176430
14:13 gehel: rolling restart of logstash for config change - T176430
13:46 zeljkof: EU SWAT finished
13:45 zfilipin@tin: Synchronized static/images/project-logos/hiwiki.png: SWAT: Make hiwiki logo a little bit bigger (T175721) (duration: 00m 46s)
13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Import sources on hr.wikibooks (T176320) (duration: 00m 46s)
13:30 addshore@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/WikimediaEventsHooks.php: SWAT: gerrit:379749 onBeforeInitializeWMDECampaign, fix early bail condition T174948 (duration: 00m 46s)
13:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several rights to eliminators in fawiki (T176553) (duration: 00m 46s)
12:52 moritzm: removed salt master from neodymium
12:38 moritzm: revoked all salt keys on neodymium
11:58 godog: powercycle ms-be2020 - blk_update_request: I/O error, dev sdb, sector in consol
11:54 moritzm: removing salt-minion/salt-common from production
11:47 Dereckson: Repopulate sites tables for Wikidata clients (T173013)
11:47 Dereckson: Create new database and tables for hi.wikivoyage (T173013)
11:45 dereckson@tin: Synchronized wmf-config/interwiki.php: Interwiki map update (+hiwikivoyage and other changes previous month, T176572) (duration: 00m 46s)
11:39 marostegui: Run redact_sanitarium for hiwikivoyage on db1095 - T173027
11:39 moritzm: upgrading tor to latest stable release (0.3.1.7)
11:37 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Initial configuration for hi.wikivoyage.org (T173013) (duration: 00m 46s)
11:37 dereckson@tin: Synchronized static/images/project-logos/: Logos for hi.wikivoyage.org (T173013) (duration: 00m 45s)
11:34 dereckson@tin: rebuilt wikiversions.php and synchronized wikiversions files: +hiwikivoyage (T173013)
11:32 dereckson@tin: Synchronized dblists: Add hiwikivoyage to the set (T173013) (duration: 00m 46s)
10:42 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 weight (duration: 00m 46s)
10:14 elukey: add mw1323 to live traffic (mw appserver) - traffic weights will go from 5 to 20 incrementally
09:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight and remove main traffic from special slaves (duration: 00m 46s)
09:07 marostegui: Add 50GB to db1036 /srv partition - T176311
09:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1036 and full repool db1101 - T176311 (duration: 00m 46s)
08:49 moritzm: uploaded php-luasandbox 2.0.14, hhvm-tidy 0.1.3 and php-wikidiff2 1.4.1 for stretch-wikimedia to apt.wikimedia.org
08:36 mobrovac@tin: Finished deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233 (duration: 10m 03s)
08:26 mobrovac@tin: Started deploy [restbase/deploy@ab24f70]: Switch mobile-sections to next-gen storage and stop storing the current-day feed in Cassandra - T169940 T176233
08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 45s)
07:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1101 - T176311 (duration: 00m 46s)
07:42 moritzm: uploaded hhvm 3.18.2+dfsg-1+wmf5+deb9u1 for stretch-wikimedia to apt.wikimedia.org
07:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1055 weight (duration: 00m 47s)
07:37 elukey: add mw132[0,2] to live traffic (mw appservers) - traffic weights will go from 5 to 20 incrementally
06:38 moritzm: installing apache security updates on logstash*
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 25 02:36:48 UTC 2017 (duration 6m 42s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 07m 17s)

2017-09-24

14:23 jynus: restarting db2047 for kernel upgrade
14:16 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 (duration: 00m 48s)
off: restarted ircecho on einsteinium

2017-09-23

21:12 hoo: Ran scap pull on mwdebug1001 to undo changes to profile https://github.com/wmde/WikibaseDataModel/pull/761/files

2017-09-22

23:36 mutante: gerrit2001 - rebooting .. wave a dead chicken http://www.catb.org/jargon/html/W/wave-a-dead-chicken.html
22:56 mutante: gerrit2001 - trying to manually start gerrit again, as opposed to puppet doing it.. debugging gerrit-ssh issue there, cobalt still untouched
22:34 mutante: gerrit2001 - stopping gerrit with gerrit.sh stop, letting puppet start it again
22:29 mutante: gerrit2001 - systemd says gerrit.service is failed. gerrit.sh start says "Already Running!!" :p - cobalt is fine
18:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.18 [keeping static files] (duration: 01m 29s)
15:21 hashar: Restarted Jenkins. Out of memory)
14:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1101 with low weight (duration: 00m 47s)
14:43 moritzm: uploaded php-luasandbox build for src:php5.5 (required for CI tests on jessie)
14:42 moritzm: updated tor packages to 0.3.1.7 (new stable series)
13:49 elukey: mw1321 (new appserver) serving traffic (going to increase its weight up to 20)
12:02 hashar: apt-get upgrade on contint1001 / contint2001
09:42 elukey: mw1319 (new appserver) serving traffic (going to increase its weight up to 20)
09:14 jynus: stop mariadb at db1055 for upgrade and maintenance
06:32 moritzm: installing emacs security updates on trusty (Debian already fixed)

2017-09-21

23:56 ejegg: updated payments-wiki from 7ed7243 to d2ce697
23:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/extension.json: T175047: Stop delivering search relevance survey javascript (duration: 00m 46s)
23:18 ebernhardson@tin: Synchronized wmf-config: T175047: Turn off human search relevance survey (duration: 00m 48s)
23:16 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T175047: Turn off human search relevance survey (duration: 00m 46s)
22:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1055 (duration: 00m 46s)
22:12 volans: uploaded cumin_1.1.0-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
22:10 ejegg: updated payments-wiki from 9da8482 to 7ed7243
21:07 demon@tin: Synchronized wmf-config/CommonSettings.php: extdist settings (duration: 00m 46s)
21:04 ejegg: updated CiviCRM from fc56901 to 45c56cf
19:32 demon@tin: Synchronized wmf-config/InitialiseSettings.php: adjust logging on csp reports (duration: 00m 46s)
19:20 demon@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: fix user/wgUser mixup (duration: 00m 46s)
19:16 gehel: upgrade to nodejs 6.11 on maps servers (including restart of tilerator / kartotherian) - T171707
19:16 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
19:04 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.19
18:56 urandom: T171772: Applying locally hacked Prometheus exporter config to RESTBase dev Cassandra instances
18:44 thcipriani@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters: SWAT: WLFilters: Do not hide .watchlistDetails while loading T176300 RCFilters: Make the interface not jump around while loading (duration: 00m 49s)
18:41 urandom: T171772: Restarting Cassandra restbase-dev1004-a to apply locally hacked Prometheus exporter config
18:34 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turning off Explore Similar AB test T175649 (duration: 00m 49s)
18:31 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Print instrumentation T176341 (duration: 00m 52s)
14:56 moritzm: uploaded php-defaults and php-redis built against src:php5.5 to component/ci for jessie-wikimedia
13:45 gehel: upgrade to nodejs 6.11 on the full maps-test cluster - T171707
13:12 zeljkof: EU SWAT finished
13:09 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
13:09 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 48s)
13:07 zfilipin@tin: Synchronized tests/cirrusTest.php: SWAT: Revert "Switch elasticsearch active cluster to codfw" (duration: 00m 49s)
12:55 _joe_: stopped long-running runjobs for refreshlinks on commons.
12:50 mobrovac@tin: Finished deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy (duration: 18m 34s)
12:32 mobrovac@tin: Started deploy [restbase/deploy@bc02191]: produce a diff for the mobile proxy
11:59 moritzm: upgrading apache on deployment servers and script runners
11:40 moritzm: upgrading apache on canary app servers
11:17 moritzm: installing apache security updates (our configurations are not affected, the upload of 2.4.10-10+deb8u11+wmf1 is mostly for the benefit of whatever people are doing in Cloud VPS, but we should still fix the underlying bug in production as well)
11:14 dcausse: reindexing all hebrew wikis to pickup the new hebmorph analyzer
11:09 dcausse: reindexing ilwikimedia in codfw to pickup and test new hebmorph analyzer
10:26 akosiaris: upload docker-ce_17.06.2~ce-0~debian_amd64.deb to apt.wikimedia.org jessie-wikimedia/thirdparty/ci T175293
09:59 dcausse: reindexing group1 wikis T176397
09:54 dcausse: reindexing group0 wikis T176397
08:35 moritzm: unbreak deployment-puppetmaster02 in deployment-prep (broken by unattended-upgrades update of apache T159254)
08:35 akosiaris: pool all of wtp1025 to wtp1048 T165520
08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1044.eqiad.wmnet
08:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1043.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1042.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1041.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1040.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1039.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1038.eqiad.wmnet
08:34 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1037.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1036.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1035.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1034.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1033.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1032.eqiad.wmnet
08:33 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1031.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1030.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1029.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1028.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1027.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1026.eqiad.wmnet
08:32 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:31 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:30 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
08:30 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: name=wtp1025.eqiad.wmnet
08:25 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 03m 18s)
08:22 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:21 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 00m 58s)
08:20 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:19 mobrovac@tin: Finished deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests (duration: 05m 22s)
08:14 mobrovac@tin: Started deploy [restbase/deploy@9fd380d]: Log only mismatches during updates, not live requests
08:08 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940 (duration: 10m 28s)
07:57 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections - T169940
07:31 _joe_: restarted aphlict on phab1001
07:27 mobrovac@tin: Finished deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940 (duration: 00m 30s)
07:27 mobrovac@tin: Started deploy [restbase/deploy@3e9bd9f]: New storage schema for mobile-sections, canary deploy for schema creation, take 2 - T169940
07:09 ema: bounce pybal on lvs1003 to clear stale alert T176388
06:59 ema: bounce pybal on lvs1006 to clear stale alert
06:55 ema: bounce pybal on lvs1009 to clear stale alert
03:24 bblack: depooled cp4026
03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 21 03:12:41 UTC 2017 (duration 6m 36s)
03:06 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 14m 44s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 50s)
00:41 eileen: update civicrm from da833ac to fc56901
00:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 53s)
00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Fix schema name for search satisfaction error logging (duration: 00m 49s)
00:09 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on commons (T124742) (duration: 00m 49s)
00:02 catrope@tin: Synchronized php-1.30.0-wmf.19/resources/src/mediawiki.rcfilters/: Lazy-load the RCFilters menu (T176250) (duration: 00m 48s)

2017-09-20

23:49 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 48s)
23:47 catrope@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/: Log JS errors during search satisfaction tests (duration: 00m 49s)
23:40 mutante: phabricator: aphlict (notification service) now running on prod server
23:36 mutante: phab1001 - re-enable puppet, run after follow-up fix, adding notification server config
23:31 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add throttle rule for John Michael Kohler Art Center (T176287) (duration: 00m 48s)
23:29 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable user email blacklist on meta (T174694) (duration: 00m 48s)
23:21 catrope@tin: Synchronized dblists/categories-rdf.dblist: Add mlwiki to categories-rdf.dblist (duration: 00m 48s)
23:14 mutante: phab2001 - deluser aphlict, delgroup aphlict, try to let puppet create it again, still fails, just won't create the group
23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Add patroller group on metawiki (T176079) (duration: 00m 48s)
23:11 mutante: phab2001 - groupadd aphlict,temp adduser aphlict to debug user creation issue
22:22 ebernhardson: elasticsearch eqiad completely recovery, all green. Returning recovery max_bytes_per_sec to 20mb
21:54 mutante: phab: disable puppet on phab1001 for a minute, apply notification server change on phab2001
21:45 eileen: update civicrm from 0fb4676 to da833ac
21:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4015.ulsfo.wmnet
20:58 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4007.ulsfo.wmnet
20:40 bsitzmann@tin: Finished deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263) (duration: 06m 05s)
20:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4014.ulsfo.wmnet
lunch: increase max recovery bytes/s to 80mb in eqiad elasitcsearch cluster
20:34 bsitzmann@tin: Started deploy [mobileapps/deploy@e23bf66]: Update mobileapps to d46860e (T175759 T176263)
20:33 demon@tin: Synchronized php-1.30.0-wmf.19/includes/diff/DifferenceEngine.php: fix some warnings (duration: 00m 51s)
19:55 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4016.ulsfo.wmnet
19:54 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster (duration: 00m 10s)
19:54 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1] (dev-cluster): Deploy 4.1.0 artifact to dev cluster
19:53 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4006.ulsfo.wmnet
19:48 eevans@tin: Finished deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release (duration: 00m 47s)
19:47 eevans@tin: Started deploy [cassandra/metrics-collector@df909a1]: Pushing out 4.1.0 release
19:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4008.ulsfo.wmnet
19:26 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.19
19:24 demon@tin: Synchronized php: symlink swap for wmf.19 (duration: 00m 49s)
19:02 catrope@tin: Finished scap: Patches for T176302, T175962, T173533 (duration: 21m 29s)
18:40 catrope@tin: Started scap: Patches for T176302, T175962, T173533
17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on terbium. Can be KILLed at any time, if needed.
17:29 hoo: Started mwscript refreshLinks.php --wiki elwiki --namespace 0 in a screen on elwiki. Can be KILLed at any time, if needed.
17:13 hoo: Ran mwscript refreshLinks.php --wiki elwiki --namespace 0 -e 30 for testing purposes
17:06 ejegg: removed limit from donation queue consumer
17:05 otto@tin: Finished deploy [eventstreams/deploy@e62ab64]: (no justification provided) (duration: 04m 44s)
17:01 otto@tin: Started deploy [eventstreams/deploy@e62ab64]: (no justification provided)
16:53 madhuvishy: Upgraded and restarted apache2 on labs-puppetmaster.wikimedia.org
16:45 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch
16:42 gehel: rolling restart of logstash ingesters after cirrus eqiad recovery
16:34 gehel: elasticsearch eqiad is finally recovering, after the restart of elastic1051
16:34 gehel: cleanup jar hell and restart elastic1051
16:19 gehel: starting elasticsearch on all nodes on eqiad
16:10 gehel: restarting elasticsearch masters on eqiad (elastic1030, 36 and 40)
16:04 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch
16:03 gehel: depooling all nodes in elasticsearch eqiad to investigate failed cluster restart (traffic is already directed to codfw)
15:46 ejegg: limited donations queue consumer to 100 per run
15:36 ejegg: updated payments-wiki from 896f16a to 9da8482
15:36 gehel: upgrading elasticsearch plugins on elasticsearch eqiad, including cold restart of the cluster - T173231
15:33 ejegg: limited donations queue consumer to 200 per run
14:03 akosiaris@tin: Synchronized wmf-config/InitialiseSettings.php: Whitelist wtp10[25-48] for Linter (duration: 00m 49s)
13:42 otto@tin: Synchronized /srv/mediawiki-staging/wmf-config/CommonSettings-labs.php: (no justification provided) (duration: 00m 49s)
13:24 ema: begin rolling codfw varnish-be restarts, 20m spacing
13:20 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 3/3) (duration: 00m 48s)
13:18 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 2/3) (duration: 00m 49s)
13:17 dereckson@tin: Synchronized tests/cirrusTest.php: Switch elasticsearch active cluster to codfw (Gerrit:378850, 1/3, no-op in prod) (duration: 00m 49s)
12:51 ema: restarted varnish-be on cp4028
12:28 gehel: upgrading nodejs to 6.11 on maps-test2003 for testing - T171707
12:19 ema: restarted varnish-be on cp4027
12:18 moritzm: uploaded apache 2.4.10-10+deb8u11+wmf1 to jessie-wikimedia (rebuild of latest security update with our local patches on top)
12:08 hoo@tin: Synchronized wmf-config/Wikibase-production.php: Enable statement usage tracking on elwiki (T151717) (duration: 00m 49s)
11:58 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940 (duration: 07m 20s)
11:56 ema: restarted varnish-be on cp4018
11:51 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: New storage schema for mobile-sections, canary deploy for schema creation - T169940
11:49 jynus: stopping replication on db1101 for faster repartition work T176311
11:45 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1101 (duration: 00m 58s)
11:34 ema: restarted varnish-be on cp4010
11:25 mobrovac@tin: Finished deploy [restbase/deploy@dea2b41]: (no justification provided) (duration: 00m 02s)
11:25 mobrovac@tin: Started deploy [restbase/deploy@dea2b41]: (no justification provided)
11:03 moritzm: rolling out new debdeploy release across the fleet
11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 19s)
11:02 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:02 ema: restarted varnish-be on cp4008
11:02 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:01 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 03s)
11:01 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
11:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee]: (no justification provided) (duration: 00m 04s)
11:00 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee]: (no justification provided)
09:55 bblack: restarted varnish-be on cp3042
09:29 bblack: restarted varnish-be on cp1052
09:23 ema: restarted varnish-be on cp3043
09:21 ema: restarted varnish-be on cp3041
09:19 ema: restarted varnish-be on cp3032
09:14 ema: restarted varnish-be on cp3040
08:42 bblack: restarted varnish-be on cp1053
08:41 bblack: restarted varnish-be on cp1068
08:40 bblack: restarted varnish-be on cp1054
08:40 bblack: restarted varnish-be on cp1065
07:11 ema: cp3030 varnish-be restart due to 503s
03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 20 03:16:25 UTC 2017 (duration 7m 31s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.19) (duration: 15m 57s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 06s)
00:55 bblack: cp1066 - backend restart, mailbox lag, text
00:54 bblack: cp4016 - backend restart, mailbox lag, text

2017-09-19

23:57 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 49s)
23:56 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/VisualEditor/ApiVisualEditor.php: SWAT: ApiVisualEditor: Fix checkbox label message handling with Message objects T176249 (duration: 00m 48s)
23:46 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 48s)
23:45 thcipriani@tin: Synchronized php-1.30.0-wmf.19/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: SWAT: Javascript timestamps are in ms, not s T174106 (duration: 00m 50s)
23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
23:36 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/specials/SpecialRecentchangeslinked.php: SWAT: SpecialRecentchangeslinked: Unconditionally join on the page table T176228 (duration: 00m 49s)
23:29 thcipriani@tin: Synchronized php-1.30.0-wmf.18/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 49s)
23:27 thcipriani@tin: Synchronized php-1.30.0-wmf.19/includes/page/PageArchive.php: SWAT: Do not attempt to copy to ip_changes in PageArchive class (duration: 00m 50s)
23:17 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable jQuery 3 on meta.wikimedia.org T124742 (duration: 00m 50s)
22:58 eileen: manually running Omnirecipient.load from 2017-09-18 with accidentally low throttle setting
22:22 mutante: gerrit2001 - stopped gerrit with init.d script. moved init.d script to /root, started with systemctl
22:21 eileen: update civicrm from 9677cf7 to 0fb4676
22:15 mutante: gerrit2001 /bin/sh /var/lib/gerrit2/review_site/bin/gerrit.sh start
22:02 mutante: gerrit2001 - systemctl start gerrit
21:51 volans: disabled puppet on labpuppetmaster1001, testing cumin T175712
21:27 MaxSem: on terbium: mwscript populateIpChanges.php --wiki=enwikivoyage --force
21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
21:23 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
21:22 bblack@neodymium: conftool action : set/pooled=yes; selector: cluster=cache_text,dc=eqiad,name=cp1066.*
21:18 MaxSem: re-ran populateIpChanges.php on group0 wikis
21:15 maxsem@tin: Synchronized php-1.30.0-wmf.19/maintenance/populateIpChanges.php: https://gerrit.wikimedia.org/r/#/c/379057/1 (duration: 00m 49s)
21:06 ema: cp4009: varnish-be restart, 503 spike
20:04 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4013.ulsfo.wmnet
20:03 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4016.ulsfo.wmnet
19:50 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_upload,name=cp4005.ulsfo.wmnet
19:49 bblack@neodymium: conftool action : set/pooled=no; selector: cluster=cache_text,name=cp4008.ulsfo.wmnet
19:17 ejegg: updated payments-wiki from 26b16ea to 896f16a
19:08 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.19
18:52 gehel: upgrading nodejs to 6.11 on maps-test2004 for testing - T171707
18:35 urandom: T160570, T169940: Upgrade restbase-ng environment to Cassandra 3.11.0-wmf5
18:26 arlolra: Updated Parsoid to 05a0965 (T122965, T175101, T173384, T173061, T172896, T174977, T159894)
18:17 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 48s)
18:15 ladsgroup@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 49s)
18:08 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/tests/phpunit/includes/Content/EntityContentTest.php: Fix undoing merge operations that turned Items into a redirects, part II (T175984) (duration: 00m 50s)
18:07 ladsgroup@tin: Synchronized php-1.30.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/includes/Content/EntityContent.php: Fix undoing merge operations that turned Items into a redirects, part I (T175984) (duration: 00m 50s)
18:00 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 12m 26s)
17:54 ebernhardson: T173464 start cirrussearch reindex of all wikis starting with zh
17:48 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
17:43 jynus: stopping db2010 and cloning to db2078
17:16 demon@tin: Finished scap: bootstrap wmf.19 (duration: 31m 04s)
16:45 demon@tin: Started scap: bootstrap wmf.19
16:44 demon@tin: scap aborted: bootstrap wmf.19 (duration: 01m 52s)
16:42 demon@tin: Started scap: bootstrap wmf.19
16:38 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1018 (duration: 00m 51s)
16:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1018 (duration: 01m 01s)
16:34 demon@tin: Pruned MediaWiki: 1.30.0-wmf.17 [keeping static files] (duration: 01m 18s)
16:31 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 (duration: 03m 15s)
16:29 jynus: disable puppet on db1009
16:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Trying that again, got an error the first time (duration: 00m 52s)
16:26 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RCFilters by default on cawiki, frwiki and hewiki (T157642) (duration: 00m 59s)
16:19 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable structured filters on watchlist on all wikis (duration: 00m 45s)
16:11 ema: pybal 1.14.0 uploaded to apt.w.o
15:54 catrope@tin: Finished scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765 (duration: 20m 32s)
15:33 catrope@tin: Started scap: core, GuidedTour and WikimediaMessages patches for T176191, T167262 and T175765
15:22 jynus: enabling firewall on db1020 (m2-master)
15:01 ema: restart varnish-be on cp1055, 503 spike
14:55 jynus: deploying firewall to m5 hosts
14:39 jynus: disabling puppet on all misc active mysql hosts
13:35 zeljkof: EU SWAT finished
13:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 00m 45s)
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 55s)
13:19 gehel: upgrading elasticsearch plugins on relforge, including cold restart of the cluster - T173231
13:17 gehel: upgrading elasticsearch plugins on elasticsearch codfw, including cold restart of the cluster - T173231
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Wikisaurus namespace on Wiktionary to "Thesaurus" (T174264) (duration: 02m 56s)
13:04 gehel: upgrading elasticsearch plugins on elastic2001 - T173231
12:55 moritzm: rebooting kubestage1001
11:25 moritzm: uploaded debdeploy 0.99.1 to apt.wikimedia.org (for trusty, jessie and stretch)
11:23 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 02m 42s)
11:20 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
10:53 volans: restarted nagios-nrpe-server on stat1005
10:14 jynus: shutting down mysql on db1018
09:42 moritzm: installing libxml2 security updates on trusty (Debian already fixed)
09:40 elukey: powercycle analytics1062 - no ssh, console com2 frozen
09:34 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the updateBetaFeaturesUserCounts logging channel - T175637 (duration: 00m 50s)
09:30 ema: cp1062 - varnish-backend-restart, mbox lag
08:40 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 52s)
08:39 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
08:09 mobrovac@tin: Finished deploy [cxserver/deploy@03c1eb1]: (no justification provided) (duration: 00m 13s)
08:09 mobrovac@tin: Started deploy [cxserver/deploy@03c1eb1]: (no justification provided)
07:58 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 00m 37s)
07:58 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
07:50 moritzm: installing bind9 regression update on trusty servers
07:35 kartik@tin: (no justification provided)
07:35 kartik@tin: Finished deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7 (duration: 01m 04s)
07:34 kartik@tin: Started deploy [cxserver/deploy@03c1eb1]: Update cxserver to 569b7b7
02:28 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 19 02:28:56 UTC 2017 (duration 7m 6s)
02:21 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 04s)
01:52 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor newline nitpick. I promise that's it (duration: 00m 46s)
01:41 demon@tin: Synchronized scap/plugins/updatewikiversions.py: Minor spacing nitpick, that's it for today (duration: 00m 45s)
01:17 demon@tin: Synchronized scap/plugins/updatewikiversions.py: add new hotness updatewikiversions.py (duration: 00m 45s)
01:15 demon@tin: Synchronized multiversion/: rm old and busted updateWikiversions (duration: 01m 07s)
00:12 foks: Remove 2FA from users Alan and Magicknight94 (retroactive log)
00:10 eileen: update civicrm from 187238f to b6f7dc3

2017-09-18

23:28 tstarling@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: (no justification provided) (duration: 00m 46s)
23:13 thcipriani@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow: SWAT: Only intercept users who can potentially create articles otherwise T176100 (duration: 00m 46s)
22:32 demon@tin: Synchronized wmf-config/CommonSettings.php: Use PrivateSettings.php directly (duration: 00m 46s)
21:57 bawolff: deployed patch T175900
21:18 krinkle@tin: Synchronized php-1.30.0-wmf.18/extensions/NavigationTiming/modules/ext.navigationTiming.js: Unbreak - T176105 (duration: 00m 48s)
21:16 mutante: repeating import of gerrit_2.13.8+git1-wmf.7 on install1002, followed by reprepro copy stretch-wikimedia jessie-wikimedia gerrit to make it available on stretch
21:06 mutante: uploaded gerrit 2.13.8+git1-wmf.7 for jessie-wikimedia on APT.wm.org
20:42 krinkle@tin: Finished deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148 (duration: 03m 23s)
20:38 krinkle@tin: Started deploy [jobrunner/jobrunner@57f5f47]: No-op sync - first time scap3 - T129148
20:17 arlolra@tin: (no justification provided)
20:15 arlolra@tin: Finished deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965 (duration: 01m 53s)
20:13 arlolra@tin: Started deploy [parsoid/deploy@a01064d]: Updating Parsoid to 05a0965
19:29 XioNoX: restarting pfw3-codfw for software upgrade
19:10 urandom: T160570: Upgrading restbase-dev100[5-6].eqiad.wmnet to Cassandra 3.11.0-wmf5
19:07 smalyshev@tin: Finished deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf (duration: 01m 20s)
19:05 smalyshev@tin: Started deploy [wdqs/wdqs@2523836]: Deploy correct prefixes conf
19:03 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4 (duration: 02m 36s)
19:02 no_justification: gerrit: restarting service, back in a moment
19:00 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 4
18:56 urandom: T160570: Upgrading restbase-dev1004.eqiad.wmnet to Cassandra 3.11.0-wmf5 (canary)
18:46 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3 (duration: 02m 25s)
18:44 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS, take 3
18:37 smalyshev@tin: (no justification provided)
18:36 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 02m 26s)
18:34 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
18:20 maxsem@tin: Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/378357/ https://gerrit.wikimedia.org/r/#/c/376791/ (duration: 00m 48s)
17:42 smalyshev@tin: Finished deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS (duration: 00m 21s)
17:42 smalyshev@tin: Started deploy [wdqs/wdqs@ecdbd0d]: Deploying Blazegraph fixes and category scripts for WDQS
17:26 demon@tin: Synchronized wmf-config/ProductionServices.php: no-op/comment fix (duration: 00m 45s)
16:46 jynus: shuting down db1100 T175973
16:40 reedy@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: perf improvements T175962 (duration: 00m 46s)
16:05 bblack: cp1063 - backend restart, mailbox lag
15:38 bblack: cp1072 - varnish backend restart, mailbox lag
15:28 bblack: cp1074 - backend restart for mailbox lag
15:25 bblack: cp1050 - backend restart for mailbox lahg
14:27 reedy@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: T175444 (duration: 00m 47s)
13:29 ppchelko@tin: Finished deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property (duration: 00m 29s)
13:28 ppchelko@tin: Started deploy [cpjobqueue/deploy@2c422f5]: Annotate the job event with pipeline property
13:25 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Meta(Talk)Namespace configuration for be.wiktionary - T175950 (duration: 00m 46s)
13:18 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Stop A/B test on enwiki and dewiki - T176068 (duration: 00m 45s)
13:10 hashar@tin: Synchronized php-1.30.0-wmf.18/extensions/BetaFeatures: Temporary log all the executions of the update job. - T175637 (duration: 00m 46s)
13:09 hashar@tin: Synchronized wmf-config: New 'abusefilter-helper' configuration for en.wikipedia - T175684 (duration: 00m 49s)
12:59 moritzm: installing libxen updates
12:36 moritzm: installing tomcat8 security updates
10:40 ema: cp1099 - backend restart, mailbox lag
06:42 moritzm: installing freexl security updates
06:18 TimStarling: testing EtcdConfig in beta cluster
05:40 tstarling@tin: Synchronized wmf-config: just for consistency, should have no effect (gerrit 375108) (duration: 00m 49s)
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 18 02:36:50 UTC 2017 (duration 7m 6s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 08m 04s)

2017-09-17

12:49 _joe_: taking a full debug dump of mw1288 after depooling it
12:40 _joe_: restarting hhvm on some api appservers, to ease the cpu overlock
09:46 elukey: restart varnish backend on cp1052 - recurrent mailbox lag
08:11 elukey: restart varnish backend on cp1053 - recurrent mailbox lag

2017-09-16

22:05 godog: compress older otrs directories to reclaim inodes - T171490
16:53 elukey: restart varnish-backend on cp1073 (cache upload) for mailbox lag
15:40 andrewbogott: rebooting labvirt1017
15:34 andrewbogott: rebooting labvirt1015 for T176044

2017-09-15

21:50 mutante: contint2001 - systemctl start zuul
21:47 mutante: contint1001 - tmp disable puppet - contint2001 - test zuul unit file (gerrit 359016)
21:43 demon@tin: Finished deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though (duration: 00m 08s)
21:43 demon@tin: Started deploy [gerrit/gerrit@48ad332]: 2.13.9 -- not being used yet though
15:46 bblack: cp1049 - backend restart, mailbox lag
15:40 bblack: cp1099 - backend restart, mailbox lag
15:38 bblack: cp1072 - backend restart, mailbox lag
15:21 _joe_: uploaded td-agent-bit (fluentd-bit) to wikimedia-stretch T175527
13:13 moritzm: pruned hhvm bytecode cache on mw2256, led to log spam for an undefined variable after being out of service for hardware maintenance
13:10 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
13:10 addshore@tin: Synchronized wmf-config/CommonSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 2/2 (duration: 00m 45s)
13:08 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: beta: $wgWikiDiff2MovedParagraphDetectionCutoff = 25 BETA ONLY 1/2 (duration: 00m 47s)
11:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: mw2256.codfw.wmnet
09:10 moritzm: installing tcpdump security updates on trusty hosts (Debian systems already fixed)
08:54 moritzm: installing libbluetooth updates
08:03 jynus: dropping mariadb analytics users on db1031, db1029
07:49 Amir1: running maintenance/updateCollation.php --force on Persian (fa) wikis (T173601)
07:39 gehel: shutting down and masking elasticsearch on elastic1020 - T175951
07:35 gehel: depooling elastic1020 - T175951
07:28 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw2256.codfw.wmnet
00:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 46s)

2017-09-14

23:51 mutante: phabricator - tested gerrit 354247 before merging using apache-fast-test from tin -restarted apache
23:37 catrope@tin: Synchronized wmf-config/: VRS config for Electron (T175868) (duration: 00m 47s)
23:34 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/lib/ve/src/ui/styles/ve.ui.TableLineContext.css: T169389 (duration: 00m 45s)
23:33 catrope@tin: Synchronized php-1.30.0-wmf.18/extensions/ContentTranslation/modules/widgets/translator/ext.cx.translator.js: Fix JS error in CX (duration: 00m 46s)
23:29 catrope@tin: Synchronized wmf-config/CommonSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
23:27 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Give sysops the flow-create-board right on Flow wikis (T175934) (duration: 00m 45s)
23:22 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Use RemexHtml on mediawikiwiki and testwiki (T175095) (duration: 00m 46s)
23:19 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=acamar.wikimedia.org
23:18 mutante: acamar is back up and running - no failed units
23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats to remove accountcreator on mediawikiwiki (T175903) (duration: 00m 46s)
23:12 mutante: acamar - staged BIOS upgrade in progress
23:09 mutante: acamar - done with upgrade - rebooting - it's depooled and removed from resolv.conf - T162850
22:56 mutante: depooled acamar (codfw dns recursor) for BIOS upgrade - installing firmware
22:56 dzahn@neodymium: conftool action : set/pooled=no; selector: name=acamar.*
22:55 maxsem@tin: Synchronized wmf-config/: Labs only: https://gerrit.wikimedia.org/r/378184 https://gerrit.wikimedia.org/r/378187 (duration: 00m 47s)
22:15 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: ACTRIAL going live: https://gerrit.wikimedia.org/r/#/c/378104/ (duration: 00m 46s)
21:22 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/378125/ (duration: 00m 47s)
20:34 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.userpage.icons/userpage.svg: I95d76339 (duration: 00m 45s)
20:18 demon@tin: Synchronized php-1.30.0-wmf.18/includes/api/ApiFeedWatchlist.php: Ibea5bd88 (duration: 00m 46s)
20:04 demon@tin: Synchronized php-1.30.0-wmf.18/extensions/OAuth/backend/MWOAuthDAO.php: I785230df (duration: 00m 46s)
20:02 ppchelko@tin: Finished deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl (duration: 00m 30s)
20:02 ppchelko@tin: Started deploy [cpjobqueue/deploy@af9b590]: Emit broker statistics more frequentl
19:58 dcausse: banning elastic1020 to see if T175951 is caused by mixed versions of the ltr plugin
19:36 demon@tin: Synchronized php-1.30.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 46s)
19:31 demon@tin: Synchronized php-1.30.0-wmf.17/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/userNormal.svg: fix svg syntax (duration: 00m 45s)
19:07 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.18
18:58 tgr@tin: Synchronized wmf-config/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 48s)
18:57 tgr@tin: Synchronized private/PrivateSettings.php: set Electron secret for T175868 (duration: 00m 49s)
18:29 twentyafterfour: Phabricator: Deploying hotfix for T175942
18:10 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Configure enwiki to use CirrusSearch MLR by default T175772 (duration: 00m 50s)
18:03 Jamesofur: disabled oauth validation on metawiki/SUL for Teles after verification
17:57 bblack: cp2001 backend restart, mailbox lag
17:56 bblack: cp1074 backend restart, mailbox lag
17:18 eevans@tin: Started restart [electron-render/deploy@8dd5f13]: (no justification provided)
17:16 gehel@tin: Finished deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3 (duration: 00m 04s)
17:16 gehel@tin: Started deploy [wikimedia/discovery/analytics@ab5d5c1]: moving discovery/analytics to scap3
16:59 krinkle@tin: Synchronized php-1.30.0-wmf.17/extensions/NavigationTiming: T104902 (duration: 01m 00s)
15:51 bblack: repool cp1063 (after wiping varnish storage + restarting all 3x main service daemons)
15:51 bblack: cp1064 backend restart, mailbox lag
15:46 bblack: depool cp1063 (upload eqiad) - seems to be having some iowait issues?
14:07 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/CheckUser/specials/SpecialCheckUser.php: Fix Special cu for ip ranges (temp) (T175898) (duration: 00m 49s)
13:34 dcausse@tin: Synchronized wmf-config/CirrusSearch-production.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 49s)
13:33 dcausse@tin: Synchronized wmf-config/CirrusSearch-common.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 48s)
13:31 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: Setup Cirrus MLR models for top 20 language AB test (duration: 00m 50s)
11:14 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1049, pool db1100 (duration: 00m 49s)
11:12 akosiaris: upload apertium-crh-tur_0.2.0~r81866-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1049 (duration: 00m 50s)
09:51 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp1025.eqiad.wmnet
09:43 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wtp1025.eqiad.wmnet
09:43 akosiaris@puppetmaster1001: conftool action : set/weight=1; selector: name=wtp1025.eqiad.wmnet
09:41 gehel: re-enabling puppet on all cassandra nodes - T171704
09:41 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 09m 30s)
09:32 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
09:02 akosiaris@tin: Finished deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520 (duration: 02m 25s)
09:00 akosiaris@tin: Started deploy [parsoid/deploy@cec7d17]: test deploy using dsh groups. T165520
08:13 gehel: merging cassandra refactoring for puppet - T171704
02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 14 02:55:10 UTC 2017 (duration 7m 28s)
02:47 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 07m 58s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 09m 27s)
00:46 MaxSem: restarted populateIpChanges to use updated code
00:26 twentyafterfour: everything appears to be up and running. Phabricator upgrade complete.
00:23 twentyafterfour: running puppet on phab1001 to restore files which get lost by scap deploy
00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 01s)
00:21 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
00:21 twentyafterfour@tin: Finished deploy [phabricator/deployment@7e3a0a8]: (no justification provided) (duration: 00m 25s)
00:20 twentyafterfour@tin: Started deploy [phabricator/deployment@7e3a0a8]: (no justification provided)
00:18 twentyafterfour: Begin phabricator upgrade #phab-2017-09-13

2017-09-13

23:48 dereckson@tin: Synchronized php-1.30.0-wmf.18/maintenance/populateIpChanges.php: Improve ip_changes update script to insert rows in batches (Gerrit:377918) (duration: 00m 49s)
23:29 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Wikidata/extensions/Wikibase/client: Split page set before constructing InjectRCRecordsJob (T175316) (duration: 00m 57s)
23:28 dereckson@tin: Synchronized php-1.30.0-wmf.18/extensions/Echo/modules/api/mw.echo.api.APIHandler.js: (no justification provided) (duration: 00m 48s)
23:26 dereckson@tin: Synchronized wmf-config/CommonSettings.php: Set $wgScoreSafeMode to false (T174413) (duration: 00m 50s)
22:06 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/EventBus/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
22:03 maxsem@tin: Synchronized php-1.30.0-wmf.18/extensions/ArticleCreationWorkflow/: https://gerrit.wikimedia.org/r/#/c/377900/1 (duration: 00m 49s)
21:55 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377874/ (duration: 00m 47s)
21:23 ejegg: updated payments-wiki from ed2e481 to 26b16ea
grrrr: populating ip_changes on group1 wikis
20:43 bblack: re-routed ulsfo->eqiad around possible codfw damage as well - https://gerrit.wikimedia.org/r/#/c/377871/
20:34 bblack: deployed https://gerrit.wikimedia.org/r/#/c/377845/ (no gerrit bot, jenkins lint failing, etc) due to possible codfw routing issues
19:39 andrewbogott: rebooting labtestvirt2001
19:35 ppchelko@tin: Finished deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule (duration: 01m 18s)
19:34 ppchelko@tin: Started deploy [changeprop/deploy@86e8103]: Separate kafka metrics by rule
19:21 urandom: T172384: Upgrading restbase-dev100[5-6] to Cassandra 3.11.0-wmf4
19:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.18
19:13 demon@tin: Synchronized php: symlink thingie (duration: 00m 49s)
19:11 urandom: T172384: Upgrading restbase-dev1004 to Cassandra 3.11.0-wmf4 (canary)
18:34 niharika29@tin: Synchronized static/images/project-logos/: Change logo for huwiktionary T175483 (duration: 00m 49s)
18:27 mutante: gerrit2002 - re-enabled puppet - had forgotten to enable it again after recently tryning to test the gerrit->logstash change
18:26 niharika29@tin: Synchronized php-1.30.0-wmf.18/extensions/VisualEditor/: Make ve.dm.Change part of core module T175828 (duration: 00m 50s)
18:25 niharika29@tin: Synchronized docroot/mediawiki/ontology/: Add setup for https://www.mediawiki.org/ontology T171807 (duration: 00m 49s)
18:12 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on frwiki T154371 (duration: 00m 50s)
17:55 gwicke: rolling restart of pdfrender service in equiad after hang T174916 T172815
16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1001.eqiad.wmnet
16:42 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wdqs1002.eqiad.wmnet
16:28 gehel: starting decommissioning of wdqs100[12] - T175595
16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1002.eqiad.wmnet
16:25 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=wdqs1001.eqiad.wmnet
16:22 andrewbogott: rebooting labtestvirt2002
16:08 ottomata: restarting druid-brokers with increase in query cache size
15:23 demon@tin: Synchronized php-1.30.0-wmf.18/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
15:22 demon@tin: Synchronized php-1.30.0-wmf.18/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
15:21 demon@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: I2ecb6030 (duration: 00m 49s)
15:20 demon@tin: Synchronized php-1.30.0-wmf.17/includes/diff/DifferenceEngine.php: I2ecb6030 (duration: 00m 49s)
15:09 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/: backporting I13051038 (duration: 00m 52s)
14:31 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=logstash
14:31 gehel: pooling new logstash servers (logstash100[7-9]) - T175045
14:23 mobrovac@tin: Finished deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210 (duration: 00m 33s)
14:22 mobrovac@tin: Started deploy [cpjobqueue/deploy@60d0a78]: Start using the EventBus infrastructure for the updateBetaFeaturesUserCounts job - T175210
13:45 akosiaris: upload apertium-crh_0.1.0~r81872-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
13:30 urandom: T169936: Converting RESTBase dev environment to size-tiered compaction
13:29 elukey: update puppet compiler's facts via ./modules/puppet_compiler/files/compiler-update-facts
13:23 gehel: initial installation of logstash100[7-9] - T175045
13:21 zeljkof: EU SWAT finished
13:13 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Temporary lift account creation limits for WM United Kingdom workshop (T175700) (duration: 00m 49s)
12:14 moritzm: rebooting mc1015 (spare) for some tests
12:07 jynus: stopping wikidata rebuild maintenance script on terbium
12:07 bblack: cp1049 - backend restart, mailbox lag
11:57 jynus@tin: Synchronized wmf-config/db-eqiad.php: Set db1097 as the main api server for s4, Remove regular load from db1070 (duration: 00m 56s)
11:39 jynus: disabling semi-sync replication on the largest s5 database servers
11:03 ariel@tin: Finished deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run (duration: 00m 02s)
11:03 ariel@tin: Started deploy [dumps/dumps@3f119ab]: monitor will cd to dumps repo before each run
11:02 joal@tin: Finished deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches) (duration: 03m 12s)
10:58 joal@tin: Started deploy [analytics/refinery@b2e8852]: Regular analytics weekly deploy (new jars, oozie patches)
10:52 volans: reboot lvs3001
10:06 akosiaris: upload apertium-cat-srd_0.9.0~r82238-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
10:06 akosiaris: upload apertium-srd-ita_0.9.5~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:36 moritzm: installing bind updates from Debian stable/oldstable SUA update (client tools only)
09:25 akosiaris: upload apertium-srd_0.10.0~r82237-1+wmf1_amd64.changes to apt.wikimedia.org/jessie-wikimedia/main T174988
09:24 akosiaris: upload apertium-tur_0.1.0~r81882-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:23 akosiaris: upload apertium-ita_0.10.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:23 akosiaris: upload apertium-cat_2.3.0~r82237-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:05 akosiaris: upload apertium_3.4.2~r68466-2+wmf2 to apt.wikimedia.org/jessie-wikimedia
08:55 elukey: restart varnish backend on cp1072 (upload) - mailbox expiry lag
08:24 moritzm: installing emacs security updates
07:56 akosiaris: bounce varnish on cp1062, mailbox lag
07:47 moritzm: rolling out hhvm-luasandbox 2.0.14 to the remaining hosts in eqiad (along with HHVM restarts) (T173705)
07:08 moritzm: installing tcpdump security updates (fixing 90 CVE IDs...)
05:18 kartik@tin: Finished deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36 (duration: 02m 56s)
05:15 kartik@tin: Started deploy [cxserver/deploy@ee0081d]: Update cxserver to 3baaa36
03:12 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 13 03:12:46 UTC 2017 (duration 7m 30s)
03:05 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.18) (duration: 16m 28s)
03:03 jynus: dropping old users from phabricator database
03:00 eileen: update civicrm from ee7dda3 to 187238f
02:41 demon@tin: Synchronized wmf-config/mobile.php: rm unused config var (duration: 00m 46s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 31s)
01:19 jynus: stopping mariadb @ db1048 to clone it to db1059
00:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:PageLanguage on mul.wikisource Fix correct wiki name for wgPageLanguageUseDB on www.wikisource.org T175622 (duration: 00m 49s)

2017-09-12

23:53 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Do not deploy Timeless on fr.wiktionary for now" (duration: 00m 49s)
23:35 thcipriani@tin: Synchronized wmf-config: SWAT: Enable Flow on Newsletter_talk on mediawiki.org T175502 (duration: 00m 52s)
22:17 chasemp: labvirt1018:~# /sbin/reboot
22:00 bsitzmann@tin: Finished deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305) (duration: 04m 28s)
21:55 bsitzmann@tin: Started deploy [mobileapps/deploy@b11b75c]: Update mobileapps to 297b048 (T174707 T175305)
21:49 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/377527/2 T175725 (duration: 00m 49s)
21:37 MaxSem: Populating ip_changes in group0 wikis
21:33 ejegg: changed contact de-duplication duty cycle to 25 minutes
21:05 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.18
20:33 demon@tin: Finished scap: bootstrap wmf.18 (for real this time) (duration: 26m 23s)
20:06 demon@tin: Started scap: bootstrap wmf.18 (for real this time)
19:53 demon@tin: Finished scap: bootstrap wmf.18 (duration: 08m 14s)
19:45 demon@tin: Started scap: bootstrap wmf.18
19:05 papaul: maintenance complete on mw2256
17:51 bblack: cp3034 - reset fetch_maxchunksize to 0.25G (default) for fe+be
17:40 cwd: turned on dedupe
16:55 niharika29@tin: Finished deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154 (duration: 00m 01s)
16:55 niharika29@tin: Started deploy [iegreview/iegreview@69c4c3f]: Deploying iegreview with scap3 T129154
16:40 ejegg: updated CiviCRM from 63288ae to ee7dda3
16:35 niharika29@tin: Finished deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134 (duration: 00m 03s)
16:35 niharika29@tin: Started deploy [scholarships/scholarships@004635d]: Deploying scholarships with scap3 T129134
16:26 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2100.codfw.wmnet
16:18 ejegg: updated SmashPig from dce4f0a to dc62203910
15:43 _joe_: restarted ircecho on einsteinium
15:35 gehel: restart elasticsearch on relforge (last check of new plugin deployment) - T158560
15:22 bblack: manually setting fetch_maxchunksize=16MB via varnishadm on cp3034 fe+be instances
15:06 chasemp: disable puppet for lab* things for designate refactor rolleout (delayed a few minutes for a meeting)
15:04 moritzm: uploaded php5.5 5.5.38 to component/ci of jessie-wikimedia (for use in CI)
14:53 papaul: shutting down mw2256 for maintenance
14:43 elukey: add kafka-jumbo IPs to the kafka term of the analytics-in4 filter on cr1/cr2 eqiad
14:42 ema: cp1074 - backend restart, mailbox lag
14:39 bblack: manually setting fetch_maxchunksize=64MB via varnishadm on cp3034 fe+be instances
14:19 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove all references to db1059 (duration: 00m 44s)
14:18 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove all references to db1059 (duration: 00m 45s)
14:10 gehel: restarting elasticsearch on elastic1020 to validate new plugin deployment - T158560
14:07 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/eqiad - T158560
14:06 volans: testing reimage of mw2100 - T166300
14:06 gehel: restarting elasticsearch on elastic2001 to validate new plugin deployment - T158560
14:04 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059, pool db1097 with low load (duration: 00m 45s)
14:02 gehel: switching elasticsearch plugin deployment to .deb instead of scap on cirrus/codfw - T158560
13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1005.eqiad.wmnet
13:37 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs1004.eqiad.wmnet
13:37 gehel: pooling wdqs100[45] - T171210
13:31 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/CentralAuth/includes/api/ApiSetGlobalAccountStatus.php: API: Unbreak setglobalaccountstatus locked=lock - T175462 (duration: 00m 44s)
13:26 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner for Russian Wikimedia chapter wiki - T175356 (duration: 00m 45s)
13:25 hashar@tin: Synchronized php-1.30.0-wmf.17/includes/changes/ChangesListBooleanFilter.php: WLFilters: Respect default values - T174725 (duration: 00m 45s)
13:21 hashar@tin: Synchronized wmf-config/CommonSettings.php: Add Extension:Newsletter permissions to CommonSettings (duration: 00m 45s)
13:18 hashar@tin: Synchronized wmf-config/throttle.php: Lift account creation restrictions for WM Taiwan 10th anniversary - T175534 (duration: 00m 45s)
13:15 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/Wikidata/extensions/Wikibase/view/resources/jquery/wikibase/themes/default/jquery.wikibase.entityselector.css: Remove red highlighting on all entity selectors - T175525 (duration: 00m 45s)
13:12 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Reduce wikiPageUpdaterDbBatchSize to 20 - T173710 (duration: 00m 45s)
09:33 moritzm: upgrading deployment servers to HHVM-Luasandbox 2.0.14
09:32 moritzm: upgrading mw2* to HHVM-Luasandbox 2.0.14
09:25 moritzm: upgrading mw1293-mw1295 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
09:24 moritzm: upgrading mw1293-mw1205 (image scalers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
08:38 gehel: deploying elasticsearch plugins on relforge - T158560 (wrong ticket number before)
08:34 gehel: deploying elasticsearch plugins on relforge - T175159
08:28 moritzm: upgrading mw1189-mw1208 (API servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
07:35 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with full weight (duration: 00m 45s)
07:15 moritzm: upgrading mw1180-mw1188, mw1209-1220 (app servers) to HHVM-Luasandbox 2.0.14 (along with HHVM restarts)
02:29 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 12 02:29:20 UTC 2017 (duration 6m 45s)
02:22 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 07m 31s)
02:10 bblack: cp1050 - backend restart, mailbox lag
02:08 bblack: cp1099 - backend restart, mailbox lagh
00:48 awight@tin: Finished deploy [ores/deploy@42c5663]: Cause ORES service restart (duration: 00m 18s)
00:48 awight@tin: Started deploy [ores/deploy@42c5663]: Cause ORES service restart

2017-09-11

23:06 mutante: jenkins-bot says: java.lang.OutofMemoryError: PermGen space
22:30 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4026.*
22:30 dereckson@tin: Synchronized wmf-config/CommonSettings-labs.php: Fix MediaWiki\Auth\AbstractPreAuthenticationProvider case (Gerrit:377281, no-op in prod) (duration: 00m 45s)
21:45 awight@tin: Finished deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster (duration: 13m 47s)
21:41 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4025.*
21:31 awight@tin: Started deploy [ores/deploy@42c5663]: Try ORES filehandle fix on new cluster
21:03 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4023.*
20:13 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 (duration: 02m 49s)
19:11 chasemp: adjust nodepool rate to 10
19:08 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 46s)
19:01 mutante: restbase-dev1004:/srv# mv /home/eevans/cassandra-1502141526-pid13410.hprof /srv/eevans/ to free disk space
18:52 mutante: restbase - java[12418]: Error: Could not find or load main class org.wikimedia.cassandra.metrics.service.Service
18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable responsive reference columns on wiktionaries, wikivoyages and enwiki https://gerrit.wikimedia.org/r/#/c/376573/, https://gerrit.wikimedia.org/r/#/c/371630/ (duration: 00m 46s)
18:33 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been complete and tested good T174475
18:29 chasemp: bump nodepool rate to 15 and max-servers to 18 (no restart as it picks up config periodically)
18:29 robh: firmware update on scs-a1-codfw and scs-c1-codfw has been started, awaiting completion T174475
18:24 robh: updating firmware on scs-a1-codfw and scs-c1-codfw
18:10 robh: scs-ulsfo has successfully updated firmware T174475
18:03 robh: upgrading firmware on scs-ulsfo
17:09 gehel@tin: Finished deploy [wdqs/wdqs@177d20a]: (no justification provided) (duration: 02m 39s)
17:07 gehel: wdqs weekly deployment (logging and throttling fixes)
17:06 gehel@tin: Started deploy [wdqs/wdqs@177d20a]: (no justification provided)
15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4022.*
15:20 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4028.*
14:40 godog: roll-restart thumbor to apply https://gerrit.wikimedia.org/r/#/c/377264/ and upgrade to 1.4 - T173580 T174997
14:24 hashar: tin.eqiad.wmnet: reverted patch "WLFilters: Respect default values" that got merged but left undeployed
14:23 hashar: tin.eqiad.wmnet: reverted all four mediawiki-config that got merged but left undeployed and rebased the workspace
14:05 hashar: European SWAT cancelled due to deploy infrastructure issue
13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
13:54 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 27s)
13:50 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
13:43 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 28s)
13:40 hashar@tin: Synchronized static/images/project-logos/huwiktionary.png: Change logo for huwiktonary - T175483 (duration: 00m 28s)
11:43 moritzm: installing file/libmagic security updates (stretch-specific, does not affect older distros)
10:30 jynus: restarting mysql @ labsdb1001
09:40 godog: roll-restart thumbor to apply changes for gif max animated area - T173580
09:39 mobrovac@tin: Finished deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281 (duration: 00m 20s)
09:39 mobrovac@tin: Started deploy [cpjobqueue/deploy@e73a10f]: Initial deploy of cpjobqueue - T175281
08:44 mobrovac@tin: Finished deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992 (duration: 00m 07s)
08:44 mobrovac@tin: Started deploy [zotero/translators@a0c41c3]: Update translators to upstream e03695273 - T174992
08:16 volans: reimage script tests continue on mc100[1-2] T166300
07:59 moritzm: installing systemd updates from stretch point update
07:41 moritzm: installing remaining libonig security updates
02:41 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 11 02:41:43 UTC 2017 (duration 7m 0s)
02:34 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 08m 10s)

2017-09-10

21:00 bawolff: disabled Jforrester's gerrit account and flush sessions
13:41 elukey: restart Varnish backend on cp1073 (cache::upload) for mailbox expiry lag
13:13 elukey: restart cp1053's varnish backend for mailbox expiry lag and 503s - T175473
00:37 chasemp: labnodepool1001:~# service nodepool restart
00:36 bd808: wmcs: purged rabbitmq ceilometer_notifications.* queues

2017-09-09

23:42 chasemp: labnodepool1001:~# service nodepool start
23:06 chasemp: freeswap && labcontrol1001:/home/rush# service rabbitmq-server restart
22:59 chasemp: restart nova-api and nova-network
22:49 chasemp: labnodepool1001:~# service nodepool stop
22:44 madhuvishy: Running service rabbitmq-server restart on labcontrol1001
20:41 legoktm@tin: Synchronized php-1.30.0-wmf.17/includes/filerepo/file/LocalFile.php: Fix issues related to comments and files in UI and API - T175443 T175444 (duration: 00m 46s)
01:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.humanSearchRelevance.js: T171740: Reduce annoyance of survey by enforcing minimum 2 days between showing survey to same browser (duration: 00m 46s)

2017-09-08

23:46 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: T171740: Fix inverted sampling rates for human relevance survey (duration: 00m 47s)
23:38 mutante: netmon1002 - terminated unused screen session
23:09 mutante: phab1001, phab2001 - terminating 3 unused screen processes that were from iridium migration / data sync
21:54 mutante: gerrit2001 - restarting gerrit to test logstash config change (while not touching "live" gerrit)
20:24 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: bugs and stuff (duration: 00m 54s)
20:14 mutante: heze - rebooting for firmware upgrade
20:10 mutante: heze (bacula storage) - installing BIOS upgrade (T162850)
20:10 demon@tin: Synchronized scap/scap.cfg: drop git_repo for now, T175041 (duration: 00m 46s)
19:43 mutante: zosma.codfw.wmnet - delete salt key, puppet node clean, puppet node deactivate, remove from Icinga,... (T138650)
19:37 mutante: removing ganeti instance 'zosma' on ganeti2001 (T138650)
18:17 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: no-op (duration: 00m 46s)
16:04 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: T175338, followup (duration: 00m 46s)
16:03 demon@tin: Synchronized php-1.30.0-wmf.17/includes/revisiondelete/RevDelLogList.php: typofix (duration: 00m 46s)
15:53 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.*
15:47 ejegg: increased cURL timeout for PayPal IPN confirmation to 14 sec
15:07 akosiaris@tin: Finished deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config (duration: 00m 04s)
15:06 akosiaris@tin: Started deploy [librenms/librenms@5554213]: Testing new scap keyholder_key config
15:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4021.*
14:58 volans: testing wmf-auto-reimage also on mc1002 T166300 T164341
12:36 _joe_: restarting a check of the md2 devices on conf200* T162013 after reducing sync speed
12:36 _joe_: reduced raid resync max speed of raid devices on conf200* to 20000
10:35 hashar@tin: Synchronized php-1.30.0-wmf.17/extensions/ProofreadPage: Restore page status buttons - T175304 (duration: 00m 50s)
10:12 _joe_: stopped all md devices syncs on conf200* machines
10:09 _joe_: etcd cluster lost consensus T162013
10:07 volans: testing wmf-auto-reimage on mc1001 T166300 T164341
09:58 _joe_: running the same command on conf2002 too
09:48 _joe_: running `/usr/share/mdadm/checkarray --cron --all --idle --quiet` on conf2001 and conf2003 trying to reproduce consensus issues in T162013
09:25 moritzm: restarting apache on tegmen/einsteinium to pick up security update
09:09 moritzm: installing libonig security updates
09:08 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940 (duration: 01m 01s)
09:07 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster, take #2 - T169940
09:01 moritzm: installing remaining gnutls updates from jessie point release
08:13 moritzm: installing libgd2 security updates
08:11 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): Use double writing for mobileapps in the dev cluster - T169940
08:11 mobrovac: restbase enabled back puppet in the dev cluster - T169940
07:53 moritzm: installing ruby2.3 security updates
07:32 mobrovac@tin: Finished deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided) (duration: 07m 02s)
07:25 mobrovac@tin: Started deploy [restbase/deploy@0d39acf] (dev-cluster): (no justification provided)
07:21 mobrovac@tin: Finished deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided) (duration: 00m 11s)
07:21 mobrovac@tin: Started deploy [restbase/deploy@e6aeeeb] (dev-cluster): (no justification provided)
00:42 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/AbuseFilter/includes/Views/AbuseFilterViewExamine.php: fix comment stuff (duration: 00m 46s)

2017-09-07

23:42 bblack: cp1066 - depooled all traffic services, seems to have some bug causing 503 spikes
23:19 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.17
23:15 XenoRyet: Updated Smashpig from 8eb98c1 to dce4f0a
22:41 niharika29@tin: Synchronized php-1.30.0-wmf.17/extensions/ArticleCreationWorkflow/: Bump extension version (duration: 00m 49s)
22:03 niharika29@tin: Finished scap: Deploying ArticleCreation Workflow on test2 wiki TT175302 (duration: 23m 00s)
21:40 niharika29@tin: Started scap: Deploying ArticleCreation Workflow on test2 wiki TT175302
21:23 demon@tin: Synchronized php-1.30.0-wmf.17/includes/api/ApiQueryRecentChanges.php: T175307 (duration: 00m 49s)
21:01 urandom: T169940: Disabling puppet and restbase service in RESTBase dev
21:00 urandom: T169940: Disabling changeprop in RESTBase dev environment
19:05 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/MobileFrontend/includes/specials/SpecialMobileHistory.php: T175161 (duration: 00m 49s)
18:48 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.17
18:47 demon@tin: Synchronized php: symlink update -> wmf.17 (duration: 00m 48s)
18:39 legoktm: restarted script to reparse all pages in parsoid for Linter (python3 parsoid_reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
17:55 ejegg: disabled queue consumers for data cleaning job
17:52 ejegg: updated CiviCRM from c1ece1e to 63288ae
17:35 otto@tin: Finished deploy [analytics/refinery@6b60f2c]: (no justification provided) (duration: 03m 23s)
17:32 otto@tin: Started deploy [analytics/refinery@6b60f2c]: (no justification provided)
17:31 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 00m 23s)
17:31 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
17:23 otto@tin: Finished deploy [analytics/refinery@bdf6754]: (no justification provided) (duration: 03m 25s)
17:20 otto@tin: Started deploy [analytics/refinery@bdf6754]: (no justification provided)
16:13 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Remove Extension:RelatedSites from zhwikivoyage (duration: 00m 50s)
15:32 mobrovac@tin: Finished deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848 (duration: 06m 18s)
15:25 mobrovac@tin: Started deploy [restbase/deploy@77961d0]: Revert using MCS for the summary end point - T168848
14:45 _joe_: manually running /usr/share/mdadm/checkarray --cron --all --idle --quiet on conf2001, trying to reproduce consensus issues in T162013
14:40 jynus: rebooting and upgrading es1019
13:54 godog: powercycle ms-be2023 - load through the roof and no login possible
13:54 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=jobrunner,name=eqiad
13:14 zeljkof: EU SWAT finished
13:13 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
13:12 zfilipin@tin: Synchronized dblists/wikidataclient.dblist: SWAT: Add English Wiktionary as a client of Wikidata (T159316) (duration: 00m 49s)
11:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1100 with 0 weight - T172679 (duration: 00m 49s)
09:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 depooled to s5 array - T172679 (duration: 00m 49s)
08:59 ariel@tin: Finished deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps (duration: 00m 02s)
08:58 ariel@tin: Started deploy [dumps/dumps@da05e9f]: add general info to siteinfo dumps
08:44 elukey: restart varnish backend on cp1063 - mailbox expiry lag
07:57 elukey: force re-mount of /mnt/hdfs on stat1005
07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1100 - T172679 (duration: 00m 48s)
07:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1100 - T172679 (duration: 00m 49s)
07:45 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848 (duration: 06m 36s)
07:39 mobrovac@tin: Started deploy [restbase/deploy@b51c58c]: Use MCS for the summary endpoint - T168848
07:35 mobrovac@tin: Finished deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint (duration: 01m 43s)
07:33 mobrovac@tin: Started deploy [restbase/deploy@b51c58c] (staging): Use MCS for the summary endpoint
07:20 oblivian@tin: Synchronized rpc/RunSingleJob.php: Adding the RunSingleJob.php rpc endpoint for jobrunners (duration: 00m 56s)
06:07 eileen: civicrm updated from 73a24b4 to c1ece1e
03:13 eileen: seconds behind=78
03:08 eileen: seconds behind 55
02:52 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Sep 7 02:52:05 UTC 2017 (duration 7m 15s)
02:44 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 06m 15s)
02:43 eileen: restarting groupmember.load after hacking out cache delete calls
02:40 eileen: update civicrm from 764bfe1 to 73a24b4
02:24 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 23s)
02:05 eileen: replag 217 behind
01:56 eileen: replag 197 seconds
01:47 eileen: replag 23 seconds
01:29 eileen: replag 0
01:22 eileen: replag 550 sec behind master
01:13 eileen: replag delay 422
01:12 eileen: job killed
00:49 eileen: 8389 contacts now
00:49 eileen: 7971 contacts so far
00:46 eileen: started Omnigroupmember.load
00:24 twentyafterfour: phabricator update deployed
00:12 twentyafterfour@tin: Finished deploy [phabricator/deployment@c265c1a]: (no justification provided) (duration: 00m 24s)
00:11 twentyafterfour@tin: Started deploy [phabricator/deployment@c265c1a]: (no justification provided)
00:07 twentyafterfour: deploying phabricator update 2017-09-06 https://phabricator.wikimedia.org/project/view/2980/

2017-09-06

23:41 twentyafterfour@tin: Synchronized php-1.30.0-wmf.16/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to all webservers refs T174387 (duration: 00m 49s)
23:28 twentyafterfour@tin: Synchronized php-1.30.0-wmf.17/extensions/WikimediaEvents/: Sync Change-Id: I7ae522 to wmf.17 refs T174387 (duration: 00m 49s)
23:23 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-common.php: CirrusSearch-common.php goes last (duration: 00m 48s)
23:21 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings.php: Now sync initializesettings (duration: 00m 49s)
23:19 twentyafterfour@tin: Synchronized wmf-config/CirrusSearch-rel-survey.php: sync the added file first (duration: 00m 49s)
23:17 twentyafterfour@tin: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
23:12 twentyafterfour@tin: Synchronized wmf-config/: roll-back due to huge error rate spike (duration: 00m 51s)
23:10 twentyafterfour@tin: Synchronized wmf-config/: deploy config for CirrusSearch human relevancy survey. Change-Id: I272c69 Bug: T174106 (duration: 00m 52s)
22:18 no_justification: prior deploy was no-op
22:18 demon@tin: Finished deploy [gerrit/gerrit@d4f9a77]: (no justification provided) (duration: 00m 07s)
22:18 demon@tin: Started deploy [gerrit/gerrit@d4f9a77]: (no justification provided)
22:11 bsitzmann@tin: Finished deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808) (duration: 04m 53s)
22:06 bsitzmann@tin: Started deploy [mobileapps/deploy@507a479]: Update mobileapps to 2cb6281 (T168848 T169277 T169274 T162179 T164033 T167921 T174698 T168848 T174808)
21:39 niharika29@tin: Synchronized wmf-config/CommonSettings-labs.php: Config for ArticleCreationWorkflow T175054 (duration: 00m 50s)
21:28 ejegg: updated payments-wiki from bf97cd2 to ed2e481
21:24 ejegg: updated CiviCRM from f9c373e to 764bfe1
21:17 ottomata: rebooting kafka-jumbo1004
20:34 arlolra: Updated Parsoid to f9d367ea (T169342)
20:27 arlolra@tin: Finished deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea (duration: 08m 27s)
20:18 arlolra@tin: Started deploy [parsoid/deploy@f07ac8c]: Updating Parsoid to f9d367ea
19:53 ottomata: reimaging kafka-jumbo* with stretch
19:48 chasemp: reboot labvirt1018
18:50 maxsem@tin: Synchronized wmf-config/: just to make sure... (duration: 00m 50s)
18:48 maxsem@tin: Synchronized wmf-config/CommonSettings.php: revert (duration: 00m 49s)
18:33 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370292/3 (duration: 00m 49s)
18:25 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370291/2 (duration: 00m 49s)
18:19 maxsem@tin: Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
18:18 maxsem@tin: Synchronized wmf-config/Wikibase-production.php: https://gerrit.wikimedia.org/r/#/c/376317/2 (duration: 00m 48s)
18:09 maxsem@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
18:09 ejegg: rolled payments-wiki back to bf97cd2
18:08 maxsem@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 50s)
18:05 ejegg: updated payments-wiki from bf97cd2 to 6911158
17:53 bblack: cp1048 - varnish backend restart for mailbox lag...
17:23 demon@tin: Synchronized wmf-config/FeaturedFeedsWMF.php: code cleanup (duration: 00m 49s)
17:20 mutante: added LDAP user schoenbaechler to WMF group (T175156)
17:16 mutante: modify-ldap-group on terbium is broken
17:02 ejegg: disabled creating CiviMail records for Thank You emails
16:51 demon@tin: Synchronized php-1.30.0-wmf.17/extensions/Flow/includes/: I284b5aa (duration: 01m 01s)
16:41 chasemp: disable puppet for cloud things to test changes
16:40 bblack: cp1072 - varnish backend restart (mailbox lag)
16:40 bblack: cp1062 - varnish backend restart (mailbox lag)
15:55 ejegg: re-enabled IPN notification at PayPal console (48 minutes ago)
15:53 akosiaris: powercycling lvs3001
15:28 papaul: firmware upgrade on mw2256
15:11 bblack: cp1049 - restart varnish backend (mailbox lag)
15:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 49s)
15:06 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2040 (duration: 00m 49s)
14:17 gehel: wdqs1005 is coming up after a few reimaging issues, expect some icinga noise...
13:19 jynus: restarting and upgrading db2040
13:13 reedy@tin: Synchronized wmf-config/Wikibase-production.php: Move some wikidata config T174962 (duration: 00m 48s)
13:12 reedy@tin: Synchronized wmf-config/Wikibase-labs.php: Move some wikidata config T174962 (duration: 00m 49s)
13:07 reedy@tin: Synchronized wmf-config/throttle.php: Throttle exception T175113 (duration: 00m 49s)
12:08 moritzm: installing libapache2-mod-perl update from jessie 8.9 update
11:41 moritzm: installing gtk+2.0 update from jessie 8.9 update
11:24 elukey: temporarily raise kafka log4j authorizer verbosity to DEBUG on kafka1012 - T173493
11:21 moritzm: installing gnutls update from jessie 8.9 and stretch 9.1 point updates
11:20 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2040 (duration: 00m 49s)
11:20 godog: reimage restbase1008 with cassandra 3 - T169939
11:15 marostegui: Disable puppet on db1100 for mydumper/myloader
10:26 moritzm: installing perl update from stretch 9.1 point release
10:25 moritzm: installing perl update from jessie 8.9 point release
10:12 godog: reimage restbase1010 with cassandra 3 - T169939
09:39 moritzm: installing libgd security updates
09:28 moritzm: installing libonig security uodates#
09:05 jynus: disabling puppet on most db hosts to merge firewall changes safely
08:53 godog: reimage restbase1009 with cassandra 3 - T169939
08:49 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T174948 Add WMDE log channel (duration: 00m 49s)
08:43 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: T110170 Enable Newsletter on mediawikiwiki (duration: 00m 51s)
07:15 marostegui: Stop MySQL on db1049 to copy its content to db1100 - https://phabricator.wikimedia.org/T172679
06:46 moritzm: installing php-luasandbox 2.0.14 on API canaries along with HHVM restart (T173705)
06:37 marostegui: Truncate l10n_cache table across production - T150306
04:08 demon@tin: Synchronized wmf-config/throttle.php: no-op (duration: 00m 49s)
03:14 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Sep 6 03:14:16 UTC 2017 (duration 7m 6s)
03:07 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.17) (duration: 14m 49s)
02:31 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 18s)
away: disabled IPN messages at the PayPal console
00:14 Reedy: that was T174747 Adjust language icon color
00:14 reedy@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue: (no justification provided) (duration: 00m 49s)
00:10 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 49s)
00:09 reedy@tin: Synchronized php-1.30.0-wmf.17/extensions/Scribunto/tests/phpunit/: Fix broken test (duration: 00m 50s)

2017-09-05

23:58 reedy@tin: Synchronized wmf-config/: Test ArticleCreationWorkflow extension on the Beta Cluster (duration: 00m 51s)
23:39 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable AbuseFilter runtime profile T161059 (duration: 00m 49s)
23:33 foks: 2FA disabled for Omshivaprakash (T175075)
22:17 robh: rebooting mw1163 as its serial console is locked up
22:07 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: kill some warnings and stuff (duration: 02m 55s)
22:06 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/FlaggedRevs/frontend/specialpages/reports/: (no justification provided) (duration: 03m 13s)
20:48 ejegg: re-enabled donation queue consumer and thank you mailer
20:40 ejegg: updated civicrm from 820bd98 to f9c373e
20:37 ejegg: disabled Donation queue consumer and thank-you mail sender
19:38 gilles@tin: Synchronized private/PrivateSettings.php: Add Thumbor username to Swift configuration (duration: 00m 48s)
19:33 jynus: rebuilding querycache table on tewiktionary.dbstore1002, it had crashed
19:28 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: wmf.17 on group0
19:05 Reedy: deleted querycache rows where qc_type = '' on all wikis T174513
18:11 ejegg: updated fundraising tools from 9554229 to 8d39806
18:11 demon@tin: Finished scap: wmf.17 bootstrap (duration: 47m 00s)
17:38 bblack: cp1074 - restart varnish backend, mailbox lag
17:24 demon@tin: Started scap: wmf.17 bootstrap
17:14 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 (duration: 02m 32s)
16:59 andrewbogott: reducing the number of concurrent nova-conductor workers. May cause hiccups as services restart
16:58 elukey: ran authdns-update from ns1.w.o after https://gerrit.wikimedia.org/r/374385 to create electcom.wikimedia.org
16:51 jynus: reloading dbproxy1005 to repoing to db1009 again- things seem stable right now
16:45 elukey: add new virtualhost electcom.wikimedia.org to the appservers apache config - https://gerrit.wikimedia.org/r/374389 (implies apache config reload)
16:33 mattflaschen@tin: Synchronized php-1.30.0-wmf.16/: Prepare to enable RCFilters (WLFilters) on Watchlist, but without i18n changes (reverted) (duration: 12m 41s)
16:14 addshore: addshore@terbium:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki mediawikiwiki --extension newsletter
16:08 akosiaris: reboot all kubernetes boxes T170119
15:43 matt_flaschen: 'Scap sync failed on i18n, so I'll deploy just the non-i18n ones'
15:36 mattflaschen@tin: scap aborted: Prepare to enable RCFilters (WLFilters) on Watchlist (duration: 08m 42s)
15:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T174509 (duration: 00m 47s)
15:27 mattflaschen@tin: Started scap: Prepare to enable RCFilters (WLFilters) on Watchlist
14:30 reedy@tin: Synchronized wmf-config/: phpcs (duration: 00m 47s)
14:28 reedy@tin: Synchronized tests/: phpcs (duration: 00m 46s)
14:26 reedy@tin: Synchronized errorpages/404.php: phpcs (duration: 00m 45s)
14:25 reedy@tin: Synchronized docroot/noc/conf/highlight.php: Fix links for dblists in highlight.php (duration: 00m 44s)
14:15 zeljkof: EU SWAT finished
14:13 zfilipin@tin: Synchronized php-1.30.0-wmf.16/includes/EditPage.php: SWAT: Re add wpScrolltop id in EditPage (T174723) (duration: 00m 45s)
13:47 zfilipin@tin: Synchronized php-1.30.0-wmf.16/extensions/ContentTranslation/: SWAT: encodeURIComponent title to escape / properly (T174792) (duration: 00m 48s)
13:30 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add abusefilter-view-private to rollbackers in zhwiki (T174978) (duration: 00m 46s)
13:28 godog: reimage restbase2005 - T169939
13:17 marostegui: Drop pr_index tables from where ProofreadPage isn't enabled - T174782
13:14 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logo for sr.wikibooks.org (T172284) (duration: 00m 46s)
12:24 hashar: ci: switched mediawiki/core mediawiki/vendor php5.5 jobs from trusty to jessie - T161882
12:15 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Enable sendEchoNotification for enwiki, dewiki, frwiki (T142102) (duration: 00m 46s)
11:54 marostegui: Drop reader_feedback, reader_feedback_history, reader_feedback_pages tables - T174586
11:07 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: (Actually) Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 45s)
11:05 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on sqwiki (T174335) (duration: 00m 46s)
10:38 hashar: Nodepool / CI are fully backup and processing queued jobs
10:25 hashar: Restarting Nodepool. I have managed to spawn instances manually for contintcloud tenant
10:18 _joe_: stopping puppet, nginx on mw2249 for experimentation
10:11 ema: reboot cp4024 T174891
10:08 godog: copy python-logstash from jessie-wikimedia to stretch-wikimedia
09:59 hashar: Stopped Nodepool on labnodepool1001.eqiad.wmnet
09:52 akosiaris: reimage kubernetes[12]00[1-4] T170119
08:49 moritzm: upgrading canary app servers to hhvm-luasandbox 2.0.14 (T173705)
08:47 moritzm: uploaded php-luasandbox/hhvm-luasandbox 2.0.14 to apt.wikimedia.org (T173705)
07:14 moritzm: installing libgd security updates on canary app servers (along with hhvm restart)
07:03 _joe_: launching manually 3 workers for refreshLinks jobs on commons, T173710
06:32 marostegui: Deploy alter table on db1083 - T174509
06:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T174509 (duration: 00m 46s)
06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T174509 (duration: 00m 46s)
06:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T174509 (duration: 00m 55s)
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Sep 5 02:30:03 UTC 2017 (duration 6m 37s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 07m 28s)

2017-09-04

23:32 Dereckson: Set e-mail to Hexabot bot account to allow account recovery (T174973)
22:07 legoktm: starting script to reparse all pages in parsoid for Linter (python2 parsoid-reparse.py http://parsoid.discovery.wmnet:8000 --sitematrix --linter-only --skip-closed https://aa.wikipedia.org/w/api.php) - T161556
17:50 moritzm: uploaded debdeploy 0.0.99-2 to apt.wikimedia.org
17:19 gehel@tin: Finished deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment (duration: 02m 46s)
17:17 gehel@tin: Started deploy [wdqs/wdqs@1caaa30]: weekly WDQS deployment
16:29 mobrovac@tin: Finished deploy [restbase/deploy@8c9c436]: (no justification provided) (duration: 06m 23s)
16:23 mobrovac@tin: Started deploy [restbase/deploy@8c9c436]: (no justification provided)
16:10 mobrovac: restbase depooled restbase10(0[89]|10) and restbase2005 for T169939
16:07 _joe_: rolling restart of pybal in codfw/eqiad low-traffic pools for picking up the new jobrunner LVS endpoint
15:42 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,service=nginx
14:51 godog: reimage restbase2003 for use with cassandra 3 / jbod - T169939
14:27 moritzm: enabled LDAP authentication for ganglia.wikimedia.org
13:37 gehel: rolling restart of elasticsearch/logstash to cleanup unused plugins - T174933
13:34 gehel: delete leftover grafana-dashboards index on elasticsearch logstash cluster
13:22 gehel: restart elasticsearch on logstash1006 to test plugin removal - T174933
12:26 moritzm: uploaded debdeploy 0.0.99-1+trusty/trusty-wikimedia to apt.wikimedia.org
12:18 marostegui: Stop MySQL on db1037 as it is going to be decommissioned - T174902
12:10 elukey: rolling restart of zookeeper on conf100[123] for jvm security updates
11:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
11:52 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)
11:45 godog: restart varnish on cp1099 - mailbox lag
10:22 ariel@tin: Finished deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression (duration: 00m 02s)
10:22 ariel@tin: Started deploy [dumps/dumps@0b32eef]: fix abstract dumps variants regression
09:51 moritzm: uploaded debdeploy 0.0.99-1+jessie/jessie-wikimedia to apt.wikimedia.org
09:41 mobrovac@tin: Started restart [mathoid/deploy@44ea6d8]: Restart for libxml update
09:36 moritzm: uploaded debdeploy 0.0.99-1/stretch-wikimedia to apt.wikimedia.org
08:52 elukey: restart zookeeper on conf2003 for jvm security updates
08:48 marostegui: Stop MySQL on db1045 as it will be decommissioned - T174806
08:47 moritzm: installing mercurial security updates
08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 46s)
08:39 moritzm: installing libgcrypt20 security updates
08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 48s)
08:33 godog: upload scap 3.7.0-1 to apt.w.o - T127762
07:39 moritzm: installing gnupg security updates
07:03 marostegui: Deploy alter table on s3 codfw master with replication on kpbwiki - T168661
07:02 _joe_: starting additional runJobs instance for htmlcacheupdate on commons T173710
04:52 marostegui: Deploy alter table on db1052 - T168661
02:44 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Sep 4 02:44:15 UTC 2017 (duration 6m 52s)
02:37 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 08m 17s)

2017-09-03

17:52 elukey: depooled cp4024 (ulsfo upload) due to kernel errors in dmesg
15:57 ema: restart varnish backend on cp1073 (mailbox lag)
09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=varnish-fe
09:41 ema@neodymium: conftool action : set/pooled=yes; selector: name=cp4024.ulsfo.wmnet,service=nginx
09:25 ema: cp4024 is down, powercycled
09:13 _joe_: manually fixed value of /conftool/v1/pools/eqiad/api_appserver/apache2/mw1208.eqiad.wmnet in codfw to allow replication to restart
00:59 legoktm: legoktm@terbium:~$ mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php --wiki=hiwikiversity --backend=local-multiwrite # T174859

2017-09-01

16:18 bblack: restart varnish backend on cp1074 (mailbox lag)
{{safesubst:SAL entry|1=15:15 urandom: Restarting Cassandra: restbase-dev100[5-6]-{a,b}}}
{{safesubst:SAL entry|1=15:06 urandom: Restarting Cassandra: restbase-dev1004-{a,b}}}
15:00 reedy@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: T174724 (duration: 00m 46s)
14:28 marostegui: Add 150G to /srv partition on labsdb1001
13:42 marostegui: Rename table pr_index on enwiki on db1089 - T174782
13:10 marostegui: Stop MySQL on db1026 as it will be decommissioned - T174763
12:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1081 original weight - T168661 (duration: 00m 43s)
12:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
12:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)
11:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1081 weight - T168661 (duration: 00m 43s)
11:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low weight - T168661 (duration: 00m 44s)
10:35 elukey: stop puppet on thorium and disable root rsyncs - T174756
10:31 marostegui: Upgrade MariaDB to 10.0.32 on db1081 - T168661
10:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 - T168661 (duration: 00m 43s)
09:10 godog: bounce graphite-web on graphite1001, problematic query? using all CPU
09:04 ema: lvs1007 upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
08:59 godog: depool restbase200[135] before reimage - T169939
07:22 elukey: restart apache2 and hue on thorium, Analytics sites down, investigating
07:06 marostegui: Power reset db2044 as it is unresponsive - T174764
03:54 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I4dfc33f66c3 - Enable jQuery 3 on nlwiki sister projects (duration: 00m 43s)
00:57 tstarling@tin: Synchronized php-1.30.0-wmf.16/includes/parser/Parser.php: (no justification provided) (duration: 00m 44s)
00:13 RainbowSprinkles: gerrit: flushed all non-login caches, things might be sluggish for the next ~15mins or so

2017-08-31

20:03 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided)
19:27 mattflaschen@tin: Finished scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030 (duration: 21m 23s)
19:17 cmjohnson1: powercycled mw1294
19:05 mattflaschen@tin: Started scap: Watchlist filters: Convert edit watchlist button to new UX and fix server-side tag filtering. T172030
18:53 mattflaschen@tin: Synchronized docroot/noc/conf/interwiki-labs.php.txt: Interwiki links on Beta Cluster: T69931 (duration: 00m 47s)
18:51 mattflaschen@tin: Synchronized wmf-config: Interwiki links on Beta Cluster: T69931 (duration: 00m 49s)
17:50 demon@tin: Synchronized php-1.30.0-wmf.16/skins/MinervaNeue/resources/skins.minerva.icons.images.scripts/watch.svg: sorta impromptu swat thing (duration: 00m 47s)
16:35 ejegg: re-enabled donation queue consumer and thank you mail sender
15:25 ejegg: updated civicrm from f00611d to 820bd98
15:20 ejegg: updated SmashPig from 8f4388d to 8eb98c1
15:19 ejegg: turned off donation queue consumer and thank you job
15:18 elukey: restart zookeeper on conf200[2,3] for jvm security updates
13:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1084 original weight - T168661 (duration: 00m 47s)
13:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1084 weight - T168661 (duration: 00m 47s)
13:07 elukey: restart zookeeper on conf2001 for security updates (canary node)
13:00 moritzm: installing augeas security updates
12:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low weight - T168661 (duration: 00m 47s)
12:14 marostegui: Upgrade MariaDB on db1084 - T168661
12:11 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T168661 (duration: 00m 47s)
12:05 akosiaris: reimage chlorine and kubernetes1004 T170119
12:04 akosiaris: reimage chlorine and kubernetes1004 T10119
11:58 moritzm: restarting nginx on dataset1001/ms1001 to pick up libxml security update
11:21 elukey: restart apache2 on bohrium for libxml + gnutls security updates
11:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1056 original weight - T168661 (duration: 00m 47s)
11:17 moritzm: restarting apache on dbmonitor* to pick up libxml security update
10:27 moritzm: restarting nginx on prometheus* to pick up libxml security update
10:22 moritzm: restarting apache on einsteinium/tegmen (Icinga hosts) to pick up libxml security update
10:14 moritzm: restarting nginx on meitnerium/archiva.wikimedia.org to pick up libxml security update
10:13 moritzm: restarting nginx on debug proxies to pick up libxml security update
10:10 moritzm: restarting apache on hafnium to pick up libxml security update
09:57 addshore: extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php run done, 6326 hashes recalculated, T172987
09:50 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 # Should upsert 6326 rows T172987
09:44 addshore: addshore@terbium:~$ mwscript extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php --wiki enwiktionary --batch-size 1000000 --dry-run # T172987
09:43 moritzm: restarting apache on auth* to pick up libxml security update
09:40 moritzm: installing libxml2 security updates
09:10 moritzm: installing experimental hhvm-luasandbox package on mw1261 (canary app server) with backported patch for T171392 / T173705
09:04 hashar: contint1001 upgrading git, restarting git-daemon and zuul-merger - T161086
09:01 godog: test reimage of restbase2001 - T169939
08:57 hashar: contint2001 upgrading git, restarting git-daemon and zuul-merger - T161086
08:45 godog: remove /srv/deployment/grafana on naos and tin, unused - T170881
08:41 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@eqiad
08:40 addshore@tin: Synchronized php-1.30.0-wmf.16/extensions/Cognate/maintenance/recalculateCognateNormalizedHashes.php: T172987 Add waitForReplication to RecalculateCognateNormalizedHashes script (duration: 00m 47s)
08:20 gehel: restart postgresql / kartotherian / tilerator / tileratorui on maps@codfw
08:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
08:01 marostegui: Rename reader_feedback, reader_feedback_history, reader_feedback_pages tables on dewiki on db1092 - T174586
07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic on db1056 - T168661 (duration: 00m 47s)
06:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1056 with low weight - T168661 (duration: 00m 46s)
06:19 marostegui: Upgrade MariaDB to 10.0.32 on db1056 - T168661
06:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
06:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 - T168661 (duration: 00m 47s)
02:55 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 31 02:55:12 UTC 2017 (duration 7m 9s)
02:48 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 06m 40s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 10m 02s)

2017-08-30

23:10 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 47s)
23:09 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Newsletter/includes/specials/SpecialNewsletter.php: T174604 (duration: 00m 48s)
23:05 demon@tin: Synchronized wmf-config/unitConversionConfig.json: swat (duration: 00m 47s)
21:05 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.16
21:03 thcipriani@tin: Synchronized php-1.30.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: ProofreadPage: Avoids a stack overflow T173520 (duration: 00m 47s)
20:45 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 back to 1.30.0-wmf.15
20:22 ppchelko@tin: Finished deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages (duration: 01m 05s)
20:21 ppchelko@tin: Started deploy [changeprop/deploy@8d5fa29]: Start rejecting deduplicated messages
20:07 urandom: T169939: Decommission Cassandra: restbase1009-c.eqiad.wmnet
19:09 ppchelko@tin: Finished deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages (duration: 01m 17s)
19:07 ppchelko@tin: Started deploy [changeprop/deploy@ff93ea8]: Don't deduplicate retry messages
19:06 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.16
19:04 demon@tin: Synchronized php: Symlink swap for group1/wmf.16 (duration: 00m 46s)
18:11 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/374399/4 (duration: 00m 47s)
17:29 urandom: T169939: Decommission Cassandra: restbase1009-b.eqiad.wmnet
17:27 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
17:12 ejegg: disabled creating CiviMail records for Thank You emails
16:49 XioNoX: rolling back vrrp priority change for the cr2<->asw-d-codfw link - T174366
16:34 XioNoX: start work on the cr2<->asw-d-codfw link - T174366
16:13 XioNoX: lowering vrrp priority for the cr2<->asw-d-codfw link - T174366
15:56 elukey: re-added analytics1055 among the hdfs/yarn worker after maintenance
15:49 bblack: cp1049 - restart varnish backend, mailbox lag
15:43 urandom: T169939: Decommission Cassandra: restbase1009-a.eqiad.wmnet
15:02 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1294.eqiad.wmnet
14:31 moritzm: restarting nginx on sodium to pick up libxml security update
14:07 elukey: restart java daemons on druid100[456] for jvm security updates
14:04 moritzm: restarting squid on install* servers to pick up libxml security update
14:04 moritzm: restarting squid on installurl downloaders to pick up libxml security update
14:02 moritzm: restarting apache on krypton to pick up libxml security update
13:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1053 original weight - T168661 (duration: 00m 46s)
13:55 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: Scale A/B test bucket sizes by 10 - T172291 (duration: 00m 46s)
13:50 moritzm: restarting squid on url downloaders to pick up libxml security update
13:34 moritzm: installing libxml2 security updates
13:23 moritzm: installing ffmpeg security upadtes
13:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 46s)
13:19 hashar: European SWAT completed
13:16 gehel: restarting wdqs-blazegraph and wdqs-updater on all wdqs nodes for logback config change - T172710
13:11 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable SandboxLink on cywiki - T173054 (duration: 00m 48s)
12:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 45s)
12:51 gehel: restart wdqs-updater on wdqs2001 for config change
12:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1053 weight - T168661 (duration: 00m 47s)
12:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1053 with low weight - T168661 (duration: 00m 46s)
12:05 marostegui: Upgrade MariaDB to 10.0.32 on db1053 - T168661
12:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1053 - T168661 (duration: 00m 47s)
11:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1055 original weight - T174265 (duration: 00m 46s)
10:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
09:49 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1059 - T168661 (duration: 00m 47s)
09:07 elukey: restart all jvm daemons on druid100[123] for security updates
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 48s)
08:06 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give db1055 more traffic - T174265 (duration: 00m 47s)
07:47 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1228.eqiad.wmnet
07:47 marostegui: Upgrade MariaDB on db1059 to 10.0.32 - T168661
07:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059 for a MariaDB upgrade - T168661 (duration: 00m 53s)
07:14 moritzm: powering down mw1294 for hardware diagnostics (T1674069
07:12 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1294.eqiad.wmnet
07:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly repool db1055 - T174265 (duration: 00m 52s)
03:16 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 30 03:16:09 UTC 2017 (duration 7m 19s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.16) (duration: 15m 42s)
02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 28s)
02:17 demon@tin: Finished deploy [gerrit/gerrit@f33c63f]: Testing, no-op (duration: 00m 07s)
02:17 demon@tin: Started deploy [gerrit/gerrit@f33c63f]: Testing, no-op
01:27 demon@tin: Synchronized wmf-config/InitialiseSettings.php: remove education program from legalteamwiki (duration: 00m 47s)
00:53 demon@tin: Synchronized wmf-config/flaggedrevs.php: killing FR on testwiki (duration: 00m 46s)
00:52 demon@tin: Synchronized dblists/flaggedrevs.dblist: killing FR on testwiki (duration: 00m 47s)
00:05 cwd: turned dedupe off

2017-08-29

23:50 cmjohnson1: reinstalling mw1228
23:34 demon@tin: Synchronized php-1.30.0-wmf.15/extensions/Popups/: swat (duration: 00m 47s)
23:33 demon@tin: Synchronized php-1.30.0-wmf.16/extensions/Popups/: swat (duration: 00m 50s)
21:33 urandom: T169939: Decommission Cassandra: restbase1008-c.eqiad.wmnet
21:16 andrewbogott: restarting designate-sink on labservices1001
21:01 ejegg: updated DjangoBannerStats from 5963e7c to ce95eab
20:56 ejegg: updated payments-wiki from 58d33b8 to bf97cd2
20:49 godog: bounce varnish on cp1074 / cp1099 / cp1072 - mailbox lag
20:38 ppchelko@tin: Finished deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2 (duration: 01m 17s)
20:37 ppchelko@tin: Started deploy [changeprop/deploy@a57c79d]: Release a redis-based deduplicator in test mode. Attempt 2
20:36 ppchelko@tin: Finished deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode (duration: 04m 28s)
20:31 ppchelko@tin: Started deploy [changeprop/deploy@ed0fadc]: Release a redis-based deduplicator in test mode
19:47 urandom: T169939: Decommission Cassandra: restbase1008-b.eqiad.wmnet
19:34 ottomata: restarting analytics kafka brokers and all mirror maker processes to apply https://gerrit.wikimedia.org/r/#/c/374610/
19:12 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.16
19:08 ppchelko@tin: Finished deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config (duration: 06m 48s)
19:02 ottomata: restarting main-eqiad -> analytics kafka mirror maker processes on analytics kafka brokers, something is not working...
19:01 XenoRyet: updated Smashpig from 98c5516 to 8f4388d
19:01 ppchelko@tin: Started deploy [restbase/deploy@7f2e55f]: Update CXServer endpoints config
18:19 robh: attempting firmware upgrade on scs-a8-eqiad
18:17 Pchelolo: depooling restbase1008,1009,1010,2003,2005 while cluster reshaping is going on
18:15 demon@tin: Finished scap: bootstrap wmf.16 (duration: 45m 27s)
18:15 urandom: T169939: Decommission Cassandra: restbase1008-a.eqiad.wmnet
17:30 demon@tin: Started scap: bootstrap wmf.16
17:23 ejegg: restarted donation queue consumer and thank you mailer
17:20 Reedy: Disabled oathauth for KartikMistry on wikitech
16:33 ejegg: updated civicrm from 7dfea0d to f00611d
16:32 ejegg: turned off donation queue consumer and thank you mailer
16:04 ejegg: restarted ingenico recurring charge job
15:54 urandom: T169939: Decommission Cassandra: restbase2005-c.codfw.wmnet
14:51 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from db1047 (archived on HDFS) - T172322
14:33 marostegui: Shutdown db1055 to replace its BBU - T174265
13:12 zeljkof: EU SWAT finished
13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add several HD logos (T150618) (duration: 00m 43s)
12:37 elukey: restart kafka daemons on kafka1014 for jvm security updates
12:31 godog: bounce pybal on lvs1003 to pick up config changes
11:49 elukey: restart kafka daemons on kafka1013 for jvm security updates
11:43 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1168.eqiad.wmnet
11:30 elukey: restart java daemons on analytics100[1,2] (Hadoop Master nodes) for jvm updates
11:08 marostegui: Stop MariaDB on db1097 to migrate it to file per table - T161088
10:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T161088 (duration: 00m 43s)
10:48 moritzm: installing libxml2 security updates
10:31 moritzm: reimaging mw1168 (video scaler) to jessie
10:16 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1028 (duration: 00m 42s)
10:15 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1028 (duration: 02m 48s)
09:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T168661 (duration: 00m 43s)
09:39 marostegui: Update MariaDB on db1064 to 10.0.32 - T168661
09:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 for a mariadb upgrade - T168661 (duration: 00m 43s)
09:33 jmm@puppetmaster1001: conftool action : set/pooled=yes; selector: mw1169.eqiad.wmnet
09:29 elukey: re-installed pmacct/librdkafka1/kafkacat on rhenium with stretch versions - T173489
09:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight - T168661 (duration: 00m 43s)
08:53 ema: upgrading cache_text to varnish 4.1.8-1wm1
08:45 akosiaris: reprepro copy calico, calico-cni from jessie-wikimedia to stretch-wikimedia (apt.wikimedia.org) T170119
08:43 hashar: Restarting Jenkins for openjdk update
08:42 elukey: restart yarn/hdfs daemons for openjdk security updates
08:42 akosiaris: upload kubernetes_1.7.4-1 to apt.wikimedia.org/stretch-wikimedia/main T170119
08:42 moritzm: restarting archiva to pick up openjdk security update
08:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase db1091 weight - T168661 (duration: 00m 43s)
08:35 gehel: restart wdqs-updater on wdqs2001
08:32 hashar: apt-get upgrade on contint1001 and contint2001
08:20 moritzm: reimaging mw1169 (video scaler) to jessie
08:09 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight - T168661 (duration: 00m 43s)
08:06 elukey: drop log.MobileWebUIClickTracking_10742159_15423246 from dbstore1002 to free space (table archived on HDFS) - T172322 T168303
07:41 marostegui: Upgrade MariaDB on db1091 to 10.0.32 - T168661
07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097 - T168661 (duration: 00m 42s)
06:56 moritzm: installing ghostscript security updates on trusty (Debian already fixed)
06:09 demon@tin: Synchronized scap/plugins/clean.py: no-op, for consistency (duration: 00m 43s)
06:05 marostegui: Restart MariaDB on db1102 and db1095 to pick up new replication filters - T174385
05:49 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 (duration: 02m 50s)
02:34 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 29 02:34:09 UTC 2017 (duration 6m 44s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 08m 00s)
00:25 urandom: T169939: Decommissioning restbase1010-a.eqiad.wmnet

2017-08-28

23:37 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/: consistency (duration: 00m 43s)
23:31 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 43s)
23:27 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: output (duration: 00m 42s)
23:23 reedy@tin: Synchronized wmf-config/CommonSettings.php: abc2ly in firejail (duration: 00m 43s)
23:22 reedy@tin: Synchronized php-1.30.0-wmf.15/extensions/Score/Score.body.php: Fix escaping (duration: 00m 43s)
23:13 cwd: re-enabled dedupe jobs
23:13 ebernhardson@tin: Synchronized php-1.30.0-wmf.15/extensions/WikimediaEvents/: Turn off two cirrus AB tests T171742 T171214 (duration: 00m 44s)
23:08 ebernhardson@tin: Synchronized docroot/noc/conf/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
23:07 ebernhardson@tin: Synchronized dblists/categories-rdf.dblist: T173892: Add list for wikis that would have categories dumped into RDF (duration: 00m 43s)
22:56 ejegg: turned fundraising orphan rectifier back on
22:46 urandom: T169939: Decommissioning restbase1010-b.eqiad.wmnet
21:48 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: I1c9c9c0e - Enable jQuery 3 on nlwiki, svwiki, plwiki (duration: 00m 44s)
21:37 reedy@tin: Synchronized wmf-config/CommonSettings.php: Disable wgScoreAbc2Ly firejail T172582 (duration: 00m 44s)
21:06 ejegg: switched nightly Ingenico audit from wr1 to wx file parsing
21:06 reedy@tin: Synchronized wmf-config/CommonSettings.php: Run Score binaries in firejail T172582 (duration: 00m 43s)
21:01 reedy@tin: Synchronized wmf-config/CommonSettings.php: Move email blacklist to meta:Email_blacklist (duration: 00m 45s)
20:53 XioNoX: pushing "aggregate defaults discard" to all the cr* routers - T174364
20:52 urandom: T169939: Decommissioning restbase1010-c.eqiad.wmnet
20:16 ayounsi@tin: Finished deploy [librenms/librenms@5fea59c]: (no justification provided) (duration: 00m 05s)
20:16 ayounsi@tin: Started deploy [librenms/librenms@5fea59c]: (no justification provided)
20:16 XioNoX: upgrading librenms to 1.31.01
20:08 bd808@tin: Finished deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes) (duration: 00m 26s)
20:07 bd808@tin: Started deploy [striker/deploy@47b5e8a]: Deploying 47b5e8a (various bug fixes)
19:24 ejegg: updated CiviCRM from 200f34b to 7dfea0d
19:24 ottomata: restarting eventbus in eqiad to apply https://gerrit.wikimedia.org/r/#/c/372179/
19:23 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Grant arbcomers abusefilter-view-private and abusefilter-log-private at cswiki T174357; Enable popups on en and de wiki for A/B test T172291 (duration: 00m 43s)
19:15 herron: scb1001: restarted pdfrender service
19:14 ottomata: restarting eventbus in codfw to verify https://gerrit.wikimedia.org/r/#/c/372179/
19:09 ottomata: rolling restart and rebalances of main-* kafka clusters for https://gerrit.wikimedia.org/r/#/c/372179/
19:03 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: Fix exception on some combination of quotes T174060 (duration: 00m 44s)
18:48 gwicke: restarted pdfrender instances in eqiad (T159922)
18:48 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: pagePreviews: remove invalidated popup sampling rate variables https://gerrit.wikimedia.org/r/#/c/373171/7 (duration: 00m 43s)
18:47 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 02s)
18:47 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
18:43 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: 2d83569397 - Fix old regression in HTMLCacheUpdate de-duplication (duration: 00m 44s)
18:37 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Fix incorrect Special:UserLogin in Popups blacklist https://gerrit.wikimedia.org/r/#/c/373696/ (duration: 00m 43s)
18:33 XioNoX: upgrading librenms to 1.31
18:32 ayounsi@tin: Finished deploy [librenms/librenms@8c9da11]: (no justification provided) (duration: 00m 05s)
18:32 ayounsi@tin: Started deploy [librenms/librenms@8c9da11]: (no justification provided)
18:30 ejegg: updated CiviCRM from c30ec13 to 200f34b
18:21 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable wgEchoPerUserBlacklist at all wikis T173838 (duration: 00m 43s)
18:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 45s)
18:14 niharika29@tin: Synchronized static/images/mobile/copyright/: Upload wikipedia wordmark zh https://gerrit.wikimedia.org/r/#/c/373948 (duration: 00m 44s)
17:59 XioNoX: pushing "aggregate defaults discard" to *ams - T174364
17:56 urandom: T169939: Decommission Cassandra: restbase2005-b.codfw.wmnet
17:31 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
17:19 gehel: restarting wdqs-blazegraph and wdqs-updater for config change
17:11 XioNoX: pushing "aggregate defaults discard" to cr2-knams - T174364
17:09 gehel@tin: Finished deploy [wdqs/wdqs@90f4e2d]: (no justification provided) (duration: 02m 27s)
17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
17:07 gehel@tin: Started deploy [wdqs/wdqs@90f4e2d]: (no justification provided)
16:46 demon@tin: Pruned MediaWiki: 1.30.0-wmf.10 (duration: 02m 41s)
16:39 demon@tin: Pruned MediaWiki: 1.30.0-wmf.14 [keeping static files] (duration: 02m 00s)
14:39 elukey: restart eventlogging_sync on dbstore1002 - issue after drop of old table
14:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db1099 with normal weight on s5 - T172679 (duration: 00m 44s)
14:31 elukey: drop PageContentSaveComplete_5588433_15423246 from the log database on db1046 (m4-master) - T170720
14:28 zeljkof: EU SWAT finished
14:28 zfilipin@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue/jobs/HTMLCacheUpdateJob.php: SWAT: Disable rebound CDN purges for backlinks in HTMLCacheUpdateJob (T173710) (duration: 00m 45s)
14:01 zeljkof: extending EU SWAT for 10 or so minutes to deploy two more patches
13:55 elukey: restart kafka* daemons on kafka1012 for openjdk security updates (canary)
13:52 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Log WikibaseQualityConstraints (T171281) (duration: 00m 44s)
13:42 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: Automatically include commons and wikidata in $wmgThrottlingExceptions (T163872) (duration: 00m 44s)
13:36 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
13:34 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
13:33 zfilipin@tin: Synchronized wmf-config/throttle-analyze.php: SWAT: throttle.php: Separate the throttling definitions from the exception values itself (T167040) (duration: 00m 44s)
13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysops to grant/remove transwiki user group in dtywiki (T174226) (duration: 00m 44s)
13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add 4 URLs to $wgCopyUploadsDomain whitelist (T174152) (duration: 00m 45s)
13:09 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Remove non-transparent background from dty.wiki logos (T174098) (duration: 00m 45s)
12:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
12:43 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw1259.eqiad.wmnet
12:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add more weight to db1099 on s5 - T172679 (duration: 00m 44s)
12:07 gehel: restarting nginx for upgrade on wdqs*
12:05 gehel: restarting nginx for upgrade on elastic* / relforge*
12:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1099 pooled with low weight to s5 - T172679 (duration: 00m 45s)
11:05 moritzm: installing libxml2 security updates
08:41 moritzm: restarting squid3 on install1002
08:16 marostegui: Ugprade MariaDB on s4 codfw master - db2051 to 10.0.32 - T168661
08:10 marostegui: Stop MySQL on db1045 - T172679
08:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1045 to clone db1099 from it - T172679 (duration: 00m 46s)
07:51 moritzm: reimaging mw1259 to jessie (T145742)
07:22 marostegui: Force re-learn cycle on db1055 - https://phabricator.wikimedia.org/T174265
07:12 moritzm: installing openjdk security updates on notebook* hosts
07:05 moritzm: installing openjdk security updates on meitnerium
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 28 02:36:23 UTC 2017 (duration 6m 54s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 09m 36s)

2017-08-27

20:35 ariel@tin: Finished deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete (duration: 00m 02s)
20:34 ariel@tin: Started deploy [dumps/dumps@39f9b52]: write output files to temp location and move into place when complete
15:46 akosiaris: upload kubernetes_1.4.6-7 to apt.wikimedia.org/jessie-wikimedia/main T170346
05:30 marostegui: Force BBU relearn on db1055 - T174265
00:45 jynus: correction last log s/db1051/db1055/
00:42 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1051, hw issues, may get lag (duration: 00m 44s)

2017-08-26

15:46 godog: bounce varnish on cp1073 - mailbox lag

2017-08-25

19:09 gehel: actually not killing the "stuck discovery report-updater process on stat1005", it is already gone - T174110
19:07 gehel: kill stuck discovery report-updater process on stat1005 - T174110
16:03 moritzm: created cn=ciadmin group in LDAP to be used for privileged Jenkins access, initial members in ticket (T169557)
14:29 godog: bounce varnish on cp1049 / cp1072 - mailbox lag
14:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db1097 - T168661 (duration: 00m 44s)
13:53 marostegui: Upgrade MariaDB on db1097 to 10.0.32 - T168661
13:53 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db1097 - T168661 (duration: 00m 44s)
13:40 marostegui: Upgrade MariaDB to 10.0.32 on db2019 - T168661
13:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2037 - T168661 (duration: 00m 45s)
13:24 marostegui: Upgrade MariaDB on db2037 to 10.0.32 - T168661
13:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2037 - T168661 (duration: 00m 44s)
12:51 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2044 - T168661 (duration: 00m 44s)
12:29 marostegui: Upgrade MariaDB on db2044 - T168661
12:29 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2044 - T168661 (duration: 00m 43s)
12:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2058 - T168661 (duration: 00m 48s)
11:48 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1028 (duration: 00m 47s)
09:19 moritzm: restart thumbor to pick up libxml2 security updates
09:12 godog: reimage restbase2001 to test new partman recipe - T169939
08:41 moritzm: restarting apache/hhvm on canary app servers to pick up libxml security update
08:00 volans: upgrading cumin to v1.0.0 on sarin/neodymium
07:55 gehel: reimaging logstash1006 after change of failed disk - T173679
07:49 jynus@tin: Synchronized wmf-config/db-eqiad.php: pool db1069 with more weight (duration: 00m 44s)
07:00 jynus@tin: Synchronized wmf-config/db-eqiad.php: decom db1033, pool db1069 with low weight (duration: 00m 44s)
06:55 jynus@tin: Synchronized wmf-config/db-codfw.php: decom db1033 (duration: 00m 43s)
06:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: formatting fixes (duration: 00m 44s)
06:11 jynus@tin: Synchronized wmf-config/db-codfw.php: repool es2013, formatting fixes (duration: 00m 44s)
06:01 marostegui: Upgrade MariaDB to 10.0.32 on db2058 - T168661
06:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2058 - T168661 (duration: 00m 43s)
05:30 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 - T168661 (duration: 00m 44s)
05:18 moritzm: re-enable mw1260 (video scaler) for active job processing
05:16 marostegui: Reboot db2065 to pick up new kernel
05:08 marostegui: Upgrade MariaDB on db2065 - T168661
05:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 for a MariaDB upgrade - T168661 (duration: 00m 44s)
01:12 mattflaschen@tin: Synchronized php-1.30.0-wmf.15/extensions/Flow/modules/engine/components/board/features/flow-board-loadmore.js: T173807: Fix Flow infinite scroll (duration: 00m 45s)

2017-08-24

23:17 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/TimedMediaHandler.php: SWAT: Disable Ogg Theora video transcodes in default config T172445 (duration: 00m 45s)
22:45 demon@tin: Pruned MediaWiki: 1.30.0-wmf.13 [keeping static files] (duration: 02m 01s)
22:24 ejegg: updated civicrm from fa882f2 to c30ec13 (pt-br and ja thank you letters)
21:59 gwicke: restarted pdfrender service instances in eqiad / T159922
21:54 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: https://gerrit.wikimedia.org/r/#/c/373701/1 (duration: 00m 45s)
21:13 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Logging: https://gerrit.wikimedia.org/r/#/c/373691/ (duration: 00m 44s)
20:07 andrewbogott: stopping, starting, restarting etc. nodepool
19:58 urandom: T169939: Decommission Cassandra: restbase2003-c.codfw.wmnet
19:24 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.15
19:14 andrewbogott: restarting rabbitmq-server on labcontrol1001
18:38 ejegg: updated payments-wiki from bd9f730 to 58d33b8
18:38 bblack: varnish backend restart - cp1074 - mailbox lag
18:24 urandom: T169939: Decommission Cassandra: restbase2003-b.codfw.wmnet
17:33 arlolra@tin: Finished deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f (duration: 12m 24s)
17:21 arlolra@tin: Started deploy [parsoid/deploy@bd12f8a]: Updating Parsoid to 538dad7f
17:01 jynus: stopping and cloning db1033 to db1069
16:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give normal weight to db1096 - T172679 (duration: 00m 47s)
16:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
16:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some more weight to db1096 - T172679 (duration: 00m 47s)
16:34 urandom: T169939: Decommission Cassandra: restbase2003-a.codfw.wmnet
16:24 andrewbogott: apt-get upgrade and updated to MW 1.29.1 on wikitech-static. Rebooting to pick up kernel updates.
15:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Give some weight to db1096 - T172679 (duration: 00m 47s)
15:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1096 depooled to s5 - T172679 (duration: 00m 47s)
14:44 ladsgroup@tin: Synchronized php-1.30.0-wmf.15/extensions/Wikidata/extensions/Wikibase/client/includes/Changes/WikiPageUpdater.php: Reduce batch size in WikiPageUpdater (T173710) (duration: 00m 48s)
14:14 phuedx: refreshLinks on bawiki (T173994)
14:13 volans: updated cumin package in our APT to version cumin_1.0.0-1
13:19 phuedx@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T171853: Re-enable Page Previews for enwiki and dewiki on the Beta Cluster (duration: 00m 47s)
13:13 phuedx@tin: Synchronized portals: (no justification provided) (duration: 00m 49s)
13:12 phuedx@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 49s)
12:45 gehel: killing discovery report updater on stats1005 (stuck since Aug 15) - T173333
12:30 marostegui: Stop MySQL on db1026 to clone db1096 from it - T172679
12:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1026 to clone db1096 from it T172679 (duration: 00m 47s)
11:36 jynus: disable puppet on db1069 in preparation for reimage
11:08 marostegui: Global rename Papa1234 → Karl-Heinz Jansen - T173859
09:56 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with full weight (duration: 00m 46s)
09:33 volans: executing https://wikitech.wikimedia.org/wiki/Cumin#Run_Puppet_only_if_last_run_failed to get rid of failed ones
08:54 jynus: 16 minute test run of rebuildTermSqlIndex.php on terbium
08:52 marostegui: Drop tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki - T173590
07:24 Amir1: starting the run for rebuildTermIndex (T171460)
02:50 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 24 02:50:40 UTC 2017 (duration 7m 4s)
02:43 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 05m 56s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 09m 50s)
01:46 aaron@tin: Synchronized php-1.30.0-wmf.15/includes/jobqueue: Deploy 752580637 (T173710) (duration: 00m 49s)

2017-08-23

23:49 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
23:47 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: SWAT fixes (duration: 00m 47s)
23:42 maxsem@tin: Synchronized php-1.30.0-wmf.14/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 47s)
23:41 maxsem@tin: Synchronized php-1.30.0-wmf.15/extensions/CodeMirror/: https://gerrit.wikimedia.org/r/#/c/373324/ (duration: 00m 49s)
23:28 maxsem@tin: Synchronized php-1.30.0-wmf.15/resources/: https://gerrit.wikimedia.org/r/#/c/373391/ (duration: 00m 48s)
23:27 maxsem@tin: Synchronized wmf-config/MetaContactPages.php: https://gerrit.wikimedia.org/r/#/c/373369/ (duration: 00m 48s)
20:59 urandom: T169939: Truncating MCS tables
20:26 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.15 (duration: 00m 46s)
20:25 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.15
20:13 jynus@tin: Synchronized wmf-config/db-eqiad.php: repool db1078 with low weight (duration: 00m 47s)
20:12 thcipriani@tin: Finished scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520 (duration: 04m 11s)
20:08 thcipriani@tin: Started scap: Revert ProofreadPage to db7507246665e69384c1d92af2aedc62263a5116 for wmf.15 T173520
19:49 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change: pagePreviews: Enable A/B test (BC-only (duration: 00m 47s)
19:45 joal@tin: Finished deploy [analytics/refinery@f467ce1]: Bug fixing deploy (duration: 03m 16s)
19:44 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend: Verify the existence of `url` key when parsing lang objects T172316 (duration: 00m 56s)
19:42 joal@tin: Started deploy [analytics/refinery@f467ce1]: Bug fixing deploy
19:36 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler: Enable WebM playback via ogv.js T172444 (duration: 00m 50s)
19:22 kaldari: foreachwiki sql.php /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
19:19 kaldari: mwscript sql.php --wiki=testwiki /srv/mediawiki/php/maintenance/archives/patch-ip_changes.sql
19:15 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Remove CitThisPage from blacklist for page previews T173865 (duration: 00m 47s)
19:13 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 49s)
19:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/TimedMediaHandler/: Enable WebM playback via ogv.js https://gerrit.wikimedia.org/r/#/c/373126/ (duration: 00m 50s)
19:00 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/MobileFrontend/: Verify the existence of key when parsing lang objects https://gerrit.wikimedia.org/r/#/c/373292/ (duration: 00m 49s)
18:57 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Log channel for LoginNotify T173888 (duration: 00m 47s)
18:55 ppchelko@tin: Finished deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains (duration: 01m 33s)
18:54 ppchelko@tin: Started deploy [changeprop/deploy@1998c10]: [Config] Disable mobile rerenders for non-wikipedia domains
18:52 niharika29@tin: Synchronized php-1.30.0-wmf.15/extensions/LoginNotify/: Log everything T173888 (duration: 00m 48s)
18:43 gehel: manually running report updater for discovery golden data on stat1005 - T173333
18:40 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
18:40 gehel: resetting permissions on stat1005:/srv/published-datasets/discovery - T173333
18:39 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Reinforce LoginNotify settings https://gerrit.wikimedia.org/r/#/c/372555/ (duration: 00m 47s)
18:32 thcipriani: jenkins restarted, zuul should pick-up queue
18:26 thcipriani: jenkins appears to be having some issues, trying to dump threads now and then restart to see if it helps
18:13 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Allow crats to add people to accountcreator group on mw.or https://gerrit.wikimedia.org/r/#/c/373317/ (duration: 00m 49s)
17:40 bblack: reboot cp4021
16:38 cwd: disabled all dedupe
16:26 demon@tin: Pruned MediaWiki: 1.30.0-wmf.11 [keeping static files] (duration: 01m 32s)
16:24 bd808@tin: Finished deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845) (duration: 00m 39s)
16:23 bd808@tin: Started deploy [striker/deploy@2f2dd7c]: Deploying 2f2dd7c "Tool account creation and more" (T128400, T149458, T159044, T164847, T167931, T168480, T173845)
16:22 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 (duration: 03m 44s)
15:35 legoktm: added addshore to "integration" gerrit group (T173233)
15:10 gehel: shutdown logstash1006 for disk replacement - T173679
14:49 moritzm: installing texlive security updates on trusty (Debian already fixed)
14:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp4027.ulsfo.wmnet
14:34 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp4027.ulsfo.wmnet
13:31 marostegui: Stop MySQL on db1041 to get it ready for decommission - T173915
13:13 zeljkof: EU SWAT finished
13:10 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Flow as a Beta feature on wawiki and wawikionary (T172947) (duration: 00m 48s)
12:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
12:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1041 to decommission it - T173915 (duration: 00m 48s)
10:39 jynus: stopping and deleting s1 and s4 from dbstore2001
10:06 joal@tin: Finished deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last) (duration: 03m 30s)
10:03 moritzm: upgrading remaining image scalers to luasandbox 2.0.13
10:02 joal@tin: Started deploy [analytics/refinery@84d6ee4]: Regular deploy (1 month since last)
09:36 godog: kill older and running tasks from gerrit queue, it was stuck
09:10 moritzm: upgrading remaining API servers in to luasandbox 2.0.13
09:08 godog: upload prometheus-blackbox-exporter 0.7.0+ds1-1~wmf1 to jessie-wikimedia, backported - T169860
08:35 marostegui: Stop Upgrade MySQL on db2073 to 10.0.32 - T168661
08:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T168661 (duration: 00m 59s)
08:18 moritzm: upgrading remaining job runners to luasandbox 2.0.13
07:12 marostegui: Global rename: Opdire657 → Sakiv - T173834
06:55 moritzm: upgrading remaining app servers in to luasandbox 2.0.13
03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 23 03:15:36 UTC 2017 (duration 7m 6s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.15) (duration: 15m 11s)
02:32 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 03s)

2017-08-22

23:39 thcipriani@tin: Synchronized php-1.30.0-wmf.15/extensions/Linter/includes/LintErrorsPager.php: SWAT: Fix up 11f4a97ba6bcd0c1de (duration: 00m 49s)
23:35 thcipriani@tin: Finished scap: SWAT: Fix typo in the notification message (duration: 21m 48s)
23:13 thcipriani@tin: Started scap: SWAT: Fix typo in the notification message
23:10 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/Popups/includes/PopupsHooks.php: SWAT: Remove aborting of BeforePageDisplay hook T173411 (duration: 00m 49s)
21:18 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Retry CodeMirror deployment T170966 (duration: 00m 49s)
20:41 ejegg: updated CiviCRM from 4aa177b to fa882f2
20:25 mutante: restbase-dev* - puppet runs fail due to E: Version '3.11.0' for 'cassandra' was not found
20:00 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.30.0-wmf.15
19:56 urandom: starting cassandra restbase2004-a and restbase2006-c, OOMs
19:46 thcipriani@tin: Finished scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache (duration: 45m 55s)
19:00 thcipriani@tin: Started scap: testwiki to php-1.30.0-wmf.15 and rebuild l10n cache
18:50 robh: labservies1002 update completed (not 1003, typo)
18:50 robh: labservies1003 update completed
18:40 robh: firmware update of labservices1002 in progress
17:54 andrewbogott: removing obsolete apache2 and puppetmaster packages from labcontrol boxes for https://phabricator.wikimedia.org/T171786
17:07 thcipriani: starting branch cut for 1.30.0-wmf.15
13:58 godog: bounce varnish on cp1074 / cp1049 / cp1073 - mailbox problems
13:48 zeljkof: EU SWAT finished
13:45 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
13:44 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: relatedArticles: Tidy up config (T165991) (duration: 00m 44s)
13:29 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wmgUseWikimediaShopLink to true for ptwiki (T173768) (duration: 00m 45s)
10:41 moritzm: upgrading hhvm-luasandbox on mw1293-mw1295 (image scalers)
10:21 Amir1: another run of rebuildTermSqlIndex (T171460)
09:54 moritzm: upgrading hhvm-luasandbox on mw1189-mw1208 (API servers)
09:42 marostegui: Renaming user Darwinius → DarwIn - T173159
09:35 moritzm: upgrading hhvm-luasandbox on mw1180-1188 and mw1209-mw1220 (app servers)
09:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1064 - T172996 (duration: 00m 44s)
08:59 akosiaris: upload apertium-bel-rus_0.2.0~r81186-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
08:55 moritzm: upgrading hhvm-luasandbox on mw1161-mw1167 (job runners)
08:43 akosiaris: upload apertium-rus_0.1.0~r81184-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
08:43 akosiaris: upload apertium-bel_0.1.0~r81357-1+wmf to apt.wikimedia.org/jessie-wikimedia/main
08:35 godog: bounce varnish on cp1072 - mailbox problems
08:33 godog: bounce varnish on cp4015 - mailbox problems
08:24 marostegui: Drop s4 from db1095 with NO replication - T172996
08:23 moritzm: upgrading hhvm-luasandbox on deployment servers / script runners
07:34 marostegui: Stop replication on db1064 sanitarium2 and sanitarium3 master to move labsdb1009,10 and 11 s4 from db1095 to db1102 - https://phabricator.wikimedia.org/T172996
07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 - T172996 (duration: 00m 52s)
07:28 moritzm: installing ruby security updates on trusty (Debian already fixed)
07:11 moritzm: installing c-ares security updates on trusty (Debian already fixed)
06:54 moritzm: installing graphite2 (the image library, not the metrics tool) security updates on trusty (Debian already fixed)
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 22 02:30:47 UTC 2017 (duration 6m 55s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 07m 11s)

2017-08-21

22:50 robh: rebooting tin for firmware update since its idle
22:20 mutante: praseodymium - installing BIOS upgrade, reboot
22:11 mutante: xenon - installing BIOS upgrade
21:58 mutante: cerium (cassandra test) - rebooting for firmware upgrade
21:54 mutante: cerium - installing Dell BIOS upgrade (T162850)
21:26 mutante: bast2001 - running Dell BIOS firmware upgrade (T162850)
21:20 arlolra@tin: Finished deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b (duration: 10m 59s)
21:09 arlolra@tin: Started deploy [parsoid/deploy@2210a38]: Updating Parsoid to 28a9a22b
18:40 niharika29@tin: Synchronized dblists/pp_stage1.dblist: Roll page previews out to all wikis except en and de wiki T162672 (duration: 00m 44s)
18:34 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
18:33 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OOjs UI EditPage on all wikis https://gerrit.wikimedia.org/r/#/c/366868/ (duration: 00m 44s)
18:30 niharika29@tin: Synchronized wmf-config/Wikibase.php: Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList T173571 (duration: 00m 44s)
18:25 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Increase AbuseFilter autodisable thresholds for Meta-Wiki T173633 (duration: 00m 44s)
18:22 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Add CookBook and Cookbook Talk NS on hiwikibooks T173398 (duration: 00m 45s)
18:16 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Administrators to add/remove 'transwiki' at nowiktionary T172365 (duration: 00m 45s)
18:12 niharika29@tin: Synchronized static/images/: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
18:11 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Set project logo for wikimania2018wiki T173042 (duration: 00m 44s)
17:22 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038 (duration: 00m 14s)
17:22 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Bring back cxserver on scb2001 to a stable state - T173038
17:12 gehel@tin: Finished deploy [wdqs/wdqs@a1c4f1f]: (no justification provided) (duration: 02m 27s)
17:10 gehel@tin: Started deploy [wdqs/wdqs@a1c4f1f]: (no justification provided)
15:40 bblack: cp1099 - varnish backend restart for mailbox lag
14:55 mobrovac: cxserver depool scb2001 to debug failed checks - T173038
14:54 mobrovac@tin: Finished deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038 (duration: 00m 18s)
14:53 mobrovac@tin: Started deploy [cxserver/deploy@1065ffe]: Deploy 1065ffe2 to canary scb2001 for debugging - T173038
14:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T163190 (duration: 00m 44s)
14:04 zeljkof: EU SWAT finished
14:04 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
14:02 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add HD logos for srwikisource, update them too (T172268) (duration: 00m 44s)
13:53 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 49s)
13:52 jynus: stop dbstore2001 mariadb@s4, start mariadb@s7
13:52 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwikinews, add HD version for them (T172255) (duration: 00m 44s)
13:46 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set X-Frame-Options: SAMEORIGIN if UploadWizard enabled (T173631) (duration: 00m 44s)
13:46 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on db1047: T170990
13:37 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 44s)
13:36 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Update logos for srwiktionary, add HD logos for srwiktionary (T172245) (duration: 00m 45s)
13:27 jynus: stop dbstore2001 mariadb@s7
13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
13:26 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Add new logo for the Baskhir Wikibooks (T173471) (duration: 00m 44s)
13:26 ottomata: adding index on (database, rev_timestamp) on mediawiki_page_create_2 table on dbstore1002: T170990
13:11 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
13:10 zfilipin@tin: Synchronized dblists/closed.dblist: SWAT: Reopen bawikibooks (T173471) (duration: 00m 44s)
12:52 marostegui: Stop replication on db1079 and db1041 to compare their data - T163190
12:52 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T163190 (duration: 00m 45s)
12:35 marostegui: Stop MySQL on db1015 to decommission it - T173570
09:50 jynus: restart dbstore2001 mariadb@s1
09:40 jynus: restart dbstore2001 mariadb@x1
09:29 moritzm: upgrading hhvm-luasandbox on mw1262-1265
09:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1015 - T173570 (duration: 00m 44s)
09:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1015 - T173570 (duration: 00m 44s)
09:01 moritzm: upgrading hhvm-luasandbox on mw1261
08:08 marostegui: Rename tables article_assessment, article_assessment_pages, article_assessment_ratings tables from testwiki on db1078 - T173590
07:42 marostegui: Drop cx_drafts table from x1 - T172364
07:13 marostegui: Stop s4 replication thread on db1095 - T172996
07:09 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2047 (duration: 00m 45s)
02:36 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 21 02:36:58 UTC 2017 (duration 7m 3s)
02:29 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 08m 57s)

2017-08-19

13:17 Amir1: another run: (Hotwc3 → HotWC3) (Lamia Bahy → Albedo11) (Monóxido de carbono → Roquetero) (PaulMichaels → PaulBenario) (Rodrigo.dst → RodrigoTavares) (Sadia Tasnim (Moyna) → মুহাম্মদ সুমন মাহমুদ) (Syou 18331322 → Ms3102) (TzvetelinaOOD1 → Tzveti1) (World Para Taekwondo → TKD at World Para Taekwondo) (Yaellerner → Ya1levy777) (平井俊光 → Toshimit) (T173419)
13:00 Amir1: running the script for ("Clopper228" "CGminded") and ("Gregory.lussier" "StevenSmith83473") (T173419)
12:47 Amir1: ladsgroup@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki "Bolsée" "Kathmandu2017" (T173419)
03:21 krinkle@tin: Synchronized wmf-config/InitialiseSettings.php: Enable jQuery 3 on test.wikidata and mediawiki.org - Id4d42d8c53 (duration: 00m 45s)

2017-08-18

21:05 mutante: gerrit2001 - restarted gerrit again after reverting gerrit:372426, using systemctl commands, not 'service' or init.d
20:55 mutante: restarting gerrit on gerrit2001
20:48 demon@tin: Synchronized scap/plugins/clean.py: Completeness (duration: 00m 44s)
20:43 RainbowSprinkles: prior thing was a no-op, testing
20:42 demon@tin: Pruned MediaWiki: 1.30.0-wmf.9 [keeping static files] (duration: 01m 43s)
19:03 bblack: cp1074 - varnish backend restart (mailbox lag)
19:01 bblack: reboot cp4021-8
18:21 bblack: puppeting cp4021-8 - expect possibility of ipsec alerts, etc...
13:32 Amir1: another run of /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item --from-id $(tail -100 /tmp/rebuildTermSqlIndex.log | grep -E "Processed up to page (\d+?)" | sed -E "s/Processed up to page //; s/ \(Q.+?//" | tail -1) >>/tmp/rebuildTermSqlIndex.log 2>&1
13:26 Amir1: ladsgroup@terbium:~$ mwscript namespaceDupes.php --wiki=hiwikiversity (T172977)
12:44 Amir1: start of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item
11:36 jynus: change topology of dbstore2001:x1 and dbstore2002:x1
11:00 elukey: reboot dbstore1002 for kernel updates
10:34 elukey: restart mysql on dbstore1002 - attempt to reclaim space after big table drop (stop slaves and el_sync, check running queries, stop mysql, check process, start mysql)
09:55 Amir1: one small pass of ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=entity (T171460)
09:46 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T171460)
09:42 moritzm: installing libmspack security updates
09:16 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2077 - T168409 (duration: 00m 44s)
07:28 marostegui: Stop MySQL on db2077 to copy it to dbstore2001 - T168409
07:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2077 - T168409 (duration: 00m 44s)
07:20 moritzm: installing openjdk-7 security updates on trusty
06:58 jynus: and nginx
06:55 jynus: systemctl restart squid3.service on install2002, apt seem stuck on some servers
06:24 jynus: stopping and upgrading db2033
05:43 jynus: upgrading and restarting all mariadb instances on dbstore2002
afk: updated fundraising dash from 696a3ff to 7dfc969
00:48 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (again) to 1k per bucket per day (duration: 00m 44s)
00:18 ebernhardson@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171213: Increase sampling rate of cirrus satisfaction schema (duration: 00m 44s)

2017-08-17

23:31 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Revert "Disable cirrus MLR ab test" (duration: 00m 44s)
23:21 thcipriani@tin: Synchronized php-1.30.0-wmf.14/resources/src/mediawiki.rcfilters/dm/mw.rcfilters.dm.FilterGroup.js: SWAT: RCFilters: Fix validation for single_option groups T173303 (duration: 00m 44s)
23:14 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Reapply "Remove temporary wgStructuredChangeFiltersEnableExperimentalViews setting" (duration: 00m 45s)
22:10 ppchelko@tin: Finished deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping (duration: 01m 14s)
22:09 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/specials/SpecialNewpages.php: Restore the newFromId() approach in SpecialNewpages::feedItemDesc T173541 (duration: 00m 46s)
22:09 ppchelko@tin: Started deploy [changeprop/deploy@2c553a6]: Lower the concurrecy for transcludes to decrease cassandra load during cluster reshaping
22:07 gwicke: restarting pdfrender on scb100* nodes
21:38 ejegg: updated dash from bec0077 to 696a3ff
21:28 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify on sites without Echo too. :|https://gerrit.wikimedia.org/r/#/c/372456/ (duration: 00m 44s)
21:19 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Enable LoginNotify finally! Yippeeeeee https://gerrit.wikimedia.org/r/#/c/372450/ (duration: 00m 45s)
21:17 bblack: cp1063: varnish backend restart (mailbox lag)
20:07 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.30.0-wmf.14
19:28 chasemp: disable puppet for cloud things for some careful refactor merging
19:14 thcipriani@tin: Synchronized php: group1 wikis to wmf.14 (duration: 00m 46s)
19:12 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to wmf.14
19:07 thcipriani@tin: Finished scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520 (duration: 06m 13s)
19:01 thcipriani@tin: Started scap: ProofReadPage Revert to db7507246665e69384c1d92af2aedc62263a5116 T173520
18:28 ebernhardson: restart logstash on logstash1003 to see why its not reading EventError messages from kafka
18:24 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/RevisionSlider/: Revert Reintroduce hover and bar clicking - https://gerrit.wikimedia.org/r/#/c/372384/ (duration: 00m 48s)
18:16 jynus: upgrading and restarting all mariadb instances on dbstore2001
18:12 niharika29@tin: Synchronized php-1.30.0-wmf.14/extensions/LoginNotify/: Log usage statistics https://gerrit.wikimedia.org/r/#/c/372214/ (duration: 00m 51s)
18:03 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/UploadWizard/UploadWizard.config.php: SWAT: Preserve array keys (language keys) when sorting the language dropdown T173522 (duration: 00m 51s)
17:53 jynus: restarting mariadb (dbstore2001:x1) to test new buffer pool configuration
17:20 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.13 now T173520
17:10 RainbowSprinkles: gerrit: cpu spikes, lots of large gc logs. Looking into it. (nb: things might be a little slow, but it is /up/)
16:54 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis back to wmf.14 now for T164173
16:51 thcipriani@tin: Synchronized php-1.30.0-wmf.14/includes/jobqueue/jobs/RefreshLinksJob.php: Avoid lock acquisition errors for multi-title refreshlinks jobs T173462 (duration: 00m 51s)
16:45 ejegg: updated SmashPig from c501f53 to 98c5516
16:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Pool db2077 - T170662 (duration: 00m 49s)
16:24 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2077 - T170662 (duration: 00m 51s)
16:16 urandom: T169939: RESTBase: converting (33) new keyspaces to time-windowed compaction
16:11 urandom: T169939: Lower eqiad compaction throughput from 20MB/s to 15MB/s
15:46 godog: reimage restbase2001 - T169939
15:42 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2001.codfw.wmnet
14:32 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp3036.*
13:30 zeljkof: EU SWAT finished
13:24 zfilipin@tin: Synchronized static/favicon/wikiversity.ico: SWAT: Update wikiversity favicon (T160491) (duration: 00m 50s)
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change $wgArticleCountMethod to any for srwikiquote (T172974) (duration: 00m 51s)
13:08 zfilipin@tin: Synchronized wmf-config/throttle.php: SWAT: Add one throttling exception (T173444) (duration: 00m 51s)
12:38 moritzm: installing libgd/libsoup security updates
12:08 urandom: T169939: Decommissioning Cassandra/restbase2001-c.codfw.wmnet
12:07 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet
10:29 gilles: Thumbor stress test finished
10:20 gilles: Load testing Thumbor
10:13 godog: roll-restart nginx on thumbor to apply https://gerrit.wikimedia.org/r/372199
08:36 godog: roll-restart thumbor to apply webp support change
08:30 godog: restart varnish on cp1062 to fix mailbox lag
08:05 godog: reboot ms-be2021, unreachable on network
06:24 marostegui: Stop slave on db2047 to fix duplicate keys - T151029
05:55 marostegui: Stop replication on db2077 to fix duplicate entries - T151029
05:48 marostegui: Stop replication in sync on db1078 and db1015 - T164488
03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 17 03:11:13 UTC 2017 (duration 7m 18s)
03:03 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 03m 58s)
02:50 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 33s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 52s)
00:45 urandom: T169939: Decommissioning Cassandra/restbase2001-b.codfw.wmnet

2017-08-16

22:30 urandom: T169939: Decommissioning Cassandra/restbase2001-a.codfw.wmnet
22:14 urandom: T169939: Cleaning up wikipedia parsoid snapshots
22:13 urandom: T169939: Rolling restart of Cassandra complete
21:38 bawolff: deleting private info from securepoll_votes that the script missed due to ref-integ issues for old elections (T173393)
21:37 thcipriani: train is on hold pending resolution of T173462
21:33 bawolff: deleting private info in enwiki arbcom1_vote table (T173393)
21:30 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack d
21:22 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: revert group1 wikis to 1.30.0-wmf.14 for T173462
21:21 thcipriani@tin: Synchronized php: revert group1 wikis to 1.30.0-wmf.14 for T173462 (duration: 00m 47s)
20:46 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack b
20:45 arlolra@tin: Finished deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e (duration: 08m 52s)
20:37 arlolra@tin: Started deploy [parsoid/deploy@a9dc803]: Updating Parsoid to 1832a78e
19:57 urandom: T169939: Rolling restart of Cassandra instances, eqiad, rack a
19:52 MaxSem: Manually cleaning up PI on enwiki (T173393)
19:37 thcipriani@tin: Synchronized php: group1 wikis to 1.30.0-wmf.14 (duration: 00m 46s)
19:35 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.14
19:06 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack d
18:57 thcipriani@tin: Synchronized wmf-config/CirrusSearch-common.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART II (duration: 00m 50s)
18:56 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: cirrus Tune ordering of crossproject search results on enwiki T171803 PART I (duration: 00m 51s)
18:41 thcipriani@tin: Synchronized wmf-config: SWAT: Apply token count limits to phrase queries on all wikis T172653 (duration: 00m 53s)
18:30 thcipriani@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 50s)
18:29 thcipriani@tin: Synchronized php-1.30.0-wmf.14/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Disable cirrus MLR ab test T171214 (duration: 00m 51s)
18:20 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable group1 T163380 (duration: 00m 54s)
18:18 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack c
17:31 urandom: T169939: Rolling restart of Cassandra instances, codfw, rack b
17:07 demon@tin: Synchronized php-1.30.0-wmf.14/extensions/AntiSpoof/SpoofUser.php: T173394 (duration: 00m 51s)
15:54 marostegui: Stop MySQL and shutdown db1078 for HW checks - T173365
15:43 elukey: drop PageContentSaveComplete_5588433_15423246 from db1047 and dbstore1002 (analytics-slaves)
15:41 godog: delete outdated CFs cassandra metrics from graphite2002 and graphite1003
15:35 marostegui: Rename cx_drafts table on db1029 - T172364
15:09 godog: restart varnish on cp1072 to clear mailbox lag
14:56 godog: restart varnish on cp1099 to clear mailbox lag
14:48 godog: restart varnish on cp1074 to clear mailbox lag
14:44 godog: restart varnish on cp1049 to clear mailbox lag
14:38 gilles: Thumbor stress test finished
14:29 gilles: Stress testing Thumbor from single IP
13:57 zeljkof: EU SWAT finished
13:35 zfilipin@tin: Synchronized dblists/pp_stage1.dblist: SWAT: pagePreviews: Deploy to next 100 stage 1 wikis (T162672) (duration: 00m 50s)
13:09 zfilipin@tin: Synchronized portals: (no justification provided) (duration: 00m 52s)
13:08 zfilipin@tin: Synchronized portals/prod/wikipedia.org/assets: (no justification provided) (duration: 00m 52s)
12:25 Amir1: ladsgroup@terbium:~$ time /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki testwikidatawiki --entity-type=property (T172776, T171460)
12:25 moritzm: powercycling cp3036
12:24 marostegui: Compressing InnoDB on db2077 - T168409
12:22 jmm@puppetmaster1001: conftool action : set/pooled=no; selector: cp3036.esams.wmnet
10:46 marostegui: Stop replication in sync on db1015 and db1078 - T164488
10:42 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2076 - T170662 (duration: 00m 51s)
10:03 godog: copy ubuntu-font-family-sources to stretch-wikimedia - T170817
08:37 marostegui: Drop wikigrok tables from s1, s3 and s5 - T172020
08:12 marostegui: Stop MySQL on db2047 to copy its content to db2077 - T170662
08:10 moritzm: bounced pdfrender on scb1004 (T159922)
08:07 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2047 - T170662 (duration: 01m 06s)
07:25 marostegui: Stop MySQL on db2076 to copy its content to dbstore2001 - T168409
07:08 elukey: executed sudo find -type f -mtime +30 -exec rm {} \; in /var/log/carbon to free some space
06:40 marostegui: Run pt-table-checksum on s3 for revision table - T164488
06:01 marostegui: Stop replication on db2076 to fix duplicate entries - T151029
03:35 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 16 03:35:43 UTC 2017 (duration 7m 21s)
03:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.14) (duration: 16m 32s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 08m 08s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 51s)
00:43 dereckson@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372088) (duration: 00m 51s)
00:42 dereckson@tin: Synchronized php-1.30.0-wmf.14/extensions/Wikidata/Wikidata.php: Explicitly load badges extension (Gerrit:372051) (duration: 00m 51s)
00:38 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable Timeless on three French wikis (T154371) + Fixes for Wikidata: Remove wbq_evaluation logging, Update Wikidata property blacklist (Gerrit:367913 and Gerrit:370846) (duration: 00m 53s)
00:02 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner on test wikis (T173388) (duration: 00m 51s)

2017-08-15

23:58 dereckson@tin: Synchronized php-1.30.0-wmf.14/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 50s)
23:55 dereckson@tin: Synchronized php-1.30.0-wmf.13/skins/Timeless/resources/screen-common.less: Fix messed up recent changes/watchlist legends (T173151) (duration: 00m 54s)
23:14 Dereckson: Ran updateArticleCount.php on sr.wikiquote (T172974)
23:12 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Create a few of namespace aliases for hiwikiversity (T172977) (duration: 00m 53s)
22:50 ppchelko@tin: Started restart [changeprop/deploy@444223d]: (no justification provided)
22:45 Pchelolo: scb - codfw: enabling puppet and starting changeprop back T169939
21:40 mobrovac: scb - codfw: disabling puppet and stopping changeprop to release load on Cassandra while dropping old tables - T169939
21:26 urandom: Starting Cassandra restbase2005-b
19:24 demon@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.14
19:19 mobrovac@tin: Finished deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939 (duration: 02m 50s)
19:16 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones), part #2 - T169939
19:14 demon@tin: Finished scap: bootstrap wmf.14 v2.0 (duration: 30m 56s)
19:04 mobrovac@tin: Started deploy [restbase/deploy@1139d00]: Use only new parsoid tables (before the removal of old ones) - T169939
19:03 mobrovac@tin: Finished deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939 (duration: 04m 39s)
18:58 mobrovac@tin: Started deploy [restbase/deploy@1139d00] (staging): Use only new parsoid tables (before the removal of old ones) - T169939
18:43 demon@tin: Started scap: bootstrap wmf.14 v2.0
18:39 demon@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_3982832257" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 03m 05s)
18:35 demon@tin: Started scap: bootstrap wmf.14
18:18 demon@tin: Pruned MediaWiki: 1.30.0-wmf.12 [keeping static files] (duration: 02m 47s)
17:59 bsitzmann@tin: Finished deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362) (duration: 04m 00s)
17:55 bsitzmann@tin: Started deploy [mobileapps/deploy@34a1304]: Update mobileapps to 33b80dd (T172829 T152441 T172021 T103362)
15:21 mobrovac: restarting pdfrender on scb1001, added some debug messages to help us diagnose T159922
13:42 gehel: cleanup of old leftover logs on deployment-logstash2:/var/log/logstash
13:21 moritzm: bounced pdfrender on scb1001 (T159922)
11:32 moritzm: installing Linux updates on stretch systems (no reboots yet)
10:45 moritzm: installing PHP security updates on trusty
10:09 jynus: rebooting db1078
09:39 jynus@tin: Synchronized wmf-config/db-eqiad.php: depool db1078 (duration: 00m 49s)
07:56 moritzm: installing cvs security updates
07:18 elukey: restart pdfrender on scb1003
02:56 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 15 02:56:14 UTC 2017 (duration 6m 42s)
02:49 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 53s)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 58s)

2017-08-14

22:16 Dereckson: Previous log entry is related to Gerrit:371960 and Gerrit:371961.
22:15 dereckson@tin: Synchronized php-1.30.0-wmf.11/extensions/EventBus/JobQueueEventBus.php: JobQueueEventBus: Populate the database field + not set properties are accessed (duration: 00m 48s)
20:44 ppchelko@tin: Finished deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation (duration: 09m 35s)
20:35 ppchelko@tin: Started deploy [restbase/deploy@4d6c706]: Temporary fallback to the new storage buckets before truncation
17:09 gehel: resetting mode on stat1005:/srv/published-datasets/discovery recursively
16:29 elukey: execute sudo find -type f -mtime +60 -exec rm {} \; in /var/log/carbon on graphite2001 to free some space in /
13:32 elukey: Execute systemctl mask nfacctd on rhenium.wikimedia.org for T172681
13:08 zeljkof: nothing for EU SWAT
12:21 jynus: stopping replication on all instances of dbstore2001 T169516
11:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2076 - T170662 T151029 (duration: 00m 47s)
11:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2076 - T170662 T151029 (duration: 01m 02s)
10:34 marostegui: Restart MySQL on db1069 to pick up new replcation filters
10:27 marostegui: Restart MySQL on db1095 to pick up new replication filters
09:16 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=thumbor,name=thumbor2001.codfw.wmnet
09:07 _joe_: stopping thumbor on thumbor2001 after depooling it for testing
09:05 oblivian@puppetmaster1001: conftool action : set/pooled=no; selector: service=thumbor,name=thumbor2001.codfw.wmnet
03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 14 03:15:31 UTC 2017 (duration 7m 6s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 06m 47s)
02:39 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 29s)

2017-08-13

22:45 ebernhardson: restart elsaticsearch on elastic1017 after setting md2 readahead to 256 to match md2 on 1032-152
19:36 godog: upload python-thumbor-wikimedia 1.2 - T161719
19:01 godog: bounce pdfrender on scb1001 and scb1003 - T159922
18:46 godog: bounce carbon and uwsgi on graphite1003
17:53 reedy@tin: Synchronized wmf-config/db-eqiad.php: fix comment (duration: 00m 47s)
17:52 reedy@tin: Synchronized wmf-config/db-codfw.php: fix comment (duration: 00m 47s)
17:51 reedy@tin: Synchronized refresh-dblist: phpcs (duration: 00m 48s)

2017-08-12

20:00 krinkle@tin: Synchronized php-1.30.0-wmf.13/includes/jobqueue/JobQueueGroup.php: T171371 - Log job pushes to bogus wikis (duration: 00m 53s)
16:09 Reedy: Deleted some bogus user languages from commonswiki.user_properties
15:25 elukey: powercycle mw2256 (able to use com2 but not to login as root, regular ssh hanging) - T163346

2017-08-11

21:47 legoktm@tin: Finished scap: (no justification provided) (duration: 07m 09s)
21:41 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: mw2256.codfw.wmnet
21:40 legoktm@tin: Started scap: (no justification provided)
21:40 legoktm@tin: scap failed: LockFailedError Failed to acquire lock "/var/lock/scap.unknown-but-probably-mediawiki.lock"; owner is "legoktm"; reason is "Deploying Timeless (try 2) - T154371" (duration: 00m 00s)
21:02 ebernhardson: unban elastic1017 from elasticsearch cluster
20:52 bblack: varnish backend restart on cp1099
20:46 legoktm@tin: Started scap: Deploying Timeless (try 2) - T154371
20:42 legoktm@tin: scap aborted: Deploying Timeless - T154371 (duration: 05m 10s)
20:36 legoktm@tin: Started scap: Deploying Timeless - T154371
20:19 bblack: varnish backend restart on cp1049 + cp1074 (mailbox lag)
19:59 ebernhardson: ban elastic1017 from eqiad search cluster
18:28 moritzm: installing subversion security updates
17:19 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 00m 47s)
17:05 twentyafterfour: Deploying phabricator security update
16:21 jynus_: stop db1069:s6 replication and dropping frwiki, jawiki, ruwiki
15:41 jynus_: stopping db2075 to clone it to dbstore2001
15:39 moritzm: installing git security updates on trusty (jessie/stretch already fixed)
15:38 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 00m 48s)
13:51 elukey: moved the eventbus scap deployment dirs on kafka[12]00[123] to deploy-service:deploy-service to allow scap to depool/pool - T171506
13:49 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 27s)
13:49 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:42 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 13s)
13:42 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:25 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 34s)
13:24 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:16 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 12s)
13:15 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
13:00 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 17s)
13:00 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
12:51 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 04s)
12:51 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
12:44 mobrovac@tin: Finished deploy [eventlogging/eventbus@41e3418]: (no justification provided) (duration: 00m 05s)
12:44 mobrovac@tin: Started deploy [eventlogging/eventbus@41e3418]: (no justification provided)
12:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T151029 (duration: 00m 48s)
11:25 jynus: stopping and upgrading labsdb1010
11:19 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
09:28 jynus: stopping and restarting es2013 for upgrade
09:07 jynus: stopping and restarting db2046 for upgrade
07:35 elukey: restart pdfrender on scb1004

2017-08-10

23:43 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/371196/2 (duration: 00m 47s)
23:26 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 47s)
23:25 maxsem@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/370310/ (duration: 00m 46s)
22:31 twentyafterfour@tin: Synchronized php-1.30.0-wmf.13/includes/specials/SpecialDoubleRedirects.php: Hopefully fix T173045 (duration: 00m 48s)
22:17 moritzm: installing git security-updates
20:51 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371 (duration: 00m 21s)
20:51 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2.5 - T137371
20:50 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371 (duration: 00m 09s)
20:50 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment, take #2 - T137371
20:35 mobrovac@tin: Finished deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371 (duration: 00m 05s)
20:34 mobrovac@tin: Started deploy [cassandra/metrics-collector@d0169ee] (staging): First Scap3 deployment - T137371
20:00 mobrovac@tin: Finished deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371 (duration: 18m 17s)
19:42 mobrovac@tin: Started deploy [cassandra/metrics-collector@5db1a43] (staging): First Scap3 deployment - T137371
19:35 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 47s)
19:33 reedy@tin: Synchronized wmf-config/: Newsletter on testwiki (duration: 00m 49s)
19:31 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340 (duration: 01m 08s)
19:30 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (aqs): first Scap3 deployment - T116340
19:30 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340 (duration: 00m 34s)
19:29 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa]: first Scap3 deployment - T116340
19:28 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340 (duration: 00m 12s)
19:28 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment (rest of the nodes) - T116340
19:27 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: All wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
19:16 godog: restart cassandra instances on xenon to test logstash-logback-encoder deploy
19:13 mobrovac@tin: Finished deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340 (duration: 00m 03s)
19:13 mobrovac@tin: Started deploy [cassandra/logstash-logback-encoder@d085ffa] (staging): first Scap3 deployment - T116340
18:56 Reedy: created newsletter tables on testwiki
18:55 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/Newsletter/: sql file updates (duration: 00m 52s)
18:53 reedy@tin: Synchronized php-1.30.0-wmf.13/extensions/WikimediaMaintenance/createExtensionTables.php: newsletter (duration: 00m 52s)
18:21 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Fix commons/commonswiki snafu (duration: 00m 52s)
17:53 kartik@tin: Finished deploy [cxserver/deploy@f43ef96]: (no justification provided) (duration: 00m 44s)
17:52 kartik@tin: Started deploy [cxserver/deploy@f43ef96]: (no justification provided)
17:42 kartik@tin: Finished deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3 (duration: 00m 40s)
17:42 kartik@tin: Started deploy [cxserver/deploy@1065ffe]: Update cxserver to 686f4f3
16:23 ema: cp1072: restart varnish backend
15:48 XioNoX: troubleshoting interface errors between pfw3-codfw and fasw-codfw
15:09 marostegui: Stop replication on db2046 to fix duplicate entries - T151029
15:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T151029 (duration: 00m 50s)
15:03 marostegui: Poweroff es2013 for maintenance - T172265
14:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Pool db2075 - T170662 (duration: 00m 51s)
14:32 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2045 - T151029 (duration: 00m 51s)
14:11 elukey: restart kafka1012 temporary with some logs to TRACE to debug T172681
away: updated civicrm from 200abc2 to 4aa177b
13:43 marostegui: Compress cebwiki on db1095 - T153058
13:32 zeljkof: EU SWAT finished
13:26 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgArticleCountMethod to any on srwikisource (T172974) (duration: 00m 53s)
13:18 zeljkof: EU SWAT, part two
13:17 marostegui: Drop m3 databases from dbstore1002 - T156758
13:16 zeljkof: EU SWAT finished
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on knwiki (T172894) (duration: 00m 52s)
12:56 marostegui: Stop MySQL on db2045 to upgrade socket location - T148507
12:07 elukey: restored varnishakafka on cp3032
11:17 elukey: disabled puppet on cp3032 and restarted varnishkafka with debug logging
09:58 jynus: continuing cloning of dbstore2002 to dbstore2001
09:30 gehel: repooling wdqs2001, long after data reload completed
09:30 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=wdqs2001.codfw.wmnet
09:27 gehel@tin: Finished deploy [wdqs/wdqs@c186e3e]: (no justification provided) (duration: 01m 31s)
09:25 gehel@tin: Started deploy [wdqs/wdqs@c186e3e]: (no justification provided)
09:17 jynus: disabling lag notification on all s4 replicas
08:59 elukey: update librdkafka1 to 0.9.4.1 on eventlog1001
08:15 elukey: add 50G to carbon lv on graphite1003 and 100G on graphite2002
07:32 jynus: enabling semisymc master replication on db1068
07:29 jynus: disabling semisync slave replication on all SSD hosts on s4
06:45 elukey: powercycle mw2256 - T163346
06:38 elukey: restart pdfrender on scb1004
06:04 Dereckson: Removed 2FA for GoldRingChip account (T172878)
03:23 bblack: cp1008, restbase-dev100[456] have puppet disabled, manual "rm /etc/ssh/userkeys/dzahn"
03:20 bblack: batched cumin puppet agent run on all hosts (not forced)
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 09m 12s)

2017-08-09

22:06 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Get baaack you broken CodeMirror! (duration: 00m 51s)
21:20 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Deploy CodeMirror https://gerrit.wikimedia.org/r/#/c/370943/ (duration: 00m 50s)
21:11 maxsem@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370904/ (duration: 00m 51s)
21:05 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 07m 30s)
21:01 godog: add 100G to carbon lv on graphite1003 and graphite2002
20:58 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
20:54 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy (duration: 07m 12s)
20:51 marostegui: Remove m3 replication from dbstore1002 - T156758
20:47 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. All tables created, finish deploy
20:43 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3 (duration: 08m 05s)
20:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2046 - T170662 (duration: 00m 49s)
20:35 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 3
20:22 twentyafterfour: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Actually sync all group1 wikis (except wikidata) to 1.30.0-wmf.13 refs T170631
18:57 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation. Attempt 2
18:39 marostegui: Stop replication on db2045 to fix duplicate keys - T151029
18:36 urandom: Restarting Cassandra, restbase2001-b.codfw.wmnet (schema jankiness)
18:27 urandom: Restarting Cassandra, restbase2001-a.codfw.wmnet (schema jankiness)
18:10 ebernhardson: restart es on elastic1018 to test interleaved numa
17:50 marostegui: Add db2076 to tendril - T170662
17:41 marostegui: Compress innodb on s6 on db2076 - T170662
17:31 legoktm: changed mediawiki/vendor to fast forward only
17:24 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Rollback on canary (duration: 01m 27s)
17:23 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Rollback on canary
16:35 ppchelko@tin: Finished deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation (duration: 04m 46s)
16:30 ppchelko@tin: Started deploy [restbase/deploy@f97beeb]: Temporary fallback to the new storage buckets before truncation
16:29 urandom: T172384: Upgrading Cassandra in RESTBase dev to 3.11.0-wmf2 (patched to disable use of FastThreadLocal)
16:15 marostegui: Stop mysql on db2046 to copy its content to db2076 - T170662
16:14 elukey: rolling restart of eventstream on scb hosts to deploy https://gerrit.wikimedia.org/r/370793
16:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2046 - T170662 (duration: 00m 50s)
16:08 gehel: config update on logstash, a few logs might be lost during restart - T172713
15:19 XioNoX: removing old pfw related config from cr1-codfw - T171970
14:08 mutante: purging req.urls with "^/resources" from varnish cluster, to fix redirect with cached 404
14:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2075 to s5 - T170662 (duration: 00m 50s)
14:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2075 to s5 - T170662 (duration: 00m 51s)
13:55 marostegui: Reboot db2076 for maintenance - T170662
13:21 marostegui: Stop replication on db2075 for maintenance - T170662
13:18 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats remove confirmed user group (T101983) (duration: 00m 51s)
13:10 gehel@tin: Finished deploy [wdqs/wdqs@6620e0f]: (no justification provided) (duration: 01m 34s)
13:09 gehel@tin: Started deploy [wdqs/wdqs@6620e0f]: (no justification provided)
12:41 gehel: restarting updater on wdqs1001 (real fix coming up soon
10:20 jynus: stopping dbstore2002's mysqls and cloning them to dbstore2001
08:39 jynus: disable puppet on dbstore2001, about to be reimaged
07:47 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1065, db1064, db1070 (duration: 01m 04s)
07:12 jynus: stopping replication in sync between db1069 and db1065, db1044, db1064, db1070
07:07 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1065, db1064, db1070 (duration: 00m 47s)
06:26 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 51s)
06:11 jynus: stopping db1069 (s7) and db1079 replication in sync
06:03 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085, depool db1079 (duration: 00m 51s)
05:42 jynus: stopping db1069 (s6) and db1085 replication in sync
05:32 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1060, depool db1085 (duration: 00m 50s)
04:56 jynus: stopping db1069 (s2) and db1060 replication in sync
04:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 00m 58s)
04:48 jynus: restarting pdfrender on scb1001, unresponsive
04:09 jynus: powercycling mw2256
03:23 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.13) (duration: 07m 33s)
02:56 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 07m 45s)
02:38 ebernhardson: restart elasticsearch1017 with niofs store
02:28 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 38s)
00:27 ebernhardson: test vm.zone_reclaim_mode=0 on elastic1017

2017-08-08

23:23 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Allow bureaucrats on WMF wikis to grant and remove 'confirmed' (T101983) (duration: 00m 51s)
21:25 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.30.0-wmf.13 refs T170631
21:08 ppchelko@tin: Finished deploy [restbase/deploy@c16fb6b]: Update summary and licensing information for pageviews API (duration: 08m 31s)
21:05 twentyafterfour@tin: Finished scap: again: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631 (duration: 42m 55s)
20:59 ppchelko@tin: Started deploy [restbase/deploy@c16fb6b]: Update summary and licensing information for pageviews API
20:32 twentyafterfour: updated mediawiki.org changelog for 1.30.0-wmf.13
20:22 twentyafterfour@tin: Started scap: again: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631
19:46 ema: restart varnish backend on cp1074
19:45 godog: bounce thumbor-instances on thumbor1003 to make sure all memory limits are applied
19:44 twentyafterfour@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="test2wiki" --outdir="/tmp/scap_l10n_224168097" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 07m 33s)
19:36 twentyafterfour@tin: Started scap: deploy 1.30.0-wmf.13 to testwikis and rebuild l10n refs T170631
18:28 XioNoX: frack-codfw moved to new infrastructure
18:27 elukey: re-enabled irc-echo after the puppet shower
18:11 elukey: stop ircecho to avoid puppet shower
16:59 twentyafterfour: Branching mediawiki/master to mediawiki/wmf/1.30.0-wmf.13 refs T170631
16:04 elukey: rolling restart of varnishkafka-webrequest to apply https://gerrit.wikimedia.org/r/#/c/370659/ (puppet automatically restarts)
15:03 XioNoX: starting pfw-codfw migration - T171970
14:19 elukey: restart of all the varnishkafka statsv/eventlogging instances on caching hosts to pick up https://gerrit.wikimedia.org/r/370644 (puppet automatic restarts)
14:16 elukey: set mw2256 pooled=inactive + downtime to allow BIOS upgrade - T163346
14:07 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: SWAT: Add copyright info for Wikidata API (T112606) (duration: 00m 47s)
13:56 gehel: dump of task API for elasticsearch eqiad - T169498
13:54 ladsgroup@tin: Synchronized static/images/project-logos: SWAT: Fix srwiki logos (T150618) (duration: 00m 48s)
13:00 elukey: restart varnishkafka-webrequest with kafka.broker.version.fallback=0.9.0.1 + kafka.api.version.request=false on cp3032 (local test, to rollback remove the lines from /etc/varnishkafka/webrequest.conf)
12:49 Amir1: start of ladsgroup@terbium:~$ /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property --rebuild-all-terms (T172776)
12:32 elukey: restart pdfrender on scb1002
12:20 Amir1: start of ladsgroup@terbium:~$ /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=property (T172776)
12:15 gehel: restarting wdqs-updater on wdqs1001
12:14 elukey: stop eventlogging on eventlog1001 to test kafka consumer failures
11:22 Amir1: start of ladsgroup@terbium:~$ timeout 3500s /usr/local/bin/mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki wikidatawiki --entity-type=item >>/tmp/rebuildTermSqlIndex.log 2>&1 (T171460)
10:12 elukey: update librdkafka1* on notebook100[12] and stat1003
09:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 as the main recentchanges/watchlist s6 role (duration: 00m 47s)
08:46 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 at 50% load (duration: 00m 46s)
08:28 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1098 with limited load (duration: 00m 46s)
07:53 jynus@tin: Synchronized wmf-config/db-codfw.php: Add db1098 (duration: 00m 47s)
07:41 elukey: stop puppet on cp3032 (cache::text) to set varnishkafka-webrequest logging to debug
07:34 Amir1: stopped the script and re-running without --deduplicate-terms (T171460)
07:21 Amir1: start of ladsgroup@terbium:~$ time mwscript extensions/Wikidata/extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --entity-type=property --deduplicate-terms (T171460)
06:12 elukey: alert users with big home directories for stat1005 disk alarms (will erase data later on only if they don't answer)
06:12 elukey: restart pdfrender on scb1003
03:55 mutante: phab1001 /usr/local/bin/community_metrics.sh | /usr/local/bin/project_changes.sh creating stats mails to admins (which failed before) (T163938)
03:25 mutante: phab1001 /srv/phab/tools/public_task_dump.py to create dump, was failed cron due to missing /srv/dumps/
02:53 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 8 02:53:13 UTC 2017 (duration 6m 53s)
02:46 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 06m 59s)
02:36 ejegg: enabled donation queue consumer
02:26 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 08m 25s)
00:46 ejegg: disabled donation queue consumer

2017-08-07

21:28 urandom: T172384: Upgrading Cassandra to 3.11.0-wmf1 in dev environment (build patched to disable in-built heap dumping)
19:51 ejegg: updated CiviCRM from f24ba78 to 200abc2
19:05 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171212 - Turn on CirrusSearch MLR AB test (duration: 00m 46s)
19:00 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-common.php: T169498 - Enable max token count for phrase rescore on zh lang wikis (step 2) (duration: 00m 46s)
18:59 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T169498 - Enable max token count for phrase rescore on zh lang wikis (step 1) (duration: 00m 46s)
18:53 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-production.php: T171212 - Update CirrusSearch AB test rescore profiles (duration: 00m 46s)
18:44 ebernhardson@tin: Synchronized wmf-config/Wikibase-labs.php: T112606 - beta only - Add copyright info for Wikidata API (duration: 00m 46s)
18:43 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T172630 - Enable wgMinervaEnableSiteNotice for kowiki (duration: 00m 46s)
18:38 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T170687 - Exclude files from Special:ShortPages on commons (duration: 00m 46s)
18:23 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/CirrusSearch/: T169498 limit phrase token count, T172464 constant boost ltr queries (duration: 00m 58s)
18:10 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T172594 - Translate sitename for nl.wikinews (duration: 00m 47s)
18:05 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: Grant autopatrol to editor in en.wikibooks - T172561 (duration: 00m 47s)
17:57 mutante: phab2001 - re-enabling puppet, but closing firewall for 80/443
17:56 jynus: stopping slave and reparitioning db1098
17:10 marostegui: Restart s7 instance on db1102 to pick up new replication filters - T172693
17:05 gehel@tin: Finished deploy [wdqs/wdqs@da33919]: (no justification provided) (duration: 02m 28s)
17:02 gehel@tin: Started deploy [wdqs/wdqs@da33919]: (no justification provided)
16:44 marostegui: Restart s7 instance on db1069 to pick up new replication filters - T172693
16:37 XioNoX: manually restarted varnish on cp1099
15:10 thcipriani: restarting jenkins for plugin upgrade
14:51 gehel: reducing elasticsearch eqiad concurrent rebalance to 4 (from 8)
14:38 elukey: updated librdkafka1 and ++1 to 0.9.4.1 on hafnium
14:32 mutante: phab2001 - stopping Apache,schedule downtime for http and puppet
14:22 herron: mx[1,2]001, fermium: Installed libmail-dkim-perl and restarted spamassassin service - T172689
13:15 jynus: reboot db1098
12:39 _joe_: restarting pdfrender on scb1001, T159922
12:39 elukey: restart kafka on kafka1018 to force it out of the kafka topic leaders - T172681
12:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2074 - T171321 (duration: 00m 45s)
12:08 gehel: deploying https://gerrit.wikimedia.org/r/#/c/299825/ - some logs will be lost during logstash restart
10:02 marostegui: Add dbstore2002:3313 to tendril - T171321
09:47 jynus: stopping db1050's mysql and cloning it to db1089
09:06 elukey: set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 (was 120) on all the analytics kafka brokers - T136094
09:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 after fixing: linter, page and watchlist tables (duration: 00m 47s)
08:12 marostegui: Force BBU re-learn on db1016 - T166344
07:02 marostegui: Stop replication on db2065 to reimport: page, linter and watchlist tables
07:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 to reimport: page, linter and watchlist tables (duration: 00m 47s)
06:38 marostegui: Stop MySQL on db2074 - T171321
06:37 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2074 - T171321 (duration: 00m 46s)
06:33 marostegui: Stop replication on db2075 - T170662
06:27 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2073 - T171321 (duration: 00m 47s)
06:20 marostegui: Force BBU re-learn on db1016 - T166344
02:57 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Aug 7 02:57:42 UTC 2017 (duration 6m 42s)
02:51 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 07m 56s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 10m 16s)

2017-08-06

13:17 elukey: powercycle mw2256 - com2 frozen - T163346
13:13 elukey: restart pdfrender on scb1002
06:18 ebernhardson@tin: Synchronized wmf-config/PoolCounterSettings.php: T169498: Reduce cirrus search pool counter to 200 parallel requests cluster wide (duration: 02m 54s)
01:28 chasemp: conf2002:~# service etcdmirror-conftool-eqiad-wmnet restart (not sure what else to do the service failed)

2017-08-05

14:40 Reedy: created oauth tables on foundationwiki T172591
14:13 reedy@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaMaintenance/createExtensionTables.php: add oauth (duration: 00m 48s)

2017-08-04

23:51 mutante: phab2001 - removed outdated /etc/hosts entries, that fixed rsync, syncing /srv/repos/ from phab1001
23:35 mutante: phab2001 rebooting
23:35 mutante: phab2001 - installing various package upgrades, apt-get autoremove old kernel images
23:12 mutante: "reserved" UID 498 for phd on https://wikitech.wikimedia.org/wiki/UID | phab2001: find -exec chown to fix all the files , restart cron
23:04 mutante: phab2001 - changing UID/GID for phd user from 997:997 to 498:498 to make it match phab1001, to fix rsync breaking permissions. (rsync forces --numeric-ids when fetching from and rsyncd configured with chroot=yes). chown -R phw:www-data /srv/repos/
22:37 ejegg: restarted donations and refund queue consumers
21:44 ejegg: stopped donations and refund queue consumers
21:24 urandom: T172384: Disabling Puppet in dev environment to prevent unattended Cassandra restarts
20:19 mutante: renewing SSL cert for status.wm.org (just like wikitech-static, but that one didnt have monitoring?)
20:02 mutante: wikitech-static-ord - apt-get install certbot
19:41 ejegg: updated CiviCRM from f1fd7f0 to f24ba78
19:38 mutante: renaming graphite varnish director/fixing config, running puppet on cache misc, tested on cp1045
18:17 andrewbogott: switched most cloud instance to new puppetmasters, as per https://phabricator.wikimedia.org/T171786
11:46 marostegui: Deploy schema change directly on s3 master for maiwikimedia - T172485
11:30 marostegui: Deploy schema change directly on s3 master for kbpwiki - T172485
11:14 marostegui: Deploy schema change directly on s3 master for dinwiki - T172485
10:14 marostegui: Deploy schema change directly on s3 master for atjwiki - T172485
10:05 marostegui: Stop replication on db2073 for maintenance
09:22 marostegui: Add dbstore2002 to tendril - T171321
09:19 marostegui: Deploy schema change directly on s3 master for techconductwiki - T172485
08:35 marostegui: Deploy schema change directly on s3 master for hiwikiversity - T172485
08:19 marostegui: Deploy schema change directly on s3 master for wikimania2018wiki - T172485
08:04 marostegui: Sanitize wikimania2018wiki on sanitarium and sanitarium2 - T155041
07:47 marostegui: Stop MySQL on db2073 to copy its data to dbstore2002 - https://phabricator.wikimedia.org/T171321
07:47 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2073 - T171321 (duration: 00m 47s)
07:07 moritzm: installing imagemagick regression security updates on trusty
06:47 marostegui: Sanitize hiwikiversity on sanitarium and sanitarium2 - T171829
05:23 mutante: phab1001 sudo ip addr del 10.64.32.186/32 dev eth0 (T172478)
02:28 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=phab1001-vcs.eqiad.wmnet
02:28 dzahn@neodymium: conftool action : set/pooled=no; selector: name=phab1001-vcs.eqiad.wmnet
02:15 mutante: phab1001 can't talk to mx servers via IPv6, but works via IPv4. iridium and other mailservers can also talk IPv6 to it. why? it did not change even when stopping ferm on client and on server it allows from anywhere. workaround for now was to hardcode IPv4 IP in phab config. (T163938)
02:13 twentyafterfour: outgoing phab mail is working again
01:16 twentyafterfour: twentyafterfour@phab1001:/srv/repos$ sudo chown -R phd:www-data /srv/repos
01:16 twentyafterfour: twentyafterfour@phab1001:/srv/repos$ sudo chmod -x /usr/local/sbin/sync-srv-repos
01:12 mutante: phab1001 - stopped and started exim, which is now running with same options as iridium
00:53 twentyafterfour: starting phd to shut up icinga
00:40 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Turn CirrusSearch MLR test back off (duration: 00m 47s)
00:35 twentyafterfour: phabricator web service is also up
00:34 mutante: phab1001 - service IPs switched - puppet ran - ssh-phab service up
00:30 twentyafterfour: phab1001 chown -R phd:phd /srv/repos
00:28 twentyafterfour: testing phd on phab1001
00:24 mutante: phab1001 sudo ip addr add 2620:0:861:103:10:64:32:186/128 dev eth0
00:24 mutante: phab1001 sudo ip addr add 10.64.32.186/32 dev eth0
00:23 mutante: iridium sudo ip addr del 2620:0:861:ed1a::3:16/128 dev lo
00:23 mutante: iridiium sudo ip addr del 208.80.154.250/32 dev lo
00:23 mutante: iridium sudo ip addr del 2620:0:861:103:10:64:32:186/128 dev eth0
00:23 mutante: iridium sudo ip addr del 10.64.32.186/32 dev eth0
00:07 twentyafterfour: stoped phd and apache on iridium
00:07 twentyafterfour: stopped ssh-phab on iridium
00:03 mutante: phab1001 - /usr/bin/rsync -av rsync://iridium.eqiad.wmnet/srv-repos /srv/repos/
00:02 twentyafterfour: Taking phabricator down for maintenance / migration to a new server: phab1001

2017-08-03

23:43 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T171212: Turn on CirrusSearch MLR AB test (duration: 00m 46s)
23:36 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/370117/2 (duration: 00m 47s)
23:14 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror/resources/ext.CodeMirror.js: Only show popup if CodeMirror button exists (duration: 00m 46s)
23:12 ebernhardson@tin: Synchronized php-1.30.0-wmf.12/extensions/WikimediaEvents/extension.json: (no justification provided) (duration: 00m 47s)
23:08 ebernhardson@tin: Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch config for MLR test (step 2) (duration: 00m 46s)
23:07 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: CirrusSearch config for MLR test (step 1) (duration: 00m 47s)
22:06 ejegg: restarted donation and refund queue consumers
21:51 maxsem@tin: Synchronized wmf-config/CommonSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/370096/ (duration: 00m 47s)
21:51 ejegg: enabled ingenico_audit drupal module
21:45 ejegg: updated civicrm from 5c741b1 to f1fd7f0
21:43 ejegg: disabled donation and refund queue consumers
21:31 chasemp: clear out nova-fullstack project so we can monitor fresh on new puppetmaster
21:30 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Enable CodeMirror https://gerrit.wikimedia.org/r/#/c/370072/ (duration: 00m 47s)
21:20 maxsem@tin: Synchronized php-1.30.0-wmf.12/extensions/CodeMirror: https://gerrit.wikimedia.org/r/#/c/370066/ https://gerrit.wikimedia.org/r/#/c/370074/ (duration: 00m 47s)
21:04 herron: copper /var/cache/pbuilder 95% full - grew /dev/copper-vg/pbuilder fs by +5G and tune2fs -m 0. now at 85% full
20:40 milimetric@tin: Finished deploy [analytics/refinery@cc40bf2]: Fix sqoop script with updated scap config (duration: 11m 51s)
20:29 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis except wikidata to 1.30.0-wmf.12, leaving wikidata behind due to T172320
20:28 milimetric@tin: Started deploy [analytics/refinery@cc40bf2]: Fix sqoop script with updated scap config
20:24 milimetric@tin: Finished deploy [analytics/refinery@cc40bf2]: Fix sqoop script (duration: 33m 17s)
19:51 milimetric@tin: Started deploy [analytics/refinery@cc40bf2]: Fix sqoop script
19:21 twentyafterfour@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata/extensions/Wikibase/client/includes/Changes: be sure https://gerrit.wikimedia.org/r/#/c/369847/ is sync'd (duration: 00m 46s)
19:12 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.12 refs T168053
19:08 twentyafterfour: deploying 1.30.0-wmf.12 to group1, will proceed to group2 after verifying the branch is stable.
18:59 arlolra: Updated Parsoid to 6e1a20d5 (T168765, T155038, T165977)
18:53 demon@tin: Synchronized wmf-config/InitialiseSettings.php: Final gerrit 369960: "pagePreviews: Deploy to first 50 of stage 1 wikis" (duration: 00m 46s)
18:52 arlolra@tin: Finished deploy [parsoid/deploy@48be65b]: Updating Parsoid to 6e1a20d5 (duration: 06m 25s)
18:51 demon@tin: Synchronized wmf-config/CommonSettings.php: new dblist for gerrit 369960 (duration: 00m 46s)
18:46 arlolra@tin: Started deploy [parsoid/deploy@48be65b]: Updating Parsoid to 6e1a20d5
18:41 demon@tin: Synchronized docroot/: gerrit 369960 (duration: 00m 47s)
18:39 demon@tin: Synchronized dblists/: gerrit 369960 (duration: 00m 47s)
18:29 demon@tin: Synchronized wmf-config/InitialiseSettings.php: last part of gerrit 351287 (duration: 00m 47s)
18:23 mobrovac@tin: Finished deploy [restbase/deploy@65af18d]: Expose the recommendation API publicly and activate hiwikiversity - T170877 T168765 (duration: 08m 33s)
18:17 demon@tin: Synchronized wmf-config/CommonSettings.php: Add new dblist for page preview stuff, basically no-op (duration: 00m 47s)
18:15 demon@tin: Synchronized docroot/noc/conf/pp_stage0.dblist: tidy up pp config (duration: 00m 46s)
18:15 mobrovac@tin: Started deploy [restbase/deploy@65af18d]: Expose the recommendation API publicly and activate hiwikiversity - T170877 T168765
18:14 demon@tin: Synchronized dblists/pp_stage0.dblist: clean up config (duration: 00m 47s)
18:11 mobrovac@tin: Finished deploy [restbase/deploy@65af18d] (staging): (no justification provided) (duration: 01m 57s)
18:09 mobrovac@tin: Started deploy [restbase/deploy@65af18d] (staging): (no justification provided)
18:03 arlolra: Updated Parsoid to 651f12c2 (T119802, T73386, T170289)
17:55 andrewbogott: restarting rabbitmq-server on labcontrol1001
17:50 arlolra@tin: Finished deploy [parsoid/deploy@612e711]: Updating Parsoid to 651f12c2 (duration: 11m 17s)
17:39 arlolra@tin: Started deploy [parsoid/deploy@612e711]: Updating Parsoid to 651f12c2
16:15 jynus: repointing dbproxy1008 back to db1043
16:06 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: This is a test, disregard (duration: 00m 03s)
16:05 demon@tin: Started deploy [gerrit/gerrit@15f1544]: This is a test, disregard
16:03 demon@tin: Synchronized wmf-config/: doc fixes, no-op (duration: 00m 48s)
16:02 demon@tin: Synchronized scap/plugins/prep.py: doc fixes, no-op (duration: 00m 47s)
16:00 godog: silence cassandra-related alerts on restbase-dev cluster, known OOMs
15:44 urandom: T172384: lower tombstone failure threshold in RESTBase dev to 1000
15:39 reedy@tin: Synchronized wmf-config/interwiki.php: Update for 3 new wikis (duration: 00m 46s)
15:31 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: techconductwiki
15:27 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: techconductwiki (duration: 00m 46s)
15:25 reedy@tin: Synchronized static/images/project-logos/: techconductwiki (duration: 00m 44s)
15:24 reedy@tin: Synchronized dblists/: techconductwiki (duration: 00m 47s)
15:03 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic1028.eqiad.wmnet
15:02 gehel: unbanning and repooling elastic1028 - T168816
14:58 reedy@tin: Synchronized static/images/project-logos/: hiwikiversity (duration: 00m 46s)
14:57 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: hiwikiversity (duration: 00m 48s)
14:55 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: hiwikiversity
14:55 reedy@tin: Synchronized dblists: hiwikiversity (duration: 00m 49s)
14:53 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(29|30|31|32).eqiad.wmnet
14:53 Dereckson: mwscript initSiteStats.php --wiki srwikiquote --update (T172241)
14:52 gehel: unbanning and repooling elastic10(29|30|31|32) - T168816
14:29 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: wikimania2018wiki (duration: 00m 47s)
14:28 reedy@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no justification provided)
14:27 reedy@tin: Synchronized dblists: wikimania2018wiki (duration: 00m 47s)
14:21 marostegui: Restart MySQL on db2019 to get the new socket location updated
14:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2045 - T170662 (duration: 00m 46s)
14:06 marostegui: Compress s5 on db2075 - T170662
14:02 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(28|29|30|31|32).eqiad.wmnet
14:01 gehel: depooling and shutting down elastic10(28|29|30|31|32) - T168816
14:00 marostegui: Add db2075 to tendril - T170662
13:44 marostegui: Enable gtid back on codfw s4 slaves - T170351
13:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Promote db2051 as s4 codfw master - T170351 (duration: 00m 46s)
13:28 elukey: drop CookieBlock backup tables for T171883
13:26 marostegui: Starting the actual s4 codfw failover db2019 -> db2051 - T170351
13:22 gehel: banning elastic1032 - T168816
13:19 chasemp: disabling puppet acros wmcs things for testing
13:14 marostegui: Start topology change for s4 in codfw, slaves will be moved under db2051 - T170351
13:10 aude@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata: Fix bug in InjectRCRecordsJob (duration: 02m 12s)
13:01 marostegui: Disable gtid on s4 codfw slaves to get ready for the topology change - T170351
12:15 marostegui: Restart MySQL on db2051 - T170351
10:21 elukey: removed /run/pacct_shadow.d on stat1005 to allow atopacct.service restart
10:01 moritzm: installing poppler security updates on trusty
09:57 _joe_: systemctl reset-failed puppetmaster.service on labpuppetmaster1002
09:52 moritzm: installing mysql 5.5 security updates (package as shipped by Debian jessie, not our internal wmf-mariadb package)
09:32 gehel: banning elastic103[01] - T168816
08:56 ema: upgrading cache_upload to varnish 4.1.8-1wm1
08:36 gehel: banning elastic102[89] - T168816
08:35 marostegui: Stop MySQL on db2045 to copy its data to db2075 - T170662
08:34 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2045 - T170662 (duration: 00m 47s)
08:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db2045 - T170662 (duration: 00m 47s)
07:58 marostegui: Add db2073 and db2074 to tendril - T170662
06:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2074 to s3 - T170662 (duration: 00m 46s)
06:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2074 to s3 - T170662 (duration: 00m 46s)
06:01 marostegui: Compress s7 on db1102 - T172169
05:57 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2051 - T170351 (duration: 00m 54s)
05:27 marostegui: Deploy alter table on enwiki - labsdb1010 - T166204
05:17 marostegui: Stop replication on labsdb1011 for maintenance - T153743
03:09 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Aug 3 03:09:22 UTC 2017 (duration 7m 18s)
03:02 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 05m 47s)
02:33 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 53s)
01:40 twentyafterfour: repositories synced for phabricator migration.
00:28 reedy@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Simply ores (duration: 00m 47s)
00:26 twentyafterfour: scheduled downtime for phabricator migration
00:12 mutante: phab1001 starting ferm service
00:11 mutante: iridium restarted ferm - rsync fragment was in config but not applied, breaking data rsync to phab1001
00:05 mutante: rsyncing /srv/repos from iridium to phab1001 (T163938)

2017-08-02

23:53 ebernhardson@tin: Finished scap: dblists/rtl.dblist T172305: Adding RTL database list for project with default RTL languages (duration: 36m 21s)
23:17 ebernhardson@tin: Started scap: dblists/rtl.dblist T172305: Adding RTL database list for project with default RTL languages
21:48 ejegg: updated SmashPig from f4ca53c to c501f53
20:24 bsitzmann@tin: Finished deploy [mobileapps/deploy@3b61ced]: Update mobileapps to 2d8e8f6 (T170325) (duration: 05m 38s)
20:19 bsitzmann@tin: Started deploy [mobileapps/deploy@3b61ced]: Update mobileapps to 2d8e8f6 (T170325)
19:50 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.11 refs T168053 - rollback due to T172320
19:24 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.30.0-wmf.12 refs refs T168053
19:23 twentyafterfour: group1 wikis to 1.30.0-wmf.12 refs T168053
18:41 thcipriani@tin: Synchronized php-1.30.0-wmf.12/includes/specials/SpecialRecentchanges.php: SWAT: Follow-up 31be7d0: send tags list if experimental mode is disabled (duration: 00m 47s)
18:21 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: beta-only change Enable HTML5 sections in betalabs (duration: 00m 46s)
18:20 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: name=elastic10(24|25|26|27).eqiad.wmnet
18:18 thcipriani@tin: Synchronized php-1.30.0-wmf.12/includes/specials/SpecialUndelete.php: SWAT: Fix Special:Undelete search - use variable and not request param (duration: 00m 46s)
18:18 gehel: un-banning and repooling elastic102[4567] - T168816
18:13 thcipriani@tin: Synchronized wmf-config/jobqueue.php: SWAT: JobQueueEventBus: Enable job events in group0 wikis T163380 Part II (duration: 00m 47s)
18:11 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: JobQueueEventBus: Enable job events in group0 wikis T163380 Part I (duration: 00m 47s)
17:56 ottomata: restart kafka1012 broker with listeners=PLAINTEXT://:9092 to verify https://gerrit.wikimedia.org/r/#/c/356232/ before merge. This should be a functional no-op
17:44 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963, take #2 (duration: 00m 16s)
17:44 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963, take #2
17:44 mobrovac@tin: Finished deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963 (duration: 00m 29s)
17:43 mobrovac@tin: Started deploy [cxserver/deploy@f43ef96]: Revert canary scb2001 to f43ef963
17:26 mobrovac@tin: Finished deploy [recommendation-api/deploy@8dfae34]: (no justification provided) (duration: 01m 36s)
17:25 mobrovac@tin: Started deploy [recommendation-api/deploy@8dfae34]: (no justification provided)
17:18 herron: installed libmail-spf-perl and restarted spamassassin service on mx[1,2]001 - T172299
16:45 gehel@puppetmaster1001: conftool action : set/pooled=no; selector: name=elastic10(24|25|26|27).eqiad.wmnet
16:43 gehel: depooling and shutting down elastic102[4567] - T168816
15:34 mobrovac@tin: Finished deploy [cxserver/deploy@cf3e280]: (no justification provided) (duration: 00m 22s)
15:34 mobrovac@tin: Started deploy [cxserver/deploy@cf3e280]: (no justification provided)
15:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db2051 IP - T169501 (duration: 00m 46s)
15:17 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db2051 IP - T169501 (duration: 00m 46s)
15:17 nschaaf@tin: Finished deploy [recommendation-api/deploy@baa11c0]: source parameter validation (duration: 03m 22s)
15:14 nschaaf@tin: Started deploy [recommendation-api/deploy@baa11c0]: source parameter validation
15:06 marostegui: Poweroff db2051 to get it move to another rack - T169501
14:46 ema: upgrading cache_misc to varnish 4.1.8-1wm1
14:31 ema: varnish 4.1.8-1wm1 (fixes VSV00001, DSA 3924-1) built and uploaded to apt.w.o
14:12 marostegui: Stop MySQL on db2051 in order to get it ready to move to another rack - T170351
13:16 hashar@tin: Synchronized wmf-config/Wikibase-production.php: Write to term_full_entity_id column in wb_terms table in prod too - T167229 (duration: 00m 46s)
13:07 hashar@tin: Synchronized php-1.30.0-wmf.12/extensions/Wikidata: fix constraint type checks - T169326 (duration: 02m 13s)
12:59 gehel: banning elastic102[67] - T168816
12:41 gehel: banning elastic102[45] - T168816
11:42 jynus: disabling autolearn on newest db1* hosts
11:39 jynus: disabling autolearn on newest db2* hosts
11:33 marostegui: Stop MySQL on db2051 for maintenance - T170351
11:29 jynus: setting all es hosts with disabled auto-learn bbu properties
11:22 kartik@tin: (no justification provided)
11:21 kartik@tin: (no justification provided)
11:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2051 - T170351 (duration: 00m 46s)
11:18 kartik@tin: Finished deploy [cxserver/deploy@cf3e280]: Update cxserver to fe03ad7 (duration: 00m 56s)
11:17 kartik@tin: Started deploy [cxserver/deploy@cf3e280]: Update cxserver to fe03ad7
11:12 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool es2013 (duration: 00m 47s)
11:02 hashar: Upgraded tox on CI to 2.5.0
10:49 marostegui: Force a re-learn cycle on es2013
10:18 marostegui: Rename wikigrok tables on enwiki on db1089 - T172020
09:03 marostegui: Drop table click_tracking_user_properties wherever it exists - T115982
09:02 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2057 - T170662 (duration: 00m 46s)
08:51 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db2073 to s4 - T170662 (duration: 00m 47s)
08:50 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db2073 to s4 - T170662 (duration: 00m 57s)
08:32 marostegui: Drop table click_tracking wherever it exists - T115982
08:15 godog: bounce pdfrender on scb1001, stuck
07:09 marostegui: Disable BBU auto-learn on es2013
06:58 marostegui: Stop MySQL on db2057 for maintenance - T148507
06:50 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1055 - T166204 (duration: 00m 45s)
05:47 demon@tin: Synchronized dblists/: No-op (duration: 00m 47s)
02:59 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Aug 2 02:59:33 UTC 2017 (duration 7m 3s)
02:52 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.12) (duration: 05m 54s)
02:36 eileen: update process-control to 3d3978a
02:27 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 35s)

2017-08-01

23:47 reedy@tin: Synchronized wmf-config/CommonSettings.php: Make Babel use databasey stuff T145366 (duration: 00m 46s)
23:46 eileen: update process control to33a0262d87aa1512888c82ac6b4ed91de3d3cac5
23:44 reedy@tin: Synchronized wmf-config/CommonSettings.php: wfLoadExtension Scribunto (duration: 00m 46s)
23:43 reedy@tin: Synchronized wmf-config/extension-list: json! (duration: 00m 46s)
23:41 reedy@tin: Synchronized wmf-config/CommonSettings.php: Unbreak ContactPage T172199 (duration: 00m 46s)
23:39 reedy@tin: Synchronized wmf-config/MetaContactPages.php: Unbreak ContactPage T172199 (duration: 00m 46s)
23:36 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: T167071 (duration: 00m 47s)
23:30 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: wikinews - T172211 (duration: 00m 46s)
23:26 reedy@tin: Synchronized php-1.30.0-wmf.12/resources/src/mediawiki.rcfilters/mw.rcfilters.Controller.js: T172156 T171514 (duration: 00m 46s)
23:24 reedy@tin: Synchronized static/images/project-logos/: optimise pngs (duration: 00m 46s)
23:23 eileen: update process_control to ecb6669
23:23 reedy@tin: Synchronized static/apple-touch/: optimise pngs (duration: 00m 46s)
23:21 reedy@tin: Synchronized docroot/noc/css/images/: optimise pngs (duration: 00m 47s)
23:13 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Disable popups on Special pages T170893 (duration: 00m 47s)
23:09 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: New wordmarks T168203 T171769 (duration: 00m 47s)
23:08 reedy@tin: Synchronized static/images/mobile/copyright/: 2 new workmarks (duration: 00m 48s)
22:57 chasemp: push procs off swap for labcontrol1001 w/ swapoff -a & swapon -a
22:20 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/369434/3 (duration: 00m 47s)
22:20 eileen: tools updated from 58bcbf3 to 9554229
22:17 eileen: update process_control to c198b7f
21:48 demon@tin: Started deploy [gerrit/gerrit@15f1544]: Initial deploy x2
21:47 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: Initial deploy (duration: 00m 40s)
21:47 demon@tin: Started deploy [gerrit/gerrit@15f1544]: Initial deploy
21:46 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
21:45 demon@tin: Finished deploy [gerrit/gerrit@15f1544]: (no justification provided) (duration: 00m 06s)
21:45 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
21:44 demon@tin: Started deploy [gerrit/gerrit@15f1544]: (no justification provided)
20:40 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.30.0-wmf.12 refs refs T168053
19:33 twentyafterfour@tin: Finished scap: Sync 1.30.0-wmf.12 and build l10n refs T168053 (duration: 29m 58s)
19:19 urandom: restarting Cassandra, restbase1004-a.eqiad.wmnet, aberrant read latency
19:03 twentyafterfour@tin: Started scap: Sync 1.30.0-wmf.12 and build l10n refs T168053
18:49 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment (duration: 00m 02s)
18:49 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment
18:47 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment (duration: 00m 16s)
18:47 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: testing phab1001 deployment
18:43 twentyafterfour@tin: Finished deploy [phabricator/deployment@3d728e1]: (no justification provided) (duration: 00m 48s)
18:42 twentyafterfour@tin: Started deploy [phabricator/deployment@3d728e1]: (no justification provided)
17:23 twentyafterfour: MediaWiki train for 1.30.0-wmf.12 - finished `scap prep` & `scap patch` refs T168053
16:41 ejegg: updated CiviCRM from 23f2bbf to 5c741b1
16:25 twentyafterfour: MediaWiki Train: Creating new branch wmf/1.30.0-wmf.12 from master. See T168053 for deployment blockers.
16:17 dcausse: restarting elastic on relforge100x servers to pick up new version of the plugins
15:54 bblack: varnish backend restart on cp1072 (mailbox lag)
15:43 bblack: rebooting lvs1002
15:42 marostegui: db1069: Migrate trwiktionary.page from TokuDB to InnoDB
15:40 bblack: stopping pybal on lvs1002 for impending reboot
15:39 bblack: stopping pybal on 1002 for impending reboot
15:33 marostegui: Stop s3 on db1069 - replication stuck
15:17 marostegui: Stop MySQL on db1055 for maintenance - https://phabricator.wikimedia.org/T148507
14:56 andrewbogott: rebooting labvirt1016
14:46 marostegui: Deploy InnoDB compression on s3 - db2074 for the following tables (revision, pagelinks and templatelinks) - T170662
14:45 ema: lvs1001-1003 (eqiad primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
14:28 ema: lvs1004-1006 (eqiad secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
14:27 bblack: restart varnish backend on cp1049 (mailbox lag)
14:12 bblack: restart varnish backend on cp1074 (mailbox lag)
13:53 ema@neodymium: conftool action : set/pooled=yes; selector: name=achernar.wikimedia.org,service=pdns_recursor
13:26 ema@neodymium: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org,service=pdns_recursor
13:10 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OOjs UI EditPage on all wikis except Commons (duration: 00m 44s)
13:06 marostegui: Compress s2 on db1102 - T172169
12:49 elukey: restart hive daemons on analytics1003 to pick up new jvm settings (bigger Xmx, JMX ports)
12:06 dcausse: 100% cpu spike on elastic1023 caused percentiles to jump for a short period of time (T169498)
12:04 elukey: stop eventlogging_sync on analytics-slaves && rename all CookieBlock* tables (log db) to CookieBlock*_backup - T171883
11:52 marostegui: Stop MySQL on db2057 to copy its data to db2074 - T170662
11:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2057 - T170662 (duration: 00m 43s)
11:10 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2065 - T170662 (duration: 00m 43s)
10:31 hashar: Enabling Zuul/CI again and reenabling puppet on contint1001
10:24 hashar: contint1001 stopped puppet agent to prevent Zuul server to come back up
10:12 hashar: Stopped Zuul / CI for mass mediawiki extension changes
08:55 ema: lvs2001-2003 (codfw primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
08:32 ema: lvs2004-2006 (codfw secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
08:03 ema: lvs3*: upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
07:40 ema: lvs4001, lvs4002 (ulsfo primaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
07:35 ema: lvs4003, lvs4004 (ulsfo secondaries): upgrade to pybal 1.13.11 - one-packet-scheduling, instrumentation fixes. T104442, T103882
07:33 ema: pybal 1.13.11 uploaded to apt.w.o T103882
06:20 marostegui: Stop MySQL on db2065 to copy its data to db2073 - T170662
06:13 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2065 - T170662 (duration: 00m 43s)
05:21 marostegui: Restart MySQL on labsdb1003 as it is totally stuck
03:50 mobrovac@tin: Finished deploy [restbase/deploy@0d12138]: Add nl.wikinews - T171897 (duration: 07m 57s)
03:42 mobrovac@tin: Started deploy [restbase/deploy@0d12138]: Add nl.wikinews - T171897
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Aug 1 02:32:17 UTC 2017 (duration 6m 36s)
02:25 l10nupdate@tin: scap sync-l10n completed (1.30.0-wmf.11) (duration: 07m 46s)

Server Admin Log

2017-09-28

2017-09-27

2017-09-26

2017-09-25

2017-09-24

2017-09-23

2017-09-22

2017-09-21

2017-09-20

2017-09-19

2017-09-18

2017-09-17

2017-09-16

2017-09-15

2017-09-14

2017-09-13

2017-09-12

2017-09-11

2017-09-10

2017-09-09

2017-09-08

2017-09-07

2017-09-06

2017-09-05

2017-09-04

2017-09-03

2017-09-01

2017-08-31

2017-08-30

2017-08-29

2017-08-28

2017-08-27

2017-08-26

2017-08-25

2017-08-24

2017-08-23

2017-08-22

2017-08-21

2017-08-19

2017-08-18

2017-08-17

2017-08-16

2017-08-15

2017-08-14

2017-08-13

2017-08-12

2017-08-11

2017-08-10

2017-08-09

2017-08-08

2017-08-07

2017-08-06

2017-08-05

2017-08-04

2017-08-03

2017-08-02

2017-08-01

Archives

Navigation menu

Search