Server Admin Log

2018-12-10

07:18 _joe_: reenabling puppet given my changes were useless
06:52 _joe_: running puppet on the puppetmasters in codfw, twice, then restarting apache to ensure cleanup of any cache
06:50 _joe_: disabled puppet across the fleet for merge of hiera change
06:47 marostegui: Stop slave on s4 on labsdb1011
06:25 marostegui: Deploy schema change on db1121 with replication (this will generate lag on labs) - T86338 T202167
06:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 T86338 T202167 (duration: 00m 49s)
06:23 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - T86338
02:53 urandom: decommissioning cassandra-b, restbase2004 -- T210843
00:55 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ic07ff9acfbe17 - T211529, T205546 (duration: 00m 47s)

2018-12-09

20:16 urandom: decommissioning cassandra-a, restbase2004 -- T210843
18:39 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Icb9ad2f554e1 - Fix ruwikinews logo config (duration: 00m 57s)
11:55 godog: decommissioning cassandra-c, restbase2003 -- T210843
03:00 urandom: decommissioning cassandra-b, restbase2003 -- T210843

2018-12-08

23:07 krinkle@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/: T185446 - I6cf29d598a11 (duration: 00m 47s)
22:21 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209794 (duration: 00m 56s)
20:59 urandom: decommissioning cassandra-a, restbase2003 -- T210843
13:54 godog: decommissioning cassandra-c, restbase2002 -- T210843

2018-12-07

23:54 ejegg: updated payments-wiki from b99cd0816e to b8acb95a2a
23:41 urandom: decommissioning cassandra-b, restbase2002 -- T210843
22:43 ejegg: re-enabled Thank You mail sender
22:31 ejegg: updated fundraising CiviCRM from 3e5d74f17e to 8e18485697
22:29 ejegg: Turned off Thank You mailing job for letter update
17:47 herron: rebooting logstash1006 for security updates
17:17 bstorm_: T207377 rebooted labstore1007 for kernel upgrades
17:15 _joe_: uploading php-tideways (rebuilt with php 7.2 support) to stretch-wikimedia thirdparty/php72 T206152
15:37 moritzm: rebooting sarin/neodymium
14:52 urandom: decommissioning cassandra-a, restbase2002 -- T210843
14:21 godog: more weight to new ms-be codfw hosts - T209395
13:37 moritzm: rolling reboot of scb in codfw (along with nodejs update)
12:18 moritzm: installing nodejs security updates on stat/notebook
12:05 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title (duration: 21m 11s)
11:43 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title
11:42 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 00m 21s)
11:42 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
11:41 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 18m 58s)
11:22 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
11:20 moritzm: rolling upgrade of nginx on swift frontends
11:19 mobrovac@deploy1001: Finished deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title (duration: 03m 49s)
11:15 mobrovac@deploy1001: Started deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title
11:09 mobrovac@deploy1001: Finished deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero (duration: 06m 19s)
11:03 mobrovac@deploy1001: Started deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero
10:58 mobrovac@deploy1001: Finished deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero (duration: 02m 42s)
10:55 mobrovac@deploy1001: Started deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero
09:40 banyek: importing back linkwatcher_linklog into database s51230__linkwatcher on host labsdb1004.eqiad.wmnet. - T211210
09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
09:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
08:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
08:21 marostegui: Stop MySQL on db1096:3315,3316 for kernel and mysql upgrade
08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315, db1096:3316 for kernel and mysql upgrade (duration: 00m 46s)
08:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1084 (duration: 00m 47s)
07:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
07:50 godog: decommissioning cassandra-c, restbase2001 -- T210843
07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
07:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1084 (duration: 00m 46s)
07:11 marostegui: Stop MySQL on db1084 for mysql and kernel upgrade
07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 49s)
00:47 XioNoX: done troubleshoting bird bfd on dns2001/cr1-codfw
00:10 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: turn off wbsearchentities ab test T209402 (duration: 00m 47s)

2018-12-06

23:47 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API T201192 (duration: 02m 06s)
23:47 XioNoX: troubleshoot bird bfd on dns2001/cr1-codfw
23:45 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API T201192
23:21 ppchelko@deploy1001: Finished deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification T201192 (duration: 22m 46s)
22:58 ppchelko@deploy1001: Started deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification T201192
22:12 urandom: decommissioning cassandra-b, restbase2001 -- T210843
21:39 gtirloni: reimaging cloudvirt1019 with jessie T196507
21:33 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event (duration: 01m 15s)
21:32 ppchelko@deploy1001: Started deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event
21:07 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA (duration: 09m 18s)
21:00 XioNoX: remove 2 esams avoid path + 4 prefered/selected transits - T194542
20:58 smalyshev@deploy1001: Started deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA
20:51 XioNoX: remove 2 eqiad avoid path - T194542
20:48 gtirloni: reimaging cloudvirt1019 with stretch T196507
20:45 XioNoX: remove codfw/eqdfw avoid path - T194542
19:32 gehel: shutting down elasticsearch on elastic2001-2024 (third time is a charm) - T211023
18:21 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642) (duration: 03m 54s)
18:17 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642)
17:31 moritzm: installing nodejs updates on proton*
17:11 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for jessie-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
16:27 urandom: decommissioning cassandra-a, restbase2001 -- T210843
16:17 gehel: shutting down elasticsearch on elastic2001-2024 (second try) - T211023
15:51 moritzm: upgrading spamassassin on mx1001/fermium
15:45 fsero@deploy1001: scap-helm zotero finished
15:45 fsero@deploy1001: scap-helm zotero cluster codfw completed
15:45 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
15:42 fsero: modifying zotero deploy CLUSTER=codfw scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero - T211322
15:42 _joe_: disabling puppet fleet-wide for a change in the role() function
15:38 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for stretch-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
15:08 gehel: restartign new elasticsearch masters on codfw - T211023
15:05 gehel: upgrade nginx on wdqs servers
14:59 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10 (duration: 00m 09s)
14:59 elukey@deploy1001: Started deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10
14:46 moritzm: uploaded nodejs 10.4.0~dfsg-1+wmf2 to apt.wikimedia.org/component/node10 (backports of recent security fixes)
14:16 moritzm: installing nginx security updates on mw in eqiad
13:46 moritzm: upgrading spamassassin on mx2001
12:56 gehel: depooling and shutting down elasticsearch on elastic2001-2024 - T211023
12:55 moritzm: installing nginx updates on mw in codfw
11:59 moritzm: installing nginx updates on mw canaries
10:59 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1096:3315 (duration: 00m 47s)
10:56 volans: disable event handler on Icinga for ms-be2047 MD Raid and MegaRAID checks, it's spamming Phabricator - T209921
10:56 banyek: repooling db1096 for schema change - T85757
10:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1096:3315 (duration: 00m 49s)
10:29 banyek: depooling db1096 for schema change - T85757
09:56 dcausse: elastic@codfw cleanup: deleting wikidatawiki_content_1537469318 index (failed reindex probably)
01:03 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
00:55 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 47s)
00:42 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
00:40 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
00:38 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
00:15 mutante: MPM prefork tweaks for high load systems are applied again (apparently they were not since a change in the past that resulted in 2 competing configs in mods-enabled and conf-enabled with the latter one being loaded last and containing the package defaults
00:13 mutante: re-enabling puppet on phabricator, applying change that adds php-fpm support on stretch ..which doesnt affect phab1001 (prod) on jessie.. BUT re-adds tuning config from the past for mpm_prefork.conf (more SpareServers etc) that was not actually applied due to a bug
00:10 urandom: bootstrapping cassandra-c, restbase2018 -- T210843

2018-12-05

23:50 ejegg: updated payments-wiki from 20595cca97 to b99cd0816e
23:40 ejegg: re-enabled fundraising queue consumer jobs
23:33 ejegg: updated fundraising CiviCRM from e757753a46 to 3e5d74f17e
23:32 ejegg: turned off fundraising queue jobs for base queue consumer logic update
22:43 jijiki: restarting pdfreder on scb* hosts in eqiad
21:44 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 02m 47s)
21:41 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362
21:41 mdholloway: mobileapps deployment failed for group default03, rolling back and retrying
21:39 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 18m 46s)
21:39 arlolra: Updated Parsoid to a6058e3 (T210647, T208360, T205333)
21:37 urandom: bootstrapping cassandra-b, restbase2018 -- T210843
21:23 arlolra@deploy1001: Finished deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3 (duration: 11m 36s)
21:20 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362
21:12 arlolra@deploy1001: Started deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3
20:08 banyek: repooling labsdb1010 - T210693
19:12 cmjohnson1: cloudvirt1019 for an all inclusive part swap by HPE
18:53 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Fix formatting of help panel links (duration: 00m 47s)
18:01 jijiki: uploaded thumbor_6.3.2+git20170607-1+deb9u1 to stretch-wikimedia
17:23 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Enable Kartographer maps on testwikidatawiki (T184933) (duration: 00m 46s)
17:22 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
17:20 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
17:19 XioNoX: remove private IPs from codfw cloud-instance-transport1-b T207663
17:11 XioNoX: add public IPs to codfw cloud-instance-transport1-b T207663
16:58 XioNoX: re-deactivate ams-ix prefix list entry on cr2-esams
16:53 XioNoX: activate ams-ix prefix list entry on cr2-esams
16:27 jijiki: uploaded python-thumbor-wikimedia_2.2-1+deb9u1 to stretch-wikimedia
16:18 akosiaris@deploy1001: scap-helm zotero finished
16:18 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
16:18 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
16:17 akosiaris@deploy1001: scap-helm zotero finished
16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
16:17 akosiaris@deploy1001: scap-helm zotero finished
16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
16:15 akosiaris@deploy1001: scap-helm zotero finished
16:15 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
16:15 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
16:08 fsero: redeploying zotero on eqiad
16:02 akosiaris@deploy1001: scap-helm zotero finished
16:02 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
16:02 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 stable/zotero [namespace: zotero, clusters: eqiad,codfw]
16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 [namespace: zotero, clusters: eqiad,codfw]
15:34 thcipriani: restarting ci jenkins for update
15:33 akosiaris: add back pods/portforward right to kubernetes deploy user. T211040
15:07 anomie: Running cleanupUsersWithNoId.php on potentially missed s3 and s7 wikis for T181731
15:03 fsero: repooling citoid mathoid eqiad
15:03 godog: bootstrap cassandra-c, restbase2017 - T210843
14:56 banyek: executing schema change on s5 codfw master replication lag could be expected - T85757
14:54 fsero: upgrading k8s on eqiad to 1.10.11
14:54 elukey: restart HDFS namenode and Yarn resource manager on an-master100[1,2] to update rack topology config - T209929
14:51 fsero: depool mathoid/eqiad: pooled changed True => False
14:51 fsero: depool citoid/eqiad: pooled changed True => False
14:43 anomie: Running cleanupUsersWithNoId.php on metawiki for T181731 / T210985
14:09 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary item/property access for all wiktionaries (T175273) (duration: 00m 47s)
13:53 akosiaris: repool citoid/mathoid codfw
12:59 onimisionipe: banning elastic2001-elastic2024 from codfw production, psi and omega clusters
12:53 jijiki: uploaded python-thumbor-community-core_0.4.0-1+deb9u1 to stretch-wikimedia
12:47 dcausse: EU SWAT done
12:45 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=(cit|math)oid,name=codfw
12:44 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] Add temp clusters but still write to the old ones 2/2 (duration: 00m 46s)
12:42 dcausse@deploy1001: Synchronized wmf-config/CommonSettings.php: T210381: [cirrus] Add temp clusters but still write to the old ones 1/2 (duration: 00m 46s)
12:31 fsero: pooling mathoid and citoid again on codfw
12:30 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T202497) (duration: 00m 46s)
12:29 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T202497) (duration: 00m 49s)
12:23 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211188: Increase autoconfirmed count for Meta-Wiki to 5 (duration: 00m 47s)
12:15 fsero: upgrading codfw k8s cluster to 1.10.11
12:15 dcausse: running namespaceDupes & cirrus indexNamespaces on yuewiktionary
12:11 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T205546: Define 2 new namespaces for yuewiktionary (duration: 00m 47s)
12:02 fsero: depooling mathoid and citoid servers on codfw for k8s upgrade
11:28 mobrovac@deploy1001: Finished deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - T211127 (duration: 05m 55s)
11:23 mobrovac@deploy1001: Started deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - T211127
11:05 akosiaris: upgrade kubernetes-client and kubernetes-master on staging to 1.10.11
11:04 godog: bootstrap cassandra-b, restbase2017 - T210843
10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 (duration: 00m 45s)
10:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 45s)
10:51 ema: cache hosts: begin nginx rolling upgrade to 1.13.6-2+wmf2
10:46 marostegui: Reboot db1090 for kernel upgrade
10:44 moritzm: uploaded jenkins 2.138.4 to jessie-wikimedia/thirdparty and stretch-wikimedia/thirdpary/ci
10:42 marostegui: Stop MySQL on db1090:3312 and db1090:3317 for MySQL upgrade
10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 46s)
10:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 46s)
10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1080 (duration: 00m 46s)
10:21 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
10:17 arturo: T205969 icinga downtime the load avg check in labstore1007 for 1 week
10:10 banyek: depooling labsdb1010 for testing materialized views - T210693
10:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1080 (duration: 00m 46s)
09:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly pool db1080 (duration: 00m 46s)
09:32 gehel: setting up new elasticsearch servers on codfw - elastic2045-2054 - T210265
09:22 marostegui: Stop MySQL on db1080 for mysql and kernel upgrade
09:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for MySQL upgrade (duration: 00m 46s)
09:07 elukey: matomo read only + upgrade to matomo 3.7.0 on matomo1001 - T209808
09:00 _joe_: disabed puppet on mw1261, used for logging tests for T211184
08:48 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207850 Define a new Wikibase log channel to use (duration: 00m 47s)
08:01 moritzm: installing pdns-recursor security update in esams
07:59 godog: bootstrap cassandra-a, restbase2017 - T210843
07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 T86338 T202167 (duration: 00m 46s)
06:55 marostegui: Deploy schema change on db1091 T86338 T202167
06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 T86338 T202167 (duration: 00m 51s)
04:03 urandom: bootstrapping cassandra-c, restbase2016 -- T210843
03:11 kartik@deploy1001: Finished deploy [cxserver/deploy@a3dd2ca]: Update cxserver to c4240e6 and enable Youdao MT (T208985, T210578) (duration: 04m 26s)
03:06 kartik@deploy1001: Started deploy [cxserver/deploy@a3dd2ca]: Update cxserver to c4240e6 and enable Youdao MT (T208985, T210578)
00:32 dcausse: Evening SWAT done
00:30 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [cirrus] prepare multi-instance services (T210381) (duration: 00m 46s)
00:28 dcausse@deploy1001: Synchronized wmf-config/ProductionServices.php: [cirrus] prepare multi-instance services (T210381) (duration: 00m 46s)
00:16 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Block notice stats on itwiki (T210452) (duration: 00m 47s)
00:10 urandom: bootstrapping cassandra-b, restbase2016 -- T210843

2018-12-04

23:46 eileen: civicrm revision changed from a411d6bd64 to e757753a46, config revision is 0e6ccc37fe
23:33 XioNoX: update prefix-list peering4 on cr1-eqsin to match jnt
22:46 XioNoX: remove neodymium/sarin from term labs-in4 on cr1/2-eqiad - T210612
22:40 ejegg: Updated payments-wiki from 7403a196b4 to 20595cca97
21:58 XioNoX: clear ethernet-swtiching table for labvirt1004:eth1's switch port
21:57 XioNoX: clear ethernet-swtiching table for labvirt1009:eth1's switch port
21:49 XioNoX: make cr1/2-codfw conform to jnt
21:44 XioNoX: make cr2-eqord/eqdfw conform to jnt
21:41 XioNoX: make cr3/4-ulsfo conform to jnt
20:04 urandom: bootstrapping cassandra-a, restbase2016 -- T210843
19:35 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@81dac18]: Install new Updater for T210044 investigation (duration: 10m 36s)
19:24 smalyshev@deploy1001: Started deploy [wdqs/wdqs@81dac18]: Install new Updater for T210044 investigation
18:04 joal@deploy1001: Finished deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint (duration: 17m 33s)
18:03 akosiaris: bump zotero pod number from 4 to 16 in eqiad/codfw
17:47 joal@deploy1001: Started deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint
17:46 ppchelko@deploy1001: Finished deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance T210465 (duration: 01m 13s)
17:45 ppchelko@deploy1001: Started deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance T210465
17:22 Reedy: created oathauth tables on punjabiwikimedia T211110
17:16 godog: bootstrap cassandra-c on restbase2015 - T210843
16:48 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure 'api-warning' log channel (duration: 00m 47s)
15:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 T86338 T202167 (duration: 00m 47s)
14:04 elukey: upgrade turnilo on analytics-tools1002 to nodejs-10 - T210705
13:56 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=bnwikisource --fix --add-prefix=T210472
13:53 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=euwiki --fix
13:52 marostegui: Deploy schema change on db1103:3314 T86338 T202167
13:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 T86338 T202167 (duration: 00m 47s)
13:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 T86338 T202167 (duration: 00m 46s)
13:34 cdanis: T210416: adding grafana 5 to wikimedia-stretch: reprepro --restrict grafana update stretch-wikimedia
13:33 moritzm: installing nodejs security updates on restbase in codfw
13:32 godog: bootstrap cassandra-b on restbase2015 - T210843
13:29 moritzm: installing nodejs security updates on proton*
13:19 marostegui: Deploy schema change on db1081 T86338 T202167
13:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 T86338 T202167 (duration: 00m 46s)
13:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 T86338 T202167 (duration: 00m 46s)
12:39 Lucas_WMDE: EU SWAT done
12:38 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create namespace "Work" on bnwikisource (T210472) (duration: 00m 46s)
12:33 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create List namespace on euwiki (T209834) (duration: 00m 47s)
12:28 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: Revert "Milestone logo for atjwiki" (T200713) (duration: 00m 47s)
12:17 moritzm: installing tiff security updates
11:49 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided) (duration: 04m 47s)
11:44 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided)
11:41 moritzm: rebooting puppetboard2001 to pick up SSBD-enabled qemu
11:38 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - T197242 (duration: 00m 21s)
11:38 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - T197242
11:37 moritzm: rebooting puppetboard1001 to pick up SSBD-enabled qemu
11:36 akosiaris: enable puppet on scb1004, run puppet T197242
11:36 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - T197242 (duration: 00m 20s)
11:35 marostegui: Deploy schema change on db1084 T86338 T202167
11:35 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - T197242
11:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 T86338 T202167 (duration: 00m 47s)
11:34 akosiaris: enable puppet on scb1003, run puppet T197242
11:33 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - T197242 (duration: 00m 28s)
11:33 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - T197242
11:31 akosiaris: enable puppet on scb1002, run puppet T197242
11:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 T86338 T202167 (duration: 00m 46s)
11:18 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - T197242 (duration: 00m 30s)
11:18 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - T197242
11:17 akosiaris: enable puppet on scb1001, run puppet T197242
11:12 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - T164201 (duration: 09m 06s)
11:04 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - T211088 T197242 (duration: 20m 59s)
11:03 elukey@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - T164201
10:59 fdans@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques (duration: 00m 37s)
10:58 fdans@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques
10:57 fdans: deploying AQS to expose offset and underestimate numbers on unique devices
10:51 moritzm: rebooting analytics-tool1003 to pick up SSBD-enabled qemu
10:47 moritzm: rebooting analytics-tool1002 to pick up SSBD-enabled qemu
10:43 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - T211088 T197242
10:41 moritzm: rebooting analytics-tool1001 to pick up SSBD-enabled qemu
10:03 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - T197242 (duration: 01m 45s)
10:02 godog: bootstrap cassandra-a on restbase2015 - T210843
10:01 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - T197242
09:59 gehel: upgrading nginx on elasticsearch eqiad
09:54 akosiaris: enable puppet on all scb2*, run puppet T197242
09:52 gehel: upgrading nginx on elasticsearch codfw
09:52 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - T197242 (duration: 00m 30s)
09:51 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - T197242
09:50 akosiaris: enable puppet on scb2001, run puppet T197242
09:46 akosiaris: disable puppet on scb for citoid migration to zoterov2 T197242
09:46 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
09:31 gehel: add elastic2039-2044 to cirrus eqiad (new server) - T210265
09:11 gehel: add elastic2038 to cirrus eqiad (new server) - T210265
09:00 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/MediaWiki/electronpdf/action # Ran https://phabricator.wikimedia.org/P7882 for T157012
08:54 marostegui: Deploy schema change on db1097:3314 T86338 T202167
08:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 T86338 T202167 (duration: 00m 47s)
08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*-*.wsp" -delete # T120639
08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*_*.wsp" -delete # T120639
08:43 marostegui: Deploy schema change on db1102:3314 T86338 T202167
08:42 marostegui: Deploy schema change on dbstore1002:s4 T86338 T202167
08:41 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/datamodel$ sudo -u _graphite rm wikipedia_references.wsp # T121521
08:36 gehel: restarting stuck tilerator on maps* - T204047
08:35 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/wbgetclaims$ sudo -u _graphite find . -type f -name "*.wsp" -delete # T140280
08:11 moritzm: installing perl security updates on jessie/trusty (stretch already updated)
07:49 godog: bootstrap cassandra-c on restbase2014 - T209615
07:26 marostegui: Deploy schema change on s4 codfw master (db2051) with replication T86338 T202167
07:22 marostegui: Deploy schema change on wikitech primary master (db1073) for labswiki and labtestwiki T86338 T202167
06:39 marostegui: Deploy schema change on s5 primary master (db1070) T86338 T202167
06:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 T86338 T202167 (duration: 00m 49s)
06:12 marostegui: Deploy schema change on db1110 T86338 T202167
06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 T86338 T202167 (duration: 00m 53s)
01:22 foks: Removing 2FA per request at https://phabricator.wikimedia.org/T210703
01:22 foks: Reset password for user "Orangemike"

2018-12-03

23:54 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/tests/: for completeness (duration: 00m 58s)
23:53 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/resources/src/mediawiki.legacy/: Restore gray coloring for autocomments (T165189 part 2) (duration: 00m 47s)
23:51 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Linker.php: Restore old HTML structure for history section links (T165189 part 1) (duration: 00m 47s)
22:56 sbassett@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/AbuseFilter/includes/api/ApiQueryAbuseLog.php: Deploy security fix for T210329 (duration: 00m 47s)
21:24 mutante: temp. disabling puppet on logstash1007 and logstash1008 to carefully deploy gerrit:476916
21:17 XioNoX: push firewall change to pfw3-eqiad - T211028
21:10 catrope@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: Fix ChangesListFilters validation errors (duration: 00m 49s)
21:00 urandom: bootstrapping restbase2014-a -- T210843
20:20 ppchelko@deploy1001: Finished deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema (duration: 01m 13s)
20:19 ppchelko@deploy1001: Started deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema
20:14 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build (duration: 09m 31s)
20:05 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build
20:05 XioNoX: push firewall change to pfw3-eqiad - T211028
19:59 ppchelko@deploy1001: Finished deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change (duration: 01m 13s)
19:58 ppchelko@deploy1001: Started deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change
19:39 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend/resources/mobile.toc/TableOfContents.js: SWAT T210869 Fix Table of contents rendering (duration: 00m 47s)
19:30 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT I3b906e8b1 CS part of setting enhanced RC (duration: 00m 46s)
19:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Ic2787309e59e IS part of setting enhanced RC (duration: 00m 47s)
19:25 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Echo/modules/styles/mw.echo.ui.PaginationWidget.less: SWAT T210487 I914b94515 (duration: 00m 47s)
19:15 jforrester@deploy1001: Synchronized wmf-config/ProductionServices.php: SWAT T210381 I73c7596818b Actual config (duration: 00m 46s)
19:09 jforrester@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: SWAT T210381 I2ae162f5 Part II (duration: 00m 47s)
19:08 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT T210381 I2ae162f5 Part I (duration: 00m 46s)
18:42 anomie@deploy1001: Synchronized wmf-config/CommonSettings.php: Updating SkinBuildSidebar hook function for T210528 (duration: 00m 47s)
18:08 gehel: add elastic2037 to cirrus eqiad (new server) - T210265
18:06 godog: bootstrap cassandra-c on restbase2013 - T209615
17:09 bstorm_: T207377 reboot labstore1006 for upgrades
16:28 godog: poweroff ms-be2021 for battery replacement - T208269
15:23 gehel: start configuration of elastic2037-2044 (new servers) - T210265
15:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 T86338 T202167 (duration: 00m 46s)
14:54 marostegui: Deploy schema change on db1100 T86338 T202167
14:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 T86338 T202167 (duration: 00m 48s)
14:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 T86338 T202167 (duration: 00m 47s)
14:17 marostegui: Deploy schema change on db1113:3315 T86338 T202167
14:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 T86338 T202167 (duration: 00m 46s)
14:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 T86338 T202167 (duration: 00m 46s)
13:47 marostegui: Deploy schema change on db1082 (sanitarium master) with replication, lag will be generated on labs (s5) T86338 T202167
13:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 T86338 T202167 (duration: 00m 47s)
13:01 Lucas_WMDE: EU SWAT done
12:44 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleaning of wgLogoHD (T150618) (duration: 00m 46s)
12:39 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for multiple projects (T150618) (duration: 00m 47s)
12:37 moritzm: installing nodejs security updates on scb1001
12:34 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Close internalwiki (T205584) (duration: 00m 46s)
12:28 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change sitename of shnwiki (T206777) (duration: 00m 47s)
12:23 godog: bootstrap cassandra-b on restbase2013 - T209615
12:19 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Perform more PHP constraint checks before falling back (T209504) (duration: 00m 48s)
12:14 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Don’t send SPARQL prefixes in WikibaseQualityConstraints (T204317) (duration: 00m 49s)
11:39 godog: more weight to new ms-be codfw hosts - T209395
11:02 moritzm: installing nodejs security updates on stat/notebook hosts
10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 T86338 T202167 (duration: 00m 47s)
10:52 moritzm: rolling upgrade of scb in codfw to nodejs security update
10:39 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
10:39 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 47s)
10:34 moritzm: installing nodejs security updates on scb2001
10:31 marostegui: Deploy schema change on db1097:3315 T86338 T202167
10:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 T86338 T202167 (duration: 00m 46s)
10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 T86338 T202167 (duration: 00m 45s)
09:48 marostegui: Deploy schema change on db1096:3315 T86338 T202167
09:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 T86338 T202167 (duration: 00m 47s)
09:27 banyek: executing schema change on db1066 (s2 master) - T85757
09:22 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1074 (duration: 00m 47s)
09:16 banyek: repooling db1074 - T85757
09:16 banyek: repooling db1074
08:50 banyek: stopping replication on db1074 - T85757
08:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1074 (duration: 00m 48s)
08:44 godog: bootstrap cassandra-a on restbase2013 - T209615
08:43 banyek: depooling db1074 - T85757
08:32 moritzm: restarted keyholder agents/proxies on netmon1002/netmon2001 to pick up removal of netbox key
08:30 marostegui: Deploy schema change on s5 codfw master (db2052) with replication, lag will be generated on codfw T86338 T202167
08:13 moritzm: rearmed keyholders on netmon1002/netmon2001
08:07 marostegui: Deploy schema change on s2 master (db1066) T86338 T202167
07:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 T86338 T202167 (duration: 00m 47s)
07:32 marostegui: Deploy schema change db1074 with replication (lag will appear on labs) T86338 T202167
07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 T86338 T202167 (duration: 00m 46s)
07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 T86338 T202167 (duration: 00m 47s)
07:09 marostegui: Stop MySQL on pc1004, pc1005 and pc1006 as they will be decommissioned - T210969
06:52 marostegui: Remove pc1004, pc1005 and pc1006 from tendril and zarcillo - T210969
06:38 marostegui: Deploy schema change db1122 T86338 T202167
06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 T86338 T202167 (duration: 00m 48s)
06:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 T86338 T202167 (duration: 00m 47s)
06:16 marostegui: Deploy schema change db1076 T86338 T202167
06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 T86338 T202167 (duration: 00m 50s)
00:34 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Title.php: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/477182 (duration: 00m 52s)

2018-12-02

22:39 addshore: addshore@mwmaint1002:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=testwikidatawiki addless # This is my account, and apparently I no longer have the 2fa for it

2018-12-01

16:48 andrewbogott: rebuilding labvirt1014 as cloudvirt1014, T210904

2018-11-30

20:23 ottomata: temporarily disabled puppet on stat1005 to test rsyncd changes
20:02 ssastry@deploy1001: Finished deploy [parsoid/deploy@9981ddf]: Update Parsoid to 310edecd (deploy-20181130 branch) (duration: 11m 38s)
19:50 ssastry@deploy1001: Started deploy [parsoid/deploy@9981ddf]: Update Parsoid to 310edecd (deploy-20181130 branch)
15:29 urandom: bootstrapping restbase2013-a -- T210843
15:22 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf4+jessie to apt.wikimedia.org/jessie-wikimedia (fixes a dependency compared to the initial jessie update)
12:55 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf3+jessie to apt.wikimedia.org/jessie-wikimedia (backporting the current security fixes)
11:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 T86338 T202167 (duration: 00m 47s)
11:07 banyek: deploy schema change on dbstore1002 - T85757
10:59 marostegui: Deploy schema change on db1105:3312 T86338 T202167
10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 T86338 T202167 (duration: 00m 45s)
09:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 T86338 T202167 (duration: 00m 46s)
09:00 marostegui: Deploy schema change on db1090:3312 T86338 T202167
09:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 T86338 T202167 (duration: 00m 47s)
08:55 moritzm: removed rutherfordium from debmonitor DB (T210036)
08:21 marostegui: Deploy schema change on dbstore1002 T86338 T202167
08:12 moritzm: installing perl security updates
07:28 marostegui: Deploy schema change on db1095:3312 T86338 T202167
07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 T86338 T202167 (duration: 00m 47s)
06:55 marostegui: Purge binary logs on pc1005
06:52 marostegui: Deploy schema change on db1103:3312 T86338 T202167
06:51 marostegui: Deploy schema change on db1103:3312 T86338 T20216
06:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 T86338 T202167 (duration: 00m 48s)
06:12 marostegui: Deploy schema change on s2 codfw master with replication (db2035), this will generate lag on codfw - T86338 T202167
02:17 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: update wbsearchentities ab test configuration T209402 (duration: 00m 47s)
01:06 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents: SWAT: wbsearchentities ab test improvements T209402 (duration: 00m 46s)
01:04 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend: SWAT: Change config flag for enabling Block Notice stats T201719 (duration: 00m 48s)
00:57 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/VisualEditor: SWAT: Rename configs for tracking block notices on visual editor ve.init.mw.ArticleTarget: Stop when we fail to load metadata T209542 (duration: 00m 48s)

2018-11-29

23:26 mutante: puppetmaster: sudo puppet cert revoke rutherfordium.eqiad.wmnet; sudo puppet node clean rutherfordium.eqiad.wmnet ; sudo puppet node deactivate rutherfordium.eqiad.wmnet ; run puppet on icinga1001.. removed host from monitoring (decom for ganeti VM) (T210036)
22:21 hashar: 1.33.0-wmf.6 is on all wikis and looks stable.
22:04 hashar: hashar@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.6
21:54 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T210636 - I9ebbc6 (duration: 00m 55s)
21:41 tzatziki: changing email for User:Mathounette
21:34 XioNoX: removed unused vc-port on asw2-c-eqiad:fpc8 - T210788
20:50 mutante: people - rsynced /home one last time, switched DNS people.eqiad CNAME over, varnish change merged (T210036)
20:42 mutante: people.wikimedia.org is switching backends from rutherfordium to people1001, please stand by during a short maintenance period.. data has been copied | https://wikitech.wikimedia.org/wiki/People.wikimedia.org#Backend_upgrade_November_2018 | T210036
20:01 XioNoX: Apply Icinga:check_vcp to all VC switches - T201097
19:42 XioNoX: remove neodymium/sarin from mgmt routers - T210612
19:16 hashar@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend/: RecordRevision::getUser() returns UserIdentity not int - T210737 (duration: 00m 55s)
19:09 ejegg: updated payments-wiki from 34250e80b5 to 7403a196b4
18:53 XioNoX: Netbox: remove Napalm integration
18:28 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting comment migration to write-new/read-new on group 0 (T166733) (duration: 00m 52s)
17:55 XioNoX: remove test netbox user from cr3-ulsfo - T205898
17:49 mforns@deploy1001: Finished deploy [analytics/refinery@40b1972]: deploying refinery to refinery-source version v0.0.81 (duration: 06m 01s)
17:43 mforns@deploy1001: Started deploy [analytics/refinery@40b1972]: deploying refinery to refinery-source version v0.0.81
17:41 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/revisiondelete/RevisionDeleteUser.php: Fix RevisionDeleteUser rev_actor query for MySQL, for real this time (T210628) (duration: 00m 53s)
17:34 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/revisiondelete/RevisionDeleteUser.php: Fix RevisionDeleteUser rev_actor query for MySQL (T210628) (duration: 00m 53s)
17:02 robh: decom of labvirt101[01] continuing
16:10 anomie@mwmaint1002: Running Wikibase/populateSitesTable.php and cleanupUsersWithNoId.php on more wiktionaries, incubatorwiki, and sourceswiki for T210732
15:54 papaul: shutting down ms-be2047 for maintenance
15:35 gtirloni: T196507 downtimed and powercycled cloudvirt1019
15:32 anomie@mwmaint1002: Running Wikibase/populateSitesTable.php and cleanupUsersWithNoId.php on several other wiktionaries for T210732
15:29 gehel: activating multiple elasticsearch instances on cirrus / eqiad - T207918
15:18 $WHO: Running Wikibase populateSitesTable.php on eswiktionary for T210732
14:31 hashar@deploy1001: rebuilt and synchronized wikiversions files: Revert all wikis to 1.33.0-wmf.6
14:29 hashar@deploy1001: scap failed: average error rate on 11/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
14:17 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.6
14:10 hashar: test stashbot
14:03 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf3 to apt.wikimedia.org/stretch-wikimedia (backporting the current security fixes)
13:45 hashar@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Wikibase: feature flag for globe coordinator formatter using kartographer - T184933 T210617 (duration: 01m 18s)
13:40 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Clarify parsercache keys section (duration: 00m 53s)
13:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Clarify parsercache keys section (duration: 00m 52s)
13:17 moritzm: rebooting certcentral1001 to pick up SSBD-enabled qemu/kernel update
13:14 moritzm: rebooting certcentral2001 to pick up SSBD-enabled qemu/kernel update
13:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool pc1009 in pc3 - T208383 (duration: 00m 53s)
13:05 marostegui: Upgrade pc3 tendril topology - T208383
13:00 mobrovac@deploy1001: Started restart [eventstreams/deploy@07033d4]: Restart ES on scb1004 due to possible memory leak (again)
12:49 dcausse: EU swat done
12:48 dcausse@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/TwoColConflict/: Fix unescaped HTML injected into conflict resolution interface (duration: 00m 53s)
12:30 jynus: run puppet on notebook1004, people1001, rutherfordium to fix failures
12:13 dcausse@deploy1001: Synchronized dblists/cirrussearch-big-indices.dblist: T210381: [cirrus] multi-instance: add cirrussearch-big-indices.dblist (duration: 00m 53s)
12:09 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] Use normal config for labswiki (duration: 00m 55s)
12:01 banyek: repooling labsdb1010 after upgrades - T209517
10:17 elukey: remove zookeeper's crontabs from conf100[1-3] to fix cronspam
10:16 arturo: T209626 icinga downtime labvirt1011 for 1 month to avoid bogus pages
10:01 banyek: depooling labsdb1010 due of maintenance - T209517
09:55 gehel: restarting prometheus-elasticsearch-exporter-9200 on all elastic cirrus nodes
09:11 akosiaris: increase nofile of process to 20k and maxclients to 15k to account for the backlog of ores scorings
08:48 _joe_: restarting uwsgi-ores on ores1003
08:04 vgutierrez: replacing TLS certificates in gerrit - T207050
06:59 marostegui: Stop MySQL on pc1006 to clone pc1009 - T208383
06:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1006 - T208383 (duration: 00m 53s)
06:36 marostegui: Deploy schema change on db1061 (s6 master) - T202167
06:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 T202167 (duration: 00m 53s)
06:31 marostegui: Deploy schema change on db1088 - T202167
06:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 T202167 (duration: 00m 56s)
06:15 reedy@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/OATHAuth: revert logging (loldeployingfromaplane) (duration: 00m 59s)
06:11 marostegui: Deploy schema change on s6 primary master - T86338
04:08 ejegg: updated fundraising internal dashboard from 5e9fb9a3ef to 1e4dd2a9ec
03:30 ejegg: updated payments-wiki from 42283c73d0 to 34250e80b5
00:08 catrope@deploy1001: Synchronized wmf-config/throttle.php: T210681 (duration: 01m 04s)

2018-11-28

23:23 tzatziki: changing a few passwords for compromised accounts
21:35 mutante: gnt-instance reboot proton1001.eqiad (stopped working, no SSH)
21:28 arlolra: Updated Parsoid to 18a98af (T209236, T210437, T184755, T187142, T208470, T207286, T206777, T205710, T205546, T204477)
21:26 mutante: rebooting proton1002
21:19 herron: restarted nagios-nrpe-server on proton1002
21:16 arlolra@deploy1001: Finished deploy [parsoid/deploy@9ed8c47]: Updating Parsoid to 18a98af (duration: 09m 35s)
21:06 arlolra@deploy1001: Started deploy [parsoid/deploy@9ed8c47]: Updating Parsoid to 18a98af
20:41 herron: rebooting logstash1005 for security updates
20:22 paravoid: neodymium: mv /srv/jnt{,.old} (use cumin1001 instead!)
19:40 herron: rebooting logstash1004 to pick up security updates
19:31 ebernhardson: start goreplay logging of port 9200 across eqiad elastic cluster to track down T208248
19:05 ejegg: updated payments-wiki from 3e0c97a969 to 6552acdc0f
18:58 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/ORES/includes/Hooks/ApiHooksHandler.php: Don't try to add scores in API where there is nothing to add (T210610) (duration: 00m 55s)
18:33 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Wikibase/lib/includes/Formatters/CachingKartographerEmbeddingHandler.php: Never return null in CachingKartographerEmbeddingHandler::getParserOutput T210617 (duration: 00m 53s)
18:26 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/EventBus/includes/EventBus.php: SWAT: Revert "Revert "Revert "Set event datetime with microsecond resolution.""" T210608 (duration: 00m 55s)
17:16 bblack: rebooting lvs1006, console was unresponsive
17:11 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable classic_entity wbsearchentities AB test T209402 T210618 (duration: 00m 55s)
17:01 chasemp: stat1004:~# aptitude install exfat-fuse exfat-utils (elukey fyi)
16:57 mutante: restarting gerrit
16:54 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaIncubator/extension.json: Revert "Replace wiki with wikipedia as wmf-config has been updated" T117023 (duration: 00m 54s)
16:53 mutante: gerrit about to restart for logging config change
16:50 gtirloni: T206916 created shnwiki views/index in labsdb replicas
15:55 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1122 (duration: 00m 53s)
15:52 banyek: repooling db1122 after schema change (T85757)
15:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 T86338 (duration: 00m 52s)
15:49 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: wikidatawiki back to 1.33.0-wmf.4
15:24 vgutierrez: use a certcentral managed TLS certificate in dumps.wm.o - T207050
15:19 banyek: Deploy schema change on db1122 - T85757
15:17 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1122 (duration: 00m 53s)
15:10 banyek: depooling db1122 due schema change (T85757)
15:09 marostegui: Deploy schema change on db1085 with replication - T86338
15:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 T86338 (duration: 00m 56s)
14:05 hashar@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.6 (duration: 00m 52s)
14:04 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.6
13:59 zeljkof: EU SWAT (finally) done
13:57 zfilipin@deploy1001: Finished scap: SWAT: Add user preference to disable the advanced interface (T210479) (duration: 38m 12s)
13:19 zfilipin@deploy1001: Started scap: SWAT: Add user preference to disable the advanced interface (T210479)
12:54 moritzm: installing serf update from stretch point release
12:43 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Cite: SWAT: Make backlink highlighting robust for community customized HTML (T205270 T210520) (duration: 00m 55s)
12:34 moritzm: installing ghostscript security updates
12:24 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make AdvancedSearch default on all wikis (T207639) (duration: 00m 54s)
11:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 T202167 (duration: 00m 53s)
11:17 marostegui: Deploy schema change on db1085 (s6) (sanitarium master) with replication - T202167
11:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 T202167 (duration: 00m 53s)
11:15 banyek: repooling labsdb1009 after maintenance
10:46 moritzm: rolling reboot of logstash1007-1009 to pick up new SSBD instructions and OpenJDK security updates
10:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool pc1008 - T208383 (duration: 00m 50s)
10:26 ladsgroup@deploy1001: Finished deploy [ores/deploy@9b9ba06]: T206333 (duration: 14m 48s)
10:11 ladsgroup@deploy1001: Started deploy [ores/deploy@9b9ba06]: T206333
10:08 jynus: disable mailing list mediation-en-l
10:00 banyek: depooling labsdb1009 (T209517)
09:55 marostegui: Update tendril topology for pc1 - T208383
09:51 marostegui: Change pc2008 to replicate from pc1008 instead of pc1005 - T208383
09:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 T86338 T202167 (duration: 00m 53s)
09:14 vgutierrez: Use a TLS certificate managed by certcentral in icinga.wm.o - T207050
09:00 marostegui: Deploy schema change db1113:3316 T86338 T202167
09:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3316 T86338 T202167 (duration: 00m 53s)
08:41 moritzm: installing git security updates on trusty (Debian already fixed)
08:28 moritzm: installing samba security updates (client libs)
08:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 T86338 (duration: 00m 53s)
08:12 elukey: apply -R 200 to memcached on mc1022 (cache wipe) - T208844
07:45 marostegui: Deploy schema change db1096:3316 T86338
07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 T86338 (duration: 00m 53s)
07:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316 T86338 T202167 (duration: 00m 54s)
07:17 marostegui: Deploy schema change db1098:3316 T86338 T202167
07:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 T86338 T202167 (duration: 00m 53s)
06:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 T86338 T202167 (duration: 00m 52s)
06:40 marostegui: Deploy schema change db1093 T86338 T202167
06:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 T86338 T202167 (duration: 00m 53s)
06:14 marostegui: Stop MySQL on pc1005 to clone pc1008 - T208383
06:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1005 - T208383 (duration: 01m 04s)
06:03 marostegui: Deploy schema change on s6 codfw - T86338 T202167
02:17 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: Fix wbsearchentities collection of non-test bucketed data (duration: 00m 56s)
02:16 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: Fix wbsearchentities collection of non-test bucketed data (duration: 00m 54s)
01:45 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/: wbsearchentities needed extension.json to be deployed as well. Sync the whole directory (duration: 00m 53s)
01:44 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/: wbsearchentities needed extension.json to be deployed as well. Sync the whole directory (duration: 00m 53s)
01:34 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209402: Start wbsearchentities ab test at 10% (duration: 00m 54s)
01:29 ebernhardson: ebernhardson@deploy1001 Synchronized php-1.33.0-wmf.6/includes/parser/Parser.php: SWAT: T209236 Protect legacy URL parameter syntax in link and alt options (duration: 00m 51s)
01:22 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/ParsoidBatchAPI/includes/ApiParsoidBatch.php: SWAT: Revert ApiParsoidBatch update (duration: 00m 54s)
01:12 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209402: Configuration for wbsearchentities AB test (duration: 00m 53s)
01:09 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: T209402: AB testing support for wbsearchentities (duration: 00m 52s)
01:08 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: T209402: AB testing support for wbsearchentities (duration: 00m 53s)
01:06 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/Wikibase/view/resources/jquery/wikibase/jquery.wikibase.entityselector.js: Allow AB test to modify entityselector api request (duration: 00m 56s)
01:02 ejegg: updated payments-wiki config to 51abff4b66
00:19 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/ParsoidBatchAPI/includes/ApiParsoidBatch.php: SWAT I4e4373a7 revert Modernize ApiParsoidBatch using ApiResult to generate prettier output (duration: 00m 54s)
00:07 ebernhardson@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: SWAT: no-op labs sync for 476027 (duration: 00m 53s)
00:06 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: no-op labs sync for 476027 (duration: 00m 55s)

2018-11-27

23:49 ejegg: updated payments-wiki config to 04f718e8f1
23:29 ejegg: updated payments-wiki config to 082eab3566
22:35 ejegg: rolled back payments-wiki to 86742dd4fd
22:33 ejegg: updated payments-wiki from 86742dd4fd to 3ba10e49b0
21:26 ejegg: updated payments-wiki config to d497b7e880
19:40 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: no op sync wikiversions to test scap 3.8.10-1
18:03 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: no op sync wikiversions to test scap 3.8.9-1
17:51 godog: upload scap 3.8.9-1 - T210469
17:40 bstorm_: T209517 rebooted labsdb1005 after upgrades
17:30 bstorm_: T209517 icinga downtime labsdb1004
17:30 XioNoX: repool uslfo
17:25 arturo: T209517 icinga downtime labsdb1005
17:23 arturo: T207377 icinga downtime labnet1001
16:17 mutante: einsteinium - apt-get remove --purge icinga nsca; apt-get autoremove ; apt-get remove --purge icinga-doc icinga-common icinga-cgi-bin icinga-cgi; apt-get remove --purge monitoring-plugin* ; rm /etc/rsync.d/frag-icinga* T209738
16:14 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.6
16:09 moritzm: rebooting planet1001 for kernel security update
16:04 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105 (duration: 00m 53s)
15:53 moritzm: rebooting planet2001 for kernel security update
15:44 moritzm: rebooting kafkamon1001 for kernel security update
15:41 mutante: einsteinium - removed icinga package
15:39 moritzm: rebooting kafkamon2001 for kernel security update
15:35 mutante: einsteinium - stopped icinga, stopped nsca, stopped rsyncd, killall -u icinga, killall -u nagios ... T209738
15:35 hashar@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.6 and rebuild l10n cache | T206660 (duration: 32m 42s)
15:21 akosiaris: create graphoid namespace on kubernetes eqiad, codfw, staging clusters T203091
15:02 hashar@deploy1001: Started scap: testwiki to php-1.33.0-wmf.6 and rebuild l10n cache | T206660
14:52 hashar@deploy1001: Pruned MediaWiki: 1.32.0-wmf.26 (duration: 09m 48s)
14:40 hashar: Applied security patches for 1.33.0-wmf.6 | T206660
14:33 hashar: scap prep 1.33.0-wmf.6 | T206660
14:04 hashar: Cutting 1.33.0-wmf.6 branches | T206660
14:00 banyek: repooling db1105 due a schema change (T85757)
13:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool pc1010 in pc1 - T208383 (duration: 00m 46s)
13:34 marostegui: Change pc2007 and pc2010 to replicate from pc1010 instead of from pc1004 - T208383
13:18 banyek: executing schema change on db1105 (T85757)
13:18 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1105 (duration: 00m 46s)
13:17 pmiazga@deploy1001: Finished deploy [proton/deploy@9efc072]: Proton: Rewrite Queue to promise-way flow (T204055) (duration: 02m 43s)
13:15 pmiazga@deploy1001: Started deploy [proton/deploy@9efc072]: Proton: Rewrite Queue to promise-way flow (T204055)
13:14 banyek: depooling db1105 due a schema change (T85757)
13:09 raynor: proton deploying 9efc072
12:39 mobrovac@deploy1001: Synchronized php-1.33.0-wmf.4/includes/page/WikiPage.php: Convert $archivedRevisionCount to integer - T210013 T210451 (duration: 00m 47s)
12:32 gilles@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/http2priorities/upload.wikimedia.org.html: T210141 Add variant of HTTP/2 priorities test pointing to upload (duration: 00m 46s)
12:25 mobrovac@deploy1001: Synchronized static/images/project-logos: Add localised logos for the Minangkabau Wikipedia - T210387 (duration: 00m 47s)
12:19 mobrovac@deploy1001: Synchronized rpc/RunSingleJob.php: RunSingleJob: Check that JobExecutor has been loaded - T208922 (duration: 00m 47s)
11:56 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1103 (duration: 00m 46s)
11:50 banyek: repooling db1103 after schema change (T85757)
11:32 banyek: executing schema change on db1103:3312 (T85757)
11:32 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1103 (duration: 00m 46s)
11:28 banyek: depooling db1103 due a schema change (T85757)
10:59 banyek: executing schema change on db1095 (T85757)
10:55 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1090 (duration: 00m 46s)
10:51 banyek: repooling db1090 after schema change (T85757)
10:51 banyek: repooling db1090 due a schema change (T85757)
10:42 gilles@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/http2priorities: T210141 HTTP/2 prioritie speed test (duration: 00m 47s)
10:36 arturo: T209948 disable puppet in all WMCS hw servers
10:30 arturo: T209948 schedule 2h icinga downtime in all WMCS hw servers
10:26 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1090 (duration: 00m 46s)
10:19 banyek: depooling db1090 due a schema change (T85757)
10:18 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1076 (duration: 00m 45s)
10:13 banyek: repooling db1076 after schema change (T85757)
10:06 banyek: executing schema change on db1076 (T85757)
10:02 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1076 (duration: 00m 46s)
09:57 banyek: depooling db1076 due a schema change (T85757)
09:35 marostegui: Stop MySQL on pc1004 to clone pc1010 - T208383
09:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1004 - T208383 (duration: 00m 46s)
09:00 banyek: executing schema change on db2035 for s2 (T85757)
08:43 elukey: roll restart of all druid daemons on druid100[1-6] for openjdk-8 upgrades
08:41 vgutierrez: Use a TLS certificate managed by certcentral in apt.wm.o - T207050
08:15 godog: more weight to new ms-be hosts in codfw - T209395
07:35 _joe_: depooling mw1261 for benchmarking, T206341
07:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 after cloning db1095:3312 (duration: 00m 46s)
07:16 marostegui: Start MySQL on db1090:3312 after recloning db1095:3312
07:16 marostegui: Start MySQL on db1095:3312 after recloning it
06:16 marostegui: Stop mysql on db1090:3312 to clone db1095:3312
06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 49s)
06:15 marostegui: Stop mysql on db1095:3312 to get it recloned
04:33 ejegg: updated payments-wiki from d2b66c5bab to 86742dd4fd
04:31 ejegg: updated fundraising CiviCRM from 013807a7b9 to a411d6bd64
04:27 ejegg: updated standalone SmashPig deployment from f65daa8550 to fb3268897b
00:38 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix weird caching issue maybe for Ian's patch (duration: 00m 46s)
00:29 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Enable SVGs in page language everywhere T208899 (duration: 00m 45s)
00:27 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable SVGs in page language everywhere T208899 (duration: 00m 46s)
00:17 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgMinervaSchemaMainMenuClickTrackingSampleRate T205008 (duration: 00m 46s)
00:12 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Delete a namespace from ruwikisource T210171 (duration: 00m 46s)
00:05 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Enable trust survey on labs T209882 (duration: 00m 46s)
00:03 niharika29@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 03s)

2018-11-26

23:50 ottomata: temporarily disabling puppet on stat1007 to copy over eventbus validation logs (not using stat1007 after all)
23:48 ottomata: temporarily disabling puppet on stat1007 to copy over eventbus validation logs
23:24 sbassett@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove unblockself right everywhere 2046d8df3 T150826 (duration: 00m 47s)
23:15 XioNoX: depool ulsfo for transport providers maintenance
20:14 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/Wikibase/repo/includes/Search/Elastic/EntitySearchElastic.php: SWAT T209402 Make wbsearchentities tie-breaker configurable (duration: 00m 47s)
19:33 ebernhardson@deploy1001: Synchronized wmf-config/WikibaseSearchSettings.php: SWAT T209402 Search profiles for wbsearchentities AB test (duration: 00m 46s)
19:25 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/includes/: SWAT T210003 T210004 EditorJourney: Adjust DeferredUpdates usage (duration: 00m 46s)
19:24 XioNoX: remove IP from blacklist on Amsterdam routers - T201411
19:24 cmjohnson1: cloudvirt1019 is going down to check something for HPE
19:16 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T209432 Enable NewUserMessage extension on tcy.wikipedia (duration: 00m 48s)
18:50 mutante: stopping icinga service on einsteinium, is a role(spare) now T209738
18:46 mutante: removed allowed sender addresses from AQL (mail2SMS gateway) portal: @einsteinium @tegmen addresses T208824 T209738
18:45 ppchelko@deploy1001: Finished deploy [changeprop/deploy@c89bff5]: Prepared for ORES error response T197000 (duration: 01m 13s)
18:44 ppchelko@deploy1001: Started deploy [changeprop/deploy@c89bff5]: Prepared for ORES error response T197000
18:41 XioNoX: re-activate BGP to AS41692 on cr2-esams
18:15 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@8bfea2b]: GUI updates and updater with dump and revision logging. (duration: 11m 13s)
18:04 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@8bfea2b]: GUI updates and updater with dump and revision logging.
17:54 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp5001.eqsin.wmnet
17:53 bblack: re-pooling cp5001 - T199675
17:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1122 (duration: 00m 46s)
17:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1122 (duration: 00m 46s)
17:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1122 (duration: 00m 46s)
17:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1122 (duration: 00m 45s)
17:23 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Reverting: Bumping portals to master (T128546) (duration: 00m 46s)
17:22 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Reverting: Bumping portals to master (T128546) (duration: 00m 46s)
17:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 and db1122 (duration: 00m 46s)
16:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 to clone db1122 (duration: 00m 46s)
16:00 marostegui: Stop MySQL on db1090:3312 to clone db1122
15:57 marostegui: Stop MySQL on db1122 - it will be recloned
15:50 ppchelko@deploy1001: Finished deploy [changeprop/deploy@77be2c6]: Change schema for revision-score events and start emitting again T197000 (duration: 01m 24s)
15:48 ppchelko@deploy1001: Started deploy [changeprop/deploy@77be2c6]: Change schema for revision-score events and start emitting again T197000
15:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 - actor table missing (duration: 00m 46s)
15:27 ppchelko@deploy1001: Finished deploy [changeprop/deploy@b97e8eb]: Temporary stop emitting revision-score events for schema change T197000 (duration: 01m 21s)
15:26 ppchelko@deploy1001: Started deploy [changeprop/deploy@b97e8eb]: Temporary stop emitting revision-score events for schema change T197000
15:24 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on group 1 (T188327) (duration: 00m 46s)
13:47 moritzm: rebooting releases1001
13:36 moritzm: rebooting releases2001
13:33 moritzm: rebooting netmon1003
12:04 _joe_: uploaded php-excimer for component thirdparty/php72 to stretch-wikimedia T205059
12:00 kartik@deploy1001: Finished deploy [cxserver/deploy@da41227]: Disable Youdao MT until service is back (duration: 04m 01s)
11:56 kartik@deploy1001: Started deploy [cxserver/deploy@da41227]: Disable Youdao MT until service is back
11:41 arturo: icinga downtime cloudcontrol1004 for some systemd slice tests
11:01 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
11:00 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 47s)
10:32 moritzm: rebooting boron
09:58 banyek: performing schema change on s6 master - db1061 for frwiki (T85757)
09:51 banyek: performing schema change on s6 master - db1061 for ruwiki (T85757)
09:44 banyek: performing schema change on s6 master - db1061 for jawiki (T85757)
08:35 moritzm: removed labvirt1015 from debmonitor DB (got renamed to cloudvirt1015)
08:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 - T202167 (duration: 00m 46s)
08:16 marostegui: Deploy schema change on db1096:3316 - T202167
08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 - T202167 (duration: 00m 47s)
08:10 moritzm: installing pixman security updates
07:37 elukey: restart memcached on mc1021 (cache wipe) to add -R 200 - T208844
07:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T86338 (duration: 00m 46s)
07:00 marostegui: Deploy schema change on db1088 - T86338
07:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T86338 (duration: 00m 46s)
06:46 marostegui: Deploy schema change on db2067 - T86338
06:36 marostegui: Deploy schema change on s3 master (db1075) - T86339
06:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T86339 (duration: 00m 46s)
06:22 marostegui: Reload haproxy on dbproxy1005
06:22 marostegui: Deploy schema change on db1077 (s3 sanitarium master) with replication - T86339
06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T86339 (duration: 00m 50s)
06:02 kartik@deploy1001: Finished deploy [cxserver/deploy@d2173ca]: Update cxserver to 218a543 (duration: 04m 21s)
05:58 kartik@deploy1001: Started deploy [cxserver/deploy@d2173ca]: Update cxserver to 218a543

2018-11-25

10:40 _joe_: restarting pdfrender on scb1002, flapping since 3:00 UTC

2018-11-23

21:15 bawolff: deploy patch for T210192
16:53 bblack: cleaned up remnants of globalsign-2017 unified cert (OCSP cache/config, unmanaged cert files, etc) on all cpNNNN - T206804
14:02 gehel: restor wdqs-updater heap to 2G - T210235
13:32 moritzm: installing confuse security updates
12:04 gehel: manually increasing wdqs-updater heap to 4G - T210235
11:37 gehel: restarting updater on all wdqs ndoes
08:41 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=.*,service=zotero,cluster=kubernetes,name=.*
08:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T86339 (duration: 00m 46s)
07:48 moritzm: installing libtirpc security updates
07:14 marostegui: Deploy schema change db1078 - T86339
07:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T86339 (duration: 00m 45s)
07:00 marostegui: Deploy schema change db1095 - T86339
06:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 - T86339 (duration: 00m 49s)
06:32 marostegui: Deploy schema change db1123 - T86339
06:31 marostegui: Deploy schema change dbstore1002:s3 - T86339
06:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 - T86339 (duration: 00m 48s)
06:15 marostegui: Deploy schema change on s3 codfw master (db2043) with replication - T86339
06:13 marostegui: Deploy schema change on db1067 (s1) master - T86339

2018-11-22

21:08 godog: disable raid handler for ms-be2021 - T208096
19:01 moritzm: installing uriparser security updates
18:46 moritzm: installing openjpeg2 security updates
18:45 arturo: enable puppet in all CloudVPS HW servers
18:38 arturo: disable puppet in all CloudVPS HW servers to test a patch (T209948)
17:46 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1016 (duration: 00m 46s)
16:49 jynus: upgrading, and restarting es1016 (but not deleting, that was a mistake)
16:49 jynus: upgrading, deleting at and restarting es1016
16:35 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1016 (duration: 00m 47s)
16:06 ema: trafficserver_8.0.0-1wm3 uploaded to stretch-wikimedia
15:23 akosiaris@deploy1001: scap-helm zotero finished
15:23 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
15:23 akosiaris@deploy1001: scap-helm zotero install --name production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: codfw]
15:22 akosiaris@deploy1001: scap-helm zotero finished
15:22 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
15:22 akosiaris@deploy1001: scap-helm zotero install --name production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
14:50 akosiaris@deploy1001: scap-helm zotero install --name production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
13:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T86339 (duration: 00m 45s)
13:52 marostegui@deploy1001: sync-file aborted: Depool db1080 - T86339 (duration: 00m 00s)
13:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T86339 (duration: 00m 46s)
13:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T86339 (duration: 00m 46s)
13:45 marostegui: Deploy schema change on s1 eqiad hosts T86339
13:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T86339 (duration: 00m 46s)
13:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T86339 (duration: 00m 45s)
13:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T86339 (duration: 00m 46s)
13:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 - T86339 (duration: 00m 45s)
13:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 - T86339 (duration: 00m 43s)
13:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 - T86339 (duration: 00m 46s)
13:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 - T86339 (duration: 00m 47s)
12:19 jynus: upgrading, deleting at and restarting dbstore2002
10:21 jynus: upgrading and restarting dbstore1001
10:01 godog: bounce rsyslog on lithium, tls listener timeout on icinga
09:54 godog: bounce rsyslog on wezen, tls listener timeout on icinga
09:49 jynus: stop and upgrade dbstore2001
09:24 moritzm: installing ruby-l18n security updates
09:20 moritzm: installing ruby-rack security updates
09:06 moritzm: installing jasper security updates
08:21 marostegui: Deploy schema change on s1 codfw master (db2048) with replication - T86339
08:19 marostegui: Deploy schema change on db1062 (s7 master) - T86339
08:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 - T86339 (duration: 00m 46s)
08:17 marostegui: Deploy schema change on db1094 - T86339
08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 - T86339 (duration: 00m 49s)
06:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 - T86339 (duration: 00m 46s)
06:55 marostegui: Deploy schema change on db1086 - T86339
06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T86339 (duration: 00m 46s)
06:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T86339 (duration: 00m 46s)
06:48 marostegui: Deploy schema change on db1079 (sanitarium master) with replication - T86339
06:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T86339 (duration: 00m 51s)
02:25 mutante: stat1005 - started nagios-nrpe-server

2018-11-21

23:34 mutante: rsyncing /home from rutherfordium.eqiad to people1001.eqiad (people.wikimedia.org) T210036
21:16 robh: cp5001 is offline running hardware tests after firmware updates to see if memory error still exists. ref: T199675
20:55 robh: cp5001 reboot for firmware update
19:54 milimetric@deploy1001: Finished deploy [analytics/aqs/deploy@e114d99]: Fixing sorting bug on top endpoints (duration: 05m 34s)
19:49 milimetric@deploy1001: Started deploy [analytics/aqs/deploy@e114d99]: Fixing sorting bug on top endpoints
19:43 ejegg: updated fundraising internal dashboard from b01458b260 to 5e9fb9a3ef
17:21 elukey: manually started systemd-journald.service on scb1001 after OOM
17:20 jynus: stop and upgrade db2081
17:04 jynus: stop and upgrade db2080
16:40 jynus: stop and upgrade db2066
16:37 bawolff: deploy patch T209794
16:25 jynus: stop and upgrade db2063
15:15 jynus: stop and upgrade db2073
15:15 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1085 (duration: 00m 46s)
15:03 banyek: repooling db1085 after schema change (T85757)
15:00 banyek: restarting replication on db1085 (T85757)
14:31 banyek: stopping replication on db1085 (T85757)
14:27 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1085 (duration: 00m 46s)
14:22 banyek: depooling db1085 due schema change (T85757)
13:39 jynus: stop and upgrade db2077
13:05 XioNoX: remove BGP session to 2603 on cr4-ulsfo
12:13 jynus: stop and upgrade db2076
12:08 banyek: running schema change on dbstore1001:3316 (T85757)
12:08 banyek: running schema change on dbstore1001 (T85757)
12:04 jynus: stop and upgrade db2075
11:53 banyek: running schema change on dbstore1002 (T85757)
11:00 akosiaris: disable puppet on ores2* ores1* for gradual rollout of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474694/1/modules/ores/manifests/web.pp
10:55 jynus: stop and upgrade db2074
10:51 _joe_: uploading prometheus-php-fpm-exporter to stretch-wikimedia main, T209573
10:44 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1096:3316 (duration: 00m 46s)
10:42 banyek: repooling db1096:3316 after schema change (T85757)
10:30 jynus: stop and upgrade db2095
10:26 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1096:3316 (duration: 00m 45s)
10:26 godog: initial weight for new ms-be2* hosts (all but ms-be2047) - T209395
10:23 banyek: depooling db1096:3316 due schema change (T85757)
10:14 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1098:3316 (duration: 00m 46s)
10:13 banyek@deploy1001: sync-file aborted: T85757: depool db1098:3316 (duration: 00m 03s)
10:11 banyek: repooling db1098 after schema change (T85757)
09:57 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1098:3316 (duration: 00m 46s)
09:52 banyek: depooling db1098:3316 due schema change (T85757)
09:49 volans: restarted pdfrender on scb1003
09:29 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1113 (duration: 00m 46s)
09:21 banyek: repooling db1113 after schema change (T85757)
09:09 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1113 (duration: 00m 46s)
09:01 banyek: depooling db1113 due schema change (T85757)
08:48 marostegui: Deploy schema change on s7 codfw master - T86339
08:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 - T86339 (duration: 00m 45s)
08:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 - T86339 (duration: 00m 45s)
08:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 - T86339 (duration: 00m 46s)
08:30 marostegui: Deploy schema changes on s8 eqiad hosts - T86339
08:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 - T86339 (duration: 00m 46s)
07:50 marostegui: Deploy schema change on s8 codfw master (db2045) with replication - T86339
07:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T86339 (duration: 00m 45s)
07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T86339 (duration: 00m 46s)
07:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 - T86339 (duration: 00m 46s)
07:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T86339 (duration: 00m 46s)
07:32 marostegui: Deploy schema change on s4 eqiad hosts - T86339
07:19 marostegui: Deploy schema change on db2051 (s4 codfw master) with replication - T86339
07:10 marostegui: Drop foundationwiki.petition_data from s3 master (db1075) with replication - T208979
07:06 marostegui: Deploy schema change on db1066 (s2 master) - T86339
07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 - T86339 (duration: 00m 46s)
07:02 marostegui: Deploy schema change on db1076 - T86339
07:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T86339 (duration: 00m 46s)
06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T86339 (duration: 00m 46s)
06:48 marostegui: Deploy schema change on db1074 (sanitarium master) - T86339
06:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T86339 (duration: 00m 46s)
06:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 - T86339 (duration: 00m 46s)
06:39 marostegui: Deploy schema change on db1122 - T86339
06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 - T86339 (duration: 00m 51s)
06:30 marostegui: Drop schema change on db1103:3312 and db1105:3312 - T86339
00:48 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:474967 Disable FlaggedRevs, enable RC patrol and add rights on srwikinews (duration: 00m 47s)
00:39 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:475005 Enable SVGs in page in group1, rest of group0 (duration: 00m 46s)
00:32 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:474976 Enable suppressredirect on srwiki (duration: 00m 47s)
00:24 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:472744 Enable RCPatrol and add some rights on srwikibooks (duration: 00m 46s)
00:07 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/GrowthExperiments/includes/Specials/SpecialWelcomeSurvey.php: gerrit:474946 WelcomeSurvey: indicate that the special page does write (duration: 00m 47s)

2018-11-20

23:01 XioNoX: create vol.ans account on switches - T208726
22:54 pmiazga@deploy1001: Synchronized wmf-config: SYNC: noop Doc: add repoConceptBaseUri comment (T209352)noop: Remove utf-8 characters from DOC comment for better readability (T209352)beta: Wikibase: override repoConceptBaseUri (T209352) (duration: 00m 49s)
22:13 XioNoX: create volans account on routers - T208726
21:12 eileen: civicrm revision changed from f4127d5316 to 013807a7b9, config revision is 684ec9b7c0
21:08 jgleeson: civicrm changed from a31dbefc61 to f4127d5316
18:56 ebernhardson: start loading dumps into elastic codfw omega and psi from mwmaint2001
18:19 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: Create Federated Wikibase instance on Beta Commons, part II (T204748) (duration: 00m 47s)
18:17 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Create Federated Wikibase instance on Beta Commons (T204748) (duration: 00m 48s)
18:08 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@7553087]: Deploy 2018 app fundraising announcement config (T204821) (duration: 03m 37s)
18:04 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@7553087]: Deploy 2018 app fundraising announcement config (T204821)
18:03 bstorm_: rebooting labsdb1006 for upgrades T209517
17:25 bstorm_: rebooting labsdb1004 for upgrades T209517
17:19 gehel: reload nginx configuration on elasticsearch codfw
16:59 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 46s)
16:53 XioNoX: rollback all BFD tests on cr1-codfw
16:10 jynus: stop and upgrade db2072
15:59 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2071, depool db2072 (duration: 00m 47s)
15:51 banyek: repooling labsdb1011 (T209517)
15:41 banyek: uploaded wmf-pt-kill_2.2.20-1+wmf5 packages to stretch-wikimedia (T209517)
15:38 vgutierrez: switching to certcentral managed TLS certificate for librenms.wikimedia.org - T209856
15:36 XioNoX: add test term allow BFD multihop on cr1-codfw loopback4 filter
15:36 ejegg: updated fundraising CiviCRM from e648be0d9e to a31dbefc61
15:20 moritzm: installing libopenmpt security updates
15:12 XioNoX: enable bfd traceoptions on cr1-codfw
15:02 XioNoX: Add BFD multihop support to Bird anycast DNS
14:55 jijiki: libthumbor_1.3.2-0+wmf1+stretch1 uploaded to stretch-wikimedia T209886
14:43 chasemp: puppet temp disable on es2001 for data transfer work
14:32 jynus: stop and upgrade db2033
13:19 jynus: stop and upgrade db2082
13:06 banyek: depooling labsdb1011 (T209517)
13:03 zeljkof: EU SWAT finished
13:03 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: deployment-prep: Update parsoid09 IP (T208101) (duration: 00m 47s)
12:55 zfilipin@deploy1001: Synchronized wmf-config/db-labs.php: SWAT: deployment-prep: Update deployment-db* IPs (T208101) (duration: 00m 47s)
12:55 banyek: setting innodb_flush_log_at_trx_commit to 2 on dbstore2002 (s3 instance only!) (T208320)
12:53 banyek: setting innodb_flush_log_at_trx_commit to 2 on dbstore2002 (T208320)
12:49 zfilipin@deploy1001: scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
12:45 zfilipin@deploy1001: Synchronized wmf-config/reverse-proxy-staging.php: SWAT: deployment-prep: Update cache-upload private IP (T208101) (duration: 00m 45s)
12:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos in InitialiseSettings.php for multiple projects (T150618) (duration: 00m 48s)
12:25 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add tboverride permission to extendedmover group on enwiki (T209753) (duration: 00m 47s)
12:19 jynus: powercycling db2087, stuck on reboot
12:13 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for multiple projects (T150618) (duration: 00m 48s)
11:55 moritzm: rolling reboot of proton hosts for kernel security update
11:27 jynus: stop and upgrade db2087
11:16 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: (now really) repool db1093 (duration: 00m 47s)
11:11 banyek: repooling db1093 (T85757)
11:05 banyek: executing schema change on db1093 (T85757)
11:00 jynus: stop and upgrade db2086
10:59 banyek: db1093 was depooled wrong message sent
10:51 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1093 (duration: 00m 47s)
10:48 banyek: depooling db1093 (T85757)
10:48 banyek: depooling db1093
10:47 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2018 (duration: 00m 46s)
10:17 jynus: upgrade and reboot es2018
10:13 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2014, depool es2018 (duration: 00m 46s)
09:34 marostegui: Deploy schema change on s2 hosts: dbstore1002, db1090:3312 and db1095:3312 - T86339
09:26 marostegui: Deploy schema change on s2 codfw master (db2035) with replication - T86339
09:25 jynus: upgrade and reboot es2014
09:23 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2011, depool es2014 (duration: 00m 46s)
09:23 godog: stress-test new ms-be hardware - T209395
09:13 marostegui: Stop MySQL on pc2004, pc2005 and pc2006 for decommission - T209858
09:05 gehel: powercycle elastic2021
09:04 marostegui: Remove pc2004, pc2005 and pc2006 from tendril and zarcillo - T209858
08:53 jynus: upgrade and reboot es2011
08:48 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2011 (duration: 00m 47s)
06:28 marostegui: Deploy schema change on db1070 (s5 master) - T86339
06:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 - T86339 (duration: 00m 47s)
06:21 marostegui: Deploy schema change on db1082 - T86339
06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T86339 (duration: 00m 52s)
00:55 catrope@deploy1001: Synchronized static/images/project-logos/: Correct logos for Sindhi Wiktionary (duration: 00m 47s)
00:51 mutante: Gerrit: added Jeena Huneidi to wmf-deployers (T209722)
00:26 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.FilterTagMultiselectWidget.js: RCFilters bug fix (T209657) (duration: 00m 47s)
00:23 XioNoX: registering librenms IRC bot
00:15 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Increase Schema.org page split test to 100% sampling (T208755) (duration: 00m 48s)

2018-11-19

23:18 ejegg: updated fundraising CiviCRM from 275716f000 to e648be0d9e
22:41 ejegg: updated fundraising CiviCRM from bbc0dddd1e to 275716f000
22:21 ejegg: updated fundraising CiviCRM from 6b279509f8 to bbc0dddd1e
21:58 XioNoX: restart bird on dns2001 to try to establish the BFD sessions
20:27 catrope@deploy1001: Finished scap: Full scap for special alias changes for GrowthExperiments (duration: 21m 03s)
20:06 catrope@deploy1001: Started scap: Full scap for special alias changes for GrowthExperiments
19:50 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable WelcomeSurvey on cswiki and kowiki (T209725) (duration: 00m 46s)
19:44 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/: EditorJourney fixes (T207307) (duration: 00m 46s)
19:36 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/GrowthExperiments/: WelcomeSurvey fixes (T206371) (duration: 00m 46s)
19:25 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable WelcomeSurvey on testwiki (T209725) (duration: 00m 49s)
18:29 cmjohnson1: connecting eqiad asw2-b fpc2 and fpc8
18:17 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@a25eb30]: GUI Update, new executor limits and new blazegraph build (duration: 08m 53s)
18:08 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@a25eb30]: GUI Update, new executor limits and new blazegraph build
15:45 ladsgroup@deploy1001: Finished deploy [ores/deploy@e957b24]: T209587 T170950 (duration: 17m 09s)
15:28 ladsgroup@deploy1001: Started deploy [ores/deploy@e957b24]: T209587 T170950
15:23 milimetric@deploy1001: Finished deploy [analytics/aqs/deploy@b399c34]: Removing empty fields from unique result (duration: 05m 17s)
15:18 milimetric@deploy1001: Started deploy [analytics/aqs/deploy@b399c34]: Removing empty fields from unique result
15:08 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on group 0 (T188327) (duration: 00m 47s)
14:02 joal@deploy1001: Finished deploy [analytics/aqs/deploy@7cde8c8]: Update unique-devices schema adding 2 fields (duration: 20m 57s)
13:55 gtirloni: T207377 reboot cloudcontrol1004
13:43 moritzm: installing chromium security update on proton* (tested new upstream release in deployment-prep)
13:41 joal@deploy1001: Started deploy [analytics/aqs/deploy@7cde8c8]: Update unique-devices schema adding 2 fields
13:39 fdans@deploy1001: Finished deploy [analytics/aqs/deploy@7cde8c8]: Deploying AQS to add two new fields to uniques (duration: 06m 18s)
13:39 akosiaris: cumin -b1 -s 300 'ores2*' 'enable-puppet "merge of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474158/" ; puppet agent -t ; service uwsgi-ores restart ; service celery-ores-worker restart'
13:36 akosiaris: disable puppet on ores1* and ores2* for slow deployment of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474158/
13:33 fdans@deploy1001: Started deploy [analytics/aqs/deploy@7cde8c8]: Deploying AQS to add two new fields to uniques
13:21 gtirloni: T207377 icinga downtime and reboot of labcontrol1001 and labservices1001
13:08 arturo: T207377 icinga downtime and reboot of cloudcontrol1003 and cloudservices1003
13:06 raynor: EU SWAT finished
13:03 pmiazga@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:474679]|In SecurePoll use gpg1 to avoid gpg-agent autostart (T209802)]] (duration: 00m 48s)
12:50 raynor: EU SWAT reopened
12:40 raynor: EU SWAT finished
12:38 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:472918]|Enable autopatrol, patrol, rollback rights and RCPatrol on srwiktionary (T209252)]] (duration: 00m 46s)
12:21 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:474124]|Remove wgMetaNamespaceTalk for shnwiki (T206777)]] (duration: 00m 46s)
12:10 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:473225]|Enable Schema.org page split test at 50% sampling (T208755)]] (duration: 00m 46s)
11:33 gtirloni: labsdb1011 upgraded packages on labsdb1011 (pre-work T209517)
11:20 elukey: restart memcached on mc1020 to apply -R 200 settings (shard wiped) - T208844
10:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1078 T209754 (duration: 00m 46s)
10:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1078 T209754 (duration: 00m 46s)
10:21 banyek: stopping replication on db2076 (T85757)
10:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1078 T209754 (duration: 00m 46s)
09:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1123 and increase weight for db1078 T209754 (duration: 00m 46s)
09:48 marostegui: Rename table foundationwiki.petition_data on db1078 - T208979
09:46 marostegui: Drop empty testwiki.petition_data from db1075 with replication - T208979
09:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1123 and db1078 T209754 (duration: 00m 46s)
09:10 Nikerabbit: Rebuilt message group stats cache for T208521
08:43 banyek: executing schema change on db2095 (T85757)
07:57 marostegui: Stop MySQL on db1123 - T209754
07:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 to clone db1078 T209754 (duration: 00m 47s)
06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Add db1078 line back to config file but depooled T209754 (duration: 00m 51s)
06:20 marostegui@deploy1001: sync-file aborted: Add db1078 line back to config file but depooled T209754 (duration: 00m 02s)

2018-11-18

17:26 andrewbogott: restarting cp1078 from mgmt console
09:00 elukey: cleaned up analytics1039 and restarted Yarn

2018-11-17

off: 'reset modified attributes' on IcingaUI for db1078 (and mgmt) and all its services
06:38 oblivian@deploy1001: Synchronized wmf-config/db-eqiad.php: Depooling db1078 (duration: 00m 59s)
02:54 RoanKattouw: Deployed patches for T208112, T208109, T208110

2018-11-16

23:13 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@ee91c41]: Deploy on test wdqs1010 (duration: 00m 23s)
23:12 smalyshev@deploy1001: Started deploy [wdqs/wdqs@ee91c41]: Deploy on test wdqs1010
16:23 Trey314159: reindexing Chinese wikis on elastic@eqiad and elastic@codfw (T209156)
15:46 moritzm: rebooting debmonitor* instances for kernel security update and to pick up SSBD
15:01 hashar: restarting zuul with 1300 events to process
14:56 marostegui: Create ipblocks_restrictions on labswiki and labtestwiki on db1073 - T209674
14:56 ema: upgrade cp-ats to 8.0.0-1wm2 T204225 T204209
14:39 ema: trafficserver 8.0.0-1wm2 uploaded to stretch-wikimedia T204225 T204209
14:36 hashar: Gracefully stopping zuul (kill -SIGUSR1)
14:29 _joe_: re-depooling mw1261 for php-fpm testing
14:28 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw126[1-6].*,dc=eqiad,cluster=appserver
14:26 _joe_: repooling the mw canaries
13:02 moritzm: installing spamassassin security update on mendelevium
12:10 godog: reboot restbase1014, nothing on console
11:13 kartik@deploy1001: Finished deploy [cxserver/deploy@473b0de]: Update cxserver to b7cdb26 (T208831, T203077, T203160, T206777) (duration: 04m 26s)
11:09 kartik@deploy1001: Started deploy [cxserver/deploy@473b0de]: Update cxserver to b7cdb26 (T208831, T203077, T203160, T206777)
10:39 akosiaris: upgrade OTRS to 5.0.32 T209691
09:31 marostegui: Set back sync_binlog=1 and trx_commit=1 after dbstore2002:3313 has caught up
09:25 moritzm: installing postgres updates on labsdb1006
09:21 moritzm: removed labvirt1016 from debmonitor db, got renamed to cloudvirt1016
08:40 moritzm: installing curl security updates on jessie
07:32 elukey: forced reboot + fsck + removal of /var/lib/hadoop/data/l from fstab on analytics1029
06:36 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Pool pc2009 in pc3 - T208383 (duration: 00m 56s)
06:28 marostegui: Set sync_binlog=0 and trx_commit=2 on dbstore2002:3313 to let it catch up
05:47 vgutierrez: uploaded certcentral 0.7 to apt.wikimedia.org (stretch) - T208967 T209475
00:55 mutante: some users reported missing files in home dirs on mwmaint1002, reversed rsyncd/ferm setup and rsynced /home from mwmaint2001 to /root on mwmaint1002, restored individually where requested, rsync is not fully automatic but puppetized with rsync::quickdatacopy
00:35 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/includes/PageViews.php: SWAT: Exclude users where getRegistration() returns null (duration: 00m 47s)
00:26 eileen: civicrm revision changed from 71755d021b to 6b279509f8, config revision is 684ec9b7c0 (lybunt report)
00:22 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/GrowthExperiments: SWAT: gerrit:473843 gerrit:473844 gerrit:473845 (duration: 00m 49s)
00:21 sbisson@deploy1001: sync aborted: php-1.33.0-wmf.4/extensions/GrowthExperiments SWAT: gerrit:473843 gerrit:473844 gerrit:473845 (duration: 06m 16s)
00:15 sbisson@deploy1001: Started scap: php-1.33.0-wmf.4/extensions/GrowthExperiments SWAT: gerrit:473843 gerrit:473844 gerrit:473845

2018-11-15

21:58 mutante: mwmaint1002 - restoring entire /home of mwmaint1001 from Bacula (job queued and to tmp dir, not directly into /home)
21:06 hashar: Deleting Nodepool instances on contintcloud T209361
21:05 hashar: Stopped nodepool on labnodepool1001.eqiad.wmnet . Service is no more used. T209361 T209642
20:22 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@UNKNOWN]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471) (duration: 03m 55s)
20:18 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@UNKNOWN]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471)
20:17 mutante: re-added Chase to pwstore, signed .users file, re-encrypted all pwstore files, git pushed
20:15 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@48a1e83]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471) (duration: 04m 26s)
20:11 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@48a1e83]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471)
20:04 urandom: dropping disused keyspaces -- T208616
20:04 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable data collection for UnderstandingFirstDay on cswiki and kowiki (duration: 00m 53s)
19:51 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure sensitive namespaces for EditorJourney schema (T207307) (duration: 00m 53s)
19:49 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/: cherry-picks for T208773 (duration: 00m 54s)
18:55 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Hot-deploy I26f2dc2e: Don't over-ride default Wikibase string limits (duration: 00m 53s)
18:50 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: Hot-deploy Ib10de2e3: Don't set Wikibase string limits when null (duration: 00m 55s)
18:31 ladsgroup@deploy1001: Finished deploy [ores/deploy@dba11e9]: Another small update (duration: 13m 42s)
18:18 ladsgroup@deploy1001: Started deploy [ores/deploy@dba11e9]: Another small update
18:17 ladsgroup@deploy1001: Finished deploy [ores/deploy@51cdf6b]: T208623 (duration: 14m 41s)
18:03 ladsgroup@deploy1001: Started deploy [ores/deploy@51cdf6b]: T208623
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 7 wikis in group 2 for T209373. This may cause lag in codfw.
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 6 wikis in group 2 for T209373. This may cause lag in codfw.
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 5 wikis in group 2 for T209373. This may cause lag in codfw.
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 3 wikis in group 2 for T209373. This may cause lag in codfw.
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 2 wikis in group 2 for T209373. This may cause lag in codfw.
17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 1 wikis in group 2 for T209373. This may cause lag in codfw.
17:20 gehel: upgrade prometheus-blazegraph-exporter on all wdqs nodes - T206123
16:41 bstorm_: rebooted labsdb1007 for upgrades
16:37 andrewbogott: rebuilding labvirt1015 and cloudvirt1015
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on wikitech for T209373. This may cause lag in codfw.
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 8 wikis in group 1 for T209373. This may cause lag in codfw.
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 7 wikis in group 1 for T209373. This may cause lag in codfw.
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 5 wikis in group 1 for T209373. This may cause lag in codfw.
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 4 wikis in group 1 for T209373. This may cause lag in codfw.
14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 3 wikis in group 1 for T209373. This may cause lag in codfw.
14:20 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 2 wikis in group 1 for T209373. This may cause lag in codfw.
14:07 gehel: plugin and JVM upgrade on elasticsearch / cirrus / eqiad completed - T209293
13:59 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.4
13:55 XioNoX: push firewall policies to pfw3-eqiad - T209421
13:50 banyek: Deploy schema change on s6 codfw master, this will generate lag on s6 codfw - T85757
13:48 moritzm: installing qemu security updates (which also backport support for SSBD passthrough) on ganeti clusters
13:48 moritzm: installing qemu security updates (which also backport support for SSBD passthrough)
13:12 mobrovac@deploy1001: Finished deploy [restbase/deploy@22cb0ec]: Add new wikis to RESTBase - T206777 T205710 T205546 T204477 (duration: 19m 56s)
13:00 Lucas_WMDE: EU SWAT finished
12:59 tarrow@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js: gerrit:473710 Fix (accidentally?) reversed blue and yellow lines SWAT T208238 T162119 again (duration: 00m 55s)
12:57 _joe_: upping pm.maxworkers to 40 on mw1261 on php7.2-fpm, benchmarking T206341
12:56 tarrow@deploy1001: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
12:53 moritzm: installing nginx security updates
12:52 mobrovac@deploy1001: Started deploy [restbase/deploy@22cb0ec]: Add new wikis to RESTBase - T206777 T205710 T205546 T204477
12:47 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove wgWBQualityConstraintsCacheCheckConstraintsResults (T207854) (duration: 00m 54s)
12:42 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Prod: increase Schema.org page split test to 25% sampling T208755 (duration: 00m 53s)
12:40 arturo: T207377 downtime and reboot labmon1001
12:38 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js: Fix (accidentally?) reversed blue and yellow lines (T162119, T208238) (duration: 00m 54s)
12:37 lucaswerkmeister-wmde@deploy1001: sync aborted: php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js Fix (accidentally?) reversed blue and yellow lines (T162119, T208238) (duration: 00m 11s)
12:37 lucaswerkmeister-wmde@deploy1001: Started scap: php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js Fix (accidentally?) reversed blue and yellow lines (T162119, T208238)
12:30 tarrow@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:473716 Read WikibaseStringLimit in Wikibase.php T154660 (duration: 00m 53s)
12:30 moritzm: draining ganeti1001 for reboot/kernel security update
12:26 tarrow@deploy1001: sync-file aborted: gerrit:473716 Read WikibaseStringLimit in Wikibase.php (duration: 00m 01s)
12:21 tarrow@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:473694 Set Wikibase string-limits for wikidata dblist T154660 (duration: 00m 54s)
12:08 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Make AdvancedSearch the default on de-, fa-, ar-, and hu-wiki T207640 (duration: 00m 55s)
11:34 _joe_: depooling mw1261 for early benchmarks of php7.2-fpm
11:04 akosiaris: enable puppet across the fleet. puppetdb1001 reboot done, ganeti migration_downtime setting applied
10:49 akosiaris: disable puppet across the fleet for puppetdb1001 reboot
10:49 moritzm: fail over ganeti master in eqiad to ganeti1003
10:34 akosiaris: set migration_downtime=2000 for puppetdb1001. Should help with migration stalls
10:15 banyek: sanitizing db1124 ( T205714 T207584 T205713 T206916 )
10:07 banyek: sanitizing db2094 ( T205714 T207584 T205713 T206916 )
09:57 moritzm: draining ganeti1002 for reboot/kernel security update
09:40 volans: restarting icinga on icinga1001
09:32 vgutierrez: restarting icinga on icinga1001
09:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 52s)
09:22 moritzm: draining ganeti1003 for reboot/kernel security update
09:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 53s)
09:16 marostegui: Stop MySQL on db1110 for upgrade
09:08 moritzm: reset failed debmonitor session in ms-be2038
09:08 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1088 (duration: 00m 53s)
09:03 banyek: repooling db1088 (T85757)
08:58 ema: upload fifo-log-demux 0.1 to stretch-wikimedia T204225
08:56 banyek: Deploy schema change on db1088 (T85757)
08:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1088 (duration: 00m 53s)
08:45 banyek: depooling db1088 due a schema change (T85757)
08:21 moritzm: draining ganeti1004 for reboot/kernel security update
07:42 marostegui: Drop site_stats.ss_total_views from labswiki - T86339
07:08 elukey: memcached on mc1019 restarted to apply -R 200 - T208844
06:57 marostegui: Stop MySQL on db2071 to upgrade MySQL and kernel
06:57 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 54s)
06:06 marostegui: Stop MySQL on pc2006 to clone pc2009 - T208383
06:06 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2006 - T208383 (duration: 00m 53s)
06:05 marostegui@deploy1001: sync-file aborted: Dool pc2006 - T208383 (duration: 00m 00s)
05:59 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Pool pc2008 in pc2 - T208383 (duration: 00m 56s)
00:50 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: increase Schema.org page split test to 5% sampling T208755 (duration: 00m 54s)

2018-11-14

22:46 ejegg: updated payments-wiki from 5751286f1c to d2b66c5bab
21:34 thcipriani: restart gerrit to load JavaMelody dependency library
21:28 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on cobalt (duration: 00m 09s)
21:28 thcipriani@deploy1001: Started deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on cobalt
21:27 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on gerrit2001 (duration: 00m 11s)
21:26 thcipriani@deploy1001: Started deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on gerrit2001
20:13 hashar@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.4 (duration: 00m 52s)
20:12 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.4
19:14 hashar@deploy1001: Synchronized php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php: Actually return the value from getRootJobCacheKey() - T209429 (duration: 00m 53s)
18:33 hashar@deploy1001: Finished scap: php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php Add trace to debug memcached bad key error - T209429 (duration: 34m 07s)
17:58 hashar@deploy1001: Started scap: php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php Add trace to debug memcached bad key error - T209429
17:53 arturo: T207377 downtime and reboot cloudnet1003 (cloudnet1004 is the active one already)
17:37 arturo: T207377 downtime and reboot cloudnet1004 (cloudnet1003 is the active one already)
17:31 bawolff_: Running importImage.php for 'Opening ceremony of First accusation protest against presumption of guilt of judicial branch.webm' per request T209495
17:26 sbisson@deploy1001: Synchronized dblists/wikidataclient.dblist: SWAT: Add incubatorwiki to wikidataclient.dblist (duration: 00m 48s)
17:19 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/MobileFrontend/resources/mobile.editor.common/schemaEditAttemptStep.js: SWAT: schemaEditAttemptStep.js: Use correct config var name for sampling rate (duration: 00m 54s)
17:12 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: SWAT: Fix EditAttemptStepSamplingRate variable export (duration: 00m 54s)
16:33 bblack: [Done] replacement of GlobalSign unified TLS cert at US edges complete - T206804
16:25 moritzm: rebooting ganeti1005 for kernel security update
16:16 moritzm: rebooting restbase-dev1006 for kernel security update and OpenJDK security update
16:10 bblack: disabling puppet as precaution on all caches (cumin A:cp) - T206804
16:09 bblack: starting replacement of GlobalSign unified TLS cert at US edges (affects all public TLS termination for US traffic edges) - T206804
16:08 moritzm: rebooting restbase-dev1005 for kernel security update and OpenJDK security update
15:58 moritzm: rebooting restbase-dev1004 for kernel security update and OpenJDK security update
15:53 jiji: Restarting pdfrender on scb*.eqiad.wmnet
15:50 godog: roll restart swift-proxy in eqiad to apply statsd changes
15:41 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T85757: repool db2046 (duration: 00m 52s)
15:39 banyek: repooling db2046 (T85757)
15:12 godog: roll-restart swift on ms-be1* to pick up statsd changes
15:07 Amir1: ladsgroup@mwmaint1002:/srv/mediawiki-staging/php-1.33.0-wmf.4$ mwscript sql.php --wiki=incubatorwiki extensions/Wikibase/client/sql/entity_usage.sql (T209207)
14:56 addshore@deploy1001: Synchronized wmf-config: Prod: Enable Schema.org page split test at 1% sampling (again) (duration: 00m 54s)
14:29 godog: roll-restart swift-proxy in codfw to pick up statsd changes
14:14 addshore@deploy1001: Synchronized wmf-config: Revert Prod: Enable Schema.org page split test at 1% sampling (duration: 00m 54s)
14:10 Reedy: Wiki created T205714 T207584 T205713 T206916
14:07 gehel: starting plugin and JVM upgrade on elasticsearch / cirrus / eqiad - T209293
14:07 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 25s)
14:02 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix shnwiki TZ (duration: 00m 54s)
13:57 reedy@deploy1001: rebuilt and synchronized wikiversions files: (no justification provided)
13:55 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Add pc2010 as spare - T208383 (duration: 00m 53s)
13:54 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: (no justification provided) (duration: 00m 53s)
13:51 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: new wikis (duration: 00m 53s)
13:51 gehel: restarting tilerator on maps1004 for config change
13:50 reedy@deploy1001: Synchronized static/images/project-logos/: (no justification provided) (duration: 00m 53s)
13:49 reedy@deploy1001: Synchronized dblists/: new wikis! (duration: 00m 53s)
13:48 reedy@deploy1001: Synchronized langlist: shn (duration: 00m 52s)
13:45 gehel: plugin and JVM upgrade completed on elasticsearch / cirrus / codfw - T209293
13:45 reedy@deploy1001: rebuilt and synchronized wikiversions files: (no justification provided)
13:07 moritzm: installing ghostscript security updates on stretch
13:01 reedy@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaMaintenance/addWiki.php: Fixing addshores code... (duration: 00m 53s)
13:00 reedy@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMaintenance/addWiki.php: Fixing addshores code... (duration: 00m 55s)
12:56 moritzm: installing gettext "security" updates for trusty
12:48 moritzm: installing python3.4 security updates on trusty (Debian already fixed)
12:40 reedy@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding wiktionary (duration: 00m 52s)
12:39 reedy@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding wiktionary (duration: 00m 53s)
12:39 moritzm: installing python security updates on trusty
12:36 Amir1: EU SWAT is done
12:35 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert the language of votewiki to English (en) (T207560) (duration: 00m 55s)
12:30 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Start reading from change_tag_def on wikidatawiki (T208846) (duration: 00m 55s)
12:19 pmiazga@deploy1001: Synchronized wmf-config: SWAT: [[gerrit:473079]|Enable Schema.org page split test at 1% sampling (T208755)]] (duration: 00m 54s)
12:02 reedy@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding TMH tables (duration: 00m 53s)
12:01 reedy@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding TMH tables (duration: 00m 55s)
11:11 moritzm: draining ganeti1005 for reboot/kernel security update
11:09 banyek: Deploy schema change on db2046 (T85757)
10:07 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T85757: depooling db2046 (duration: 00m 55s)
09:59 banyek: depooling db2046 (T85757)
09:22 moritzm: updated stretch netinst image for 9.6 point release
09:17 marostegui: Deploy schema change on db2053 - T86339
08:24 marostegui: Deploy schema change on s5 codfw master, this will generate lag on s5 codfw - T205913
08:22 marostegui: Deploy schema change on s2 codfw master, this will generate lag on s7 codfw - T205913
08:19 marostegui: Deploy schema change on s2 codfw master, this will generate lag on s2 codfw - T205913
08:17 marostegui: Deploy schema change on s6 codfw master, this will generate lag on s6 codfw - T205913
08:14 marostegui: Deploy schema change on s4 codfw master, this will generate lag on s4 codfw - T205913
08:08 marostegui: Deploy schema change on s3 codfw master, this will generate lag on s3 codfw - T205913
08:07 godog: rollout rsyslog_exporter to eqiad
07:42 marostegui: Deploy schema change on s3 codfw master, this will generate lag on s3 codfw - T203709
07:19 marostegui: Deploy schema change on s7 codfw master, this will generate lag on s7 codfw - T203709
07:07 marostegui: Deploy schema change on s2 codfw master, this will generate lag on s2 codfw - T203709
06:52 marostegui: Deploy schema change on s4 codfw master, this will generate lag on s4 codfw - T203709
06:40 marostegui: Deploy schema change on s6 codfw master, this will generate lag on s6 codfw -T203709
06:32 marostegui: Stop MySQL on pc2005 to clone it to pc2008 - T208383
06:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2005 - T208383 (duration: 01m 04s)
05:46 _joe_: restarting gerrit
01:02 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/Wikibase/client/includes: SWAT: Update: use wikibase-debug logger instead of "PageRandomLookup" T208796 (duration: 00m 56s)
00:42 mutante: restarted smokeping on netmon1002 and netmon2001

2018-11-13

22:42 XioNoX: restart librenms irc bot
22:24 XioNoX: add term labnet-nova-api to cloud-in4 on cr1/2-eqiad - T209424
20:22 herron: updated labs realm smarthosts (via hiera) to mx-out0[12].wmflabs.org T41785
19:49 otto@deploy1001: Finished deploy [analytics/refinery@62d6f4b]: Deploy hive jars from CDH 5.10.0 to workaround Refine bug: T209407 (duration: 05m 57s)
19:43 otto@deploy1001: Started deploy [analytics/refinery@62d6f4b]: Deploy hive jars from CDH 5.10.0 to workaround Refine bug: T209407
19:31 herron: uploaded librdkafka_0.11.6-1~bpo9+1+wikimedia1 packages to stretch-wikimedia T209300
18:11 mutante: the CUSTOM message from ores.svc.codfw was the (one-time) test of the new Icinga server
18:03 mutante: icinga migration has concluded, we are now on stretch and icinga1001, einsteinium is passive (T202782)
17:27 mutante: re-enabled puppet on icinga1001, einsteinium becoming passive
17:21 mutante: ran puppet on einsteniumr; e-enabling puppet on tegmen and icinga1001
17:13 bstorm_: Added 172.16.0.0/21 to the allowed connections for wikilabels postgresql on labsdb1004
17:04 mutante: disabled puppet on all 3 icinga servers, re-enabling on einsteinium , going through https://wikitech.wikimedia.org/wiki/Icinga#Failover_Icinga_between_the_active_and_passive_servers
17:02 ejegg: updated payments-wiki from 20542c9184 to 5751286f1c
17:01 mutante: starting migration of icinga server - maintenance windows
16:33 thcipriani: restarting gerrit service for upgrade to 2.15.6
16:32 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@d2763c6]: v2.15.6 to cobalt (duration: 00m 10s)
16:32 thcipriani@deploy1001: Started deploy [gerrit/gerrit@d2763c6]: v2.15.6 to cobalt
16:29 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@d2763c6]: v2.15.6 to gerrit2001 (duration: 00m 11s)
16:29 thcipriani@deploy1001: Started deploy [gerrit/gerrit@d2763c6]: v2.15.6 to gerrit2001
16:22 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on test wikis and mediawikiwiki (T188327) (duration: 00m 54s)
16:07 anomie@mwmaint1002: Running refreshExternallinksIndex.php on labtestwiki for T209373
16:07 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 3 wikis in group 0 for T209373
15:48 _joe_: upgrading extensions on all appservers / jobrunners while upgrading to php 7.2
15:45 gehel: restart tilerator on maps1004
15:21 moritzm: draining ganeti1006 for reboot/kernel security update
15:18 marostegui: Restore replication consistency options on dbstore2002:3313 as it has caught up - T208320
14:59 akosiaris: increase the migration downtime for kafkamon1001. It should make live migration of these VMs easier and without the need for manual fiddling
14:54 hashar@deploy1001: rebuilt and synchronized wikiversions files: group to 1.33.0-wmf.4 | T206658
14:40 hashar@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.4 | T206658 (duration: 19m 34s)
14:27 moritzm: draining ganeti1007 for reboot/kernel security update
14:20 hashar@deploy1001: Started scap: testwiki to php-1.33.0-wmf.4 | T206658
14:20 akosiaris: reboot logstash1007, logstash1008, logstash1009 with 500 secs of sleep between them for the migration_downtime ganeti setting to be applied
14:18 akosiaris: increase the migration downtime for logstash1007, logstash1008, logstash1009. It should make live migration of these VMs easier and without the need for manual fiddling
14:15 hashar@deploy1001: Pruned MediaWiki: 1.32.0-wmf.24 (duration: 08m 55s)
14:03 hashar: Applied security patches to 1.33.0-wmf.4 | T206658
14:03 gehel: start plugin and JVM upgrade on elasticsearch / cirrus / codfw - T209293
14:00 hashar: scap prep 1.33.0-wmf.4 # T206658
13:58 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Pool pc2007 to replace pc2004 (duration: 00m 48s)
13:41 marostegui: Deploy schema change on s8 codfw master (db2045) this will generate lag on s8 codfw - T203709
13:40 hashar: Cutting wmf/1.33.0-wmf.4 branch | T206658
13:30 moritzm: draining ganeti1008 for reboot/kernel security update
12:51 phuedx: European Mid-day SWAT finished
12:50 phuedx@deploy1001: Finished scap: SWAT: Define WikimediaMessages for Wikibase SEO change l18n refresh (duration: 21m 43s)
12:28 phuedx@deploy1001: Started scap: SWAT: Define WikimediaMessages for Wikibase SEO change l18n refresh
12:22 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMessages/: SWAT: Define WikimediaMessages for Wikibase SEO change (T208755) (duration: 00m 56s)
10:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 52s)
10:47 marostegui: Deploy schema change on db1116:3318 T203709
10:40 godog: stop sending metrics to old graphite hardware
10:15 gehel: restart elasticsearch on relforge for plugin upgrade - T209293
09:54 moritzm: restarting jenkins on releases1001 to pick up Java security update
09:25 _joe_: uploading new versions of php-msgpack, php-geoip compatible with both php 7.0 and php 7.2 to thirdparty/php72 T208433
09:23 marostegui: Deploy schema change on db1092 T203709
09:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 52s)
09:20 elukey: rollout new prometheus-mcrouter-exporter to mw* - previous rollout didn't work as expected
09:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 (duration: 00m 55s)
08:37 moritzm: updating remaining rsyslog on stretch to 8.38.0-1~bpo9+1wmf1
07:21 marostegui: Deploy schema change on db1104 T203709
07:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 (duration: 00m 53s)
07:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 54s)
07:05 elukey: powercycle lvs2006 - mgmt/serial console blank, not responsive since hours ago
06:02 marostegui: Add ipb_sitewide column to db1073:labtestwiki
05:43 marostegui: Stop MySQL on pc2004 to transfer its data to pc2007 - T208383
05:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2004 - T208383 (duration: 00m 53s)
05:39 marostegui: Deploy schema change on db2048 (s1 codfw master), this will create lag on s1 codfw - T114117
05:34 marostegui: Deploy schema change on db1109 T203709
05:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 55s)

2018-11-12

19:22 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T208663 4ff32d1df - Enable moving files for users with patrol and rollbacker rights on srwiki (duration: 00m 54s)
18:29 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@ee91c41]: GUI update, New Thesaurus endpoint, New updater build and blazegraph update (duration: 11m 28s)
18:17 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@ee91c41]: GUI update, New Thesaurus endpoint, New updater build and blazegraph update
18:03 elukey: rolling restart of aqs on aqs* to pick up new druid datasource settings
17:44 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Optimize s2 for throughput (duration: 00m 53s)
17:19 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool more resources into s2 api (duration: 00m 54s)
17:15 _joe_: restarting HHVM on the high-cpu api hosts in eqiad, to ease the pressure and latencies
17:10 _joe_: depooling mw1222 for debug
16:41 banyek: disabling puppet on parsercache hosts (T208383)
16:14 elukey: upgrade prometheus-mcrouter-exporter on all the mw* hosts to the new version
16:09 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for large wikis
16:02 volans: restarted proton on proton1002
15:45 jynus: stop and upgrade db2094
15:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 53s)
14:49 banyek: disabling puppet on parsercache hosts - pc[12]00[456] (T208383)
14:17 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for medium wikis
14:08 moritzm: updating rsyslog on stretch to 8.38.0-1~bpo9+1wmf1
13:59 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for small wikis (small.dblist)
13:59 marostegui: Deploy schema change on db1101:3318 - T203709
13:55 hashar: Upgrading Jenkins on contint1001 , contint2001, releases1001 and releases2002 | T209264
13:46 moritzm: updating libfastjson on stretch to 0.99.8-1~bpo9+1wmf1
13:41 gehel: starting rolling restart of elasticsearch codfw for JVM upgrade
13:32 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for mediawikiwiki
13:23 phuedx: phuedx@mwmaint1002 running resetPageRandom.php maintenance script for testwiki
13:17 zeljkof: EU SWAT finished
13:16 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/maintenance/resetPageRandom.php: SWAT: Provide a script to reset the page_random column (T208909) (duration: 00m 53s)
13:16 moritzm: updating liblognorm on stretch to 2.0.3-1~bpo9+1wmf1
13:14 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/autoload.php: SWAT: Provide a script to reset the page_random column (T208909) (duration: 00m 55s)
13:12 elukey: upgrade the Hadoop Analytics cluster to CDH 5.15 (downtime required)
12:54 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add new throttle rule for Wikipedia event in Ireland on 2018-11-13 (T209037) (duration: 00m 53s)
12:15 jiji: Restarting nutcracker on scb200[1-6] - T206450
12:00 moritzm: uploaded jenkins 2.138.3 to apt.wikimedia.org (jessie and stretch)
11:49 hashar: updating puppet CI job for mtail upgrade https://gerrit.wikimedia.org/r/#/c/integration/config/+/472962/
11:37 hashar: contint1001 : cleaning disk | T209123 ?
11:26 moritzm: installing Java security updates on elastic*
10:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 00m 55s)
10:48 godog: upload mtail 3.0.0~rc5-1~bpo9+1wmf1 to stretch-wikimedia
10:45 marostegui: Deploy schema change on db2048 (s1 codfw master), this will generate lag on s1 codfw - T51191
10:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 (duration: 00m 53s)
10:32 elukey: upload mcrouter exporter 0.0.0+git20181106 to stretch-wikimedia
09:57 elukey: upgraded cdh packages (cdh 5.10 -> 5.15) for thirdparty/cloudera in jessie/stretch-wikimedia
09:12 marostegui: Deploy schema change on db2048 (s1 codfw master) (replication will be stopped) - T67448
08:53 marostegui: Deploy schema change on db1099:3318 - T203709
08:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 (duration: 01m 01s)
08:41 marostegui: Change sync_binlog to 0 and trx_commit to 2 on dbstore2002:3313 to let it catch up
{{safesubst:SAL entry|1=08:26 _joe_: uploading new php-{luasandbox,wikidiff2} to stretch main component, rebuild php-{luasandbox,wikidiff2,geoip,msgpack} for php 7.2, upload to stretch component php72, T208433}}
08:23 godog: temporarily disable puppet in codfw before enabling rsyslog_exporter

2018-11-10

01:10 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 39s)
01:10 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting
01:07 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 04s)
01:07 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting

2018-11-09

21:46 SMalyshev: repooled wdqs1004 - looks like other servers feel worse so probably makes sense to share the load equally
21:16 jiji: Reimaging rdb2003, rdb2004 - T206450
20:46 SMalyshev: depooled wdqs1004 to let it catch up
20:40 legoktm@deploy1001: Synchronized docroot/mediawiki/keys/: Add my (20after4) PGP key to mediawiki.org/keys/keys.(txt|html) (duration: 00m 55s)
20:08 andrewbogott: restarted neutron-linuxbridge-agent on cloudvirt1018 and cloudvirt1023
away: repooling labsdb1011 (T189158)
15:24 banyek: depooling labsdb1011 (T189158)
15:23 banyek: depooling labsdb1011
15:08 banyek: repooling labsdb1009 (T189158)
15:06 bblack: cp1008/pinkunicorn: puppet disabled, public-facing testing of new globalsign 2018 certs
15:04 ladsgroup@deploy1001: Finished deploy [ores/deploy@bb39f4b]: T191842 T209060, try II (duration: 14m 43s)
14:50 andrewbogott: rebooting cloudvirt1024 to (I hope) cause a page
14:49 ladsgroup@deploy1001: Started deploy [ores/deploy@bb39f4b]: T191842 T209060, try II
14:48 ladsgroup@deploy1001: deploy aborted: T191842 T209060 (duration: 09m 32s)
14:39 ladsgroup@deploy1001: Started deploy [ores/deploy@0728805]: T191842 T209060
14:18 addshore@deploy1001: Synchronized wmf-config: BETA ONLY: Enable SSR termbox for wikibase on beta - T209143 (duration: 00m 56s)
13:32 moritzm: rebooting acrab for some qemu tests
13:21 godog: upload graphite-web_1.0.2+debian-2.1wmf1 to stretch-wikimedia - T208782
13:10 moritzm: upgrading qemu on ganeti2001 (packages supporting SSBD passthrough)
12:40 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T189158: repool db1106 (duration: 00m 53s)
12:34 banyek: repooling db1106 (T208954)
12:17 kartik@deploy1001: Finished deploy [cxserver/deploy@fc21164]: Update cxserver to 01686f6 (T208831) (duration: 01m 09s)
12:16 kartik@deploy1001: Started deploy [cxserver/deploy@fc21164]: Update cxserver to 01686f6 (T208831)
11:45 banyek: data load finished restarting replication on db1106 (T208954)
11:43 akosiaris: set previous normal wait for scb1001 for apertium service T206439
11:39 akosiaris@puppetmaster1001: conftool action : set/weight=8; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
11:30 akosiaris: upgrade apertium apertium-cat apertium-fra apertium-fra-cat apertium-lex-tools apertium-separable cg3 libapertium3-3.5-1 libcg3-1 lttoolbox on all scb boxes and restart apertium-apy
11:26 akosiaris: upgrade apertium apertium-cat apertium-fra apertium-fra-cat apertium-lex-tools apertium-separable cg3 libapertium3-3.5-1 libcg3-1 lttoolbox on scb1002
11:22 jiji: switch scb*.eqiad.wmnet nutcracker rdb1003:6382 with rdb1005:6379
10:51 vgutierrez: uploaded certcentral 0.6 to apt.wikimedia.org (stretch) - T208859 T208948 T208967 T208970
09:48 ema: repool cp2018, cp2025 (cache_upload) T208588
09:45 banyek: truncating enwiki.archive on db1124 and labsdb hosts too (T208954)
09:21 banyek: stopping replication on db1106 (T208954)
09:21 banyek: stopping replication on db1106 (T208672)
09:08 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T189158: depool db1106 (duration: 00m 55s)
09:02 banyek: depooling db1106 (T208954)
08:28 moritzm: installing nginx security updates
08:05 ema: repool cp2006, cp2012 (cache_text) T208588
04:33 ejegg: restarted recurring donation charge jobs
04:24 ejegg: updated fundraising CiviCRM from 1154cca3f2 to 71755d021b
03:25 ejegg: updated fundraising CiviCRM from 02cc1f80d4 to 1154cca3f2
00:07 ejegg: updated fundraising CiviCRM from 07183ed7cc to 02cc1f80d4
00:03 ejegg: updated payments-wiki from 983ce3af0f to 20542c9184

2018-11-08

22:48 mutante: gerrit - adding Thomas Arrow to 'wmf-deployment' group for +2 on mw-config for T208491 access request
22:37 mutante: gerrit - adding Lucas Werkmeister (WMDE) to 'wmf-deployment' group for +2 on mw-config for T208518 access request
20:28 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.3
19:35 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Disable wmgUseTwoColConflict everywhere" T205942 T208840 T209012 T209036 (duration: 00m 54s)
18:19 shdubsh: update statsd-proxy to 0.0.9-2 on graphite1004
17:29 banyek: depooling labsdb1009 (T189158)
17:24 banyek: repooling labsdb1010 (T189158)
17:07 godog: upload libfastjson 0.99.8-1~bpo9+1wmf1 version bump only
16:59 akosiaris@deploy1001: scap-helm zotero finished
16:59 akosiaris@deploy1001: scap-helm zotero cluster staging completed
16:59 akosiaris@deploy1001: scap-helm zotero [namespace: zotero, clusters: staging]
16:50 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
16:50 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
16:29 XioNoX: enable Zayo transit on cr3-ulsfo
15:42 chasemp: disable /etc/logrotate.d/udp2log-mw for a bit on mwlog1001
15:25 Amir1: rolling restart of celery on ores nodes (T209060)
15:20 akosiaris: 'cd /srv/deployment/ores/deploy/submodules/wheels && sudo -u deploy-service git lfs pull' on all ores1* and ores2* hosts T209060
15:07 XioNoX: zeroize asw-c8-codfw (decom)
14:12 moritzm: rebooting releases2001 for some tests with ssbd for KVM
13:52 moritzm: installing postgres updates on labsdb1006/1007
13:38 jiji: Done reimaging rdb1006 - T206450
13:37 moritzm: draining ganeti2001 for reboot/kernel security update
13:36 moritzm: failing over ganeti master in codfw from ganeti2001 to ganeti2003
13:13 godog: upload rsyslog 8.38.0-1~bpo9+1wmf1 to stretch-wikimedia, version bump only
13:07 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-enable wmgUseTwoColConflict on dewiki - T205942 T208840 T209012 T209036 (duration: 00m 53s)
12:56 akosiaris: increase weight of scb1001 for apertium to 99+%
12:56 akosiaris@puppetmaster1001: conftool action : set/weight=3800; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
12:53 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-enable wmgUseTwoColConflict on group0 only T205942 T208840 T209012 T209036 (duration: 00m 54s)
12:52 moritzm: draining ganeti2002 for reboot/kernel security update
12:41 jiji: Shutdown and reimage rdb200[56] - T206450
12:31 moritzm: draining ganeti2003 for reboot/kernel security update
12:30 zeljkof: EU SWAT finished
12:29 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/TwoColConflict: SWAT: Fix harmless edits turning into conflicts (T205942 T208840 T209012 T209036) (duration: 00m 55s)
12:19 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set AdvancedSearch to default on group0 wikis (T207641) (duration: 00m 55s)
12:18 moritzm: draining ganeti2004 for reboot/kernel security update
11:57 moritzm: draining ganeti2005 for reboot/kernel security update
11:51 akosiaris: increase weight of scb1001 for apertium to 50%
11:50 akosiaris@puppetmaster1001: conftool action : set/weight=38; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
11:41 moritzm: draining ganeti2006 for reboot/kernel security update
11:18 moritzm: draining ganeti2007 for reboot/kernel security update
11:05 moritzm: draining ganeti2008 for reboot/kernel security update
10:52 jiji: Reimaging rdb1006 to stretch - T206450
10:52 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable wmgUseTwoColConflict everywhere T209012 T208840 T195724 (duration: 00m 58s)
10:26 elukey: restart memcached on mc2029 (was depooled yesterday for network maintenance)
10:23 jiji: restarting pdfrender on scb1003
10:19 volans: restarting pdfrender on scb1004
10:18 volans: restarting pdfrender on scb1002
10:18 _joe_: restarting pdfrender on scb1001
10:02 moritzm: installing ppp security updates on trusty
09:37 godog: keep 2x not 3x copies of older (>15d) logstash elasticsearch indices
09:29 moritzm: installing curl security updates
09:29 godog: temporarily set elasticsearch logstash watermark to low:0.85 and high:0.9
06:34 bawolff@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/OpenStackManager/special/SpecialNovaSudoer.php: T203885 (duration: 00m 54s)
05:30 bawolff: deployed patch T208881
01:18 mutante: scb1004 - systemctl restart pdfrender (T174916)
00:43 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/includes/resourceloader/ResourceLoader.php: ResourceLoader: Fail less hard when JSON serialization of config fails I673f59d93 (duration: 00m 53s)
00:33 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T205368 Enable BotPasswords on Governance wiki (duration: 00m 55s)
00:32 ejegg: updated fundraising CiviCRM from 769dcf6456 to 07183ed7cc
00:26 James_F: Created the bot_passwords table for Governance wiki T205368
00:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208449 Disable wgWelcomeSurveyEnabled everywhere in production (duration: 00m 54s)
00:18 jforrester@deploy1001: Synchronized wmf-config/extension-list: T208081 Drop the Petition extension from extension-list (duration: 00m 53s)
00:16 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208081 Drop the Petition extension from InitialiseSettings (duration: 00m 52s)
00:14 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T208081 Drop the Petition extension from CommonSettings (duration: 00m 53s)
00:12 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208899 Enabling wgMediaInTargetLanguage for testwiki (duration: 00m 54s)
00:00 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208081 Disable the Petition extension in production (duration: 00m 52s)

2018-11-07

23:48 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Make GrowthExperiments flag operative in CommonSettings (duration: 00m 53s)
23:44 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add flag for GrowthExperiments to InitialiseSettings (duration: 00m 53s)
23:37 catrope@deploy1001: Finished scap: Full scap to rebuild i18n for the addition of the GrowthExperiments extension (duration: 39m 40s)
23:21 jiji: Disabled nagios checks on rdb1006 and rdb2005 due to rdb1005 reimaging - T206450
22:57 catrope@deploy1001: Started scap: Full scap to rebuild i18n for the addition of the GrowthExperiments extension
22:13 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Revert "labswiki rollback to 1.33.0-wmf.2"
22:07 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/LdapAuthentication/LdapAuthenticationPlugin.php: Expose methods used by OpenStackManager T208995 (duration: 00m 54s)
22:06 XenoRyet: updated payments-wiki from 34506ce636 to 983ce3af0f
22:02 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis T208986 (duration: 00m 54s)
22:02 arlolra: Updated Parsoid to 970751a (T206940)
21:54 arlolra@deploy1001: Finished deploy [parsoid/deploy@4edc771]: Updating Parsoid to 970751a (duration: 09m 34s)
21:45 arlolra@deploy1001: Started deploy [parsoid/deploy@4edc771]: Updating Parsoid to 970751a
21:21 ladsgroup@deploy1001: Finished deploy [ores/deploy@25dfa4f]: T191842 T197096 (duration: 17m 24s)
21:18 krinkle@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/AbuseFilter/includes/AbuseFilter.php: T208144 - I0fdda5 (duration: 00m 53s)
21:16 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/VipsScaler: Id9f82afd (duration: 00m 55s)
21:06 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/AbuseFilter/includes/AbuseFilter.php: T208144 - I0fdda51010243 (duration: 00m 53s)
21:06 banyek: stopping replication on db2072 (T208954)
21:04 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/includes/jobqueue/jobs/RefreshLinksJob.php: T208147 -I7f5fafe9439d8a7b4 (duration: 00m 54s)
21:03 ladsgroup@deploy1001: Started deploy [ores/deploy@25dfa4f]: T191842 T197096
20:55 banyek: depool labsdb1010 (T189158)
20:24 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: rollback labswiki to 1.33.0-wmf.2
20:12 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.3 (duration: 00m 53s)
20:11 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.3
20:00 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T206173 Adding namespaces to Governance wiki (duration: 00m 55s)
19:50 chasemp: labstore1007:~# mkdir /srv/security/
19:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-sync for skipped apaches due to maintenance (whoops) (duration: 00m 55s)
19:48 XioNoX: Revert "Redirect eqsin/ulsfo caches to eqiad" - T208272
19:47 XioNoX: repool codfw - T208272
19:45 XioNoX: asw-c-codfw maintenance finished successfuly - T208272
18:51 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T201285: Disable wgRawHTML on Governance wiki (duration: 05m 12s)
18:31 onimisionipe: restarting relforge-eqiad and relforge-eqiad-small-alpha clusters on relforge100[1-2]
18:21 XioNoX: power down asw-c4-codfw - T208272
17:31 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add sourceswiki to wikidata clients (duration: 00m 53s)
17:25 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Inject wikidata rc records on wikidata itself (duration: 00m 53s)
17:21 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WMEUnderstandingFirstDay on testwiki (duration: 00m 53s)
16:37 XioNoX: remove asw-c-codfw FPC8 from config - T208272
16:35 XioNoX: shutdown asw-c-codfw FPC8 - T208272
16:33 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.ui.less: T208898 Hot-deploy to wmf.3 (duration: 00m 53s)
16:22 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/Echo/modules/styles: Hot-deploy T208930 to wmf.3 (duration: 00m 54s)
16:20 XioNoX: Enable all VC ports (except uplinks) on spines - T208272
15:58 XioNoX: Redirect eqsin/ulsfo caches to eqiad - T208272
15:57 XioNoX: depool codfw for row C maintenance - T208272
15:36 moritzm: installing Java security updates on relforge*
15:29 jiji@deploy1001: Synchronized wmf-config/ProductionServices.php: Remove jobqueue_redis references, T198220 (duration: 00m 54s)
15:21 akosiaris: T206439 direct 30% of the apertium.svc.eqiad.wmnet traffic to scb1001. Will increase tomorrow to 50%
15:20 akosiaris@puppetmaster1001: conftool action : set/weight=15; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
15:16 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
15:15 akosiaris: T206439 pool upgraded scb1001 to apertium.svc.eqiad.wmnet as a form of canary
15:13 moritzm: uploaded nginx 1.13.6-2+wmf2 to apt.wikimedia.org/stretch-wikimedia
14:55 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
14:37 oblivian@deploy1001: Synchronized docroot/wwwportal/w/search-redirect.php: Fixing redirects if no language is specified (duration: 00m 54s)
14:33 moritzm: uploaded nginx 1.13.6-2+wmf2~jessie1 to apt.wikimedia.org/jessie-wikimedia
14:32 akosiaris: T206439 upload apertium-cat_2.6.0-1+wmf1 apertium-fra-cat_1.5.0-1+wmf1 apertium-fra_1.5.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
14:26 bblack: rebooting graphite1004
14:16 akosiaris: T206439 upload apertium-separable_0.3.2-1+wmf1 apertium-lex-tools_0.2.1-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
14:07 akosiaris: T206439 upload apertium_3.5.2-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
13:43 bblack: hi
13:21 Amir1: ladsgroup@mwmaint1002:/srv/mediawiki/php-1.33.0-wmf.1$ mwscript sql.php --wiki=sourceswiki extensions/Wikibase/client/sql/entity_usage.sql (T208858)
12:38 zeljkof: EU SWAT finished
12:36 zfilipin@deploy1001: Synchronized wmf-config: SWAT: BC: Enable Schema.org page split test (T208763) (duration: 00m 54s)
12:35 akosiaris: T206439 upload hfst-ospell_0.5.0-1+wmf1to apt.wikimedia.org/jessie-wikimedia/main
12:27 akosiaris: T206439 upload cg3_1.1.7-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
12:16 zfilipin@deploy1001: Synchronized wmf-config/InterwikiSortOrders.php: SWAT: Add dty, gor, inh, kbp and lfn to InterwikiSortOrders (T208217) (duration: 00m 53s)
12:12 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Art&Feminism event in Chile (T208866) (duration: 00m 54s)
12:12 akosiaris: T206439 upload hfst_3.15.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
12:08 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Remove expired throttle rules (duration: 01m 05s)
11:49 akosiaris: T206439 upload lttoolbox_3.5.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
11:49 akosiaris: T206439 upload lttoolbox_3.5.0-1+wmf1
10:06 hashar: CI: switched operations/puppet job to be based on Stretch ( T208422 ) and to add python3 ( T208873 )
09:25 _joe_: run systemctl reset-failed on ms-be1029, had a failed debmonitor session
08:16 kartik@deploy1001: Finished deploy [cxserver/deploy@6f97d25]: Update cxserver to f9ffd24 (duration: 04m 59s)
08:11 kartik@deploy1001: Started deploy [cxserver/deploy@6f97d25]: Update cxserver to f9ffd24
02:55 ejegg: disabled recurring charge jobs
00:46 mutante: tegmen - shutting down for renaming and reinstall (T208824)
00:11 dereckson@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 44s)

2018-11-06

23:55 mutante: cp1084 - network went down, powercycled, probably T203194
22:49 ejegg: updated fundraising CiviCRM from e0742d2210 to 769dcf6456
21:50 mutante: icinga1001-"MediaWiki EtcdConfig up-to-date" checks were all UNKNOWN because systemd unit update-etcd-mw-config-lastindex was present but service not running. it was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" and the checks all work (T202782)
21:49 mutante: icinga1001 - the "MediaWiki EtcdConfig up-to-date" checks were all unknown on the new icinga server, this was because systemd unit update-etcd-mw-config-lastindex was present but service not running. that was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" to start it and the checks
21:40 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Group0 wikis to 1.33.0-wmf.3
20:19 thcipriani@deploy1001: Finished scap: testwiki to 1.33.0-wmf.3 and rebuild l10n cache (duration: 34m 06s)
19:45 thcipriani@deploy1001: Started scap: testwiki to 1.33.0-wmf.3 and rebuild l10n cache
19:44 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.23 (duration: 07m 19s)
18:36 thcipriani: cutting branch for MediaWiki and extensions version 1.33.0-wmf.3
17:37 XioNoX: add vlan-analytics1-a-eqiad interface-range on asw2-a-eqiad
16:47 XioNoX: enable cr4-ulsfo zayo transport to cr1-codfw
16:23 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
16:23 akosiaris@deploy1001: scap-helm zotero install --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
16:12 mutante: einsteinium - temp disabling icinga notifications and puppet, reloading icinga (for extra caution while deploying global NRPE change)
16:07 mutante: planet1001 - disabling puppet, editing NRPE config, testing allowed_hosts change
16:07 akosiaris@deploy1001: scap-helm -h finished
16:07 akosiaris@deploy1001: scap-helm -h cluster staging completed
16:07 akosiaris@deploy1001: scap-helm -h [namespace: -h, clusters: staging]
15:59 banyek: updating facts for the puppet compilers
15:37 akosiaris: create zotero namespace in eqiad, codfw, staging cluster T201611
15:20 godog: switch all graphite read traffic to graphite1004
15:16 XioNoX: push `lldp port-id-subtype interface-name` to all compatible switches - T208630
15:14 jiji: scb1001/scb1002 switched nutcracker redis from rdb1001:6382 to rdb1009:6379
15:08 XioNoX: push `lldp port-id-subtype interface-name` to all routers - T208630
14:40 jynus_: reducing consistenct temp. on db2048 to avoid lagging
14:30 moritzm: installing Ruby 2.1 security updates
14:22 godog: add graphite1004 to graphite cluster for reads
14:21 moritzm: installing clamav security updates on mendelevium/ticket.wikimedia.org
13:25 moritzm: restart HHVM on canaries to pick up new curl
13:10 XioNoX: zeroize asw-b-eqiad (decom) - T208788
12:33 moritzm: installing curl security updates
12:17 moritzm: installing chromium security updates on proton* (new upstream release tested in deployment-prep)
12:12 zeljkof: EU SWAT finished
12:10 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update several Wikidata-related configs (duration: 00m 55s)
11:37 moritzm: installing Ruby 2.3 security updates
11:37 moritzm: installing Ruby 2.3 security updates on trusty
11:30 moritzm: installing Mono security updates
11:23 moritzm: installing Ruby 1.9 security updates on trusty
11:07 banyek: stopping replication on db2077 (T208672)
07:25 _joe_: also restarting on the other eqiad nodes
07:25 _joe_: restarting tilerator on maps1002
04:47 kartik@deploy1001: Finished deploy [cxserver/deploy@ddb0031]: Update cxserver to 17f9a10 (T144467, T198699, T208386) (duration: 05m 26s)
04:42 kartik@deploy1001: Started deploy [cxserver/deploy@ddb0031]: Update cxserver to 17f9a10 (T144467, T198699, T208386)
03:40 eileen: civicrm revision changed from 99895316de to e0742d2210, config revision is e832b5a04a
00:26 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471875/ (duration: 00m 51s)
00:07 maxsem@deploy1001: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471874/ (duration: 00m 54s)
00:03 eileen: civicrm revision changed from 042eeaeca9 to 99895316de, config revision is e832b5a04a

2018-11-05

22:41 mutante: sodium - reboot after disk replacement (T202705)
21:51 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@a92fce5]: Increase cirrusSearchLinksUpdate concurrency to 100 (duration: 01m 02s)
21:50 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@a92fce5]: Increase cirrusSearchLinksUpdate concurrency to 100
21:49 arlolra: Updated Parsoid to 8ed698b (T205334, T208360)
21:42 mobrovac@deploy1001: Started restart [zotero/translation-server@50f216a]: Free up some memory
21:41 arlolra@deploy1001: Finished deploy [parsoid/deploy@96d739b]: Updating Parsoid to 8ed698b (duration: 10m 59s)
21:38 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: Removed decomissioned citoid url T133001 (duration: 00m 53s)
21:30 arlolra@deploy1001: Started deploy [parsoid/deploy@96d739b]: Updating Parsoid to 8ed698b
21:23 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 take 2 (duration: 09m 18s)
21:14 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 take 2
21:10 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 (duration: 12m 15s)
21:09 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Ensure all wikis on 1.33.0-wmf.2
20:58 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324
20:38 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c] (dev-cluster): Update deps, removed sections table (duration: 03m 40s)
20:35 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c] (dev-cluster): Update deps, removed sections table
19:53 akosiaris: do a depool, scap deploy, scap wikiversions-compile, hhvm restart and then a pool in eqiad mediawiki servers
{{safesubst:SAL entry|1=19:50 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:470868|Disable page issues A/B test (duration: 00m 53s)}}
19:44 sbisson@deploy1001: Synchronized php-1.33.0-wmf.2/includes/block/BlockRestriction.php: SWAT: BlockRestriction::update() unnecessarily does a SELECT on the page table. (duration: 01m 00s)
19:19 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable error logging for WikimediaEvents (duration: 00m 52s)
19:12 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable GrowthExperiments logging channel (duration: 00m 53s)
18:57 _joe_: restarting hhvm on mwdebug1002
18:44 godog: pool graphite1004 for reads - T196484
18:43 XioNoX: delete asw2-b - asw-b interface - T183585
18:41 XioNoX: remove asw-b-eqiad from LibreNMS - T183585
18:37 XioNoX: remove vrrp priority 70 on cr2-eqiad:ae2 to failback VIPs to cr2 - T183585
18:26 XioNoX: re-enable ae2 on cr2-eqiad - T183585
18:21 thcipriani: rollback mwdebug1001 group2 wikis
18:13 thcipriani: testing php-1.33.0-wmf.2 on group2 wikis on mwdebug1001
18:05 XioNoX: disable ae2 on cr2-eqiad - T183585
18:02 XioNoX: set vrrp priority 70 on cr2-eqiad:ae2 to failover VIP to cr1 - T183585
16:49 XioNoX: Update LLDP config on cr3-ulsfo - T208630
16:48 vgutierrez: uploaded certcentral 0.5 to apt.wikimedia.org (stretch) - T208572 T208378
16:06 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting MCR to read-new on all wikis (T198308) (duration: 00m 55s)
13:57 jynus_: increase consistency of db2050, dbstore2002 s3 after them catching up replication T208462
12:33 ladsgroup@deploy1001: Finished deploy [ores/deploy@096ffb3]: T208577 T181632 T208608 (duration: 22m 58s)
12:23 zeljkof: EU SWAT finished
12:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Increase wikidata dispatchers to 3 (duration: 00m 54s)
12:16 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgForeignUploadTargets to [] for zhwiki (T208397) (duration: 00m 54s)
12:10 ladsgroup@deploy1001: Started deploy [ores/deploy@096ffb3]: T208577 T181632 T208608
12:05 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Revert "Anniversary logo for cswiki" (T207589) (duration: 00m 58s)
10:02 godog: reformat xfs filesystems on ms-be1040 - T199198
09:17 elukey@deploy1001: Finished deploy [analytics/refinery@9d39efa]: fixing stat1004 (duration: 00m 04s)
09:17 elukey@deploy1001: Started deploy [analytics/refinery@9d39efa]: fixing stat1004
09:08 joal@deploy1001: Finished deploy [analytics/refinery@9d39efa]: regular analytics weekly deploy (duration: 05m 21s)
09:02 joal@deploy1001: Started deploy [analytics/refinery@9d39efa]: regular analytics weekly deploy

2018-11-04

23:42 jynus_: deleting the same row on all s8 broken servers
23:39 jynus_: deleting one row on db1104
20:38 krinkle@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: T176232 - Ia43626584e (duration: 01m 17s)
18:32 jynus_: reduce temp. consistency level of s4, s5, and s6 codfw masters to prevent excessive lagging due to ongoing mediawiki core maintenance
08:42 eileen: process-control config revision is e832b5a04a renable running job list (all jobs on again now0
08:38 eileen: process-control config revision is e16b2c1c61 renable jobs
02:00 eileen: I think I got the rest of the jobs off process-control config revision is 4422254128
01:52 eileen: process-control config revision is 6ec67b3d01 - also turn off omnirecipient repair job
01:40 eileen: process-control config revision is 5b72cfe874 - reapply turn off q jobs

2018-11-03

09:35 elukey: run tcpdump on mc1035 to grab memcache traffic (rotating pcaps, ~30G maximum)

2018-11-02

17:04 thcipriani: rollback group2 wikis to 1.33.0-wmf.1 on mwdebug100{1,2}
16:54 thcipriani: deploying 1.33.0-wmf.2 to group2 wikis on mwdebug1002
16:43 _joe_: live-hacking removal of time limit on mwdebug1001
16:32 thcipriani: deploying 1.33.0-wmf.2 to group2 wikis on mwdebug1001
15:12 jynus: restarting replication @ db2074 after db2094:s3 table fix T208565
15:00 jynus: stopping replication on db2074 to fix db2094:s3 T208565
14:01 vgutierrez: reimaging eeden.wikimedia.org as jessie test system - T208583
11:43 jynus: ignoring cawikimedia.archive replication on db2094:s3 until a reimport happens T208565
11:29 jijiki: Rebooting mw2244 (spare system) for maintenance
10:52 ema: restart varnish-be on cp3032 T208574
08:19 jynus: performing alter table on dbstore2002 s3 and reducing consistency to improve recovery time T208462 T204006
08:01 jynus: reducing consitency on db2050 to improve recovery time T208462
07:59 jynus: performing alter table on db2050 T208462 T204006
07:38 godog: reformat ms-be1043 xfs filesystems - T199198
07:38 jynus: reducing consistency temporarily (flush, binlog sync) at db2040 to prevent lagging
07:26 jynus: reducing consistency temporarily (flush, binlog sync) at db2035 to prevent lagging

2018-11-01

23:01 shdubsh: restart hhvm on mw1261
22:29 ejegg: restarted fundraising queue consumer jobs
22:21 ejegg: updated fundraising CiviCRM from 65130ef3dd to 042eeaeca9
22:18 ejegg: turned off fundraising queue jobs for civi update
22:12 _joe_: rolling restart of hhvm on appservers and api in eqiad
22:09 shdubsh: cumin -b 2 -s 30 "O:mediawiki::appserver and *.eqiad.wmnet" "restart-hhvm"
22:05 _joe_: restarting hhvm on mw1238,1240
22:02 _joe_: restart hhvm on mw1244
21:52 shdubsh: restart hhvm on mw1247
21:49 _joe_: depooling mw1238 for debugging
21:09 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group2 back to 1.33.0-wmf.1
20:55 hoo: Restarted hhvm on mwdebug2002
19:40 hoo: Ran "UPDATE wb_changes_dispatch SET chd_seen = '775203911' WHERE chd_site LIKE '%wikt%' AND chd_seen < '775180000';" on wikidata master (dispatching for wiktionaries)
19:00 hoo@deploy1001: Synchronized php-1.33.0-wmf.1/includes/export/WikiExporter.php: Fix for missing end tag </page> on some exports (T207974) (duration: 01m 01s)
18:38 hoo@deploy1001: Synchronized php-1.33.0-wmf.2/includes/export/WikiExporter.php: Fix for missing end tag </page> on some exports (T207974) (duration: 00m 55s)
18:25 jijiki: Enabling puppet on mw servers (T206923)
18:19 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove now redundant Wikidata config for wiktionary (T208317) (duration: 00m 54s)
18:12 hoo@deploy1001: Synchronized dblists/wikidataclient.dblist: Add all wiktionaries to wikidataclient.dblist, sort list (T208317) (duration: 00m 57s)
18:02 gehel: restart nginx on relforge100*
17:57 jijiki: Disabling puppet on mw servers (T206923)
16:07 anomie@mwmaint1002: Running migrateComments.php on section 4 wikis for T166733
13:46 anomie@mwmaint1002: Running migrateComments.php on remaining section 3 wikis for T166733
13:37 anomie@mwmaint1002: Running migrateComments.php on section 7 wikis for T166733
13:37 anomie@mwmaint1002: Running migrateComments.php on wikitech for T166733
13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on wikitech for T188132
13:37 anomie@mwmaint1002: Running migrateComments.php on section 6 wikis for T166733
13:37 anomie@mwmaint1002: Running migrateComments.php on section 8 wikis for T166733
13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 8 wikis for T188132
13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 7 wikis for T188132
13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 6 wikis for T188132
13:36 anomie@mwmaint1002: Running migrateComments.php on section 5 wikis for T166733
13:36 anomie@mwmaint1002: Running migrateComments.php on section 1 wikis for T166733
13:36 anomie@mwmaint1002: Running migrateComments.php on section 2 wikis for T166733
13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 5 wikis for T188132
13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 4 wikis for T188132
13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on remaining section 3 wikis for T188132
13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 2 wikis for T188132
13:35 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 1 wikis for T188132
12:50 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: List wikidataclient-test in CS.php dblists T208488 (duration: 00m 57s)
09:10 elukey: added a tmux session on mw1314m mw1344, mw1316 that checks mcrouter stats every 10s
00:58 onimisionipe: repooling wdqs1004. It has caught up on lag with others
00:22 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/SyntaxHighlight_GeSHi/extension.json: SWAT: Follow-up I3daca6fb: Fix exception thrown when inserting new code block (invalidate RL cache) (duration: 00m 53s)
00:20 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/SyntaxHighlight_GeSHi/modules/ve-syntaxhighlight/ve.ui.MWSyntaxHighlightWindow.js: SWAT: Follow-up I3daca6fb: Fix exception thrown when inserting new code block (duration: 00m 54s)
00:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@e8f3a85] (dev-cluster): Add title normalisation and remove Accept-Language header duplicates (duration: 03m 00s)
00:10 mobrovac@deploy1001: Started deploy [restbase/deploy@e8f3a85] (dev-cluster): Add title normalisation and remove Accept-Language header duplicates
00:07 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Move auth logging to different channels for easier counting (T150300, T123243) (duration: 00m 53s)
00:05 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Move auth logging to different channels for easier counting (T150300, T123243) (duration: 00m 53s)

Server Admin Log

2018-12-10

2018-12-09

2018-12-08

2018-12-07

2018-12-06

2018-12-05

2018-12-04

2018-12-03

2018-12-02

2018-12-01

2018-11-30

2018-11-29

2018-11-28

2018-11-27

2018-11-26

2018-11-25

2018-11-23

2018-11-22

2018-11-21

2018-11-20

2018-11-19

2018-11-18

2018-11-17

2018-11-16

2018-11-15

2018-11-14

2018-11-13

2018-11-12

2018-11-10

2018-11-09

2018-11-08

2018-11-07

2018-11-06

2018-11-05

2018-11-04

2018-11-03

2018-11-02

2018-11-01

Archives

Navigation menu

Search