Server Admin Log

From Wikitech
(Redirected from Server admin log)
Jump to navigation Jump to search

2018-12-10

  • 09:07 marostegui: Deploy schema change on s8 codfw master with replication (db2045) - lag will be generated on codfw - T202167 T86338
  • 09:01 marostegui: Depool labsdb1011 - T86338
  • 08:58 moritzm: installing chromium security updates on proton*
  • 08:57 marostegui: Repool labsdb1010 - T86338
  • 08:39 elukey: roll restart of aqs on aqs100* to pick up new Druid backend settings
  • 08:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 T86338 T202167 (duration: 00m 51s)
  • 08:23 godog: final round of weight addition to new ms-be codfw hosts - T209395
  • 07:18 _joe_: reenabling puppet given my changes were useless
  • 06:52 _joe_: running puppet on the puppetmasters in codfw, twice, then restarting apache to ensure cleanup of any cache
  • 06:50 _joe_: disabled puppet across the fleet for merge of hiera change
  • 06:47 marostegui: Stop slave on s4 on labsdb1011
  • 06:25 marostegui: Deploy schema change on db1121 with replication (this will generate lag on labs) - T86338 T202167
  • 06:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 T86338 T202167 (duration: 00m 49s)
  • 06:23 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - T86338
  • 02:53 urandom: decommissioning cassandra-b, restbase2004 -- T210843
  • 00:55 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ic07ff9acfbe17 - T211529, T205546 (duration: 00m 47s)

2018-12-09

  • 20:16 urandom: decommissioning cassandra-a, restbase2004 -- T210843
  • 18:39 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Icb9ad2f554e1 - Fix ruwikinews logo config (duration: 00m 57s)
  • 11:55 godog: decommissioning cassandra-c, restbase2003 -- T210843
  • 03:00 urandom: decommissioning cassandra-b, restbase2003 -- T210843

2018-12-08

  • 23:07 krinkle@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/: T185446 - I6cf29d598a11 (duration: 00m 47s)
  • 22:21 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209794 (duration: 00m 56s)
  • 20:59 urandom: decommissioning cassandra-a, restbase2003 -- T210843
  • 13:54 godog: decommissioning cassandra-c, restbase2002 -- T210843

2018-12-07

  • 23:54 ejegg: updated payments-wiki from b99cd0816e to b8acb95a2a
  • 23:41 urandom: decommissioning cassandra-b, restbase2002 -- T210843
  • 22:43 ejegg: re-enabled Thank You mail sender
  • 22:31 ejegg: updated fundraising CiviCRM from 3e5d74f17e to 8e18485697
  • 22:29 ejegg: Turned off Thank You mailing job for letter update
  • 17:47 herron: rebooting logstash1006 for security updates
  • 17:17 bstorm_: T207377 rebooted labstore1007 for kernel upgrades
  • 17:15 _joe_: uploading php-tideways (rebuilt with php 7.2 support) to stretch-wikimedia thirdparty/php72 T206152
  • 15:37 moritzm: rebooting sarin/neodymium
  • 14:52 urandom: decommissioning cassandra-a, restbase2002 -- T210843
  • 14:21 godog: more weight to new ms-be codfw hosts - T209395
  • 13:37 moritzm: rolling reboot of scb in codfw (along with nodejs update)
  • 12:18 moritzm: installing nodejs security updates on stat/notebook
  • 12:05 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title (duration: 21m 11s)
  • 11:43 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title
  • 11:42 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 00m 21s)
  • 11:42 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
  • 11:41 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 18m 58s)
  • 11:22 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
  • 11:20 moritzm: rolling upgrade of nginx on swift frontends
  • 11:19 mobrovac@deploy1001: Finished deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title (duration: 03m 49s)
  • 11:15 mobrovac@deploy1001: Started deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title
  • 11:09 mobrovac@deploy1001: Finished deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero (duration: 06m 19s)
  • 11:03 mobrovac@deploy1001: Started deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero
  • 10:58 mobrovac@deploy1001: Finished deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero (duration: 02m 42s)
  • 10:55 mobrovac@deploy1001: Started deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero
  • 09:40 banyek: importing back linkwatcher_linklog into database s51230__linkwatcher on host labsdb1004.eqiad.wmnet. - T211210
  • 09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
  • 09:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
  • 08:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
  • 08:21 marostegui: Stop MySQL on db1096:3315,3316 for kernel and mysql upgrade
  • 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315, db1096:3316 for kernel and mysql upgrade (duration: 00m 46s)
  • 08:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1084 (duration: 00m 47s)
  • 07:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
  • 07:50 godog: decommissioning cassandra-c, restbase2001 -- T210843
  • 07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
  • 07:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1084 (duration: 00m 46s)
  • 07:11 marostegui: Stop MySQL on db1084 for mysql and kernel upgrade
  • 07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 49s)
  • 00:47 XioNoX: done troubleshoting bird bfd on dns2001/cr1-codfw
  • 00:10 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: turn off wbsearchentities ab test T209402 (duration: 00m 47s)

2018-12-06

  • 23:47 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API T201192 (duration: 02m 06s)
  • 23:47 XioNoX: troubleshoot bird bfd on dns2001/cr1-codfw
  • 23:45 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API T201192
  • 23:21 ppchelko@deploy1001: Finished deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification T201192 (duration: 22m 46s)
  • 22:58 ppchelko@deploy1001: Started deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification T201192
  • 22:12 urandom: decommissioning cassandra-b, restbase2001 -- T210843
  • 21:39 gtirloni: reimaging cloudvirt1019 with jessie T196507
  • 21:33 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event (duration: 01m 15s)
  • 21:32 ppchelko@deploy1001: Started deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event
  • 21:07 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA (duration: 09m 18s)
  • 21:00 XioNoX: remove 2 esams avoid path + 4 prefered/selected transits - T194542
  • 20:58 smalyshev@deploy1001: Started deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA
  • 20:51 XioNoX: remove 2 eqiad avoid path - T194542
  • 20:48 gtirloni: reimaging cloudvirt1019 with stretch T196507
  • 20:45 XioNoX: remove codfw/eqdfw avoid path - T194542
  • 19:32 gehel: shutting down elasticsearch on elastic2001-2024 (third time is a charm) - T211023
  • 18:21 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642) (duration: 03m 54s)
  • 18:17 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642)
  • 17:31 moritzm: installing nodejs updates on proton*
  • 17:11 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for jessie-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
  • 16:27 urandom: decommissioning cassandra-a, restbase2001 -- T210843
  • 16:17 gehel: shutting down elasticsearch on elastic2001-2024 (second try) - T211023
  • 15:51 moritzm: upgrading spamassassin on mx1001/fermium
  • 15:45 fsero@deploy1001: scap-helm zotero finished
  • 15:45 fsero@deploy1001: scap-helm zotero cluster codfw completed
  • 15:45 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 15:42 fsero: modifying zotero deploy CLUSTER=codfw scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero - T211322
  • 15:42 _joe_: disabling puppet fleet-wide for a change in the role() function
  • 15:38 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for stretch-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
  • 15:08 gehel: restartign new elasticsearch masters on codfw - T211023
  • 15:05 gehel: upgrade nginx on wdqs servers
  • 14:59 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10 (duration: 00m 09s)
  • 14:59 elukey@deploy1001: Started deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10
  • 14:46 moritzm: uploaded nodejs 10.4.0~dfsg-1+wmf2 to apt.wikimedia.org/component/node10 (backports of recent security fixes)
  • 14:16 moritzm: installing nginx security updates on mw in eqiad
  • 13:46 moritzm: upgrading spamassassin on mx2001
  • 12:56 gehel: depooling and shutting down elasticsearch on elastic2001-2024 - T211023
  • 12:55 moritzm: installing nginx updates on mw in codfw
  • 11:59 moritzm: installing nginx updates on mw canaries
  • 10:59 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1096:3315 (duration: 00m 47s)
  • 10:56 volans: disable event handler on Icinga for ms-be2047 MD Raid and MegaRAID checks, it's spamming Phabricator - T209921
  • 10:56 banyek: repooling db1096 for schema change - T85757
  • 10:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1096:3315 (duration: 00m 49s)
  • 10:29 banyek: depooling db1096 for schema change - T85757
  • 09:56 dcausse: elastic@codfw cleanup: deleting wikidatawiki_content_1537469318 index (failed reindex probably)
  • 01:03 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
  • 00:55 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 47s)
  • 00:42 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
  • 00:40 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
  • 00:38 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
  • 00:15 mutante: MPM prefork tweaks for high load systems are applied again (apparently they were not since a change in the past that resulted in 2 competing configs in mods-enabled and conf-enabled with the latter one being loaded last and containing the package defaults
  • 00:13 mutante: re-enabling puppet on phabricator, applying change that adds php-fpm support on stretch ..which doesnt affect phab1001 (prod) on jessie.. BUT re-adds tuning config from the past for mpm_prefork.conf (more SpareServers etc) that was not actually applied due to a bug
  • 00:10 urandom: bootstrapping cassandra-c, restbase2018 -- T210843

2018-12-05

  • 23:50 ejegg: updated payments-wiki from 20595cca97 to b99cd0816e
  • 23:40 ejegg: re-enabled fundraising queue consumer jobs
  • 23:33 ejegg: updated fundraising CiviCRM from e757753a46 to 3e5d74f17e
  • 23:32 ejegg: turned off fundraising queue jobs for base queue consumer logic update
  • 22:43 jijiki: restarting pdfreder on scb* hosts in eqiad
  • 21:44 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 02m 47s)
  • 21:41 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362
  • 21:41 mdholloway: mobileapps deployment failed for group default03, rolling back and retrying
  • 21:39 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 18m 46s)
  • 21:39 arlolra: Updated Parsoid to a6058e3 (T210647, T208360, T205333)
  • 21:37 urandom: bootstrapping cassandra-b, restbase2018 -- T210843
  • 21:23 arlolra@deploy1001: Finished deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3 (duration: 11m 36s)
  • 21:20 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362
  • 21:12 arlolra@deploy1001: Started deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3
  • 20:08 banyek: repooling labsdb1010 - T210693
  • 19:12 cmjohnson1: cloudvirt1019 for an all inclusive part swap by HPE
  • 18:53 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Fix formatting of help panel links (duration: 00m 47s)
  • 18:01 jijiki: uploaded thumbor_6.3.2+git20170607-1+deb9u1 to stretch-wikimedia
  • 17:23 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Enable Kartographer maps on testwikidatawiki (T184933) (duration: 00m 46s)
  • 17:22 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
  • 17:20 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
  • 17:19 XioNoX: remove private IPs from codfw cloud-instance-transport1-b T207663
  • 17:11 XioNoX: add public IPs to codfw cloud-instance-transport1-b T207663
  • 16:58 XioNoX: re-deactivate ams-ix prefix list entry on cr2-esams
  • 16:53 XioNoX: activate ams-ix prefix list entry on cr2-esams
  • 16:27 jijiki: uploaded python-thumbor-wikimedia_2.2-1+deb9u1 to stretch-wikimedia
  • 16:18 akosiaris@deploy1001: scap-helm zotero finished
  • 16:18 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 16:18 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 16:17 akosiaris@deploy1001: scap-helm zotero finished
  • 16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 16:17 akosiaris@deploy1001: scap-helm zotero finished
  • 16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 16:15 akosiaris@deploy1001: scap-helm zotero finished
  • 16:15 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:15 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 16:08 fsero: redeploying zotero on eqiad
  • 16:02 akosiaris@deploy1001: scap-helm zotero finished
  • 16:02 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 16:02 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 stable/zotero [namespace: zotero, clusters: eqiad,codfw]
  • 16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 [namespace: zotero, clusters: eqiad,codfw]
  • 15:34 thcipriani: restarting ci jenkins for update
  • 15:33 akosiaris: add back pods/portforward right to kubernetes deploy user. T211040
  • 15:07 anomie: Running cleanupUsersWithNoId.php on potentially missed s3 and s7 wikis for T181731
  • 15:03 fsero: repooling citoid mathoid eqiad
  • 15:03 godog: bootstrap cassandra-c, restbase2017 - T210843
  • 14:56 banyek: executing schema change on s5 codfw master replication lag could be expected - T85757
  • 14:54 fsero: upgrading k8s on eqiad to 1.10.11
  • 14:54 elukey: restart HDFS namenode and Yarn resource manager on an-master100[1,2] to update rack topology config - T209929
  • 14:51 fsero: depool mathoid/eqiad: pooled changed True => False
  • 14:51 fsero: depool citoid/eqiad: pooled changed True => False
  • 14:43 anomie: Running cleanupUsersWithNoId.php on metawiki for T181731 / T210985
  • 14:09 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary item/property access for all wiktionaries (T175273) (duration: 00m 47s)
  • 13:53 akosiaris: repool citoid/mathoid codfw
  • 12:59 onimisionipe: banning elastic2001-elastic2024 from codfw production, psi and omega clusters
  • 12:53 jijiki: uploaded python-thumbor-community-core_0.4.0-1+deb9u1 to stretch-wikimedia
  • 12:47 dcausse: EU SWAT done
  • 12:45 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=(cit|math)oid,name=codfw
  • 12:44 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] Add temp clusters but still write to the old ones 2/2 (duration: 00m 46s)
  • 12:42 dcausse@deploy1001: Synchronized wmf-config/CommonSettings.php: T210381: [cirrus] Add temp clusters but still write to the old ones 1/2 (duration: 00m 46s)
  • 12:31 fsero: pooling mathoid and citoid again on codfw
  • 12:30 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T202497) (duration: 00m 46s)
  • 12:29 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T202497) (duration: 00m 49s)
  • 12:23 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211188: Increase autoconfirmed count for Meta-Wiki to 5 (duration: 00m 47s)
  • 12:15 fsero: upgrading codfw k8s cluster to 1.10.11
  • 12:15 dcausse: running namespaceDupes & cirrus indexNamespaces on yuewiktionary
  • 12:11 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T205546: Define 2 new namespaces for yuewiktionary (duration: 00m 47s)
  • 12:02 fsero: depooling mathoid and citoid servers on codfw for k8s upgrade
  • 11:28 mobrovac@deploy1001: Finished deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - T211127 (duration: 05m 55s)
  • 11:23 mobrovac@deploy1001: Started deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - T211127
  • 11:05 akosiaris: upgrade kubernetes-client and kubernetes-master on staging to 1.10.11
  • 11:04 godog: bootstrap cassandra-b, restbase2017 - T210843
  • 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 (duration: 00m 45s)
  • 10:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 45s)
  • 10:51 ema: cache hosts: begin nginx rolling upgrade to 1.13.6-2+wmf2
  • 10:46 marostegui: Reboot db1090 for kernel upgrade
  • 10:44 moritzm: uploaded jenkins 2.138.4 to jessie-wikimedia/thirdparty and stretch-wikimedia/thirdpary/ci
  • 10:42 marostegui: Stop MySQL on db1090:3312 and db1090:3317 for MySQL upgrade
  • 10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 46s)
  • 10:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 46s)
  • 10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1080 (duration: 00m 46s)
  • 10:21 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
  • 10:17 arturo: T205969 icinga downtime the load avg check in labstore1007 for 1 week
  • 10:10 banyek: depooling labsdb1010 for testing materialized views - T210693
  • 10:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1080 (duration: 00m 46s)
  • 09:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly pool db1080 (duration: 00m 46s)
  • 09:32 gehel: setting up new elasticsearch servers on codfw - elastic2045-2054 - T210265
  • 09:22 marostegui: Stop MySQL on db1080 for mysql and kernel upgrade
  • 09:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for MySQL upgrade (duration: 00m 46s)
  • 09:07 elukey: matomo read only + upgrade to matomo 3.7.0 on matomo1001 - T209808
  • 09:00 _joe_: disabed puppet on mw1261, used for logging tests for T211184
  • 08:48 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207850 Define a new Wikibase log channel to use (duration: 00m 47s)
  • 08:01 moritzm: installing pdns-recursor security update in esams
  • 07:59 godog: bootstrap cassandra-a, restbase2017 - T210843
  • 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 T86338 T202167 (duration: 00m 46s)
  • 06:55 marostegui: Deploy schema change on db1091 T86338 T202167
  • 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 T86338 T202167 (duration: 00m 51s)
  • 04:03 urandom: bootstrapping cassandra-c, restbase2016 -- T210843
  • 03:11 kartik@deploy1001: Finished deploy [cxserver/deploy@a3dd2ca]: Update cxserver to c4240e6 and enable Youdao MT (T208985, T210578) (duration: 04m 26s)
  • 03:06 kartik@deploy1001: Started deploy [cxserver/deploy@a3dd2ca]: Update cxserver to c4240e6 and enable Youdao MT (T208985, T210578)
  • 00:32 dcausse: Evening SWAT done
  • 00:30 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [cirrus] prepare multi-instance services (T210381) (duration: 00m 46s)
  • 00:28 dcausse@deploy1001: Synchronized wmf-config/ProductionServices.php: [cirrus] prepare multi-instance services (T210381) (duration: 00m 46s)
  • 00:16 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Block notice stats on itwiki (T210452) (duration: 00m 47s)
  • 00:10 urandom: bootstrapping cassandra-b, restbase2016 -- T210843

2018-12-04

  • 23:46 eileen: civicrm revision changed from a411d6bd64 to e757753a46, config revision is 0e6ccc37fe
  • 23:33 XioNoX: update prefix-list peering4 on cr1-eqsin to match jnt
  • 22:46 XioNoX: remove neodymium/sarin from term labs-in4 on cr1/2-eqiad - T210612
  • 22:40 ejegg: Updated payments-wiki from 7403a196b4 to 20595cca97
  • 21:58 XioNoX: clear ethernet-swtiching table for labvirt1004:eth1's switch port
  • 21:57 XioNoX: clear ethernet-swtiching table for labvirt1009:eth1's switch port
  • 21:49 XioNoX: make cr1/2-codfw conform to jnt
  • 21:44 XioNoX: make cr2-eqord/eqdfw conform to jnt
  • 21:41 XioNoX: make cr3/4-ulsfo conform to jnt
  • 20:04 urandom: bootstrapping cassandra-a, restbase2016 -- T210843
  • 19:35 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@81dac18]: Install new Updater for T210044 investigation (duration: 10m 36s)
  • 19:24 smalyshev@deploy1001: Started deploy [wdqs/wdqs@81dac18]: Install new Updater for T210044 investigation
  • 18:04 joal@deploy1001: Finished deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint (duration: 17m 33s)
  • 18:03 akosiaris: bump zotero pod number from 4 to 16 in eqiad/codfw
  • 17:47 joal@deploy1001: Started deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint
  • 17:46 ppchelko@deploy1001: Finished deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance T210465 (duration: 01m 13s)
  • 17:45 ppchelko@deploy1001: Started deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance T210465
  • 17:22 Reedy: created oathauth tables on punjabiwikimedia T211110
  • 17:16 godog: bootstrap cassandra-c on restbase2015 - T210843
  • 16:48 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure 'api-warning' log channel (duration: 00m 47s)
  • 15:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 T86338 T202167 (duration: 00m 47s)
  • 14:04 elukey: upgrade turnilo on analytics-tools1002 to nodejs-10 - T210705
  • 13:56 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=bnwikisource --fix --add-prefix=T210472
  • 13:53 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=euwiki --fix
  • 13:52 marostegui: Deploy schema change on db1103:3314 T86338 T202167
  • 13:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 T86338 T202167 (duration: 00m 47s)
  • 13:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 T86338 T202167 (duration: 00m 46s)
  • 13:34 cdanis: T210416: adding grafana 5 to wikimedia-stretch: reprepro --restrict grafana update stretch-wikimedia
  • 13:33 moritzm: installing nodejs security updates on restbase in codfw
  • 13:32 godog: bootstrap cassandra-b on restbase2015 - T210843
  • 13:29 moritzm: installing nodejs security updates on proton*
  • 13:19 marostegui: Deploy schema change on db1081 T86338 T202167
  • 13:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 T86338 T202167 (duration: 00m 46s)
  • 13:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 T86338 T202167 (duration: 00m 46s)
  • 12:39 Lucas_WMDE: EU SWAT done
  • 12:38 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create namespace "Work" on bnwikisource (T210472) (duration: 00m 46s)
  • 12:33 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create List namespace on euwiki (T209834) (duration: 00m 47s)
  • 12:28 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: Revert "Milestone logo for atjwiki" (T200713) (duration: 00m 47s)
  • 12:17 moritzm: installing tiff security updates
  • 11:49 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided) (duration: 04m 47s)
  • 11:44 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided)
  • 11:41 moritzm: rebooting puppetboard2001 to pick up SSBD-enabled qemu
  • 11:38 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - T197242 (duration: 00m 21s)
  • 11:38 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - T197242
  • 11:37 moritzm: rebooting puppetboard1001 to pick up SSBD-enabled qemu
  • 11:36 akosiaris: enable puppet on scb1004, run puppet T197242
  • 11:36 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - T197242 (duration: 00m 20s)
  • 11:35 marostegui: Deploy schema change on db1084 T86338 T202167
  • 11:35 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - T197242
  • 11:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 T86338 T202167 (duration: 00m 47s)
  • 11:34 akosiaris: enable puppet on scb1003, run puppet T197242
  • 11:33 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - T197242 (duration: 00m 28s)
  • 11:33 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - T197242
  • 11:31 akosiaris: enable puppet on scb1002, run puppet T197242
  • 11:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 T86338 T202167 (duration: 00m 46s)
  • 11:18 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - T197242 (duration: 00m 30s)
  • 11:18 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - T197242
  • 11:17 akosiaris: enable puppet on scb1001, run puppet T197242
  • 11:12 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - T164201 (duration: 09m 06s)
  • 11:04 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - T211088 T197242 (duration: 20m 59s)
  • 11:03 elukey@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - T164201
  • 10:59 fdans@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques (duration: 00m 37s)
  • 10:58 fdans@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques
  • 10:57 fdans: deploying AQS to expose offset and underestimate numbers on unique devices
  • 10:51 moritzm: rebooting analytics-tool1003 to pick up SSBD-enabled qemu
  • 10:47 moritzm: rebooting analytics-tool1002 to pick up SSBD-enabled qemu
  • 10:43 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - T211088 T197242
  • 10:41 moritzm: rebooting analytics-tool1001 to pick up SSBD-enabled qemu
  • 10:03 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - T197242 (duration: 01m 45s)
  • 10:02 godog: bootstrap cassandra-a on restbase2015 - T210843
  • 10:01 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - T197242
  • 09:59 gehel: upgrading nginx on elasticsearch eqiad
  • 09:54 akosiaris: enable puppet on all scb2*, run puppet T197242
  • 09:52 gehel: upgrading nginx on elasticsearch codfw
  • 09:52 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - T197242 (duration: 00m 30s)
  • 09:51 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - T197242
  • 09:50 akosiaris: enable puppet on scb2001, run puppet T197242
  • 09:46 akosiaris: disable puppet on scb for citoid migration to zoterov2 T197242
  • 09:46 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
  • 09:31 gehel: add elastic2039-2044 to cirrus eqiad (new server) - T210265
  • 09:11 gehel: add elastic2038 to cirrus eqiad (new server) - T210265
  • 09:00 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/MediaWiki/electronpdf/action # Ran https://phabricator.wikimedia.org/P7882 for T157012
  • 08:54 marostegui: Deploy schema change on db1097:3314 T86338 T202167
  • 08:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 T86338 T202167 (duration: 00m 47s)
  • 08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*-*.wsp" -delete # T120639
  • 08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*_*.wsp" -delete # T120639
  • 08:43 marostegui: Deploy schema change on db1102:3314 T86338 T202167
  • 08:42 marostegui: Deploy schema change on dbstore1002:s4 T86338 T202167
  • 08:41 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/datamodel$ sudo -u _graphite rm wikipedia_references.wsp # T121521
  • 08:36 gehel: restarting stuck tilerator on maps* - T204047
  • 08:35 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/wbgetclaims$ sudo -u _graphite find . -type f -name "*.wsp" -delete # T140280
  • 08:11 moritzm: installing perl security updates on jessie/trusty (stretch already updated)
  • 07:49 godog: bootstrap cassandra-c on restbase2014 - T209615
  • 07:26 marostegui: Deploy schema change on s4 codfw master (db2051) with replication T86338 T202167
  • 07:22 marostegui: Deploy schema change on wikitech primary master (db1073) for labswiki and labtestwiki T86338 T202167
  • 06:39 marostegui: Deploy schema change on s5 primary master (db1070) T86338 T202167
  • 06:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 T86338 T202167 (duration: 00m 49s)
  • 06:12 marostegui: Deploy schema change on db1110 T86338 T202167
  • 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 T86338 T202167 (duration: 00m 53s)
  • 01:22 foks: Removing 2FA per request at https://phabricator.wikimedia.org/T210703
  • 01:22 foks: Reset password for user "Orangemike"

2018-12-03

  • 23:54 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/tests/: for completeness (duration: 00m 58s)
  • 23:53 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/resources/src/mediawiki.legacy/: Restore gray coloring for autocomments (T165189 part 2) (duration: 00m 47s)
  • 23:51 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Linker.php: Restore old HTML structure for history section links (T165189 part 1) (duration: 00m 47s)
  • 22:56 sbassett@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/AbuseFilter/includes/api/ApiQueryAbuseLog.php: Deploy security fix for T210329 (duration: 00m 47s)
  • 21:24 mutante: temp. disabling puppet on logstash1007 and logstash1008 to carefully deploy gerrit:476916
  • 21:17 XioNoX: push firewall change to pfw3-eqiad - T211028
  • 21:10 catrope@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: Fix ChangesListFilters validation errors (duration: 00m 49s)
  • 21:00 urandom: bootstrapping restbase2014-a -- T210843
  • 20:20 ppchelko@deploy1001: Finished deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema (duration: 01m 13s)
  • 20:19 ppchelko@deploy1001: Started deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema
  • 20:14 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build (duration: 09m 31s)
  • 20:05 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build
  • 20:05 XioNoX: push firewall change to pfw3-eqiad - T211028
  • 19:59 ppchelko@deploy1001: Finished deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change (duration: 01m 13s)
  • 19:58 ppchelko@deploy1001: Started deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change
  • 19:39 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend/resources/mobile.toc/TableOfContents.js: SWAT T210869 Fix Table of contents rendering (duration: 00m 47s)
  • 19:30 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT I3b906e8b1 CS part of setting enhanced RC (duration: 00m 46s)
  • 19:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Ic2787309e59e IS part of setting enhanced RC (duration: 00m 47s)
  • 19:25 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Echo/modules/styles/mw.echo.ui.PaginationWidget.less: SWAT T210487 I914b94515 (duration: 00m 47s)
  • 19:15 jforrester@deploy1001: Synchronized wmf-config/ProductionServices.php: SWAT T210381 I73c7596818b Actual config (duration: 00m 46s)
  • 19:09 jforrester@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: SWAT T210381 I2ae162f5 Part II (duration: 00m 47s)
  • 19:08 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT T210381 I2ae162f5 Part I (duration: 00m 46s)
  • 18:42 anomie@deploy1001: Synchronized wmf-config/CommonSettings.php: Updating SkinBuildSidebar hook function for T210528 (duration: 00m 47s)
  • 18:08 gehel: add elastic2037 to cirrus eqiad (new server) - T210265
  • 18:06 godog: bootstrap cassandra-c on restbase2013 - T209615
  • 17:09 bstorm_: T207377 reboot labstore1006 for upgrades
  • 16:28 godog: poweroff ms-be2021 for battery replacement - T208269
  • 15:23 gehel: start configuration of elastic2037-2044 (new servers) - T210265
  • 15:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 T86338 T202167 (duration: 00m 46s)
  • 14:54 marostegui: Deploy schema change on db1100 T86338 T202167
  • 14:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 T86338 T202167 (duration: 00m 48s)
  • 14:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 T86338 T202167 (duration: 00m 47s)
  • 14:17 marostegui: Deploy schema change on db1113:3315 T86338 T202167
  • 14:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 T86338 T202167 (duration: 00m 46s)
  • 14:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 T86338 T202167 (duration: 00m 46s)
  • 13:47 marostegui: Deploy schema change on db1082 (sanitarium master) with replication, lag will be generated on labs (s5) T86338 T202167
  • 13:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 T86338 T202167 (duration: 00m 47s)
  • 13:01 Lucas_WMDE: EU SWAT done
  • 12:44 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleaning of wgLogoHD (T150618) (duration: 00m 46s)
  • 12:39 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for multiple projects (T150618) (duration: 00m 47s)
  • 12:37 moritzm: installing nodejs security updates on scb1001
  • 12:34 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Close internalwiki (T205584) (duration: 00m 46s)
  • 12:28 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change sitename of shnwiki (T206777) (duration: 00m 47s)
  • 12:23 godog: bootstrap cassandra-b on restbase2013 - T209615
  • 12:19 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Perform more PHP constraint checks before falling back (T209504) (duration: 00m 48s)
  • 12:14 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Don’t send SPARQL prefixes in WikibaseQualityConstraints (T204317) (duration: 00m 49s)
  • 11:39 godog: more weight to new ms-be codfw hosts - T209395
  • 11:02 moritzm: installing nodejs security updates on stat/notebook hosts
  • 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 T86338 T202167 (duration: 00m 47s)
  • 10:52 moritzm: rolling upgrade of scb in codfw to nodejs security update
  • 10:39 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 10:39 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 47s)
  • 10:34 moritzm: installing nodejs security updates on scb2001
  • 10:31 marostegui: Deploy schema change on db1097:3315 T86338 T202167
  • 10:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 T86338 T202167 (duration: 00m 46s)
  • 10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 T86338 T202167 (duration: 00m 45s)
  • 09:48 marostegui: Deploy schema change on db1096:3315 T86338 T202167
  • 09:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 T86338 T202167 (duration: 00m 47s)
  • 09:27 banyek: executing schema change on db1066 (s2 master) - T85757
  • 09:22 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1074 (duration: 00m 47s)
  • 09:16 banyek: repooling db1074 - T85757
  • 09:16 banyek: repooling db1074
  • 08:50 banyek: stopping replication on db1074 - T85757
  • 08:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1074 (duration: 00m 48s)
  • 08:44 godog: bootstrap cassandra-a on restbase2013 - T209615
  • 08:43 banyek: depooling db1074 - T85757
  • 08:32 moritzm: restarted keyholder agents/proxies on netmon1002/netmon2001 to pick up removal of netbox key
  • 08:30 marostegui: Deploy schema change on s5 codfw master (db2052) with replication, lag will be generated on codfw T86338 T202167
  • 08:13 moritzm: rearmed keyholders on netmon1002/netmon2001
  • 08:07 marostegui: Deploy schema change on s2 master (db1066) T86338 T202167
  • 07:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 T86338 T202167 (duration: 00m 47s)
  • 07:32 marostegui: Deploy schema change db1074 with replication (lag will appear on labs) T86338 T202167
  • 07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 T86338 T202167 (duration: 00m 46s)
  • 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 T86338 T202167 (duration: 00m 47s)
  • 07:09 marostegui: Stop MySQL on pc1004, pc1005 and pc1006 as they will be decommissioned - T210969
  • 06:52 marostegui: Remove pc1004, pc1005 and pc1006 from tendril and zarcillo - T210969
  • 06:38 marostegui: Deploy schema change db1122 T86338 T202167
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 T86338 T202167 (duration: 00m 48s)
  • 06:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 T86338 T202167 (duration: 00m 47s)
  • 06:16 marostegui: Deploy schema change db1076 T86338 T202167
  • 06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 T86338 T202167 (duration: 00m 50s)
  • 00:34 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Title.php: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/477182 (duration: 00m 52s)

2018-12-02

  • 22:39 addshore: addshore@mwmaint1002:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=testwikidatawiki addless # This is my account, and apparently I no longer have the 2fa for it

2018-12-01

  • 16:48 andrewbogott: rebuilding labvirt1014 as cloudvirt1014, T210904

2018-11-30

  • 20:23 ottomata: temporarily disabled puppet on stat1005 to test rsyncd changes
  • 20:02 ssastry@deploy1001: Finished deploy [parsoid/deploy@9981ddf]: Update Parsoid to 310edecd (deploy-20181130 branch) (duration: 11m 38s)
  • 19:50 ssastry@deploy1001: Started deploy [parsoid/deploy@9981ddf]: Update Parsoid to 310edecd (deploy-20181130 branch)
  • 15:29 urandom: bootstrapping restbase2013-a -- T210843
  • 15:22 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf4+jessie to apt.wikimedia.org/jessie-wikimedia (fixes a dependency compared to the initial jessie update)
  • 12:55 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf3+jessie to apt.wikimedia.org/jessie-wikimedia (backporting the current security fixes)
  • 11:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 T86338 T202167 (duration: 00m 47s)
  • 11:07 banyek: deploy schema change on dbstore1002 - T85757
  • 10:59 marostegui: Deploy schema change on db1105:3312 T86338 T202167
  • 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 T86338 T202167 (duration: 00m 45s)
  • 09:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 T86338 T202167 (duration: 00m 46s)
  • 09:00 marostegui: Deploy schema change on db1090:3312 T86338 T202167
  • 09:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 T86338 T202167 (duration: 00m 47s)
  • 08:55 moritzm: removed rutherfordium from debmonitor DB (T210036)
  • 08:21 marostegui: Deploy schema change on dbstore1002 T86338 T202167
  • 08:12 moritzm: installing perl security updates
  • 07:28 marostegui: Deploy schema change on db1095:3312 T86338 T202167
  • 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 T86338 T202167 (duration: 00m 47s)
  • 06:55 marostegui: Purge binary logs on pc1005
  • 06:52 marostegui: Deploy schema change on db1103:3312 T86338 T202167
  • 06:51 marostegui: Deploy schema change on db1103:3312 T86338 T20216
  • 06:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 T86338 T202167 (duration: 00m 48s)
  • 06:12 marostegui: Deploy schema change on s2 codfw master with replication (db2035), this will generate lag on codfw - T86338 T202167
  • 02:17 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: update wbsearchentities ab test configuration T209402 (duration: 00m 47s)
  • 01:06 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents: SWAT: wbsearchentities ab test improvements T209402 (duration: 00m 46s)
  • 01:04 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend: SWAT: Change config flag for enabling Block Notice stats T201719 (duration: 00m 48s)
  • 00:57 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/VisualEditor: SWAT: Rename configs for tracking block notices on visual editor ve.init.mw.ArticleTarget: Stop when we fail to load metadata T209542 (duration: 00m 48s)

2018-11-29

  • 23:26 mutante: puppetmaster: sudo puppet cert revoke rutherfordium.eqiad.wmnet; sudo puppet node clean rutherfordium.eqiad.wmnet ; sudo puppet node deactivate rutherfordium.eqiad.wmnet ; run puppet on icinga1001.. removed host from monitoring (decom for ganeti VM) (T210036)
  • 22:21 hashar: 1.33.0-wmf.6 is on all wikis and looks stable.
  • 22:04 hashar: hashar@deploy1001 rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.6
  • 21:54 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T210636 - I9ebbc6 (duration: 00m 55s)
  • 21:41 tzatziki: changing email for User:Mathounette
  • 21:34 XioNoX: removed unused vc-port on asw2-c-eqiad:fpc8 - T210788
  • 20:50 mutante: people - rsynced /home one last time, switched DNS people.eqiad CNAME over, varnish change merged (T210036)
  • 20:42 mutante: people.wikimedia.org is switching backends from rutherfordium to people1001, please stand by during a short maintenance period.. data has been copied | https://wikitech.wikimedia.org/wiki/People.wikimedia.org#Backend_upgrade_November_2018 | T210036
  • 20:01 XioNoX: Apply Icinga:check_vcp to all VC switches - T201097
  • 19:42 XioNoX: remove neodymium/sarin from mgmt routers - T210612
  • 19:16 hashar@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend/: RecordRevision::getUser() returns UserIdentity not int - T210737 (duration: 00m 55s)
  • 19:09 ejegg: updated payments-wiki from 34250e80b5 to 7403a196b4
  • 18:53 XioNoX: Netbox: remove Napalm integration
  • 18:28 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting comment migration to write-new/read-new on group 0 (T166733) (duration: 00m 52s)
  • 17:55 XioNoX: remove test netbox user from cr3-ulsfo - T205898
  • 17:49 mforns@deploy1001: Finished deploy [analytics/refinery@40b1972]: deploying refinery to refinery-source version v0.0.81 (duration: 06m 01s)
  • 17:43 mforns@deploy1001: Started deploy [analytics/refinery@40b1972]: deploying refinery to refinery-source version v0.0.81
  • 17:41 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/revisiondelete/RevisionDeleteUser.php: Fix RevisionDeleteUser rev_actor query for MySQL, for real this time (T210628) (duration: 00m 53s)
  • 17:34 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/revisiondelete/RevisionDeleteUser.php: Fix RevisionDeleteUser rev_actor query for MySQL (T210628) (duration: 00m 53s)
  • 17:02 robh: decom of labvirt101[01] continuing
  • 16:10 anomie@mwmaint1002: Running Wikibase/populateSitesTable.php and cleanupUsersWithNoId.php on more wiktionaries, incubatorwiki, and sourceswiki for T210732
  • 15:54 papaul: shutting down ms-be2047 for maintenance
  • 15:35 gtirloni: T196507 downtimed and powercycled cloudvirt1019
  • 15:32 anomie@mwmaint1002: Running Wikibase/populateSitesTable.php and cleanupUsersWithNoId.php on several other wiktionaries for T210732
  • 15:29 gehel: activating multiple elasticsearch instances on cirrus / eqiad - T207918
  • 15:18 $WHO: Running Wikibase populateSitesTable.php on eswiktionary for T210732
  • 14:31 hashar@deploy1001: rebuilt and synchronized wikiversions files: Revert all wikis to 1.33.0-wmf.6
  • 14:29 hashar@deploy1001: scap failed: average error rate on 11/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
  • 14:17 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.6
  • 14:10 hashar: test stashbot
  • 14:03 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf3 to apt.wikimedia.org/stretch-wikimedia (backporting the current security fixes)
  • 13:45 hashar@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Wikibase: feature flag for globe coordinator formatter using kartographer - T184933 T210617 (duration: 01m 18s)
  • 13:40 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Clarify parsercache keys section (duration: 00m 53s)
  • 13:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Clarify parsercache keys section (duration: 00m 52s)
  • 13:17 moritzm: rebooting certcentral1001 to pick up SSBD-enabled qemu/kernel update
  • 13:14 moritzm: rebooting certcentral2001 to pick up SSBD-enabled qemu/kernel update
  • 13:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool pc1009 in pc3 - T208383 (duration: 00m 53s)
  • 13:05 marostegui: Upgrade pc3 tendril topology - T208383
  • 13:00 mobrovac@deploy1001: Started restart [eventstreams/deploy@07033d4]: Restart ES on scb1004 due to possible memory leak (again)
  • 12:49 dcausse: EU swat done
  • 12:48 dcausse@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/TwoColConflict/: Fix unescaped HTML injected into conflict resolution interface (duration: 00m 53s)
  • 12:30 jynus: run puppet on notebook1004, people1001, rutherfordium to fix failures
  • 12:13 dcausse@deploy1001: Synchronized dblists/cirrussearch-big-indices.dblist: T210381: [cirrus] multi-instance: add cirrussearch-big-indices.dblist (duration: 00m 53s)
  • 12:09 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] Use normal config for labswiki (duration: 00m 55s)
  • 12:01 banyek: repooling labsdb1010 after upgrades - T209517
  • 10:17 elukey: remove zookeeper's crontabs from conf100[1-3] to fix cronspam
  • 10:16 arturo: T209626 icinga downtime labvirt1011 for 1 month to avoid bogus pages
  • 10:01 banyek: depooling labsdb1010 due of maintenance - T209517
  • 09:55 gehel: restarting prometheus-elasticsearch-exporter-9200 on all elastic cirrus nodes
  • 09:11 akosiaris: increase nofile of process to 20k and maxclients to 15k to account for the backlog of ores scorings
  • 08:48 _joe_: restarting uwsgi-ores on ores1003
  • 08:04 vgutierrez: replacing TLS certificates in gerrit - T207050
  • 06:59 marostegui: Stop MySQL on pc1006 to clone pc1009 - T208383
  • 06:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1006 - T208383 (duration: 00m 53s)
  • 06:36 marostegui: Deploy schema change on db1061 (s6 master) - T202167
  • 06:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 T202167 (duration: 00m 53s)
  • 06:31 marostegui: Deploy schema change on db1088 - T202167
  • 06:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 T202167 (duration: 00m 56s)
  • 06:15 reedy@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/OATHAuth: revert logging (loldeployingfromaplane) (duration: 00m 59s)
  • 06:11 marostegui: Deploy schema change on s6 primary master - T86338
  • 04:08 ejegg: updated fundraising internal dashboard from 5e9fb9a3ef to 1e4dd2a9ec
  • 03:30 ejegg: updated payments-wiki from 42283c73d0 to 34250e80b5
  • 00:08 catrope@deploy1001: Synchronized wmf-config/throttle.php: T210681 (duration: 01m 04s)

2018-11-28

  • 23:23 tzatziki: changing a few passwords for compromised accounts
  • 21:35 mutante: gnt-instance reboot proton1001.eqiad (stopped working, no SSH)
  • 21:28 arlolra: Updated Parsoid to 18a98af (T209236, T210437, T184755, T187142, T208470, T207286, T206777, T205710, T205546, T204477)
  • 21:26 mutante: rebooting proton1002
  • 21:19 herron: restarted nagios-nrpe-server on proton1002
  • 21:16 arlolra@deploy1001: Finished deploy [parsoid/deploy@9ed8c47]: Updating Parsoid to 18a98af (duration: 09m 35s)
  • 21:06 arlolra@deploy1001: Started deploy [parsoid/deploy@9ed8c47]: Updating Parsoid to 18a98af
  • 20:41 herron: rebooting logstash1005 for security updates
  • 20:22 paravoid: neodymium: mv /srv/jnt{,.old} (use cumin1001 instead!)
  • 19:40 herron: rebooting logstash1004 to pick up security updates
  • 19:31 ebernhardson: start goreplay logging of port 9200 across eqiad elastic cluster to track down T208248
  • 19:05 ejegg: updated payments-wiki from 3e0c97a969 to 6552acdc0f
  • 18:58 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/ORES/includes/Hooks/ApiHooksHandler.php: Don't try to add scores in API where there is nothing to add (T210610) (duration: 00m 55s)
  • 18:33 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Wikibase/lib/includes/Formatters/CachingKartographerEmbeddingHandler.php: Never return null in CachingKartographerEmbeddingHandler::getParserOutput T210617 (duration: 00m 53s)
  • 18:26 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/EventBus/includes/EventBus.php: SWAT: Revert "Revert "Revert "Set event datetime with microsecond resolution.""" T210608 (duration: 00m 55s)
  • 17:16 bblack: rebooting lvs1006, console was unresponsive
  • 17:11 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable classic_entity wbsearchentities AB test T209402 T210618 (duration: 00m 55s)
  • 17:01 chasemp: stat1004:~# aptitude install exfat-fuse exfat-utils (elukey fyi)
  • 16:57 mutante: restarting gerrit
  • 16:54 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaIncubator/extension.json: Revert "Replace wiki with wikipedia as wmf-config has been updated" T117023 (duration: 00m 54s)
  • 16:53 mutante: gerrit about to restart for logging config change
  • 16:50 gtirloni: T206916 created shnwiki views/index in labsdb replicas
  • 15:55 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1122 (duration: 00m 53s)
  • 15:52 banyek: repooling db1122 after schema change (T85757)
  • 15:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 T86338 (duration: 00m 52s)
  • 15:49 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: wikidatawiki back to 1.33.0-wmf.4
  • 15:24 vgutierrez: use a certcentral managed TLS certificate in dumps.wm.o - T207050
  • 15:19 banyek: Deploy schema change on db1122 - T85757
  • 15:17 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1122 (duration: 00m 53s)
  • 15:10 banyek: depooling db1122 due schema change (T85757)
  • 15:09 marostegui: Deploy schema change on db1085 with replication - T86338
  • 15:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 T86338 (duration: 00m 56s)
  • 14:05 hashar@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.6 (duration: 00m 52s)
  • 14:04 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.6
  • 13:59 zeljkof: EU SWAT (finally) done
  • 13:57 zfilipin@deploy1001: Finished scap: SWAT: Add user preference to disable the advanced interface (T210479) (duration: 38m 12s)
  • 13:19 zfilipin@deploy1001: Started scap: SWAT: Add user preference to disable the advanced interface (T210479)
  • 12:54 moritzm: installing serf update from stretch point release
  • 12:43 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Cite: SWAT: Make backlink highlighting robust for community customized HTML (T205270 T210520) (duration: 00m 55s)
  • 12:34 moritzm: installing ghostscript security updates
  • 12:24 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make AdvancedSearch default on all wikis (T207639) (duration: 00m 54s)
  • 11:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 T202167 (duration: 00m 53s)
  • 11:17 marostegui: Deploy schema change on db1085 (s6) (sanitarium master) with replication - T202167
  • 11:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 T202167 (duration: 00m 53s)
  • 11:15 banyek: repooling labsdb1009 after maintenance
  • 10:46 moritzm: rolling reboot of logstash1007-1009 to pick up new SSBD instructions and OpenJDK security updates
  • 10:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool pc1008 - T208383 (duration: 00m 50s)
  • 10:26 ladsgroup@deploy1001: Finished deploy [ores/deploy@9b9ba06]: T206333 (duration: 14m 48s)
  • 10:11 ladsgroup@deploy1001: Started deploy [ores/deploy@9b9ba06]: T206333
  • 10:08 jynus: disable mailing list mediation-en-l
  • 10:00 banyek: depooling labsdb1009 (T209517)
  • 09:55 marostegui: Update tendril topology for pc1 - T208383
  • 09:51 marostegui: Change pc2008 to replicate from pc1008 instead of pc1005 - T208383
  • 09:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 T86338 T202167 (duration: 00m 53s)
  • 09:14 vgutierrez: Use a TLS certificate managed by certcentral in icinga.wm.o - T207050
  • 09:00 marostegui: Deploy schema change db1113:3316 T86338 T202167
  • 09:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3316 T86338 T202167 (duration: 00m 53s)
  • 08:41 moritzm: installing git security updates on trusty (Debian already fixed)
  • 08:28 moritzm: installing samba security updates (client libs)
  • 08:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 T86338 (duration: 00m 53s)
  • 08:12 elukey: apply -R 200 to memcached on mc1022 (cache wipe) - T208844
  • 07:45 marostegui: Deploy schema change db1096:3316 T86338
  • 07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 T86338 (duration: 00m 53s)
  • 07:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316 T86338 T202167 (duration: 00m 54s)
  • 07:17 marostegui: Deploy schema change db1098:3316 T86338 T202167
  • 07:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 T86338 T202167 (duration: 00m 53s)
  • 06:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 T86338 T202167 (duration: 00m 52s)
  • 06:40 marostegui: Deploy schema change db1093 T86338 T202167
  • 06:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 T86338 T202167 (duration: 00m 53s)
  • 06:14 marostegui: Stop MySQL on pc1005 to clone pc1008 - T208383
  • 06:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1005 - T208383 (duration: 01m 04s)
  • 06:03 marostegui: Deploy schema change on s6 codfw - T86338 T202167
  • 02:17 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: Fix wbsearchentities collection of non-test bucketed data (duration: 00m 56s)
  • 02:16 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: Fix wbsearchentities collection of non-test bucketed data (duration: 00m 54s)
  • 01:45 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/: wbsearchentities needed extension.json to be deployed as well. Sync the whole directory (duration: 00m 53s)
  • 01:44 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/: wbsearchentities needed extension.json to be deployed as well. Sync the whole directory (duration: 00m 53s)
  • 01:34 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209402: Start wbsearchentities ab test at 10% (duration: 00m 54s)
  • 01:29 ebernhardson: ebernhardson@deploy1001 Synchronized php-1.33.0-wmf.6/includes/parser/Parser.php: SWAT: T209236 Protect legacy URL parameter syntax in link and alt options (duration: 00m 51s)
  • 01:22 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/ParsoidBatchAPI/includes/ApiParsoidBatch.php: SWAT: Revert ApiParsoidBatch update (duration: 00m 54s)
  • 01:12 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209402: Configuration for wbsearchentities AB test (duration: 00m 53s)
  • 01:09 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: T209402: AB testing support for wbsearchentities (duration: 00m 52s)
  • 01:08 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/modules/wikibase/ext.wikimediaEvents.completionClicks.js: T209402: AB testing support for wbsearchentities (duration: 00m 53s)
  • 01:06 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/Wikibase/view/resources/jquery/wikibase/jquery.wikibase.entityselector.js: Allow AB test to modify entityselector api request (duration: 00m 56s)
  • 01:02 ejegg: updated payments-wiki config to 51abff4b66
  • 00:19 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/ParsoidBatchAPI/includes/ApiParsoidBatch.php: SWAT I4e4373a7 revert Modernize ApiParsoidBatch using ApiResult to generate prettier output (duration: 00m 54s)
  • 00:07 ebernhardson@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: SWAT: no-op labs sync for 476027 (duration: 00m 53s)
  • 00:06 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: no-op labs sync for 476027 (duration: 00m 55s)

2018-11-27

  • 23:49 ejegg: updated payments-wiki config to 04f718e8f1
  • 23:29 ejegg: updated payments-wiki config to 082eab3566
  • 22:35 ejegg: rolled back payments-wiki to 86742dd4fd
  • 22:33 ejegg: updated payments-wiki from 86742dd4fd to 3ba10e49b0
  • 21:26 ejegg: updated payments-wiki config to d497b7e880
  • 19:40 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: no op sync wikiversions to test scap 3.8.10-1
  • 18:03 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: no op sync wikiversions to test scap 3.8.9-1
  • 17:51 godog: upload scap 3.8.9-1 - T210469
  • 17:40 bstorm_: T209517 rebooted labsdb1005 after upgrades
  • 17:30 bstorm_: T209517 icinga downtime labsdb1004
  • 17:30 XioNoX: repool uslfo
  • 17:25 arturo: T209517 icinga downtime labsdb1005
  • 17:23 arturo: T207377 icinga downtime labnet1001
  • 16:17 mutante: einsteinium - apt-get remove --purge icinga nsca; apt-get autoremove ; apt-get remove --purge icinga-doc icinga-common icinga-cgi-bin icinga-cgi; apt-get remove --purge monitoring-plugin* ; rm /etc/rsync.d/frag-icinga* T209738
  • 16:14 hashar@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.6
  • 16:09 moritzm: rebooting planet1001 for kernel security update
  • 16:04 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105 (duration: 00m 53s)
  • 15:53 moritzm: rebooting planet2001 for kernel security update
  • 15:44 moritzm: rebooting kafkamon1001 for kernel security update
  • 15:41 mutante: einsteinium - removed icinga package
  • 15:39 moritzm: rebooting kafkamon2001 for kernel security update
  • 15:35 mutante: einsteinium - stopped icinga, stopped nsca, stopped rsyncd, killall -u icinga, killall -u nagios ... T209738
  • 15:35 hashar@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.6 and rebuild l10n cache | T206660 (duration: 32m 42s)
  • 15:21 akosiaris: create graphoid namespace on kubernetes eqiad, codfw, staging clusters T203091
  • 15:02 hashar@deploy1001: Started scap: testwiki to php-1.33.0-wmf.6 and rebuild l10n cache | T206660
  • 14:52 hashar@deploy1001: Pruned MediaWiki: 1.32.0-wmf.26 (duration: 09m 48s)
  • 14:40 hashar: Applied security patches for 1.33.0-wmf.6 | T206660
  • 14:33 hashar: scap prep 1.33.0-wmf.6 | T206660
  • 14:04 hashar: Cutting 1.33.0-wmf.6 branches | T206660
  • 14:00 banyek: repooling db1105 due a schema change (T85757)
  • 13:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool pc1010 in pc1 - T208383 (duration: 00m 46s)
  • 13:34 marostegui: Change pc2007 and pc2010 to replicate from pc1010 instead of from pc1004 - T208383
  • 13:18 banyek: executing schema change on db1105 (T85757)
  • 13:18 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1105 (duration: 00m 46s)
  • 13:17 pmiazga@deploy1001: Finished deploy [proton/deploy@9efc072]: Proton: Rewrite Queue to promise-way flow (T204055) (duration: 02m 43s)
  • 13:15 pmiazga@deploy1001: Started deploy [proton/deploy@9efc072]: Proton: Rewrite Queue to promise-way flow (T204055)
  • 13:14 banyek: depooling db1105 due a schema change (T85757)
  • 13:09 raynor: proton deploying 9efc072
  • 12:39 mobrovac@deploy1001: Synchronized php-1.33.0-wmf.4/includes/page/WikiPage.php: Convert $archivedRevisionCount to integer - T210013 T210451 (duration: 00m 47s)
  • 12:32 gilles@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/http2priorities/upload.wikimedia.org.html: T210141 Add variant of HTTP/2 priorities test pointing to upload (duration: 00m 46s)
  • 12:25 mobrovac@deploy1001: Synchronized static/images/project-logos: Add localised logos for the Minangkabau Wikipedia - T210387 (duration: 00m 47s)
  • 12:19 mobrovac@deploy1001: Synchronized rpc/RunSingleJob.php: RunSingleJob: Check that JobExecutor has been loaded - T208922 (duration: 00m 47s)
  • 11:56 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1103 (duration: 00m 46s)
  • 11:50 banyek: repooling db1103 after schema change (T85757)
  • 11:32 banyek: executing schema change on db1103:3312 (T85757)
  • 11:32 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1103 (duration: 00m 46s)
  • 11:28 banyek: depooling db1103 due a schema change (T85757)
  • 10:59 banyek: executing schema change on db1095 (T85757)
  • 10:55 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1090 (duration: 00m 46s)
  • 10:51 banyek: repooling db1090 after schema change (T85757)
  • 10:51 banyek: repooling db1090 due a schema change (T85757)
  • 10:42 gilles@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/http2priorities: T210141 HTTP/2 prioritie speed test (duration: 00m 47s)
  • 10:36 arturo: T209948 disable puppet in all WMCS hw servers
  • 10:30 arturo: T209948 schedule 2h icinga downtime in all WMCS hw servers
  • 10:26 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1090 (duration: 00m 46s)
  • 10:19 banyek: depooling db1090 due a schema change (T85757)
  • 10:18 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1076 (duration: 00m 45s)
  • 10:13 banyek: repooling db1076 after schema change (T85757)
  • 10:06 banyek: executing schema change on db1076 (T85757)
  • 10:02 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1076 (duration: 00m 46s)
  • 09:57 banyek: depooling db1076 due a schema change (T85757)
  • 09:35 marostegui: Stop MySQL on pc1004 to clone pc1010 - T208383
  • 09:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool pc1004 - T208383 (duration: 00m 46s)
  • 09:00 banyek: executing schema change on db2035 for s2 (T85757)
  • 08:43 elukey: roll restart of all druid daemons on druid100[1-6] for openjdk-8 upgrades
  • 08:41 vgutierrez: Use a TLS certificate managed by certcentral in apt.wm.o - T207050
  • 08:15 godog: more weight to new ms-be hosts in codfw - T209395
  • 07:35 _joe_: depooling mw1261 for benchmarking, T206341
  • 07:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 after cloning db1095:3312 (duration: 00m 46s)
  • 07:16 marostegui: Start MySQL on db1090:3312 after recloning db1095:3312
  • 07:16 marostegui: Start MySQL on db1095:3312 after recloning it
  • 06:16 marostegui: Stop mysql on db1090:3312 to clone db1095:3312
  • 06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 49s)
  • 06:15 marostegui: Stop mysql on db1095:3312 to get it recloned
  • 04:33 ejegg: updated payments-wiki from d2b66c5bab to 86742dd4fd
  • 04:31 ejegg: updated fundraising CiviCRM from 013807a7b9 to a411d6bd64
  • 04:27 ejegg: updated standalone SmashPig deployment from f65daa8550 to fb3268897b
  • 00:38 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix weird caching issue maybe for Ian's patch (duration: 00m 46s)
  • 00:29 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Enable SVGs in page language everywhere T208899 (duration: 00m 45s)
  • 00:27 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable SVGs in page language everywhere T208899 (duration: 00m 46s)
  • 00:17 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgMinervaSchemaMainMenuClickTrackingSampleRate T205008 (duration: 00m 46s)
  • 00:12 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Delete a namespace from ruwikisource T210171 (duration: 00m 46s)
  • 00:05 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Enable trust survey on labs T209882 (duration: 00m 46s)
  • 00:03 niharika29@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 03s)

2018-11-26

  • 23:50 ottomata: temporarily disabling puppet on stat1007 to copy over eventbus validation logs (not using stat1007 after all)
  • 23:48 ottomata: temporarily disabling puppet on stat1007 to copy over eventbus validation logs
  • 23:24 sbassett@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove unblockself right everywhere 2046d8df3 T150826 (duration: 00m 47s)
  • 23:15 XioNoX: depool ulsfo for transport providers maintenance
  • 20:14 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/Wikibase/repo/includes/Search/Elastic/EntitySearchElastic.php: SWAT T209402 Make wbsearchentities tie-breaker configurable (duration: 00m 47s)
  • 19:33 ebernhardson@deploy1001: Synchronized wmf-config/WikibaseSearchSettings.php: SWAT T209402 Search profiles for wbsearchentities AB test (duration: 00m 46s)
  • 19:25 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/includes/: SWAT T210003 T210004 EditorJourney: Adjust DeferredUpdates usage (duration: 00m 46s)
  • 19:24 XioNoX: remove IP from blacklist on Amsterdam routers - T201411
  • 19:24 cmjohnson1: cloudvirt1019 is going down to check something for HPE
  • 19:16 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T209432 Enable NewUserMessage extension on tcy.wikipedia (duration: 00m 48s)
  • 18:50 mutante: stopping icinga service on einsteinium, is a role(spare) now T209738
  • 18:46 mutante: removed allowed sender addresses from AQL (mail2SMS gateway) portal: @einsteinium @tegmen addresses T208824 T209738
  • 18:45 ppchelko@deploy1001: Finished deploy [changeprop/deploy@c89bff5]: Prepared for ORES error response T197000 (duration: 01m 13s)
  • 18:44 ppchelko@deploy1001: Started deploy [changeprop/deploy@c89bff5]: Prepared for ORES error response T197000
  • 18:41 XioNoX: re-activate BGP to AS41692 on cr2-esams
  • 18:15 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@8bfea2b]: GUI updates and updater with dump and revision logging. (duration: 11m 13s)
  • 18:04 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@8bfea2b]: GUI updates and updater with dump and revision logging.
  • 17:54 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp5001.eqsin.wmnet
  • 17:53 bblack: re-pooling cp5001 - T199675
  • 17:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1122 (duration: 00m 46s)
  • 17:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1122 (duration: 00m 46s)
  • 17:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1122 (duration: 00m 46s)
  • 17:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1122 (duration: 00m 45s)
  • 17:23 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Reverting: Bumping portals to master (T128546) (duration: 00m 46s)
  • 17:22 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Reverting: Bumping portals to master (T128546) (duration: 00m 46s)
  • 17:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 and db1122 (duration: 00m 46s)
  • 16:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 to clone db1122 (duration: 00m 46s)
  • 16:00 marostegui: Stop MySQL on db1090:3312 to clone db1122
  • 15:57 marostegui: Stop MySQL on db1122 - it will be recloned
  • 15:50 ppchelko@deploy1001: Finished deploy [changeprop/deploy@77be2c6]: Change schema for revision-score events and start emitting again T197000 (duration: 01m 24s)
  • 15:48 ppchelko@deploy1001: Started deploy [changeprop/deploy@77be2c6]: Change schema for revision-score events and start emitting again T197000
  • 15:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 - actor table missing (duration: 00m 46s)
  • 15:27 ppchelko@deploy1001: Finished deploy [changeprop/deploy@b97e8eb]: Temporary stop emitting revision-score events for schema change T197000 (duration: 01m 21s)
  • 15:26 ppchelko@deploy1001: Started deploy [changeprop/deploy@b97e8eb]: Temporary stop emitting revision-score events for schema change T197000
  • 15:24 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on group 1 (T188327) (duration: 00m 46s)
  • 13:47 moritzm: rebooting releases1001
  • 13:36 moritzm: rebooting releases2001
  • 13:33 moritzm: rebooting netmon1003
  • 12:04 _joe_: uploaded php-excimer for component thirdparty/php72 to stretch-wikimedia T205059
  • 12:00 kartik@deploy1001: Finished deploy [cxserver/deploy@da41227]: Disable Youdao MT until service is back (duration: 04m 01s)
  • 11:56 kartik@deploy1001: Started deploy [cxserver/deploy@da41227]: Disable Youdao MT until service is back
  • 11:41 arturo: icinga downtime cloudcontrol1004 for some systemd slice tests
  • 11:01 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 11:00 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 47s)
  • 10:32 moritzm: rebooting boron
  • 09:58 banyek: performing schema change on s6 master - db1061 for frwiki (T85757)
  • 09:51 banyek: performing schema change on s6 master - db1061 for ruwiki (T85757)
  • 09:44 banyek: performing schema change on s6 master - db1061 for jawiki (T85757)
  • 08:35 moritzm: removed labvirt1015 from debmonitor DB (got renamed to cloudvirt1015)
  • 08:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 - T202167 (duration: 00m 46s)
  • 08:16 marostegui: Deploy schema change on db1096:3316 - T202167
  • 08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 - T202167 (duration: 00m 47s)
  • 08:10 moritzm: installing pixman security updates
  • 07:37 elukey: restart memcached on mc1021 (cache wipe) to add -R 200 - T208844
  • 07:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 - T86338 (duration: 00m 46s)
  • 07:00 marostegui: Deploy schema change on db1088 - T86338
  • 07:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 - T86338 (duration: 00m 46s)
  • 06:46 marostegui: Deploy schema change on db2067 - T86338
  • 06:36 marostegui: Deploy schema change on s3 master (db1075) - T86339
  • 06:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 - T86339 (duration: 00m 46s)
  • 06:22 marostegui: Reload haproxy on dbproxy1005
  • 06:22 marostegui: Deploy schema change on db1077 (s3 sanitarium master) with replication - T86339
  • 06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 - T86339 (duration: 00m 50s)
  • 06:02 kartik@deploy1001: Finished deploy [cxserver/deploy@d2173ca]: Update cxserver to 218a543 (duration: 04m 21s)
  • 05:58 kartik@deploy1001: Started deploy [cxserver/deploy@d2173ca]: Update cxserver to 218a543

2018-11-25

  • 10:40 _joe_: restarting pdfrender on scb1002, flapping since 3:00 UTC

2018-11-23

  • 21:15 bawolff: deploy patch for T210192
  • 16:53 bblack: cleaned up remnants of globalsign-2017 unified cert (OCSP cache/config, unmanaged cert files, etc) on all cpNNNN - T206804
  • 14:02 gehel: restor wdqs-updater heap to 2G - T210235
  • 13:32 moritzm: installing confuse security updates
  • 12:04 gehel: manually increasing wdqs-updater heap to 4G - T210235
  • 11:37 gehel: restarting updater on all wdqs ndoes
  • 08:41 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=.*,service=zotero,cluster=kubernetes,name=.*
  • 08:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T86339 (duration: 00m 46s)
  • 07:48 moritzm: installing libtirpc security updates
  • 07:14 marostegui: Deploy schema change db1078 - T86339
  • 07:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T86339 (duration: 00m 45s)
  • 07:00 marostegui: Deploy schema change db1095 - T86339
  • 06:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 - T86339 (duration: 00m 49s)
  • 06:32 marostegui: Deploy schema change db1123 - T86339
  • 06:31 marostegui: Deploy schema change dbstore1002:s3 - T86339
  • 06:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 - T86339 (duration: 00m 48s)
  • 06:15 marostegui: Deploy schema change on s3 codfw master (db2043) with replication - T86339
  • 06:13 marostegui: Deploy schema change on db1067 (s1) master - T86339

2018-11-22

  • 21:08 godog: disable raid handler for ms-be2021 - T208096
  • 19:01 moritzm: installing uriparser security updates
  • 18:46 moritzm: installing openjpeg2 security updates
  • 18:45 arturo: enable puppet in all CloudVPS HW servers
  • 18:38 arturo: disable puppet in all CloudVPS HW servers to test a patch (T209948)
  • 17:46 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1016 (duration: 00m 46s)
  • 16:49 jynus: upgrading, and restarting es1016 (but not deleting, that was a mistake)
  • 16:49 jynus: upgrading, deleting at and restarting es1016
  • 16:35 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1016 (duration: 00m 47s)
  • 16:06 ema: trafficserver_8.0.0-1wm3 uploaded to stretch-wikimedia
  • 15:23 akosiaris@deploy1001: scap-helm zotero finished
  • 15:23 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 15:23 akosiaris@deploy1001: scap-helm zotero install --name production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 15:22 akosiaris@deploy1001: scap-helm zotero finished
  • 15:22 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 15:22 akosiaris@deploy1001: scap-helm zotero install --name production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 14:50 akosiaris@deploy1001: scap-helm zotero install --name production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 13:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 - T86339 (duration: 00m 45s)
  • 13:52 marostegui@deploy1001: sync-file aborted: Depool db1080 - T86339 (duration: 00m 00s)
  • 13:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 - T86339 (duration: 00m 46s)
  • 13:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 - T86339 (duration: 00m 46s)
  • 13:45 marostegui: Deploy schema change on s1 eqiad hosts T86339
  • 13:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 - T86339 (duration: 00m 46s)
  • 13:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 - T86339 (duration: 00m 45s)
  • 13:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 - T86339 (duration: 00m 46s)
  • 13:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 - T86339 (duration: 00m 45s)
  • 13:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 - T86339 (duration: 00m 43s)
  • 13:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 - T86339 (duration: 00m 46s)
  • 13:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 - T86339 (duration: 00m 47s)
  • 12:19 jynus: upgrading, deleting at and restarting dbstore2002
  • 10:21 jynus: upgrading and restarting dbstore1001
  • 10:01 godog: bounce rsyslog on lithium, tls listener timeout on icinga
  • 09:54 godog: bounce rsyslog on wezen, tls listener timeout on icinga
  • 09:49 jynus: stop and upgrade dbstore2001
  • 09:24 moritzm: installing ruby-l18n security updates
  • 09:20 moritzm: installing ruby-rack security updates
  • 09:06 moritzm: installing jasper security updates
  • 08:21 marostegui: Deploy schema change on s1 codfw master (db2048) with replication - T86339
  • 08:19 marostegui: Deploy schema change on db1062 (s7 master) - T86339
  • 08:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 - T86339 (duration: 00m 46s)
  • 08:17 marostegui: Deploy schema change on db1094 - T86339
  • 08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 - T86339 (duration: 00m 49s)
  • 06:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 - T86339 (duration: 00m 46s)
  • 06:55 marostegui: Deploy schema change on db1086 - T86339
  • 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 - T86339 (duration: 00m 46s)
  • 06:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 - T86339 (duration: 00m 46s)
  • 06:48 marostegui: Deploy schema change on db1079 (sanitarium master) with replication - T86339
  • 06:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 - T86339 (duration: 00m 51s)
  • 02:25 mutante: stat1005 - started nagios-nrpe-server

2018-11-21

  • 23:34 mutante: rsyncing /home from rutherfordium.eqiad to people1001.eqiad (people.wikimedia.org) T210036
  • 21:16 robh: cp5001 is offline running hardware tests after firmware updates to see if memory error still exists. ref: T199675
  • 20:55 robh: cp5001 reboot for firmware update
  • 19:54 milimetric@deploy1001: Finished deploy [analytics/aqs/deploy@e114d99]: Fixing sorting bug on top endpoints (duration: 05m 34s)
  • 19:49 milimetric@deploy1001: Started deploy [analytics/aqs/deploy@e114d99]: Fixing sorting bug on top endpoints
  • 19:43 ejegg: updated fundraising internal dashboard from b01458b260 to 5e9fb9a3ef
  • 17:21 elukey: manually started systemd-journald.service on scb1001 after OOM
  • 17:20 jynus: stop and upgrade db2081
  • 17:04 jynus: stop and upgrade db2080
  • 16:40 jynus: stop and upgrade db2066
  • 16:37 bawolff: deploy patch T209794
  • 16:25 jynus: stop and upgrade db2063
  • 15:15 jynus: stop and upgrade db2073
  • 15:15 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1085 (duration: 00m 46s)
  • 15:03 banyek: repooling db1085 after schema change (T85757)
  • 15:00 banyek: restarting replication on db1085 (T85757)
  • 14:31 banyek: stopping replication on db1085 (T85757)
  • 14:27 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1085 (duration: 00m 46s)
  • 14:22 banyek: depooling db1085 due schema change (T85757)
  • 13:39 jynus: stop and upgrade db2077
  • 13:05 XioNoX: remove BGP session to 2603 on cr4-ulsfo
  • 12:13 jynus: stop and upgrade db2076
  • 12:08 banyek: running schema change on dbstore1001:3316 (T85757)
  • 12:08 banyek: running schema change on dbstore1001 (T85757)
  • 12:04 jynus: stop and upgrade db2075
  • 11:53 banyek: running schema change on dbstore1002 (T85757)
  • 11:00 akosiaris: disable puppet on ores2* ores1* for gradual rollout of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474694/1/modules/ores/manifests/web.pp
  • 10:55 jynus: stop and upgrade db2074
  • 10:51 _joe_: uploading prometheus-php-fpm-exporter to stretch-wikimedia main, T209573
  • 10:44 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1096:3316 (duration: 00m 46s)
  • 10:42 banyek: repooling db1096:3316 after schema change (T85757)
  • 10:30 jynus: stop and upgrade db2095
  • 10:26 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1096:3316 (duration: 00m 45s)
  • 10:26 godog: initial weight for new ms-be2* hosts (all but ms-be2047) - T209395
  • 10:23 banyek: depooling db1096:3316 due schema change (T85757)
  • 10:14 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1098:3316 (duration: 00m 46s)
  • 10:13 banyek@deploy1001: sync-file aborted: T85757: depool db1098:3316 (duration: 00m 03s)
  • 10:11 banyek: repooling db1098 after schema change (T85757)
  • 09:57 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1098:3316 (duration: 00m 46s)
  • 09:52 banyek: depooling db1098:3316 due schema change (T85757)
  • 09:49 volans: restarted pdfrender on scb1003
  • 09:29 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1113 (duration: 00m 46s)
  • 09:21 banyek: repooling db1113 after schema change (T85757)
  • 09:09 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1113 (duration: 00m 46s)
  • 09:01 banyek: depooling db1113 due schema change (T85757)
  • 08:48 marostegui: Deploy schema change on s7 codfw master - T86339
  • 08:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 - T86339 (duration: 00m 45s)
  • 08:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 - T86339 (duration: 00m 45s)
  • 08:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 - T86339 (duration: 00m 46s)
  • 08:30 marostegui: Deploy schema changes on s8 eqiad hosts - T86339
  • 08:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 - T86339 (duration: 00m 46s)
  • 07:50 marostegui: Deploy schema change on s8 codfw master (db2045) with replication - T86339
  • 07:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 - T86339 (duration: 00m 45s)
  • 07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 - T86339 (duration: 00m 46s)
  • 07:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 - T86339 (duration: 00m 46s)
  • 07:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 - T86339 (duration: 00m 46s)
  • 07:32 marostegui: Deploy schema change on s4 eqiad hosts - T86339
  • 07:19 marostegui: Deploy schema change on db2051 (s4 codfw master) with replication - T86339
  • 07:10 marostegui: Drop foundationwiki.petition_data from s3 master (db1075) with replication - T208979
  • 07:06 marostegui: Deploy schema change on db1066 (s2 master) - T86339
  • 07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 - T86339 (duration: 00m 46s)
  • 07:02 marostegui: Deploy schema change on db1076 - T86339
  • 07:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 - T86339 (duration: 00m 46s)
  • 06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T86339 (duration: 00m 46s)
  • 06:48 marostegui: Deploy schema change on db1074 (sanitarium master) - T86339
  • 06:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T86339 (duration: 00m 46s)
  • 06:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 - T86339 (duration: 00m 46s)
  • 06:39 marostegui: Deploy schema change on db1122 - T86339
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 - T86339 (duration: 00m 51s)
  • 06:30 marostegui: Drop schema change on db1103:3312 and db1105:3312 - T86339
  • 00:48 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:474967 Disable FlaggedRevs, enable RC patrol and add rights on srwikinews (duration: 00m 47s)
  • 00:39 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:475005 Enable SVGs in page in group1, rest of group0 (duration: 00m 46s)
  • 00:32 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:474976 Enable suppressredirect on srwiki (duration: 00m 47s)
  • 00:24 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:472744 Enable RCPatrol and add some rights on srwikibooks (duration: 00m 46s)
  • 00:07 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/GrowthExperiments/includes/Specials/SpecialWelcomeSurvey.php: gerrit:474946 WelcomeSurvey: indicate that the special page does write (duration: 00m 47s)

2018-11-20

  • 23:01 XioNoX: create vol.ans account on switches - T208726
  • 22:54 pmiazga@deploy1001: Synchronized wmf-config: SYNC: noop Doc: add repoConceptBaseUri comment (T209352)noop: Remove utf-8 characters from DOC comment for better readability (T209352)beta: Wikibase: override repoConceptBaseUri (T209352) (duration: 00m 49s)
  • 22:13 XioNoX: create volans account on routers - T208726
  • 21:12 eileen: civicrm revision changed from f4127d5316 to 013807a7b9, config revision is 684ec9b7c0
  • 21:08 jgleeson: civicrm changed from a31dbefc61 to f4127d5316
  • 18:56 ebernhardson: start loading dumps into elastic codfw omega and psi from mwmaint2001
  • 18:19 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: Create Federated Wikibase instance on Beta Commons, part II (T204748) (duration: 00m 47s)
  • 18:17 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Create Federated Wikibase instance on Beta Commons (T204748) (duration: 00m 48s)
  • 18:08 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@7553087]: Deploy 2018 app fundraising announcement config (T204821) (duration: 03m 37s)
  • 18:04 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@7553087]: Deploy 2018 app fundraising announcement config (T204821)
  • 18:03 bstorm_: rebooting labsdb1006 for upgrades T209517
  • 17:25 bstorm_: rebooting labsdb1004 for upgrades T209517
  • 17:19 gehel: reload nginx configuration on elasticsearch codfw
  • 16:59 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2072 (duration: 00m 46s)
  • 16:53 XioNoX: rollback all BFD tests on cr1-codfw
  • 16:10 jynus: stop and upgrade db2072
  • 15:59 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2071, depool db2072 (duration: 00m 47s)
  • 15:51 banyek: repooling labsdb1011 (T209517)
  • 15:41 banyek: uploaded wmf-pt-kill_2.2.20-1+wmf5 packages to stretch-wikimedia (T209517)
  • 15:38 vgutierrez: switching to certcentral managed TLS certificate for librenms.wikimedia.org - T209856
  • 15:36 XioNoX: add test term allow BFD multihop on cr1-codfw loopback4 filter
  • 15:36 ejegg: updated fundraising CiviCRM from e648be0d9e to a31dbefc61
  • 15:20 moritzm: installing libopenmpt security updates
  • 15:12 XioNoX: enable bfd traceoptions on cr1-codfw
  • 15:02 XioNoX: Add BFD multihop support to Bird anycast DNS
  • 14:55 jijiki: libthumbor_1.3.2-0+wmf1+stretch1 uploaded to stretch-wikimedia T209886
  • 14:43 chasemp: puppet temp disable on es2001 for data transfer work
  • 14:32 jynus: stop and upgrade db2033
  • 13:19 jynus: stop and upgrade db2082
  • 13:06 banyek: depooling labsdb1011 (T209517)
  • 13:03 zeljkof: EU SWAT finished
  • 13:03 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: deployment-prep: Update parsoid09 IP (T208101) (duration: 00m 47s)
  • 12:55 zfilipin@deploy1001: Synchronized wmf-config/db-labs.php: SWAT: deployment-prep: Update deployment-db* IPs (T208101) (duration: 00m 47s)
  • 12:55 banyek: setting innodb_flush_log_at_trx_commit to 2 on dbstore2002 (s3 instance only!) (T208320)
  • 12:53 banyek: setting innodb_flush_log_at_trx_commit to 2 on dbstore2002 (T208320)
  • 12:49 zfilipin@deploy1001: scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
  • 12:45 zfilipin@deploy1001: Synchronized wmf-config/reverse-proxy-staging.php: SWAT: deployment-prep: Update cache-upload private IP (T208101) (duration: 00m 45s)
  • 12:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos in InitialiseSettings.php for multiple projects (T150618) (duration: 00m 48s)
  • 12:25 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add tboverride permission to extendedmover group on enwiki (T209753) (duration: 00m 47s)
  • 12:19 jynus: powercycling db2087, stuck on reboot
  • 12:13 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for multiple projects (T150618) (duration: 00m 48s)
  • 11:55 moritzm: rolling reboot of proton hosts for kernel security update
  • 11:27 jynus: stop and upgrade db2087
  • 11:16 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: (now really) repool db1093 (duration: 00m 47s)
  • 11:11 banyek: repooling db1093 (T85757)
  • 11:05 banyek: executing schema change on db1093 (T85757)
  • 11:00 jynus: stop and upgrade db2086
  • 10:59 banyek: db1093 was depooled wrong message sent
  • 10:51 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1093 (duration: 00m 47s)
  • 10:48 banyek: depooling db1093 (T85757)
  • 10:48 banyek: depooling db1093
  • 10:47 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2018 (duration: 00m 46s)
  • 10:17 jynus: upgrade and reboot es2018
  • 10:13 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2014, depool es2018 (duration: 00m 46s)
  • 09:34 marostegui: Deploy schema change on s2 hosts: dbstore1002, db1090:3312 and db1095:3312 - T86339
  • 09:26 marostegui: Deploy schema change on s2 codfw master (db2035) with replication - T86339
  • 09:25 jynus: upgrade and reboot es2014
  • 09:23 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2011, depool es2014 (duration: 00m 46s)
  • 09:23 godog: stress-test new ms-be hardware - T209395
  • 09:13 marostegui: Stop MySQL on pc2004, pc2005 and pc2006 for decommission - T209858
  • 09:05 gehel: powercycle elastic2021
  • 09:04 marostegui: Remove pc2004, pc2005 and pc2006 from tendril and zarcillo - T209858
  • 08:53 jynus: upgrade and reboot es2011
  • 08:48 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2011 (duration: 00m 47s)
  • 06:28 marostegui: Deploy schema change on db1070 (s5 master) - T86339
  • 06:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 - T86339 (duration: 00m 47s)
  • 06:21 marostegui: Deploy schema change on db1082 - T86339
  • 06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 - T86339 (duration: 00m 52s)
  • 00:55 catrope@deploy1001: Synchronized static/images/project-logos/: Correct logos for Sindhi Wiktionary (duration: 00m 47s)
  • 00:51 mutante: Gerrit: added Jeena Huneidi to wmf-deployers (T209722)
  • 00:26 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/resources/src/mediawiki.rcfilters/ui/mw.rcfilters.ui.FilterTagMultiselectWidget.js: RCFilters bug fix (T209657) (duration: 00m 47s)
  • 00:23 XioNoX: registering librenms IRC bot
  • 00:15 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Increase Schema.org page split test to 100% sampling (T208755) (duration: 00m 48s)

2018-11-19

  • 23:18 ejegg: updated fundraising CiviCRM from 275716f000 to e648be0d9e
  • 22:41 ejegg: updated fundraising CiviCRM from bbc0dddd1e to 275716f000
  • 22:21 ejegg: updated fundraising CiviCRM from 6b279509f8 to bbc0dddd1e
  • 21:58 XioNoX: restart bird on dns2001 to try to establish the BFD sessions
  • 20:27 catrope@deploy1001: Finished scap: Full scap for special alias changes for GrowthExperiments (duration: 21m 03s)
  • 20:06 catrope@deploy1001: Started scap: Full scap for special alias changes for GrowthExperiments
  • 19:50 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable WelcomeSurvey on cswiki and kowiki (T209725) (duration: 00m 46s)
  • 19:44 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/: EditorJourney fixes (T207307) (duration: 00m 46s)
  • 19:36 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/GrowthExperiments/: WelcomeSurvey fixes (T206371) (duration: 00m 46s)
  • 19:25 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable WelcomeSurvey on testwiki (T209725) (duration: 00m 49s)
  • 18:29 cmjohnson1: connecting eqiad asw2-b fpc2 and fpc8
  • 18:17 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@a25eb30]: GUI Update, new executor limits and new blazegraph build (duration: 08m 53s)
  • 18:08 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@a25eb30]: GUI Update, new executor limits and new blazegraph build
  • 15:45 ladsgroup@deploy1001: Finished deploy [ores/deploy@e957b24]: T209587 T170950 (duration: 17m 09s)
  • 15:28 ladsgroup@deploy1001: Started deploy [ores/deploy@e957b24]: T209587 T170950
  • 15:23 milimetric@deploy1001: Finished deploy [analytics/aqs/deploy@b399c34]: Removing empty fields from unique result (duration: 05m 17s)
  • 15:18 milimetric@deploy1001: Started deploy [analytics/aqs/deploy@b399c34]: Removing empty fields from unique result
  • 15:08 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on group 0 (T188327) (duration: 00m 47s)
  • 14:02 joal@deploy1001: Finished deploy [analytics/aqs/deploy@7cde8c8]: Update unique-devices schema adding 2 fields (duration: 20m 57s)
  • 13:55 gtirloni: T207377 reboot cloudcontrol1004
  • 13:43 moritzm: installing chromium security update on proton* (tested new upstream release in deployment-prep)
  • 13:41 joal@deploy1001: Started deploy [analytics/aqs/deploy@7cde8c8]: Update unique-devices schema adding 2 fields
  • 13:39 fdans@deploy1001: Finished deploy [analytics/aqs/deploy@7cde8c8]: Deploying AQS to add two new fields to uniques (duration: 06m 18s)
  • 13:39 akosiaris: cumin -b1 -s 300 'ores2*' 'enable-puppet "merge of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474158/" ; puppet agent -t ; service uwsgi-ores restart ; service celery-ores-worker restart'
  • 13:36 akosiaris: disable puppet on ores1* and ores2* for slow deployment of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/474158/
  • 13:33 fdans@deploy1001: Started deploy [analytics/aqs/deploy@7cde8c8]: Deploying AQS to add two new fields to uniques
  • 13:21 gtirloni: T207377 icinga downtime and reboot of labcontrol1001 and labservices1001
  • 13:08 arturo: T207377 icinga downtime and reboot of cloudcontrol1003 and cloudservices1003
  • 13:06 raynor: EU SWAT finished
  • 13:03 pmiazga@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:474679]|In SecurePoll use gpg1 to avoid gpg-agent autostart (T209802)]] (duration: 00m 48s)
  • 12:50 raynor: EU SWAT reopened
  • 12:40 raynor: EU SWAT finished
  • 12:38 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:472918]|Enable autopatrol, patrol, rollback rights and RCPatrol on srwiktionary (T209252)]] (duration: 00m 46s)
  • 12:21 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:474124]|Remove wgMetaNamespaceTalk for shnwiki (T206777)]] (duration: 00m 46s)
  • 12:10 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:473225]|Enable Schema.org page split test at 50% sampling (T208755)]] (duration: 00m 46s)
  • 11:33 gtirloni: labsdb1011 upgraded packages on labsdb1011 (pre-work T209517)
  • 11:20 elukey: restart memcached on mc1020 to apply -R 200 settings (shard wiped) - T208844
  • 10:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1078 T209754 (duration: 00m 46s)
  • 10:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1078 T209754 (duration: 00m 46s)
  • 10:21 banyek: stopping replication on db2076 (T85757)
  • 10:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1078 T209754 (duration: 00m 46s)
  • 09:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1123 and increase weight for db1078 T209754 (duration: 00m 46s)
  • 09:48 marostegui: Rename table foundationwiki.petition_data on db1078 - T208979
  • 09:46 marostegui: Drop empty testwiki.petition_data from db1075 with replication - T208979
  • 09:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1123 and db1078 T209754 (duration: 00m 46s)
  • 09:10 Nikerabbit: Rebuilt message group stats cache for T208521
  • 08:43 banyek: executing schema change on db2095 (T85757)
  • 07:57 marostegui: Stop MySQL on db1123 - T209754
  • 07:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 to clone db1078 T209754 (duration: 00m 47s)
  • 06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Add db1078 line back to config file but depooled T209754 (duration: 00m 51s)
  • 06:20 marostegui@deploy1001: sync-file aborted: Add db1078 line back to config file but depooled T209754 (duration: 00m 02s)

2018-11-18

  • 17:26 andrewbogott: restarting cp1078 from mgmt console
  • 09:00 elukey: cleaned up analytics1039 and restarted Yarn

2018-11-17

  • off: 'reset modified attributes' on IcingaUI for db1078 (and mgmt) and all its services
  • 06:38 oblivian@deploy1001: Synchronized wmf-config/db-eqiad.php: Depooling db1078 (duration: 00m 59s)
  • 02:54 RoanKattouw: Deployed patches for T208112, T208109, T208110

2018-11-16

  • 23:13 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@ee91c41]: Deploy on test wdqs1010 (duration: 00m 23s)
  • 23:12 smalyshev@deploy1001: Started deploy [wdqs/wdqs@ee91c41]: Deploy on test wdqs1010
  • 16:23 Trey314159: reindexing Chinese wikis on elastic@eqiad and elastic@codfw (T209156)
  • 15:46 moritzm: rebooting debmonitor* instances for kernel security update and to pick up SSBD
  • 15:01 hashar: restarting zuul with 1300 events to process
  • 14:56 marostegui: Create ipblocks_restrictions on labswiki and labtestwiki on db1073 - T209674
  • 14:56 ema: upgrade cp-ats to 8.0.0-1wm2 T204225 T204209
  • 14:39 ema: trafficserver 8.0.0-1wm2 uploaded to stretch-wikimedia T204225 T204209
  • 14:36 hashar: Gracefully stopping zuul (kill -SIGUSR1)
  • 14:29 _joe_: re-depooling mw1261 for php-fpm testing
  • 14:28 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw126[1-6].*,dc=eqiad,cluster=appserver
  • 14:26 _joe_: repooling the mw canaries
  • 13:02 moritzm: installing spamassassin security update on mendelevium
  • 12:10 godog: reboot restbase1014, nothing on console
  • 11:13 kartik@deploy1001: Finished deploy [cxserver/deploy@473b0de]: Update cxserver to b7cdb26 (T208831, T203077, T203160, T206777) (duration: 04m 26s)
  • 11:09 kartik@deploy1001: Started deploy [cxserver/deploy@473b0de]: Update cxserver to b7cdb26 (T208831, T203077, T203160, T206777)
  • 10:39 akosiaris: upgrade OTRS to 5.0.32 T209691
  • 09:31 marostegui: Set back sync_binlog=1 and trx_commit=1 after dbstore2002:3313 has caught up
  • 09:25 moritzm: installing postgres updates on labsdb1006
  • 09:21 moritzm: removed labvirt1016 from debmonitor db, got renamed to cloudvirt1016
  • 08:40 moritzm: installing curl security updates on jessie
  • 07:32 elukey: forced reboot + fsck + removal of /var/lib/hadoop/data/l from fstab on analytics1029
  • 06:36 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Pool pc2009 in pc3 - T208383 (duration: 00m 56s)
  • 06:28 marostegui: Set sync_binlog=0 and trx_commit=2 on dbstore2002:3313 to let it catch up
  • 05:47 vgutierrez: uploaded certcentral 0.7 to apt.wikimedia.org (stretch) - T208967 T209475
  • 00:55 mutante: some users reported missing files in home dirs on mwmaint1002, reversed rsyncd/ferm setup and rsynced /home from mwmaint2001 to /root on mwmaint1002, restored individually where requested, rsync is not fully automatic but puppetized with rsync::quickdatacopy
  • 00:35 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/includes/PageViews.php: SWAT: Exclude users where getRegistration() returns null (duration: 00m 47s)
  • 00:26 eileen: civicrm revision changed from 71755d021b to 6b279509f8, config revision is 684ec9b7c0 (lybunt report)
  • 00:22 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/GrowthExperiments: SWAT: gerrit:473843 gerrit:473844 gerrit:473845 (duration: 00m 49s)
  • 00:21 sbisson@deploy1001: sync aborted: php-1.33.0-wmf.4/extensions/GrowthExperiments SWAT: gerrit:473843 gerrit:473844 gerrit:473845 (duration: 06m 16s)
  • 00:15 sbisson@deploy1001: Started scap: php-1.33.0-wmf.4/extensions/GrowthExperiments SWAT: gerrit:473843 gerrit:473844 gerrit:473845

2018-11-15

  • 21:58 mutante: mwmaint1002 - restoring entire /home of mwmaint1001 from Bacula (job queued and to tmp dir, not directly into /home)
  • 21:06 hashar: Deleting Nodepool instances on contintcloud T209361
  • 21:05 hashar: Stopped nodepool on labnodepool1001.eqiad.wmnet . Service is no more used. T209361 T209642
  • 20:22 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@UNKNOWN]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471) (duration: 03m 55s)
  • 20:18 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@UNKNOWN]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471)
  • 20:17 mutante: re-added Chase to pwstore, signed .users file, re-encrypted all pwstore files, git pushed
  • 20:15 mholloway-shell@deploy1001: Finished deploy [kartotherian/deploy@48a1e83]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471) (duration: 04m 26s)
  • 20:11 mholloway-shell@deploy1001: Started deploy [kartotherian/deploy@48a1e83]: Fix: Loosen WDQS content-type header check to unbreak maps (T209471)
  • 20:04 urandom: dropping disused keyspaces -- T208616
  • 20:04 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable data collection for UnderstandingFirstDay on cswiki and kowiki (duration: 00m 53s)
  • 19:51 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure sensitive namespaces for EditorJourney schema (T207307) (duration: 00m 53s)
  • 19:49 catrope@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/: cherry-picks for T208773 (duration: 00m 54s)
  • 18:55 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Hot-deploy I26f2dc2e: Don't over-ride default Wikibase string limits (duration: 00m 53s)
  • 18:50 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: Hot-deploy Ib10de2e3: Don't set Wikibase string limits when null (duration: 00m 55s)
  • 18:31 ladsgroup@deploy1001: Finished deploy [ores/deploy@dba11e9]: Another small update (duration: 13m 42s)
  • 18:18 ladsgroup@deploy1001: Started deploy [ores/deploy@dba11e9]: Another small update
  • 18:17 ladsgroup@deploy1001: Finished deploy [ores/deploy@51cdf6b]: T208623 (duration: 14m 41s)
  • 18:03 ladsgroup@deploy1001: Started deploy [ores/deploy@51cdf6b]: T208623
  • 17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 7 wikis in group 2 for T209373. This may cause lag in codfw.
  • 17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 6 wikis in group 2 for T209373. This may cause lag in codfw.
  • 17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 5 wikis in group 2 for T209373. This may cause lag in codfw.
  • 17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 3 wikis in group 2 for T209373. This may cause lag in codfw.
  • 17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 2 wikis in group 2 for T209373. This may cause lag in codfw.
  • 17:27 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 1 wikis in group 2 for T209373. This may cause lag in codfw.
  • 17:20 gehel: upgrade prometheus-blazegraph-exporter on all wdqs nodes - T206123
  • 16:41 bstorm_: rebooted labsdb1007 for upgrades
  • 16:37 andrewbogott: rebuilding labvirt1015 and cloudvirt1015
  • 14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on wikitech for T209373. This may cause lag in codfw.
  • 14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 8 wikis in group 1 for T209373. This may cause lag in codfw.
  • 14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 7 wikis in group 1 for T209373. This may cause lag in codfw.
  • 14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 5 wikis in group 1 for T209373. This may cause lag in codfw.
  • 14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 4 wikis in group 1 for T209373. This may cause lag in codfw.
  • 14:21 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 3 wikis in group 1 for T209373. This may cause lag in codfw.
  • 14:20 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 2 wikis in group 1 for T209373. This may cause lag in codfw.
  • 14:07 gehel: plugin and JVM upgrade on elasticsearch / cirrus / eqiad completed - T209293
  • 13:59 hashar@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.4
  • 13:55 XioNoX: push firewall policies to pfw3-eqiad - T209421
  • 13:50 banyek: Deploy schema change on s6 codfw master, this will generate lag on s6 codfw - T85757
  • 13:48 moritzm: installing qemu security updates (which also backport support for SSBD passthrough) on ganeti clusters
  • 13:48 moritzm: installing qemu security updates (which also backport support for SSBD passthrough)
  • 13:12 mobrovac@deploy1001: Finished deploy [restbase/deploy@22cb0ec]: Add new wikis to RESTBase - T206777 T205710 T205546 T204477 (duration: 19m 56s)
  • 13:00 Lucas_WMDE: EU SWAT finished
  • 12:59 tarrow@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js: gerrit:473710 Fix (accidentally?) reversed blue and yellow lines SWAT T208238 T162119 again (duration: 00m 55s)
  • 12:57 _joe_: upping pm.maxworkers to 40 on mw1261 on php7.2-fpm, benchmarking T206341
  • 12:56 tarrow@deploy1001: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
  • 12:53 moritzm: installing nginx security updates
  • 12:52 mobrovac@deploy1001: Started deploy [restbase/deploy@22cb0ec]: Add new wikis to RESTBase - T206777 T205710 T205546 T204477
  • 12:47 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove wgWBQualityConstraintsCacheCheckConstraintsResults (T207854) (duration: 00m 54s)
  • 12:42 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Prod: increase Schema.org page split test to 25% sampling T208755 (duration: 00m 53s)
  • 12:40 arturo: T207377 downtime and reboot labmon1001
  • 12:38 lucaswerkmeister-wmde@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js: Fix (accidentally?) reversed blue and yellow lines (T162119, T208238) (duration: 00m 54s)
  • 12:37 lucaswerkmeister-wmde@deploy1001: sync aborted: php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js Fix (accidentally?) reversed blue and yellow lines (T162119, T208238) (duration: 00m 11s)
  • 12:37 lucaswerkmeister-wmde@deploy1001: Started scap: php-1.33.0-wmf.3/extensions/RevisionSlider/modules/ext.RevisionSlider.SliderView.js Fix (accidentally?) reversed blue and yellow lines (T162119, T208238)
  • 12:30 tarrow@deploy1001: Synchronized wmf-config/Wikibase.php: gerrit:473716 Read WikibaseStringLimit in Wikibase.php T154660 (duration: 00m 53s)
  • 12:30 moritzm: draining ganeti1001 for reboot/kernel security update
  • 12:26 tarrow@deploy1001: sync-file aborted: gerrit:473716 Read WikibaseStringLimit in Wikibase.php (duration: 00m 01s)
  • 12:21 tarrow@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:473694 Set Wikibase string-limits for wikidata dblist T154660 (duration: 00m 54s)
  • 12:08 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Make AdvancedSearch the default on de-, fa-, ar-, and hu-wiki T207640 (duration: 00m 55s)
  • 11:34 _joe_: depooling mw1261 for early benchmarks of php7.2-fpm
  • 11:04 akosiaris: enable puppet across the fleet. puppetdb1001 reboot done, ganeti migration_downtime setting applied
  • 10:49 akosiaris: disable puppet across the fleet for puppetdb1001 reboot
  • 10:49 moritzm: fail over ganeti master in eqiad to ganeti1003
  • 10:34 akosiaris: set migration_downtime=2000 for puppetdb1001. Should help with migration stalls
  • 10:15 banyek: sanitizing db1124 ( T205714 T207584 T205713 T206916 )
  • 10:07 banyek: sanitizing db2094 ( T205714 T207584 T205713 T206916 )
  • 09:57 moritzm: draining ganeti1002 for reboot/kernel security update
  • 09:40 volans: restarting icinga on icinga1001
  • 09:32 vgutierrez: restarting icinga on icinga1001
  • 09:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 (duration: 00m 52s)
  • 09:22 moritzm: draining ganeti1003 for reboot/kernel security update
  • 09:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 (duration: 00m 53s)
  • 09:16 marostegui: Stop MySQL on db1110 for upgrade
  • 09:08 moritzm: reset failed debmonitor session in ms-be2038
  • 09:08 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1088 (duration: 00m 53s)
  • 09:03 banyek: repooling db1088 (T85757)
  • 08:58 ema: upload fifo-log-demux 0.1 to stretch-wikimedia T204225
  • 08:56 banyek: Deploy schema change on db1088 (T85757)
  • 08:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1088 (duration: 00m 53s)
  • 08:45 banyek: depooling db1088 due a schema change (T85757)
  • 08:21 moritzm: draining ganeti1004 for reboot/kernel security update
  • 07:42 marostegui: Drop site_stats.ss_total_views from labswiki - T86339
  • 07:08 elukey: memcached on mc1019 restarted to apply -R 200 - T208844
  • 06:57 marostegui: Stop MySQL on db2071 to upgrade MySQL and kernel
  • 06:57 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 54s)
  • 06:06 marostegui: Stop MySQL on pc2006 to clone pc2009 - T208383
  • 06:06 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2006 - T208383 (duration: 00m 53s)
  • 06:05 marostegui@deploy1001: sync-file aborted: Dool pc2006 - T208383 (duration: 00m 00s)
  • 05:59 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Pool pc2008 in pc2 - T208383 (duration: 00m 56s)
  • 00:50 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: increase Schema.org page split test to 5% sampling T208755 (duration: 00m 54s)

2018-11-14

  • 22:46 ejegg: updated payments-wiki from 5751286f1c to d2b66c5bab
  • 21:34 thcipriani: restart gerrit to load JavaMelody dependency library
  • 21:28 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on cobalt (duration: 00m 09s)
  • 21:28 thcipriani@deploy1001: Started deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on cobalt
  • 21:27 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on gerrit2001 (duration: 00m 11s)
  • 21:26 thcipriani@deploy1001: Started deploy [gerrit/gerrit@ab2fa18]: deploy javamelody on gerrit2001
  • 20:13 hashar@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.4 (duration: 00m 52s)
  • 20:12 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.4
  • 19:14 hashar@deploy1001: Synchronized php-1.33.0-wmf.4/includes/jobqueue/JobQueue.php: Actually return the value from getRootJobCacheKey() - T209429 (duration: 00m 53s)
  • 18:33 hashar@deploy1001: Finished scap: php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php Add trace to debug memcached bad key error - T209429 (duration: 34m 07s)
  • 17:58 hashar@deploy1001: Started scap: php-1.33.0-wmf.4/includes/libs/objectcache/MemcachedPeclBagOStuff.php Add trace to debug memcached bad key error - T209429
  • 17:53 arturo: T207377 downtime and reboot cloudnet1003 (cloudnet1004 is the active one already)
  • 17:37 arturo: T207377 downtime and reboot cloudnet1004 (cloudnet1003 is the active one already)
  • 17:31 bawolff_: Running importImage.php for 'Opening ceremony of First accusation protest against presumption of guilt of judicial branch.webm' per request T209495
  • 17:26 sbisson@deploy1001: Synchronized dblists/wikidataclient.dblist: SWAT: Add incubatorwiki to wikidataclient.dblist (duration: 00m 48s)
  • 17:19 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/MobileFrontend/resources/mobile.editor.common/schemaEditAttemptStep.js: SWAT: schemaEditAttemptStep.js: Use correct config var name for sampling rate (duration: 00m 54s)
  • 17:12 sbisson@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: SWAT: Fix EditAttemptStepSamplingRate variable export (duration: 00m 54s)
  • 16:33 bblack: [Done] replacement of GlobalSign unified TLS cert at US edges complete - T206804
  • 16:25 moritzm: rebooting ganeti1005 for kernel security update
  • 16:16 moritzm: rebooting restbase-dev1006 for kernel security update and OpenJDK security update
  • 16:10 bblack: disabling puppet as precaution on all caches (cumin A:cp) - T206804
  • 16:09 bblack: starting replacement of GlobalSign unified TLS cert at US edges (affects all public TLS termination for US traffic edges) - T206804
  • 16:08 moritzm: rebooting restbase-dev1005 for kernel security update and OpenJDK security update
  • 15:58 moritzm: rebooting restbase-dev1004 for kernel security update and OpenJDK security update
  • 15:53 jiji: Restarting pdfrender on scb*.eqiad.wmnet
  • 15:50 godog: roll restart swift-proxy in eqiad to apply statsd changes
  • 15:41 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T85757: repool db2046 (duration: 00m 52s)
  • 15:39 banyek: repooling db2046 (T85757)
  • 15:12 godog: roll-restart swift on ms-be1* to pick up statsd changes
  • 15:07 Amir1: ladsgroup@mwmaint1002:/srv/mediawiki-staging/php-1.33.0-wmf.4$ mwscript sql.php --wiki=incubatorwiki extensions/Wikibase/client/sql/entity_usage.sql (T209207)
  • 14:56 addshore@deploy1001: Synchronized wmf-config: Prod: Enable Schema.org page split test at 1% sampling (again) (duration: 00m 54s)
  • 14:29 godog: roll-restart swift-proxy in codfw to pick up statsd changes
  • 14:14 addshore@deploy1001: Synchronized wmf-config: Revert Prod: Enable Schema.org page split test at 1% sampling (duration: 00m 54s)
  • 14:10 Reedy: Wiki created T205714 T207584 T205713 T206916
  • 14:07 gehel: starting plugin and JVM upgrade on elasticsearch / cirrus / eqiad - T209293
  • 14:07 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 25s)
  • 14:02 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix shnwiki TZ (duration: 00m 54s)
  • 13:57 reedy@deploy1001: rebuilt and synchronized wikiversions files: (no justification provided)
  • 13:55 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Add pc2010 as spare - T208383 (duration: 00m 53s)
  • 13:54 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: (no justification provided) (duration: 00m 53s)
  • 13:51 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: new wikis (duration: 00m 53s)
  • 13:51 gehel: restarting tilerator on maps1004 for config change
  • 13:50 reedy@deploy1001: Synchronized static/images/project-logos/: (no justification provided) (duration: 00m 53s)
  • 13:49 reedy@deploy1001: Synchronized dblists/: new wikis! (duration: 00m 53s)
  • 13:48 reedy@deploy1001: Synchronized langlist: shn (duration: 00m 52s)
  • 13:45 gehel: plugin and JVM upgrade completed on elasticsearch / cirrus / codfw - T209293
  • 13:45 reedy@deploy1001: rebuilt and synchronized wikiversions files: (no justification provided)
  • 13:07 moritzm: installing ghostscript security updates on stretch
  • 13:01 reedy@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaMaintenance/addWiki.php: Fixing addshores code... (duration: 00m 53s)
  • 13:00 reedy@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMaintenance/addWiki.php: Fixing addshores code... (duration: 00m 55s)
  • 12:56 moritzm: installing gettext "security" updates for trusty
  • 12:48 moritzm: installing python3.4 security updates on trusty (Debian already fixed)
  • 12:40 reedy@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding wiktionary (duration: 00m 52s)
  • 12:39 reedy@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding wiktionary (duration: 00m 53s)
  • 12:39 moritzm: installing python security updates on trusty
  • 12:36 Amir1: EU SWAT is done
  • 12:35 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert the language of votewiki to English (en) (T207560) (duration: 00m 55s)
  • 12:30 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Start reading from change_tag_def on wikidatawiki (T208846) (duration: 00m 55s)
  • 12:19 pmiazga@deploy1001: Synchronized wmf-config: SWAT: [[gerrit:473079]|Enable Schema.org page split test at 1% sampling (T208755)]] (duration: 00m 54s)
  • 12:02 reedy@deploy1001: Synchronized php-1.33.0-wmf.4/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding TMH tables (duration: 00m 53s)
  • 12:01 reedy@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMaintenance/addWiki.php: Unbreak adding TMH tables (duration: 00m 55s)
  • 11:11 moritzm: draining ganeti1005 for reboot/kernel security update
  • 11:09 banyek: Deploy schema change on db2046 (T85757)
  • 10:07 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: T85757: depooling db2046 (duration: 00m 55s)
  • 09:59 banyek: depooling db2046 (T85757)
  • 09:22 moritzm: updated stretch netinst image for 9.6 point release
  • 09:17 marostegui: Deploy schema change on db2053 - T86339
  • 08:24 marostegui: Deploy schema change on s5 codfw master, this will generate lag on s5 codfw - T205913
  • 08:22 marostegui: Deploy schema change on s2 codfw master, this will generate lag on s7 codfw - T205913
  • 08:19 marostegui: Deploy schema change on s2 codfw master, this will generate lag on s2 codfw - T205913
  • 08:17 marostegui: Deploy schema change on s6 codfw master, this will generate lag on s6 codfw - T205913
  • 08:14 marostegui: Deploy schema change on s4 codfw master, this will generate lag on s4 codfw - T205913
  • 08:08 marostegui: Deploy schema change on s3 codfw master, this will generate lag on s3 codfw - T205913
  • 08:07 godog: rollout rsyslog_exporter to eqiad
  • 07:42 marostegui: Deploy schema change on s3 codfw master, this will generate lag on s3 codfw - T203709
  • 07:19 marostegui: Deploy schema change on s7 codfw master, this will generate lag on s7 codfw - T203709
  • 07:07 marostegui: Deploy schema change on s2 codfw master, this will generate lag on s2 codfw - T203709
  • 06:52 marostegui: Deploy schema change on s4 codfw master, this will generate lag on s4 codfw - T203709
  • 06:40 marostegui: Deploy schema change on s6 codfw master, this will generate lag on s6 codfw -T203709
  • 06:32 marostegui: Stop MySQL on pc2005 to clone it to pc2008 - T208383
  • 06:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2005 - T208383 (duration: 01m 04s)
  • 05:46 _joe_: restarting gerrit
  • 01:02 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/Wikibase/client/includes: SWAT: Update: use wikibase-debug logger instead of "PageRandomLookup" T208796 (duration: 00m 56s)
  • 00:42 mutante: restarted smokeping on netmon1002 and netmon2001

2018-11-13

  • 22:42 XioNoX: restart librenms irc bot
  • 22:24 XioNoX: add term labnet-nova-api to cloud-in4 on cr1/2-eqiad - T209424
  • 20:22 herron: updated labs realm smarthosts (via hiera) to mx-out0[12].wmflabs.org T41785
  • 19:49 otto@deploy1001: Finished deploy [analytics/refinery@62d6f4b]: Deploy hive jars from CDH 5.10.0 to workaround Refine bug: T209407 (duration: 05m 57s)
  • 19:43 otto@deploy1001: Started deploy [analytics/refinery@62d6f4b]: Deploy hive jars from CDH 5.10.0 to workaround Refine bug: T209407
  • 19:31 herron: uploaded librdkafka_0.11.6-1~bpo9+1+wikimedia1 packages to stretch-wikimedia T209300
  • 18:11 mutante: the CUSTOM message from ores.svc.codfw was the (one-time) test of the new Icinga server
  • 18:03 mutante: icinga migration has concluded, we are now on stretch and icinga1001, einsteinium is passive (T202782)
  • 17:27 mutante: re-enabled puppet on icinga1001, einsteinium becoming passive
  • 17:21 mutante: ran puppet on einsteniumr; e-enabling puppet on tegmen and icinga1001
  • 17:13 bstorm_: Added 172.16.0.0/21 to the allowed connections for wikilabels postgresql on labsdb1004
  • 17:04 mutante: disabled puppet on all 3 icinga servers, re-enabling on einsteinium , going through https://wikitech.wikimedia.org/wiki/Icinga#Failover_Icinga_between_the_active_and_passive_servers
  • 17:02 ejegg: updated payments-wiki from 20542c9184 to 5751286f1c
  • 17:01 mutante: starting migration of icinga server - maintenance windows
  • 16:33 thcipriani: restarting gerrit service for upgrade to 2.15.6
  • 16:32 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@d2763c6]: v2.15.6 to cobalt (duration: 00m 10s)
  • 16:32 thcipriani@deploy1001: Started deploy [gerrit/gerrit@d2763c6]: v2.15.6 to cobalt
  • 16:29 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@d2763c6]: v2.15.6 to gerrit2001 (duration: 00m 11s)
  • 16:29 thcipriani@deploy1001: Started deploy [gerrit/gerrit@d2763c6]: v2.15.6 to gerrit2001
  • 16:22 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on test wikis and mediawikiwiki (T188327) (duration: 00m 54s)
  • 16:07 anomie@mwmaint1002: Running refreshExternallinksIndex.php on labtestwiki for T209373
  • 16:07 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 3 wikis in group 0 for T209373
  • 15:48 _joe_: upgrading extensions on all appservers / jobrunners while upgrading to php 7.2
  • 15:45 gehel: restart tilerator on maps1004
  • 15:21 moritzm: draining ganeti1006 for reboot/kernel security update
  • 15:18 marostegui: Restore replication consistency options on dbstore2002:3313 as it has caught up - T208320
  • 14:59 akosiaris: increase the migration downtime for kafkamon1001. It should make live migration of these VMs easier and without the need for manual fiddling
  • 14:54 hashar@deploy1001: rebuilt and synchronized wikiversions files: group to 1.33.0-wmf.4 | T206658
  • 14:40 hashar@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.4 | T206658 (duration: 19m 34s)
  • 14:27 moritzm: draining ganeti1007 for reboot/kernel security update
  • 14:20 hashar@deploy1001: Started scap: testwiki to php-1.33.0-wmf.4 | T206658
  • 14:20 akosiaris: reboot logstash1007, logstash1008, logstash1009 with 500 secs of sleep between them for the migration_downtime ganeti setting to be applied
  • 14:18 akosiaris: increase the migration downtime for logstash1007, logstash1008, logstash1009. It should make live migration of these VMs easier and without the need for manual fiddling
  • 14:15 hashar@deploy1001: Pruned MediaWiki: 1.32.0-wmf.24 (duration: 08m 55s)
  • 14:03 hashar: Applied security patches to 1.33.0-wmf.4 | T206658
  • 14:03 gehel: start plugin and JVM upgrade on elasticsearch / cirrus / codfw - T209293
  • 14:00 hashar: scap prep 1.33.0-wmf.4 # T206658
  • 13:58 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Pool pc2007 to replace pc2004 (duration: 00m 48s)
  • 13:41 marostegui: Deploy schema change on s8 codfw master (db2045) this will generate lag on s8 codfw - T203709
  • 13:40 hashar: Cutting wmf/1.33.0-wmf.4 branch | T206658
  • 13:30 moritzm: draining ganeti1008 for reboot/kernel security update
  • 12:51 phuedx: European Mid-day SWAT finished
  • 12:50 phuedx@deploy1001: Finished scap: SWAT: Define WikimediaMessages for Wikibase SEO change l18n refresh (duration: 21m 43s)
  • 12:28 phuedx@deploy1001: Started scap: SWAT: Define WikimediaMessages for Wikibase SEO change l18n refresh
  • 12:22 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMessages/: SWAT: Define WikimediaMessages for Wikibase SEO change (T208755) (duration: 00m 56s)
  • 10:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 52s)
  • 10:47 marostegui: Deploy schema change on db1116:3318 T203709
  • 10:40 godog: stop sending metrics to old graphite hardware
  • 10:15 gehel: restart elasticsearch on relforge for plugin upgrade - T209293
  • 09:54 moritzm: restarting jenkins on releases1001 to pick up Java security update
  • 09:25 _joe_: uploading new versions of php-msgpack, php-geoip compatible with both php 7.0 and php 7.2 to thirdparty/php72 T208433
  • 09:23 marostegui: Deploy schema change on db1092 T203709
  • 09:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 52s)
  • 09:20 elukey: rollout new prometheus-mcrouter-exporter to mw* - previous rollout didn't work as expected
  • 09:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 (duration: 00m 55s)
  • 08:37 moritzm: updating remaining rsyslog on stretch to 8.38.0-1~bpo9+1wmf1
  • 07:21 marostegui: Deploy schema change on db1104 T203709
  • 07:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 (duration: 00m 53s)
  • 07:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 54s)
  • 07:05 elukey: powercycle lvs2006 - mgmt/serial console blank, not responsive since hours ago
  • 06:02 marostegui: Add ipb_sitewide column to db1073:labtestwiki
  • 05:43 marostegui: Stop MySQL on pc2004 to transfer its data to pc2007 - T208383
  • 05:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2004 - T208383 (duration: 00m 53s)
  • 05:39 marostegui: Deploy schema change on db2048 (s1 codfw master), this will create lag on s1 codfw - T114117
  • 05:34 marostegui: Deploy schema change on db1109 T203709
  • 05:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 55s)

2018-11-12

  • 19:22 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T208663 4ff32d1df - Enable moving files for users with patrol and rollbacker rights on srwiki (duration: 00m 54s)
  • 18:29 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@ee91c41]: GUI update, New Thesaurus endpoint, New updater build and blazegraph update (duration: 11m 28s)
  • 18:17 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@ee91c41]: GUI update, New Thesaurus endpoint, New updater build and blazegraph update
  • 18:03 elukey: rolling restart of aqs on aqs* to pick up new druid datasource settings
  • 17:44 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Optimize s2 for throughput (duration: 00m 53s)
  • 17:19 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool more resources into s2 api (duration: 00m 54s)
  • 17:15 _joe_: restarting HHVM on the high-cpu api hosts in eqiad, to ease the pressure and latencies
  • 17:10 _joe_: depooling mw1222 for debug
  • 16:41 banyek: disabling puppet on parsercache hosts (T208383)
  • 16:14 elukey: upgrade prometheus-mcrouter-exporter on all the mw* hosts to the new version
  • 16:09 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for large wikis
  • 16:02 volans: restarted proton on proton1002
  • 15:45 jynus: stop and upgrade db2094
  • 15:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 53s)
  • 14:49 banyek: disabling puppet on parsercache hosts - pc[12]00[456] (T208383)
  • 14:17 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for medium wikis
  • 14:08 moritzm: updating rsyslog on stretch to 8.38.0-1~bpo9+1wmf1
  • 13:59 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for small wikis (small.dblist)
  • 13:59 marostegui: Deploy schema change on db1101:3318 - T203709
  • 13:55 hashar: Upgrading Jenkins on contint1001 , contint2001, releases1001 and releases2002 | T209264
  • 13:46 moritzm: updating libfastjson on stretch to 0.99.8-1~bpo9+1wmf1
  • 13:41 gehel: starting rolling restart of elasticsearch codfw for JVM upgrade
  • 13:32 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for mediawikiwiki
  • 13:23 phuedx: phuedx@mwmaint1002 running resetPageRandom.php maintenance script for testwiki
  • 13:17 zeljkof: EU SWAT finished
  • 13:16 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/maintenance/resetPageRandom.php: SWAT: Provide a script to reset the page_random column (T208909) (duration: 00m 53s)
  • 13:16 moritzm: updating liblognorm on stretch to 2.0.3-1~bpo9+1wmf1
  • 13:14 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/autoload.php: SWAT: Provide a script to reset the page_random column (T208909) (duration: 00m 55s)
  • 13:12 elukey: upgrade the Hadoop Analytics cluster to CDH 5.15 (downtime required)
  • 12:54 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add new throttle rule for Wikipedia event in Ireland on 2018-11-13 (T209037) (duration: 00m 53s)
  • 12:15 jiji: Restarting nutcracker on scb200[1-6] - T206450
  • 12:00 moritzm: uploaded jenkins 2.138.3 to apt.wikimedia.org (jessie and stretch)
  • 11:49 hashar: updating puppet CI job for mtail upgrade https://gerrit.wikimedia.org/r/#/c/integration/config/+/472962/
  • 11:37 hashar: contint1001 : cleaning disk | T209123 ?
  • 11:26 moritzm: installing Java security updates on elastic*
  • 10:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 00m 55s)
  • 10:48 godog: upload mtail 3.0.0~rc5-1~bpo9+1wmf1 to stretch-wikimedia
  • 10:45 marostegui: Deploy schema change on db2048 (s1 codfw master), this will generate lag on s1 codfw - T51191
  • 10:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 (duration: 00m 53s)
  • 10:32 elukey: upload mcrouter exporter 0.0.0+git20181106 to stretch-wikimedia
  • 09:57 elukey: upgraded cdh packages (cdh 5.10 -> 5.15) for thirdparty/cloudera in jessie/stretch-wikimedia
  • 09:12 marostegui: Deploy schema change on db2048 (s1 codfw master) (replication will be stopped) - T67448
  • 08:53 marostegui: Deploy schema change on db1099:3318 - T203709
  • 08:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 (duration: 01m 01s)
  • 08:41 marostegui: Change sync_binlog to 0 and trx_commit to 2 on dbstore2002:3313 to let it catch up
  • {{safesubst:SAL entry|1=08:26 _joe_: uploading new php-{luasandbox,wikidiff2} to stretch main component, rebuild php-{luasandbox,wikidiff2,geoip,msgpack} for php 7.2, upload to stretch component php72, T208433}}
  • 08:23 godog: temporarily disable puppet in codfw before enabling rsyslog_exporter

2018-11-10

  • 01:10 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 39s)
  • 01:10 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting
  • 01:07 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 04s)
  • 01:07 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting

2018-11-09

  • 21:46 SMalyshev: repooled wdqs1004 - looks like other servers feel worse so probably makes sense to share the load equally
  • 21:16 jiji: Reimaging rdb2003, rdb2004 - T206450
  • 20:46 SMalyshev: depooled wdqs1004 to let it catch up
  • 20:40 legoktm@deploy1001: Synchronized docroot/mediawiki/keys/: Add my (20after4) PGP key to mediawiki.org/keys/keys.(txt|html) (duration: 00m 55s)
  • 20:08 andrewbogott: restarted neutron-linuxbridge-agent on cloudvirt1018 and cloudvirt1023
  • away: repooling labsdb1011 (T189158)
  • 15:24 banyek: depooling labsdb1011 (T189158)
  • 15:23 banyek: depooling labsdb1011
  • 15:08 banyek: repooling labsdb1009 (T189158)
  • 15:06 bblack: cp1008/pinkunicorn: puppet disabled, public-facing testing of new globalsign 2018 certs
  • 15:04 ladsgroup@deploy1001: Finished deploy [ores/deploy@bb39f4b]: T191842 T209060, try II (duration: 14m 43s)
  • 14:50 andrewbogott: rebooting cloudvirt1024 to (I hope) cause a page
  • 14:49 ladsgroup@deploy1001: Started deploy [ores/deploy@bb39f4b]: T191842 T209060, try II
  • 14:48 ladsgroup@deploy1001: deploy aborted: T191842 T209060 (duration: 09m 32s)
  • 14:39 ladsgroup@deploy1001: Started deploy [ores/deploy@0728805]: T191842 T209060
  • 14:18 addshore@deploy1001: Synchronized wmf-config: BETA ONLY: Enable SSR termbox for wikibase on beta - T209143 (duration: 00m 56s)
  • 13:32 moritzm: rebooting acrab for some qemu tests
  • 13:21 godog: upload graphite-web_1.0.2+debian-2.1wmf1 to stretch-wikimedia - T208782
  • 13:10 moritzm: upgrading qemu on ganeti2001 (packages supporting SSBD passthrough)
  • 12:40 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T189158: repool db1106 (duration: 00m 53s)
  • 12:34 banyek: repooling db1106 (T208954)
  • 12:17 kartik@deploy1001: Finished deploy [cxserver/deploy@fc21164]: Update cxserver to 01686f6 (T208831) (duration: 01m 09s)
  • 12:16 kartik@deploy1001: Started deploy [cxserver/deploy@fc21164]: Update cxserver to 01686f6 (T208831)
  • 11:45 banyek: data load finished restarting replication on db1106 (T208954)
  • 11:43 akosiaris: set previous normal wait for scb1001 for apertium service T206439
  • 11:39 akosiaris@puppetmaster1001: conftool action : set/weight=8; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 11:30 akosiaris: upgrade apertium apertium-cat apertium-fra apertium-fra-cat apertium-lex-tools apertium-separable cg3 libapertium3-3.5-1 libcg3-1 lttoolbox on all scb boxes and restart apertium-apy
  • 11:26 akosiaris: upgrade apertium apertium-cat apertium-fra apertium-fra-cat apertium-lex-tools apertium-separable cg3 libapertium3-3.5-1 libcg3-1 lttoolbox on scb1002
  • 11:22 jiji: switch scb*.eqiad.wmnet nutcracker rdb1003:6382 with rdb1005:6379
  • 10:51 vgutierrez: uploaded certcentral 0.6 to apt.wikimedia.org (stretch) - T208859 T208948 T208967 T208970
  • 09:48 ema: repool cp2018, cp2025 (cache_upload) T208588
  • 09:45 banyek: truncating enwiki.archive on db1124 and labsdb hosts too (T208954)
  • 09:21 banyek: stopping replication on db1106 (T208954)
  • 09:21 banyek: stopping replication on db1106 (T208672)
  • 09:08 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T189158: depool db1106 (duration: 00m 55s)
  • 09:02 banyek: depooling db1106 (T208954)
  • 08:28 moritzm: installing nginx security updates
  • 08:05 ema: repool cp2006, cp2012 (cache_text) T208588
  • 04:33 ejegg: restarted recurring donation charge jobs
  • 04:24 ejegg: updated fundraising CiviCRM from 1154cca3f2 to 71755d021b
  • 03:25 ejegg: updated fundraising CiviCRM from 02cc1f80d4 to 1154cca3f2
  • 00:07 ejegg: updated fundraising CiviCRM from 07183ed7cc to 02cc1f80d4
  • 00:03 ejegg: updated payments-wiki from 983ce3af0f to 20542c9184

2018-11-08

  • 22:48 mutante: gerrit - adding Thomas Arrow to 'wmf-deployment' group for +2 on mw-config for T208491 access request
  • 22:37 mutante: gerrit - adding Lucas Werkmeister (WMDE) to 'wmf-deployment' group for +2 on mw-config for T208518 access request
  • 20:28 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.3
  • 19:35 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Disable wmgUseTwoColConflict everywhere" T205942 T208840 T209012 T209036 (duration: 00m 54s)
  • 18:19 shdubsh: update statsd-proxy to 0.0.9-2 on graphite1004
  • 17:29 banyek: depooling labsdb1009 (T189158)
  • 17:24 banyek: repooling labsdb1010 (T189158)
  • 17:07 godog: upload libfastjson 0.99.8-1~bpo9+1wmf1 version bump only
  • 16:59 akosiaris@deploy1001: scap-helm zotero finished
  • 16:59 akosiaris@deploy1001: scap-helm zotero cluster staging completed
  • 16:59 akosiaris@deploy1001: scap-helm zotero [namespace: zotero, clusters: staging]
  • 16:50 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
  • 16:50 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
  • 16:29 XioNoX: enable Zayo transit on cr3-ulsfo
  • 15:42 chasemp: disable /etc/logrotate.d/udp2log-mw for a bit on mwlog1001
  • 15:25 Amir1: rolling restart of celery on ores nodes (T209060)
  • 15:20 akosiaris: 'cd /srv/deployment/ores/deploy/submodules/wheels && sudo -u deploy-service git lfs pull' on all ores1* and ores2* hosts T209060
  • 15:07 XioNoX: zeroize asw-c8-codfw (decom)
  • 14:12 moritzm: rebooting releases2001 for some tests with ssbd for KVM
  • 13:52 moritzm: installing postgres updates on labsdb1006/1007
  • 13:38 jiji: Done reimaging rdb1006 - T206450
  • 13:37 moritzm: draining ganeti2001 for reboot/kernel security update
  • 13:36 moritzm: failing over ganeti master in codfw from ganeti2001 to ganeti2003
  • 13:13 godog: upload rsyslog 8.38.0-1~bpo9+1wmf1 to stretch-wikimedia, version bump only
  • 13:07 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-enable wmgUseTwoColConflict on dewiki - T205942 T208840 T209012 T209036 (duration: 00m 53s)
  • 12:56 akosiaris: increase weight of scb1001 for apertium to 99+%
  • 12:56 akosiaris@puppetmaster1001: conftool action : set/weight=3800; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 12:53 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-enable wmgUseTwoColConflict on group0 only T205942 T208840 T209012 T209036 (duration: 00m 54s)
  • 12:52 moritzm: draining ganeti2002 for reboot/kernel security update
  • 12:41 jiji: Shutdown and reimage rdb200[56] - T206450
  • 12:31 moritzm: draining ganeti2003 for reboot/kernel security update
  • 12:30 zeljkof: EU SWAT finished
  • 12:29 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/TwoColConflict: SWAT: Fix harmless edits turning into conflicts (T205942 T208840 T209012 T209036) (duration: 00m 55s)
  • 12:19 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set AdvancedSearch to default on group0 wikis (T207641) (duration: 00m 55s)
  • 12:18 moritzm: draining ganeti2004 for reboot/kernel security update
  • 11:57 moritzm: draining ganeti2005 for reboot/kernel security update
  • 11:51 akosiaris: increase weight of scb1001 for apertium to 50%
  • 11:50 akosiaris@puppetmaster1001: conftool action : set/weight=38; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 11:41 moritzm: draining ganeti2006 for reboot/kernel security update
  • 11:18 moritzm: draining ganeti2007 for reboot/kernel security update
  • 11:05 moritzm: draining ganeti2008 for reboot/kernel security update
  • 10:52 jiji: Reimaging rdb1006 to stretch - T206450
  • 10:52 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable wmgUseTwoColConflict everywhere T209012 T208840 T195724 (duration: 00m 58s)
  • 10:26 elukey: restart memcached on mc2029 (was depooled yesterday for network maintenance)
  • 10:23 jiji: restarting pdfrender on scb1003
  • 10:19 volans: restarting pdfrender on scb1004
  • 10:18 volans: restarting pdfrender on scb1002
  • 10:18 _joe_: restarting pdfrender on scb1001
  • 10:02 moritzm: installing ppp security updates on trusty
  • 09:37 godog: keep 2x not 3x copies of older (>15d) logstash elasticsearch indices
  • 09:29 moritzm: installing curl security updates
  • 09:29 godog: temporarily set elasticsearch logstash watermark to low:0.85 and high:0.9
  • 06:34 bawolff@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/OpenStackManager/special/SpecialNovaSudoer.php: T203885 (duration: 00m 54s)
  • 05:30 bawolff: deployed patch T208881
  • 01:18 mutante: scb1004 - systemctl restart pdfrender (T174916)
  • 00:43 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/includes/resourceloader/ResourceLoader.php: ResourceLoader: Fail less hard when JSON serialization of config fails I673f59d93 (duration: 00m 53s)
  • 00:33 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T205368 Enable BotPasswords on Governance wiki (duration: 00m 55s)
  • 00:32 ejegg: updated fundraising CiviCRM from 769dcf6456 to 07183ed7cc
  • 00:26 James_F: Created the bot_passwords table for Governance wiki T205368
  • 00:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208449 Disable wgWelcomeSurveyEnabled everywhere in production (duration: 00m 54s)
  • 00:18 jforrester@deploy1001: Synchronized wmf-config/extension-list: T208081 Drop the Petition extension from extension-list (duration: 00m 53s)
  • 00:16 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208081 Drop the Petition extension from InitialiseSettings (duration: 00m 52s)
  • 00:14 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T208081 Drop the Petition extension from CommonSettings (duration: 00m 53s)
  • 00:12 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208899 Enabling wgMediaInTargetLanguage for testwiki (duration: 00m 54s)
  • 00:00 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208081 Disable the Petition extension in production (duration: 00m 52s)

2018-11-07

  • 23:48 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Make GrowthExperiments flag operative in CommonSettings (duration: 00m 53s)
  • 23:44 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add flag for GrowthExperiments to InitialiseSettings (duration: 00m 53s)
  • 23:37 catrope@deploy1001: Finished scap: Full scap to rebuild i18n for the addition of the GrowthExperiments extension (duration: 39m 40s)
  • 23:21 jiji: Disabled nagios checks on rdb1006 and rdb2005 due to rdb1005 reimaging - T206450
  • 22:57 catrope@deploy1001: Started scap: Full scap to rebuild i18n for the addition of the GrowthExperiments extension
  • 22:13 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Revert "labswiki rollback to 1.33.0-wmf.2"
  • 22:07 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/LdapAuthentication/LdapAuthenticationPlugin.php: Expose methods used by OpenStackManager T208995 (duration: 00m 54s)
  • 22:06 XenoRyet: updated payments-wiki from 34506ce636 to 983ce3af0f
  • 22:02 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis T208986 (duration: 00m 54s)
  • 22:02 arlolra: Updated Parsoid to 970751a (T206940)
  • 21:54 arlolra@deploy1001: Finished deploy [parsoid/deploy@4edc771]: Updating Parsoid to 970751a (duration: 09m 34s)
  • 21:45 arlolra@deploy1001: Started deploy [parsoid/deploy@4edc771]: Updating Parsoid to 970751a
  • 21:21 ladsgroup@deploy1001: Finished deploy [ores/deploy@25dfa4f]: T191842 T197096 (duration: 17m 24s)
  • 21:18 krinkle@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/AbuseFilter/includes/AbuseFilter.php: T208144 - I0fdda5 (duration: 00m 53s)
  • 21:16 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/VipsScaler: Id9f82afd (duration: 00m 55s)
  • 21:06 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/AbuseFilter/includes/AbuseFilter.php: T208144 - I0fdda51010243 (duration: 00m 53s)
  • 21:06 banyek: stopping replication on db2072 (T208954)
  • 21:04 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/includes/jobqueue/jobs/RefreshLinksJob.php: T208147 -I7f5fafe9439d8a7b4 (duration: 00m 54s)
  • 21:03 ladsgroup@deploy1001: Started deploy [ores/deploy@25dfa4f]: T191842 T197096
  • 20:55 banyek: depool labsdb1010 (T189158)
  • 20:24 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: rollback labswiki to 1.33.0-wmf.2
  • 20:12 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.3 (duration: 00m 53s)
  • 20:11 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.3
  • 20:00 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T206173 Adding namespaces to Governance wiki (duration: 00m 55s)
  • 19:50 chasemp: labstore1007:~# mkdir /srv/security/
  • 19:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-sync for skipped apaches due to maintenance (whoops) (duration: 00m 55s)
  • 19:48 XioNoX: Revert "Redirect eqsin/ulsfo caches to eqiad" - T208272
  • 19:47 XioNoX: repool codfw - T208272
  • 19:45 XioNoX: asw-c-codfw maintenance finished successfuly - T208272
  • 18:51 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T201285: Disable wgRawHTML on Governance wiki (duration: 05m 12s)
  • 18:31 onimisionipe: restarting relforge-eqiad and relforge-eqiad-small-alpha clusters on relforge100[1-2]
  • 18:21 XioNoX: power down asw-c4-codfw - T208272
  • 17:31 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add sourceswiki to wikidata clients (duration: 00m 53s)
  • 17:25 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Inject wikidata rc records on wikidata itself (duration: 00m 53s)
  • 17:21 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WMEUnderstandingFirstDay on testwiki (duration: 00m 53s)
  • 16:37 XioNoX: remove asw-c-codfw FPC8 from config - T208272
  • 16:35 XioNoX: shutdown asw-c-codfw FPC8 - T208272
  • 16:33 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.ui.less: T208898 Hot-deploy to wmf.3 (duration: 00m 53s)
  • 16:22 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/Echo/modules/styles: Hot-deploy T208930 to wmf.3 (duration: 00m 54s)
  • 16:20 XioNoX: Enable all VC ports (except uplinks) on spines - T208272
  • 15:58 XioNoX: Redirect eqsin/ulsfo caches to eqiad - T208272
  • 15:57 XioNoX: depool codfw for row C maintenance - T208272
  • 15:36 moritzm: installing Java security updates on relforge*
  • 15:29 jiji@deploy1001: Synchronized wmf-config/ProductionServices.php: Remove jobqueue_redis references, T198220 (duration: 00m 54s)
  • 15:21 akosiaris: T206439 direct 30% of the apertium.svc.eqiad.wmnet traffic to scb1001. Will increase tomorrow to 50%
  • 15:20 akosiaris@puppetmaster1001: conftool action : set/weight=15; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 15:16 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 15:15 akosiaris: T206439 pool upgraded scb1001 to apertium.svc.eqiad.wmnet as a form of canary
  • 15:13 moritzm: uploaded nginx 1.13.6-2+wmf2 to apt.wikimedia.org/stretch-wikimedia
  • 14:55 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 14:37 oblivian@deploy1001: Synchronized docroot/wwwportal/w/search-redirect.php: Fixing redirects if no language is specified (duration: 00m 54s)
  • 14:33 moritzm: uploaded nginx 1.13.6-2+wmf2~jessie1 to apt.wikimedia.org/jessie-wikimedia
  • 14:32 akosiaris: T206439 upload apertium-cat_2.6.0-1+wmf1 apertium-fra-cat_1.5.0-1+wmf1 apertium-fra_1.5.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 14:26 bblack: rebooting graphite1004
  • 14:16 akosiaris: T206439 upload apertium-separable_0.3.2-1+wmf1 apertium-lex-tools_0.2.1-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 14:07 akosiaris: T206439 upload apertium_3.5.2-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 13:43 bblack: hi
  • 13:21 Amir1: ladsgroup@mwmaint1002:/srv/mediawiki/php-1.33.0-wmf.1$ mwscript sql.php --wiki=sourceswiki extensions/Wikibase/client/sql/entity_usage.sql (T208858)
  • 12:38 zeljkof: EU SWAT finished
  • 12:36 zfilipin@deploy1001: Synchronized wmf-config: SWAT: BC: Enable Schema.org page split test (T208763) (duration: 00m 54s)
  • 12:35 akosiaris: T206439 upload hfst-ospell_0.5.0-1+wmf1to apt.wikimedia.org/jessie-wikimedia/main
  • 12:27 akosiaris: T206439 upload cg3_1.1.7-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 12:16 zfilipin@deploy1001: Synchronized wmf-config/InterwikiSortOrders.php: SWAT: Add dty, gor, inh, kbp and lfn to InterwikiSortOrders (T208217) (duration: 00m 53s)
  • 12:12 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Art&Feminism event in Chile (T208866) (duration: 00m 54s)
  • 12:12 akosiaris: T206439 upload hfst_3.15.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 12:08 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Remove expired throttle rules (duration: 01m 05s)
  • 11:49 akosiaris: T206439 upload lttoolbox_3.5.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 11:49 akosiaris: T206439 upload lttoolbox_3.5.0-1+wmf1
  • 10:06 hashar: CI: switched operations/puppet job to be based on Stretch ( T208422 ) and to add python3 ( T208873 )
  • 09:25 _joe_: run systemctl reset-failed on ms-be1029, had a failed debmonitor session
  • 08:16 kartik@deploy1001: Finished deploy [cxserver/deploy@6f97d25]: Update cxserver to f9ffd24 (duration: 04m 59s)
  • 08:11 kartik@deploy1001: Started deploy [cxserver/deploy@6f97d25]: Update cxserver to f9ffd24
  • 02:55 ejegg: disabled recurring charge jobs
  • 00:46 mutante: tegmen - shutting down for renaming and reinstall (T208824)
  • 00:11 dereckson@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 44s)

2018-11-06

  • 23:55 mutante: cp1084 - network went down, powercycled, probably T203194
  • 22:49 ejegg: updated fundraising CiviCRM from e0742d2210 to 769dcf6456
  • 21:50 mutante: icinga1001-"MediaWiki EtcdConfig up-to-date" checks were all UNKNOWN because systemd unit update-etcd-mw-config-lastindex was present but service not running. it was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" and the checks all work (T202782)
  • 21:49 mutante: icinga1001 - the "MediaWiki EtcdConfig up-to-date" checks were all unknown on the new icinga server, this was because systemd unit update-etcd-mw-config-lastindex was present but service not running. that was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" to start it and the checks
  • 21:40 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Group0 wikis to 1.33.0-wmf.3
  • 20:19 thcipriani@deploy1001: Finished scap: testwiki to 1.33.0-wmf.3 and rebuild l10n cache (duration: 34m 06s)
  • 19:45 thcipriani@deploy1001: Started scap: testwiki to 1.33.0-wmf.3 and rebuild l10n cache
  • 19:44 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.23 (duration: 07m 19s)
  • 18:36 thcipriani: cutting branch for MediaWiki and extensions version 1.33.0-wmf.3
  • 17:37 XioNoX: add vlan-analytics1-a-eqiad interface-range on asw2-a-eqiad
  • 16:47 XioNoX: enable cr4-ulsfo zayo transport to cr1-codfw
  • 16:23 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
  • 16:23 akosiaris@deploy1001: scap-helm zotero install --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
  • 16:12 mutante: einsteinium - temp disabling icinga notifications and puppet, reloading icinga (for extra caution while deploying global NRPE change)
  • 16:07 mutante: planet1001 - disabling puppet, editing NRPE config, testing allowed_hosts change
  • 16:07 akosiaris@deploy1001: scap-helm -h finished
  • 16:07 akosiaris@deploy1001: scap-helm -h cluster staging completed
  • 16:07 akosiaris@deploy1001: scap-helm -h [namespace: -h, clusters: staging]
  • 15:59 banyek: updating facts for the puppet compilers
  • 15:37 akosiaris: create zotero namespace in eqiad, codfw, staging cluster T201611
  • 15:20 godog: switch all graphite read traffic to graphite1004
  • 15:16 XioNoX: push `lldp port-id-subtype interface-name` to all compatible switches - T208630
  • 15:14 jiji: scb1001/scb1002 switched nutcracker redis from rdb1001:6382 to rdb1009:6379
  • 15:08 XioNoX: push `lldp port-id-subtype interface-name` to all routers - T208630
  • 14:40 jynus_: reducing consistenct temp. on db2048 to avoid lagging
  • 14:30 moritzm: installing Ruby 2.1 security updates
  • 14:22 godog: add graphite1004 to graphite cluster for reads
  • 14:21 moritzm: installing clamav security updates on mendelevium/ticket.wikimedia.org
  • 13:25 moritzm: restart HHVM on canaries to pick up new curl
  • 13:10 XioNoX: zeroize asw-b-eqiad (decom) - T208788
  • 12:33 moritzm: installing curl security updates
  • 12:17 moritzm: installing chromium security updates on proton* (new upstream release tested in deployment-prep)
  • 12:12 zeljkof: EU SWAT finished
  • 12:10 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update several Wikidata-related configs (duration: 00m 55s)
  • 11:37 moritzm: installing Ruby 2.3 security updates
  • 11:37 moritzm: installing Ruby 2.3 security updates on trusty
  • 11:30 moritzm: installing Mono security updates
  • 11:23 moritzm: installing Ruby 1.9 security updates on trusty
  • 11:07 banyek: stopping replication on db2077 (T208672)
  • 07:25 _joe_: also restarting on the other eqiad nodes
  • 07:25 _joe_: restarting tilerator on maps1002
  • 04:47 kartik@deploy1001: Finished deploy [cxserver/deploy@ddb0031]: Update cxserver to 17f9a10 (T144467, T198699, T208386) (duration: 05m 26s)
  • 04:42 kartik@deploy1001: Started deploy [cxserver/deploy@ddb0031]: Update cxserver to 17f9a10 (T144467, T198699, T208386)
  • 03:40 eileen: civicrm revision changed from 99895316de to e0742d2210, config revision is e832b5a04a
  • 00:26 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471875/ (duration: 00m 51s)
  • 00:07 maxsem@deploy1001: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471874/ (duration: 00m 54s)
  • 00:03 eileen: civicrm revision changed from 042eeaeca9 to 99895316de, config revision is e832b5a04a

2018-11-05

  • 22:41 mutante: sodium - reboot after disk replacement (T202705)
  • 21:51 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@a92fce5]: Increase cirrusSearchLinksUpdate concurrency to 100 (duration: 01m 02s)
  • 21:50 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@a92fce5]: Increase cirrusSearchLinksUpdate concurrency to 100
  • 21:49 arlolra: Updated Parsoid to 8ed698b (T205334, T208360)
  • 21:42 mobrovac@deploy1001: Started restart [zotero/translation-server@50f216a]: Free up some memory
  • 21:41 arlolra@deploy1001: Finished deploy [parsoid/deploy@96d739b]: Updating Parsoid to 8ed698b (duration: 10m 59s)
  • 21:38 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: Removed decomissioned citoid url T133001 (duration: 00m 53s)
  • 21:30 arlolra@deploy1001: Started deploy [parsoid/deploy@96d739b]: Updating Parsoid to 8ed698b
  • 21:23 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 take 2 (duration: 09m 18s)
  • 21:14 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 take 2
  • 21:10 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 (duration: 12m 15s)
  • 21:09 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Ensure all wikis on 1.33.0-wmf.2
  • 20:58 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324
  • 20:38 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c] (dev-cluster): Update deps, removed sections table (duration: 03m 40s)
  • 20:35 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c] (dev-cluster): Update deps, removed sections table
  • 19:53 akosiaris: do a depool, scap deploy, scap wikiversions-compile, hhvm restart and then a pool in eqiad mediawiki servers
  • {{safesubst:SAL entry|1=19:50 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:470868|Disable page issues A/B test (duration: 00m 53s)}}
  • 19:44 sbisson@deploy1001: Synchronized php-1.33.0-wmf.2/includes/block/BlockRestriction.php: SWAT: BlockRestriction::update() unnecessarily does a SELECT on the page table. (duration: 01m 00s)
  • 19:19 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable error logging for WikimediaEvents (duration: 00m 52s)
  • 19:12 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable GrowthExperiments logging channel (duration: 00m 53s)
  • 18:57 _joe_: restarting hhvm on mwdebug1002
  • 18:44 godog: pool graphite1004 for reads - T196484
  • 18:43 XioNoX: delete asw2-b - asw-b interface - T183585
  • 18:41 XioNoX: remove asw-b-eqiad from LibreNMS - T183585
  • 18:37 XioNoX: remove vrrp priority 70 on cr2-eqiad:ae2 to failback VIPs to cr2 - T183585
  • 18:26 XioNoX: re-enable ae2 on cr2-eqiad - T183585
  • 18:21 thcipriani: rollback mwdebug1001 group2 wikis
  • 18:13 thcipriani: testing php-1.33.0-wmf.2 on group2 wikis on mwdebug1001
  • 18:05 XioNoX: disable ae2 on cr2-eqiad - T183585
  • 18:02 XioNoX: set vrrp priority 70 on cr2-eqiad:ae2 to failover VIP to cr1 - T183585
  • 16:49 XioNoX: Update LLDP config on cr3-ulsfo - T208630
  • 16:48 vgutierrez: uploaded certcentral 0.5 to apt.wikimedia.org (stretch) - T208572 T208378
  • 16:06 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting MCR to read-new on all wikis (T198308) (duration: 00m 55s)
  • 13:57 jynus_: increase consistency of db2050, dbstore2002 s3 after them catching up replication T208462
  • 12:33 ladsgroup@deploy1001: Finished deploy [ores/deploy@096ffb3]: T208577 T181632 T208608 (duration: 22m 58s)
  • 12:23 zeljkof: EU SWAT finished
  • 12:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Increase wikidata dispatchers to 3 (duration: 00m 54s)
  • 12:16 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgForeignUploadTargets to [] for zhwiki (T208397) (duration: 00m 54s)
  • 12:10 ladsgroup@deploy1001: Started deploy [ores/deploy@096ffb3]: T208577 T181632 T208608
  • 12:05 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Revert "Anniversary logo for cswiki" (T207589) (duration: 00m 58s)
  • 10:02 godog: reformat xfs filesystems on ms-be1040 - T199198
  • 09:17 elukey@deploy1001: Finished deploy [analytics/refinery@9d39efa]: fixing stat1004 (duration: 00m 04s)
  • 09:17 elukey@deploy1001: Started deploy [analytics/refinery@9d39efa]: fixing stat1004
  • 09:08 joal@deploy1001: Finished deploy [analytics/refinery@9d39efa]: regular analytics weekly deploy (duration: 05m 21s)
  • 09:02 joal@deploy1001: Started deploy [analytics/refinery@9d39efa]: regular analytics weekly deploy

2018-11-04

  • 23:42 jynus_: deleting the same row on all s8 broken servers
  • 23:39 jynus_: deleting one row on db1104
  • 20:38 krinkle@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: T176232 - Ia43626584e (duration: 01m 17s)
  • 18:32 jynus_: reduce temp. consistency level of s4, s5, and s6 codfw masters to prevent excessive lagging due to ongoing mediawiki core maintenance
  • 08:42 eileen: process-control config revision is e832b5a04a renable running job list (all jobs on again now0
  • 08:38 eileen: process-control config revision is e16b2c1c61 renable jobs
  • 02:00 eileen: I think I got the rest of the jobs off process-control config revision is 4422254128
  • 01:52 eileen: process-control config revision is 6ec67b3d01 - also turn off omnirecipient repair job
  • 01:40 eileen: process-control config revision is 5b72cfe874 - reapply turn off q jobs

2018-11-03

  • 09:35 elukey: run tcpdump on mc1035 to grab memcache traffic (rotating pcaps, ~30G maximum)

2018-11-02

  • 17:04 thcipriani: rollback group2 wikis to 1.33.0-wmf.1 on mwdebug100{1,2}
  • 16:54 thcipriani: deploying 1.33.0-wmf.2 to group2 wikis on mwdebug1002
  • 16:43 _joe_: live-hacking removal of time limit on mwdebug1001
  • 16:32 thcipriani: deploying 1.33.0-wmf.2 to group2 wikis on mwdebug1001
  • 15:12 jynus: restarting replication @ db2074 after db2094:s3 table fix T208565
  • 15:00 jynus: stopping replication on db2074 to fix db2094:s3 T208565
  • 14:01 vgutierrez: reimaging eeden.wikimedia.org as jessie test system - T208583
  • 11:43 jynus: ignoring cawikimedia.archive replication on db2094:s3 until a reimport happens T208565
  • 11:29 jijiki: Rebooting mw2244 (spare system) for maintenance
  • 10:52 ema: restart varnish-be on cp3032 T208574
  • 08:19 jynus: performing alter table on dbstore2002 s3 and reducing consistency to improve recovery time T208462 T204006
  • 08:01 jynus: reducing consitency on db2050 to improve recovery time T208462
  • 07:59 jynus: performing alter table on db2050 T208462 T204006
  • 07:38 godog: reformat ms-be1043 xfs filesystems - T199198
  • 07:38 jynus: reducing consistency temporarily (flush, binlog sync) at db2040 to prevent lagging
  • 07:26 jynus: reducing consistency temporarily (flush, binlog sync) at db2035 to prevent lagging

2018-11-01

  • 23:01 shdubsh: restart hhvm on mw1261
  • 22:29 ejegg: restarted fundraising queue consumer jobs
  • 22:21 ejegg: updated fundraising CiviCRM from 65130ef3dd to 042eeaeca9
  • 22:18 ejegg: turned off fundraising queue jobs for civi update
  • 22:12 _joe_: rolling restart of hhvm on appservers and api in eqiad
  • 22:09 shdubsh: cumin -b 2 -s 30 "O:mediawiki::appserver and *.eqiad.wmnet" "restart-hhvm"
  • 22:05 _joe_: restarting hhvm on mw1238,1240
  • 22:02 _joe_: restart hhvm on mw1244
  • 21:52 shdubsh: restart hhvm on mw1247
  • 21:49 _joe_: depooling mw1238 for debugging
  • 21:09 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group2 back to 1.33.0-wmf.1
  • 20:55 hoo: Restarted hhvm on mwdebug2002
  • 19:40 hoo: Ran "UPDATE wb_changes_dispatch SET chd_seen = '775203911' WHERE chd_site LIKE '%wikt%' AND chd_seen < '775180000';" on wikidata master (dispatching for wiktionaries)
  • 19:00 hoo@deploy1001: Synchronized php-1.33.0-wmf.1/includes/export/WikiExporter.php: Fix for missing end tag </page> on some exports (T207974) (duration: 01m 01s)
  • 18:38 hoo@deploy1001: Synchronized php-1.33.0-wmf.2/includes/export/WikiExporter.php: Fix for missing end tag </page> on some exports (T207974) (duration: 00m 55s)
  • 18:25 jijiki: Enabling puppet on mw servers (T206923)
  • 18:19 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove now redundant Wikidata config for wiktionary (T208317) (duration: 00m 54s)
  • 18:12 hoo@deploy1001: Synchronized dblists/wikidataclient.dblist: Add all wiktionaries to wikidataclient.dblist, sort list (T208317) (duration: 00m 57s)
  • 18:02 gehel: restart nginx on relforge100*
  • 17:57 jijiki: Disabling puppet on mw servers (T206923)
  • 16:07 anomie@mwmaint1002: Running migrateComments.php on section 4 wikis for T166733
  • 13:46 anomie@mwmaint1002: Running migrateComments.php on remaining section 3 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on section 7 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on wikitech for T166733
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on wikitech for T188132
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on section 6 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on section 8 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 8 wikis for T188132
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 7 wikis for T188132
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 6 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateComments.php on section 5 wikis for T166733
  • 13:36 anomie@mwmaint1002: Running migrateComments.php on section 1 wikis for T166733
  • 13:36 anomie@mwmaint1002: Running migrateComments.php on section 2 wikis for T166733
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 5 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 4 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on remaining section 3 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 2 wikis for T188132
  • 13:35 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 1 wikis for T188132
  • 12:50 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: List wikidataclient-test in CS.php dblists T208488 (duration: 00m 57s)
  • 09:10 elukey: added a tmux session on mw1314m mw1344, mw1316 that checks mcrouter stats every 10s
  • 00:58 onimisionipe: repooling wdqs1004. It has caught up on lag with others
  • 00:22 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/SyntaxHighlight_GeSHi/extension.json: SWAT: Follow-up I3daca6fb: Fix exception thrown when inserting new code block (invalidate RL cache) (duration: 00m 53s)
  • 00:20 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/SyntaxHighlight_GeSHi/modules/ve-syntaxhighlight/ve.ui.MWSyntaxHighlightWindow.js: SWAT: Follow-up I3daca6fb: Fix exception thrown when inserting new code block (duration: 00m 54s)
  • 00:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@e8f3a85] (dev-cluster): Add title normalisation and remove Accept-Language header duplicates (duration: 03m 00s)
  • 00:10 mobrovac@deploy1001: Started deploy [restbase/deploy@e8f3a85] (dev-cluster): Add title normalisation and remove Accept-Language header duplicates
  • 00:07 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Move auth logging to different channels for easier counting (T150300, T123243) (duration: 00m 53s)
  • 00:05 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Move auth logging to different channels for easier counting (T150300, T123243) (duration: 00m 53s)


Archives

See Server admin log/Archives.