Server Admin Log

2019-02-19

10:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1093 after kernel upgrade (duration: 00m 46s)
09:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: More traffic to db1093 after kernel upgrade (duration: 00m 46s)
09:42 mforns@deploy1001: Finished deploy [analytics/refinery@0d7ec19]: deploying refinery to update EL sanitization whitelist (duration: 07m 49s)
09:34 mforns@deploy1001: Started deploy [analytics/refinery@0d7ec19]: deploying refinery to update EL sanitization whitelist
09:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 T210713 (duration: 00m 45s)
09:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 on API after kernel upgrade (duration: 00m 46s)
09:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1093 after kernel upgrade (duration: 00m 46s)
08:56 _joe_: experimenting with php-fpm configuration on mwdebug1001 for T176916
08:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 for kernel upgrade (duration: 00m 45s)
08:55 hashar: Cleaning contint1001 / partition
08:50 marostegui: Deploy schema change on db1089 - T210713
08:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 T210713 (duration: 00m 46s)
08:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 T210713 (duration: 00m 49s)
07:51 marostegui: Drop ep_* tables on s1 - T174802
07:50 moritzm: installing systemd security updates on stretch
07:46 marostegui: Reboot db1106 for kernel upgrade (and remove debug from kernel) T216240 T216273
07:21 marostegui: Drop ep_* tables on s3 - T174802
06:56 marostegui: Deploy schema change on db1106 - this will generate lag on labsdb:s1 T210713
06:56 marostegui: Deploy schema change on db1106 - T210713
06:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 T210713 (duration: 00m 52s)
05:31 XioNoX: delete local pref for peering sessions in ulsfo - T204281
05:17 XioNoX: deleted previously deactivated BGP_community_actions terms - T204281
00:01 XioNoX: disable BGP to Zayo on cr2-codfw for intrusive testing - T215193

2019-02-18

20:19 gtirloni: icinga2001 ran puppet ahead of schedule (enable tools-checker-toolsdb monitor)
18:26 jynus: setting clouddb1001 in read_write mode
18:14 volans: upgraded to spicerack 0.0.16-1 cumin[12]001
18:12 volans: uploaded spicerack_0.0.16-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
18:08 jynus: killing mysql on labsdb1005
18:08 jynus: disabled puppet and edited my.cnf on labsdb1005
17:56 jynus: restarting labsdb1004
17:53 jynus: set clouddb1001 in read_only=1
17:50 jijiki: Reimaging thumbor1004 to stretch - T214597
17:49 jijiki: Reimaging thumbor1004 to stretc - T214597
15:41 jynus: performing es2 & es3 backups into es2002
15:21 jynus: move logical backups to subdirectory T210292
14:29 moritzm: rebooting mw2167 for kernel tests
13:59 marostegui: Drop ep_* tables from s7 - T174802
13:25 jijiki: Depooling thumbor1004 to check if the rest of our hosts can handle the load without it - T214597
12:34 moritzm: installing brltty bugfix update from stretch point release
12:31 moritzm: installing upgrading stat1005 to buster
12:28 XioNoX: update clouddb_return term from cloud-in4 on cr1/2-eqiad - T216353
11:53 moritzm: installing hdparm bugfix update from stretch point release
11:36 moritzm: installing uriparser security updates
11:11 moritzm: installing c3p0 security updates
10:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T210713 (duration: 00m 46s)
10:54 jijiki: Reimaging thumbor2002 to stretch - T214597
10:40 marostegui: Drop tables ep_* from s2 (cswiki nlwiki ptwiki svwiki) T174802
09:50 marostegui: Deploy schema change on db1105:3311 T210713
09:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 T210713 (duration: 00m 46s)
09:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 T210713 (duration: 00m 46s)
09:28 marostegui: Drop ep_* from s6 (ruwiki) - T174802
09:16 marostegui: Deploy schema change on db1099:3311 - T210713
09:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 T210713 (duration: 00m 48s)
09:08 marostegui: Deploy schema change on dbstore1003:3311 and dbstore1001:3311 - T210713
08:27 marostegui: Drop ep_* tables from s5 (srwiki) - T174802
08:23 marostegui: Deploy schema change on s1 codfw master (db2048), lag will be generated on s1 codfw - T210713
07:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1119 after mysql upgrade (duration: 00m 46s)
06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1119 into API service after mysql upgrade (duration: 00m 46s)
06:49 marostegui: Reboot db2085 to disable debug mode on kernel T216273
06:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1119 after mysql upgrade (duration: 00m 46s)
06:29 marostegui: Stop MySQL on db1119 for mysql and kernel upgrade
06:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 for mysql upgrade (duration: 01m 01s)
05:55 marostegui: Deploy schema change on s8 primary master (db1071) - T210713
05:52 marostegui: Set dbstore1002 on read only to start the migration T210478 T215589

2019-02-17

21:20 bstorm_: The slave of labsdb1005.eqiad.wmnet is now clouddb1001.clouddb-services.eqiad.wmflabs
13:14 XioNoX: add term labsdb_return to cloud-in4 - T216353

2019-02-16

16:26 ariel@deploy1001: Finished deploy [dumps/dumps@8f83eea]: fix up multistream index file recombines for large files; better errors for misc dumps failures (duration: 00m 03s)
16:25 ariel@deploy1001: Started deploy [dumps/dumps@8f83eea]: fix up multistream index file recombines for large files; better errors for misc dumps failures
14:21 arturo: T194855 cloudvirt1020 is poweroff, waiting for disk setup before installing
00:20 XioNoX: add port 22 in cloud-in4 term labsdb

2019-02-15

20:40 andrewbogott: enabled virtualization (all three settings) on cloudvirt1019
19:41 arturo: T193264 reimaging cloudvirt1019 to get mitaka/stretch
18:51 arturo: T193264 icinga downtime cloudvirt1019 for 1 week
18:44 bstorm_: stopped replication and then mariadb on labsdb1004
16:52 cdanis: correction, needed to increment version; adding backported rasdaemon 0.6.0-1.2+deb8u2 to jessie-wikimedia
16:48 cdanis: adding backported rasdaemon 0.6.0-1.2+deb8u1 to jessie-wikimedia
16:29 bblack: reprepro: uploaded gdnsd-3.0.0-1~wmf1 to stretch-wikimedia
15:45 moritzm: rebooting auth1001 for kernel security update
14:50 moritzm: installing unbound update from stretch point release
14:45 moritzm: removed labvirt1012 from debmonitor (got renamed to cloudvirt1012) (T216190)
14:06 moritzm: rebooting mwlog1001 for kernel security update
13:54 moritzm: rebooting mwlog2001 for kernel security update
13:46 jbond42: install tar security updates
13:19 moritzm: rolling reboot of mwdebug servers in eqiad to pick up SSBD-enabled qemu
13:12 gtirloni: reboot cloudvirt1020
13:11 arturo: T216239 labvirt1019 has been drained of any workload
13:06 moritzm: installing NSS security updates
12:42 moritzm: installing squid3 security updates
12:30 jynus: stop db2089 mysql instances for reboot testing T216240
12:30 arturo: T216239 schedule 1week of icinga downtime for labvirt1019
10:48 akosiaris: upgrade docker on contint2001 to 18.06.2 T216236
10:42 akosiaris: upgrade docker on contint1001 to 18.06.2 T216236
10:35 gtirloni: reboot cloudvirt1019
09:44 gehel: repool maps100[12]
09:33 moritzm: imported php-defaults debs to thirdparty/php72
08:42 akosiaris: restart gerrit to pick up https://gerrit.wikimedia.org/r/490640 T177868
08:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 46s)
08:28 moritzm: rolling restart of apertium to pick up Python 3.4 security update
07:55 godog: bounce prometheus@ops on prometheus2004 to take a snapshot
06:41 marostegui: Stop puppet on labsdb1005 to leave "max_user_connections" on my.cnf - T216170 T216208
06:39 marostegui: Restart labsdb1005 with max_user_connections = 20 T216208
06:17 marostegui: Deploy schema change on db1109 - T210713
06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 49s)
06:13 marostegui: Reload haproxy on dbproxy11 to repool labsdb1009
00:39 mutante: puppetmaster1001: sudo puppet node clean bast3003.wikimedia.org ; sudo puppet node deactivate bast3003.wikimedia.org (T216199)
00:15 jynus: setting labsdb1005 back into read-write

2019-02-14

23:47 jynus: restarting labsdb1005 mysql in read only mode
23:37 niharika29@deploy1001: Finished deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302 (duration: 00m 02s)
23:37 niharika29@deploy1001: Started deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302
22:07 andrewbogott: rebuilding labvirt1012 as cloudvirt1012, T216190
20:38 bstorm_: Restarted mariadb on labsdb1005 for https://wikitech.wikimedia.org/wiki/Incident_documentation/20190214-labsdb1005
20:09 ejegg: updated fundraising CiviCRM from 02ea871b88 to 165fbf5894
19:42 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.17/extensions/GrowthExperiments/modules/help: SWAT: Help Panel: Fix IME broken in help panel search T216131 (duration: 00m 54s)
19:14 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Stop NavPopups gadget conflict with PagePreviews on Wikivoyage T214878 (duration: 00m 54s)
19:01 mutante: scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev
18:52 mutante: scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev
18:12 mutante: scandium - deleting parsoid clone dir and running puppet
18:03 fsero: upgrading tiller to 2.12.2 on eqiad
17:34 godog: bounce rsyslog on wezen/lithium, tls listener timeout in icinga
16:59 moritzm: restarting apertium-apy on scb1001 to pick up Python security update
16:39 marostegui: Depool labsdb1009 - T210713
16:26 fsero: upgrading tiller on codfw
16:11 fsero: updating tiller version on staging cluster
16:10 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2085 - T214840 (duration: 00m 52s)
15:50 fsero: building and publishing new tiller docker image on boron
15:50 END: (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) (volans@cumin1001)
15:43 START: - Cookbook sre.hosts.upgrade-and-reboot (volans@cumin1001)
15:28 volans: upgraded spicerack to v0.0.15 on cumin[12]001
15:26 volans: uploaded spicerack_0.0.15-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
15:12 marostegui: Clear idrac logs from db2085 - T214840
14:45 godog: depool and stop logstash1009 for stretch reimage - T213898
14:20 marostegui: Stop MySQL on db2085 for on-site maintenance - T214840
14:12 jijiki: Enabling puppet on thumbor* servers - T214597
13:39 arturo: T215892 icinga downtime cloudvirt1024 for 2 weeks
12:22 zeljkof: EU SWAT finished
12:21 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.17/extensions/ExternalGuidance/: SWAT: Fix the eventlogging schema definition as per manifest_version=2 (duration: 00m 55s)
11:43 _joe_: restarting hhvm on mw1338, hot tc exhausted T216084
11:04 _joe_: upgrading python3-etcd on stretch T209136
11:03 jbond42: rolling security updates for curl
11:02 jijiki: Disabling puppet on thumbor* servers - T214597
10:59 moritzm: installing python3.4 security updates
10:53 godog: bounce prometheus instances on prometheus2004 to take a snapshot
08:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 T214840 (duration: 00m 52s)
07:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 T210713 (duration: 00m 54s)
07:36 marostegui: Stop MySQL on db1106 for reboot - T214840
06:10 marostegui: Deploy schema change on db1087 with replication, lag will be generated on labsdb:s8 T210713
06:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 T210713 (duration: 00m 55s)
01:52 mutante: scandium - removing parsoid deploy dir and letting puppet re-clone it after merging gerrit fix 484602 - replace manual clone with proper puppetization (T201366)
01:52 mutante: scandium - removing parsoid deploy dir and letting puppet re-clone it after merging gerrit fix 484602 - replace manual hack with proper puppet
01:15 mutante: phab1001 - phabricator mail config converted to cluster.mailers to adjust to upstream change (T212989)
00:36 bd808@deploy1001: Finished deploy [scholarships/scholarships@1d89fe2]: Live hack PHPMailer namespace T215302 (duration: 00m 02s)
00:36 bd808@deploy1001: Started deploy [scholarships/scholarships@1d89fe2]: Live hack PHPMailer namespace T215302
00:32 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES (damaging only) on itwiki (T211032) (duration: 00m 53s)
00:24 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable help panel search on cswiki and kowiki (T209301) (duration: 00m 55s)

2019-02-13

23:42 niharika29@deploy1001: Finished deploy [scholarships/scholarships@1d89fe2]: Update scholarships app for 2019 cycle T215302 (duration: 00m 02s)
23:42 niharika29@deploy1001: Started deploy [scholarships/scholarships@1d89fe2]: Update scholarships app for 2019 cycle T215302
21:31 jijiki: Restarting nutcracker on scb100*.eqiad.wmnet
20:54 mutante: ruthenium - shell access for parsoid-testers revoked by puppet, please use scandium.eqiad.wmnet (T201366)
20:44 otto@deploy1001: Started restart [eventstreams/deploy@07033d4]: bouncing eventstreams to apply page-links-change stream config
20:43 mutante: ms-be2021 - powercycling
20:09 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.17 (duration: 00m 53s)
20:08 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.17
19:55 mforns@deploy1001: Finished deploy [analytics/refinery@5f1461e]: Deploying analytics refinery with refinery-source v0.0.85 jars (duration: 07m 36s)
19:48 mforns@deploy1001: Started deploy [analytics/refinery@5f1461e]: Deploying analytics refinery with refinery-source v0.0.85 jars
18:13 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1014 (duration: 00m 52s)
18:06 godog: reimage prometheus2003 - T187987
18:01 krinkle@deploy1001: Synchronized php-1.33.0-wmf.17/includes/libs/rdbms/loadbalancer/LoadBalancer.php: Id70fdfa62ef / T215611 (duration: 00m 55s)
17:49 marostegui: Stop MYSQL on db1114 for onsite maintenance - T214720
17:25 jijiki: Pooling mw1299 back - T215569
17:06 cmjohnson1: db1106, troubleshooting idrac issue and updating f/w
16:58 otto@deploy1001: scap-helm eventgate-analytics finished
16:58 otto@deploy1001: scap-helm eventgate-analytics cluster staging completed
16:58 otto@deploy1001: scap-helm eventgate-analytics upgrade staging stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
16:30 elukey: reimage stat1005 to Debian Buster (again)
16:22 otto@deploy1001: scap-helm list finished
16:22 otto@deploy1001: scap-helm list cluster staging completed
16:22 otto@deploy1001: scap-helm list [namespace: list, clusters: staging]
16:13 otto@deploy1001: scap-helm eventgate-analytics finished
16:13 otto@deploy1001: scap-helm eventgate-analytics cluster staging completed
16:13 otto@deploy1001: scap-helm eventgate-analytics upgrade staging stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
15:46 marostegui: Stop MySQL on db1106 for onsite maintenance - this will generate lag on s1 labs - T214840
15:28 jynus: stop and upgrade es1014
15:27 otto@deploy1001: scap-helm eventgate-analytics finished
15:27 otto@deploy1001: scap-helm eventgate-analytics cluster staging completed
15:27 otto@deploy1001: scap-helm eventgate-analytics upgrade staging stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
15:27 otto@deploy1001: scap-helm eventgate-analytics upgrade staging stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad,codfw]
15:17 akosiaris@deploy1001: scap-helm eventgate-analytics finished
15:17 akosiaris@deploy1001: scap-helm eventgate-analytics cluster staging completed
15:17 akosiaris@deploy1001: scap-helm eventgate-analytics install -f /srv/scap-helm/eventgate/eventgate-analytics-staging-values.yaml --set service.port=31193 ../ [namespace: eventgate-analytics, clusters: staging]
15:16 moritzm: updated thirdparty/php72 component to PHP 7.2.15
15:10 akosiaris@deploy1001: scap-helm eventgate-analytics finished
15:10 akosiaris@deploy1001: scap-helm eventgate-analytics cluster staging completed
15:10 akosiaris@deploy1001: scap-helm eventgate-analytics install -f /srv/scap-helm/eventgate/eventgate-analytics-staging-values.yaml --set service.port=31193 ../ [namespace: eventgate-analytics, clusters: staging]
15:09 akosiaris@deploy1001: scap-helm eventgate-analytics install -f /srv/scap-helm/eventgate/eventgate-analytics-staging-values.yaml ../ [namespace: eventgate-analytics, clusters: staging]
15:08 akosiaris@deploy1001: scap-helm eventgate-analytics finished
15:08 akosiaris@deploy1001: scap-helm eventgate-analytics cluster staging completed
15:08 akosiaris@deploy1001: scap-helm eventgate-analytics install --dry-run --debug -f /srv/scap-helm/eventgate/eventgate-analytics-staging-values.yaml ../ [namespace: eventgate-analytics, clusters: staging]
15:05 akosiaris@deploy1001: scap-helm eventgate-analytics finished
15:05 akosiaris@deploy1001: scap-helm eventgate-analytics cluster staging completed
15:05 akosiaris@deploy1001: scap-helm eventgate-analytics install --dry-run --debug -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
15:05 akosiaris@deploy1001: scap-helm eventgate-analytics install -n staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics --dry-run --debug [namespace: eventgate-analytics, clusters: staging]
14:53 otto@deploy1001: scap-helm eventgate-analytics finished
14:53 otto@deploy1001: scap-helm eventgate-analytics cluster staging completed
14:53 otto@deploy1001: scap-helm eventgate-analytics install -n staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
14:25 elukey: reimage stat1005 back to stretch to test GPU drivers
14:06 godog: cancel https://integration.wikimedia.org/ci/job/operations-mw-config-composer-test-docker/12236 to unblock test-prio zuul queue
14:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1120, depool es1014 (duration: 00m 52s)
12:34 arturo: T216030 icinga downtime cloudvirt1018 for 2 weeks
12:32 arturo: T216030 T216004 rebooting cloudvirt1018
11:55 moritzm: installing avahi security updates
11:49 jynus: stop and upgrade db1120
11:43 moritzm: installing golang updates on jessie
11:41 volans: upgraded spicerack on cumin[12]001 to v0.0.14
11:38 volans: uploaded spicerack_0.0.14-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
11:33 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1120 (duration: 00m 53s)
11:11 moritzm: installing postgis security updates
09:46 moritzm: installing golang security updates
09:33 gtirloni: labsdb1005 rebooted server
09:26 gtirloni: labsdb1005 stopped mysql
09:22 marostegui: Stop MySQL on db1106 - T214840
09:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 T214840 (duration: 00m 53s)
08:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 (duration: 00m 53s)
06:46 vgutierrez: uploaded acme-chief 0.10 to apt.wikimedia.org (buster) - T215925
06:18 marostegui: Deploy schema change on db1104 - T210713
06:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 (duration: 01m 07s)
06:12 marostegui: Stop MySQL on db2085 to keep debugging kernel issues - T214840
01:31 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Add ExternalGuidance extension T213076 (part 3) (duration: 00m 53s)
01:30 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add ExternalGuidance extension T213076 (part 2) (duration: 00m 53s)
01:15 thcipriani@deploy1001: Finished scap: SWAT: Add ExternalGuidance extension T213076 (part I: build l10n and sync code) (duration: 27m 51s)
00:47 thcipriani@deploy1001: Started scap: SWAT: Add ExternalGuidance extension T213076 (part I: build l10n and sync code)
00:41 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.17/extensions/Thanks/modules/ext.thanks.mobilediff.css: SWAT: Follow ups to I807f729c1b1a9e9b5952685bb18f540f81d70f47 (duration: 00m 55s)
00:27 XioNoX: merge VRRP Icinga Check

2019-02-12

23:14 jforrester@deploy1001: Finished scap: Another full scap, hoping to find the new i18n in RL for T214482 T215471 T215472 (duration: 06m 01s)
23:09 foks: removed 4 files for legal compliance
23:08 jforrester@deploy1001: Started scap: Another full scap, hoping to find the new i18n in RL for T214482 T215471 T215472
22:47 jforrester@deploy1001: Finished scap: Full scap for new i18n and code for T214482 T215471 T215472 (duration: 18m 03s)
22:29 jforrester@deploy1001: Started scap: Full scap for new i18n and code for T214482 T215471 T215472
21:38 robh: icinga1001 in hardware testing, dont mess with it T214760
21:10 robh: working on troubleshooting icinga1001 via T214760
20:58 jforrester@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/Wikibase/view/resources/resources.php: Hot-deploy I74f6389ae for other code, file 2 (duration: 00m 52s)
20:57 jforrester@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/Wikibase/view/lib/resources.php: Hot-deploy I74f6389ae for other code, file 1 (duration: 00m 51s)
20:52 jforrester@deploy1001: Synchronized php-1.33.0-wmf.16/resources/Resources.php: Hot-deploy If0d7b687e for other code (duration: 00m 54s)
20:06 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Group0 to 1.33.0-wmf.17
19:59 thcipriani@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.17 and rebuild l10n (duration: 18m 54s)
19:40 thcipriani@deploy1001: Started scap: testwiki to php-1.33.0-wmf.17 and rebuild l10n
19:37 thcipriani@deploy1001: Pruned MediaWiki: 1.33.0-wmf.12 (duration: 03m 10s)
19:32 thcipriani@deploy1001: Pruned MediaWiki: 1.33.0-wmf.9 (duration: 10m 05s)
18:48 thcipriani: make-wmf-branch 1.33.0-wmf.17
17:54 chaomodus: notebook1003 - restarted nagios-nrpe-server T212824
17:04 marostegui: Start MySQL again on db2085 for s1 and s8 - T214840
16:18 akosiaris: refresh kubernetes default egress policy T211247
15:58 akosiaris@deploy1001: scap-helm eventgate-analytics finished
15:58 akosiaris@deploy1001: scap-helm eventgate-analytics cluster codfw completed
15:58 akosiaris@deploy1001: scap-helm eventgate-analytics cluster eqiad completed
15:58 akosiaris@deploy1001: scap-helm eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad,codfw]
15:46 akosiaris: create namespaces for eventgate-analytics on eqiad/codfw/staging cluster T211247 T213194
15:45 moritzm: rebooting db2085 for some tests
15:38 marostegui: Stop MySQL on db2085 - T214840
15:38 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2085 - T214840 (duration: 00m 47s)
15:30 otto@deploy1001: scap-helm --help finished
15:30 otto@deploy1001: scap-helm --help cluster codfw completed
15:30 otto@deploy1001: scap-helm --help cluster eqiad completed
15:30 otto@deploy1001: scap-helm --help [namespace: --help, clusters: eqiad,codfw]
15:03 ejegg: updated fundraising CiviCRM from a541a83cb2 to 02ea871b88
14:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1092 (duration: 00m 46s)
14:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: More api traffic to db1092 (duration: 00m 44s)
14:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: More traffic to db1092 (duration: 00m 46s)
13:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Give some api traffic to db1092 (duration: 00m 46s)
13:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1092 (duration: 00m 47s)
13:39 vgutierrez: uploaded acme-chief 0.9 to apt.wikimedia.org (stretch) - T207389 T213737
12:57 moritzm: installing openssl1.0 security updates
12:30 zeljkof: EU SWAT finished
12:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add https://polona.pl/ to $wgCopyUploadsDomains (T215501) (duration: 00m 46s)
12:19 moritzm: install ghostscript security updates on scb*
12:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create extendedconfirmed user group for viwiki (T215493) (duration: 00m 47s)
12:10 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Rollbackers User Group Right on azwiki (T215200) (duration: 00m 47s)
12:03 marostegui: Stop MySQL on db1092 to upgrade mysql and kernel
11:27 moritzm: rebooting stat1005
11:20 moritzm: installing ghostscript security updates on remaining thumbor hosts
10:25 marostegui: Deploy schema change on db1092 T210713
10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 46s)
10:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 46s)
10:00 moritzm: installing ghostscript security updates on thumbor1001
09:36 moritzm: reimaging stat1005 to buster
08:20 marostegui: Deploy schema change on db1101:3318 - T210713
08:20 marostegui: Depool db1101:3318 - T210713
08:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 00m 46s)
08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 (duration: 00m 49s)
07:49 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@125354e]: maintain symlink for old venv path with new virtualenv deploy script (duration: 03m 55s)
07:46 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@125354e]: maintain symlink for old venv path with new virtualenv deploy script
07:40 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@125354e]: testing simplified virtualenv deploy (take 2) (duration: 04m 14s)
07:35 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@125354e]: testing simplified virtualenv deploy (take 2)
07:31 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@125354e]: testing simplified virtualenv deploy (duration: 01m 07s)
07:30 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@125354e]: testing simplified virtualenv deploy
07:26 elukey: update analytics-in4 term mysql-dbstore on cr1/cr2 eqiad
07:09 marostegui: Rename ep_* tables on db1089 (s1) - T174802
06:33 kart_: Finished fourth manual run of unpublished draft purge script (T203059)
06:14 marostegui: Deploy schema change on db1099:3318 T210713
06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 (duration: 00m 52s)
06:04 kart_: Fourth manual run of unpublished draft purge script (T203059)
02:18 thcipriani: restarting gerrit due to high load
00:49 ebernhardson@deploy1001: Finished scap: SWAT: full sync for gerrit:489309 i18n (duration: 18m 20s)
00:30 ebernhardson@deploy1001: Started scap: SWAT: full sync for gerrit:489309 i18n
00:28 ebernhardson@deploy1001: Synchronized wmf-config/WikibaseSearchSettings.php: gerrit:489780 T214515 Promote new wbsearchentities profiles to default in de, fr, es (duration: 00m 46s)
00:13 jforrester@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/CentralNotice/: SWAT Merge branch 'master' into wmf_deploy I8e52d222eb (duration: 00m 49s)
00:05 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Stop setting wgSessionsInObjectCache, it's being removed from MW I2946b5b9a (duration: 00m 47s)

2019-02-11

23:22 cdanis: T214760 icinga2001% sudo killall nsca
22:53 cdanis: icinga.w.o-->icinga2001 DNS change deployed T214760
22:40 cdanis: icinga1001 now passive T214760
22:34 cdanis: failing over icinga to icinga2001
21:33 arlolra: Updated Parsoid to b4b9603 (T208901, T215537, T213468, T215638)
21:24 arlolra@deploy1001: Finished deploy [parsoid/deploy@4e9b142]: Updating Parsoid to b4b9603 (duration: 09m 33s)
21:22 otto@deploy1001: Synchronized wmf-config/CommonSettings.php: Use newer RCFeed config for EventBus based recentchange event - T215834 (duration: 00m 47s)
21:20 ottomata: deploying mediawiki-config change for update to EventBus RCFeed config (no-op)
21:16 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@c6a6285]: Weekly GUI deploy (duration: 11m 54s)
21:14 arlolra@deploy1001: Started deploy [parsoid/deploy@4e9b142]: Updating Parsoid to b4b9603
21:13 mobrovac@deploy1001: Finished deploy [citoid/deploy@0b91bea]: Use Zotero for DOIs and pass it the A-L header - T214766 T210806 T215755 (duration: 03m 47s)
21:09 mobrovac@deploy1001: Started deploy [citoid/deploy@0b91bea]: Use Zotero for DOIs and pass it the A-L header - T214766 T210806 T215755
21:04 smalyshev@deploy1001: Started deploy [wdqs/wdqs@c6a6285]: Weekly GUI deploy
20:08 ppchelko@deploy1001: Finished deploy [changeprop/deploy@bdb4740]: Update dependencies, minor refactor, safer deduplication, T207329 (duration: 01m 37s)
20:07 ppchelko@deploy1001: Started deploy [changeprop/deploy@bdb4740]: Update dependencies, minor refactor, safer deduplication, T207329
19:42 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106, db1118 with full weight (duration: 00m 46s)
19:34 catrope@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: Remove main page special casing from lawiki (T215709) (duration: 00m 46s)
19:28 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgRestrictionLevels on Serbian projects (T215653) (duration: 00m 46s)
19:16 catrope@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/GrowthExperiments/: Help panel search instrumentation (T211166) (duration: 00m 47s)
19:08 catrope@deploy1001: Synchronized wmf-config/throttle.php: Lift account creation cap for edit-a-thon (T215069) (duration: 00m 47s)
19:08 jijiki: Repooled thumbor1004 - T215411
18:50 robh: thumbor1004 rebooted and updated firmware T215411
18:50 robh: thumbor1004 rebooted and updated firmware
16:49 jynus: stop, upgrade and restart db1106
16:36 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1011
16:31 marostegui: Reverse password for globaldev user on dbstore1002 - T200801
16:29 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 (duration: 00m 52s)
15:49 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1118 (duration: 00m 48s)
15:24 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY (duration: 00m 47s)
15:23 marostegui: Relohad haproxy on dbproxy1010 to depool labsdb1011 - https://phabricator.wikimedia.org/T212308
15:21 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase.php, add conditional setting of useEntitySourceBasedFederation (duration: 00m 47s)
15:20 marostegui: Repool labsdb1010 - T212308
15:19 jynus: add missing grants to db1118
15:07 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Revert, second try (duration: 00m 47s)
15:00 addshore@deploy1001: sync-file aborted: Wikibase.php, add conditional setting of useEntitySourceBasedFederation (duration: 00m 01s)
14:55 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Revert (duration: 00m 45s)
14:53 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool db1118 for the first time (duration: 00m 47s)
14:51 mbsantos@deploy1001: Finished deploy [tilerator/deploy@d546183] (stretch): Updating maps2004 tilerator for the stretch migration work (duration: 00m 39s)
14:50 mbsantos@deploy1001: Started deploy [tilerator/deploy@d546183] (stretch): Updating maps2004 tilerator for the stretch migration work
14:48 mbsantos@deploy1001: Finished deploy [kartotherian/deploy@173adbe] (stretch): Updating maps2004 kartotherian for the stretch migration work (duration: 00m 21s)
14:48 mbsantos@deploy1001: Started deploy [kartotherian/deploy@173adbe] (stretch): Updating maps2004 kartotherian for the stretch migration work
14:47 moritzm: installing curl security updates on trusty
14:21 marostegui: Remove staging from dbstore1003 - T210478
14:16 godog: depool and take a snapshot of prometheus data for all instances on prometheus2003 - T187987
14:09 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - T212308
14:08 marostegui: Deploy schema change on db1116:3318 - T210713
12:21 godog: bounce rsyslogd on lithium / wezen, syslog tls listener stuck
12:19 zeljkof: EU SWAT finished
12:18 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Senior Citizens Write Wikipedia course (T215618) (duration: 00m 48s)
12:14 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Clean expired throttle rules (duration: 00m 48s)
10:47 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
10:46 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 48s)
10:41 jynus: upgrading mariadb client on cumin* hosts
10:27 mvolz@deploy1001: scap-helm zotero finished
10:27 mvolz@deploy1001: scap-helm zotero cluster codfw completed
10:27 mvolz@deploy1001: scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
10:24 mvolz@deploy1001: scap-helm zotero finished
10:24 mvolz@deploy1001: scap-helm zotero cluster eqiad completed
10:24 mvolz@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
10:19 marostegui: Add dbstore1005:3350 to tendril and zarcillo - T210478
10:17 mvolz@deploy1001: scap-helm zotero finished
10:17 mvolz@deploy1001: scap-helm zotero cluster staging completed
10:17 mvolz@deploy1001: scap-helm zotero upgrade staging -f zotero-values-staging.yaml --version=0.0.1 stable/zotero [namespace: zotero, clusters: staging]
10:17 jynus: restart db1114
09:38 marostegui: Stop all mysql instances on dbstore1005 for reboot
09:11 marostegui: Stop all mysql instances on dbstore1003 for reboot
08:17 moritzm: removed cloudcontrol2001-dev.codfw.wmnet from debmonitor (actual hostname in use is cloudcontrol2001-dev.wikimedia.org)
08:07 marostegui: Deploy schema change on s8 codfw master (db2045) - this will generate lag on codfw T210713
07:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1100 (duration: 00m 46s)
07:39 marostegui: Deploy schema change on s7 primary master (db1062) - T210713
07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Give api traffic to db1100 (duration: 00m 46s)
07:18 marostegui: Stop all mysql instances on dbstore1004 for a reboot
07:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 with low weight (duration: 00m 46s)
07:06 marostegui: Upgrade MySQL on db1100
07:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for mysql upgrade (duration: 00m 47s)
07:00 marostegui: Restart icinga on icinga1001 - checks went awol
06:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 48s)
06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 48s)
06:14 marostegui@deploy1001: sync-file aborted: Depool db0179 (duration: 00m 01s)
04:23 TimStarling: on mwmaint1002: running normalizeThrottleParameters.php --dry-run on all wikis (T209565)
04:19 tstarling@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: maintenance script update for new dry run (duration: 00m 47s)
04:19 tstarling@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/WikimediaEvents/tests/phpunit/PageViewsTest.php: test-only undeployed change (duration: 00m 46s)
04:18 tstarling@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/NavigationTiming/tests/ext.navigationTiming.test.js: test-only undeployed change (duration: 00m 51s)
04:10 tstarling@deploy1001: sync-file aborted: test-only undeployed change (duration: 00m 12s)
03:05 kartik@deploy1001: Finished deploy [cxserver/deploy@ee4a15a]: Update cxserver to 8928852 (T213256) (duration: 04m 08s)
03:01 kartik@deploy1001: Started deploy [cxserver/deploy@ee4a15a]: Update cxserver to 8928852 (T213256)

2019-02-10

off: force rebooting mw1299, stuck again - T215569
off: forcing reboot of icinga1001 because it's stuck again (no ping, no ssh, CPU stuck messages on console) - T214760
09:25 marostegui: Disable notifications for lag checks on dbstore1002 - T210478

2019-02-09

21:42 Reedy: running `foreachwiki refreshImageMetadata.php --mediatype BITMAP --mime image/vnd.djvu --force` on mwmaint1002 T215635
21:41 Reedy: refreshImageMetadata.php for commonswiki done T215635
16:51 Jeff_Green: restarted icinga process on icinga1001 because of passive check alert-storm

2019-02-08

23:23 Reedy: running `refreshImageMetadata.php --mediatype BITMAP --mime image/vnd.djvu --force` against commonswiki on mwmaint1002 T215635 (this time we mean it)
22:56 Reedy: running `refreshImageMetadata.php --mediatype BITMAP --mime image/vnd.djvu` against commonswiki on mwmaint1002 T215635
21:25 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: Move variable (duration: 00m 49s)
19:50 krinkle@deploy1001: Synchronized w/touch.php: Ia1e610a5f (duration: 00m 46s)
19:49 krinkle@deploy1001: Synchronized w/robots.php: Ia1e610a5f (duration: 00m 46s)
19:48 krinkle@deploy1001: Synchronized w/favicon.php: Ia1e610a5f (duration: 00m 46s)
19:47 krinkle@deploy1001: Synchronized w/extract2.php: Ia1e610a5f (duration: 00m 48s)
18:14 gtirloni: T213527 graphite2002 disabled puppet and commented prometheus_puppet_agent_stats cronjob due to cronspam
18:08 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for s1 rc slaves (duration: 00m 49s)
17:55 mutante: phab1001 - restart aphlict service
17:52 mutante: phab1001 - restarting phd service
17:49 arturo: T215605 add prometheus-openstack-exporter 0.0.8-4 to stretch-wikimedia
17:47 mutante: phab1001 - restarting apache2 service for library upgrade
17:42 mutante: graceful reload of apache on phabricator prod server (phab1001)
17:27 XioNoX: merge Icinga: add ping check for ulsfo PDUs
16:50 ejegg: updated payments-wiki-staging from 52a271e681 to 31647bc97e
16:09 jynus: stopping s1 replication on dbstore1001 to speed up cloning T214720
16:08 moritzm: imported git-fat 0.1.3-2+deb10u1 to buster-wikimedia (T213527)
15:46 marostegui: Repool labsdb1009 - T212308
15:33 _joe_: apt-get upgrade on mwmaint2001 to fix the php installation T215376
15:31 moritzm: imported debmonitor 0.1.5-1+deb10u1 to buster-wikimedia (T213527)
15:31 _joe_: upgraded all php extensions to php 7.2 compatible versions on mwmaint1002
15:10 jijiki: Upgrading php-redis 4.1.1 to mwmaint1002 - T215376
14:51 marostegui: Reload haproxy on dbproxy1011 to depool labsdb1009 - https://phabricator.wikimedia.org/T212308
13:56 moritzm: updated firmware-enriched buster netboot image to 20190208 daily build, the alpha5 image no longer works as Linux 4.19.16-1 bumped the ABI and migrated to testing yesterday
13:45 jynus: racadm serveraction powercycle db1114
13:39 onimisionipe: starting osm-initial-import for maps2004 which is the newly migrated to stretch master - T198622
13:37 elukey: roll restart of aqs on aqs1* to pick up new druid backend changes
13:05 arturo: T209029 reimaging cloudelastic1004
12:54 ejegg: updated fundraising CiviCRM from 3a1bb82373 to a541a83cb2
12:51 jynus: disabling notifications on db1114
12:44 elukey@deploy1001: Synchronized wmf-config/db-eqiad.php: depooling db1114, host down (duration: 00m 47s)
11:36 moritzm: reimage graphite2002 to buster
11:08 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 fully (duration: 00m 47s)
10:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099 (duration: 00m 47s)
10:27 jijiki: Restarting memcached on mc1026 to apply '-R 200' - T208844
10:23 godog: swift codfw-prod: more weight to ms-be2047 - T209395 T209921
10:15 jynus: stop and upgrade db1099
10:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 47s)
09:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 46s)
09:28 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099 (duration: 00m 46s)
09:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1086 (duration: 00m 46s)
09:16 moritzm: installing rssh security updates
09:06 moritzm: installing libarchive security updates
09:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: More traffic to db1086 (duration: 00m 47s)
08:53 moritzm: reimage graphite2002 to buster
08:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 with low load (duration: 00m 46s)
08:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1086 (duration: 00m 47s)
08:24 jynus: stop and upgrade db1083
08:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 00m 47s)
08:15 marostegui: Upgrade MySQL on db1086
08:05 marostegui: Upgrade MySQL on db1086 and deploy schema change
08:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 46s)
07:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Full repool db1094 (duration: 00m 47s)
07:45 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw1299.eqiad.wmnet
07:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: More traffic to db1094 (duration: 02m 55s)
07:27 marostegui@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1299.eqiad.wmnet
07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1094 (duration: 02m 56s)
07:12 marostegui: Upgrade mysql and kernel on db1094
06:58 marostegui: Deploy schema change on db1094 T210713
06:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 46s)
06:54 marostegui: Take a mysqldump from staging on dbstore1003 from dbstore1002 - T210478
06:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 (duration: 00m 49s)
06:29 marostegui: powercycle mw1299 - T215569
06:21 marostegui: Deploy schema change on db1098:3317
06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 (duration: 02m 58s)
06:07 marostegui: Drop staging.mep_word_persistence from dbstore1002 T215450 T213706
02:34 ejegg: updated fundraising CiviCRM from 08be00e87f to 3a1bb82373
01:37 dzahn@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1299.eqiad.wmnet
01:10 mutante: mw1299 has been down about 8 hours, does it need deployment.. depooling
01:08 mutante: powercycle crashed mw1299 via mgmt (garbled console output) (T215569)
00:22 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:488588 phab:T214515 Turn off wikidata wbsearchentities ab test in de, fr, es (duration: 02m 55s)
00:16 ebernhardson: scap sync timed out on mw1299.eqiad.wmnet
00:15 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:483044 T209873 Give protect right to centralnoticeadmin on Meta (duration: 02m 56s)

2019-02-07

23:29 XioNoX: restart ps1-22-ulsfo
23:23 reedy@deploy1001: Synchronized tests/dblistTest.php: Sync test (duration: 02m 55s)
23:18 reedy@deploy1001: Synchronized README: must be up to date (duration: 02m 54s)
22:48 reedy@deploy1001: Synchronized dblists/: alphasort dblists (duration: 02m 56s)
21:43 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group2 wikis to 1.33.0-wmf.16 refs T206670
21:38 robh: updating firmware on ps1-23-ulsfo via T209101 ps1-22-ulsfo update completed
21:22 robh: updating firmware on ps1-22-ulsfo via T209101
20:55 twentyafterfour: train status: deploying 1.33.0-wmf.16 to group2
20:19 sbisson@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/WikibaseLexeme/src/DataAccess/Search/LexemeFulltextResult.php: SWAT: Fix fatal error - EmptySet does not exist anymore (duration: 03m 03s)
19:45 sbisson@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/GrowthExperiments/: SWAT: Help Panel: Fix iOS scroll bug (duration: 03m 02s)
19:28 sbisson@deploy1001: sync-file aborted: SWAT: GrowthExperiments: Enable search for help panel on testwiki (duration: 02m 22s)
19:25 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: GrowthExperiments: Enable search for help panel on testwiki (duration: 03m 04s)
18:32 mutante: LDAP - adding raz-shuty to group nda (T214488)
17:06 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 (duration: 03m 03s)
16:03 jynus: restart db1085, temporary s6 lag on wikireplicas
15:55 gehel: starting reimage of maps2004 - T198622
15:51 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 (duration: 00m 58s)
15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on wikitech for T215464. This may cause lag in codfw.
15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 8 wikis for T215464. This may cause lag in codfw.
15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 7 wikis for T215464. This may cause lag in codfw.
15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 6 wikis for T215464. This may cause lag in codfw.
15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 5 wikis for T215464. This may cause lag in codfw.
15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 4 wikis for T215464. This may cause lag in codfw.
15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on remaining section 3 wikis for T215464. This may cause lag in codfw.
15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 2 wikis for T215464. This may cause lag in codfw.
15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 1 wikis for T215464. This may cause lag in codfw.
15:07 anomie@mwmaint1002: Fixing log_search after migrateActors.php on test wikis and mediawikiwiki for T215464. This may cause lag in codfw.
15:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1101 (duration: 00m 55s)
14:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1101 after alter and mysql upgrade (duration: 00m 55s)
14:34 jbond42: deploying security updates for libgd3
12:42 Amir1: EU SWAT is done
12:42 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: SWAT: Set EntityUsageTable addUsage batch size to 300, Part II (duration: 00m 54s)
12:42 marostegui: Set dbstore1002 as IDEMPOTENT - T213670
12:39 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set EntityUsageTable addUsage batch size to 300 (T215146), Part I (duration: 00m 55s)
12:34 marostegui: Powercycle mw1299 as it is down and not responding
12:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1101 after alter and mysql upgrade (duration: 03m 02s)
12:26 ladsgroup@deploy1001: Synchronized wmf-config/interwiki.php: SWAT: Update interwiki cache to have yuewiktionary instead of zh-yue (T214400) (duration: 03m 04s)
12:06 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4026.ulsfo.wmnet
12:03 arturo: T214448 reimaging again cloudvirt200[1-3]-dev.codfw.wmnet
11:55 marostegui: Stop MySQL on db1101:3317 and db1101:3318 for mysql upgrade
11:37 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2055 (duration: 03m 02s)
11:17 fsero: upgrade helm to 2.12.2 on deploy{1001,2001} and contint{1001,2001} T215244
11:16 fsero: upgrade helm to 2.12.2 on deploy{1001,2001} and contint{1001,2001}
10:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101 for alter and mysql upgrade (duration: 00m 56s)
10:43 marostegui: Run mysqldump from dbstore1003 to dump dbstore1002:staging.mep_word_persistence - T215450
09:49 marostegui: Deploy schema change on db1116 - T210713
09:41 akosiaris: reboot mwdebug1001, mwdebug1002, mwdebug2001, mwdebug2002 for VCPU upgrade. T212955
09:23 jynus: running alter table on db2055 for perforamance testing T212092
09:15 fsero: uploading helm and tiller 2.12.2 deb package to stretch and jessie
08:53 marostegui: Deploy schema change on s7 codfw master (db2047), this will generate lag on s7 codfw - T210713
08:34 godog: swift codfw-prod: more weight to ms-be2047 - T209395 T209921
08:14 marostegui: Deploy schema change on s4 primary master (db1068) - T210713
08:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 (duration: 00m 54s)
07:50 marostegui: Deploy schema change on db1081
07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 (duration: 00m 53s)
07:48 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 20s)
07:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 00m 53s)
07:42 reedy@deploy1001: Synchronized dblists/: Wikimania T215486 (duration: 00m 54s)
07:03 marostegui: Deploy schema change on db1084 - T210713
07:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 55s)
06:48 marostegui: Restore consistency options on db2051
06:14 marostegui: Ease consistency options on db2051 (s4 master) to let it catch up on replication
04:35 tstarling@deploy1001: Synchronized wmf-config/set-time-limit.php: (no justification provided) (duration: 00m 54s)
04:00 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable EP namespaces on wikis with no EP pages (duration: 00m 57s)
01:31 eileen: civicrm revision changed from c5aec3ae76 to 08be00e87f, config revision is 306b4de48f
01:24 eileen: civicrm revision changed from 6161a021c0 to c5aec3ae76, config revision is 306b4de48f
01:05 twentyafterfour: US Evening SWAT is complete
01:04 twentyafterfour: no phabricator deployment tonight
01:04 eileen: civicrm revision changed from 613b388916 to 6161a021c0, config revision is 306b4de48f
00:57 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT config change for Bug: T214003 (duration: 00m 53s)
00:53 twentyafterfour@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/VisualEditor/: SWAT f89e12f to fix bug: T209610 (duration: 00m 55s)
00:48 twentyafterfour@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/MobileFrontend/: SWAT dd8654a (duration: 01m 00s)
00:47 twentyafterfour: syncing commit dd8654a for Bug: T209052
00:24 twentyafterfour: running `mwscript migrateUserGroup.php commonswiki extended-uploader autopatrolled` on deploy1001

2019-02-06

23:58 mutante: restarting icinga on icinga1001 to pick up new check command ?
22:22 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.16 refs T206670 (duration: 00m 53s)
22:22 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.16 refs T206670
21:45 mutante: LDAP - adding brennen to wmf, releng, ciadmin - Welcome Brennen Bearnes, Software Engineer in Release Engineering (T215365 T214556)
21:05 arlolra@deploy1001: Finished deploy [parsoid/deploy@a4acfa6]: Updating Parsoid to fb67a71 (duration: 03m 43s)
21:04 Krinkle: krinkle@webperf1002 Kill xenon-log (pid 449). It seems its Redis TCP socket to mwlog1001 has been stuck since Dec 13, causing the process to indefinitely hang on listen()/socket.recv()
21:01 arlolra@deploy1001: Started deploy [parsoid/deploy@a4acfa6]: Updating Parsoid to fb67a71
20:49 mutante: LDAP - adding h78na to wmf - welcome Hana Worku, developer on the multimedia team (T215352)
20:40 mutante: LDAP - adding egardner to wmf - welcome Eric Gardner , software engineer in Audiences (T214654)
20:35 twentyafterfour: 1.33.0-wmf.16 has a significantly higher rate of "entire web request took longer than 60 seconds and timed out"
20:03 twentyafterfour: Resuming the MediaWiki train for version 1.33.0-wmf.16. Will deploy Group0 wikis first and then catch up to group1 after a few minutes monitoring logs for stability.
19:50 robh: updated firmware on cp4026 and re-seated (already well seated) dimm b3. errors have cleared for now T214516
19:24 milimetric@deploy1001: Finished deploy [analytics/refinery@cd413dd]: Small bug fix for history checker (duration: 12m 45s)
19:13 robh: taking cp4026 offline to flash firmware and reseat dimm for testing on T214516
19:12 milimetric@deploy1001: Started deploy [analytics/refinery@cd413dd]: Small bug fix for history checker
19:11 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@3272a46]: Add healthcheck plugin (no restart) cobalt T214326 (duration: 00m 09s)
19:11 thcipriani@deploy1001: Started deploy [gerrit/gerrit@3272a46]: Add healthcheck plugin (no restart) cobalt T214326
19:09 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@3272a46]: Add healthcheck plugin (no restart) gerrit2001 first (duration: 00m 10s)
19:09 mutante: LDAP - adding afandian2 and toddleroux to nda (T214727)
19:09 thcipriani@deploy1001: Started deploy [gerrit/gerrit@3272a46]: Add healthcheck plugin (no restart) gerrit2001 first
19:04 jforrester@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/Flow/includes/Conversion/Utils.php: I405dd193 Update Parsoid Accept header to 2.0.0 so service can deploy (duration: 00m 54s)
19:03 jforrester@deploy1001: Synchronized php-1.33.0-wmf.14/extensions/Flow/includes/Conversion/Utils.php: I405dd193 Update Parsoid Accept header to 2.0.0 so service can deploy (duration: 00m 56s)
18:03 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Removed namespace Коментар, added namespace Портал on srwikinews T214561 T214563 (duration: 00m 53s)
18:01 mutante: LDAP - adding alaasarhan to wmde (T215066)
17:57 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Changed wgImportSources for srwikinews T214562 (duration: 00m 53s)
17:53 thcipriani@deploy1001: Synchronized dblists/s3.dblist: SWAT: dblists/s3.dblist: Fix sorting of list of wikis per alphabetical order (duration: 00m 54s)
17:49 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/MobileFrontend: SWAT: VE: Load HTML in parallel with modules T209052 (duration: 00m 57s)
17:40 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/MobileFrontend: SWAT: EditorOverlay: Pass constructor of itself to VisualEditorOverlay, not instance T215408 (duration: 00m 57s)
17:10 jynus: setting db1111 in read-write mode
16:24 moritzm: reimaging graphite2002 to buster
16:19 jynus: running alter table on db2055 T93564
16:14 gehel@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad
15:44 papaul: powering down thumbor2002 for disk replacement
15:42 moritzm: installing spice security updates
15:41 andrewbogott: rebooting cloudvirt1015 to make sure that nothing drastic changes once libguestfs is installed T215423
15:11 moritzm: installing libav security updates
15:08 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2055 for performance testing T93564 (duration: 00m 55s)
14:50 moritzm: draining restbase1018 for eventual reboot for kernel security update (bundled with Java update)
14:36 moritzm: draining restbase1017 for eventual reboot for kernel security update (bundled with Java update)
14:29 elukey: add term mysql-dbstore to analytics-in4/6 on cr1/2-eqiad to allow tcp connections to dbstore100[3-5] - T210478
12:30 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3032.esams.wmnet
12:29 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3032.esams.wmnet
12:28 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3033.esams.wmnet
12:26 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3033.esams.wmnet
12:25 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3040.esams.wmnet
12:24 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3040.esams.wmnet
12:22 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3041.esams.wmnet
12:22 Amir1: EU SWAT is done
12:21 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3041.esams.wmnet
12:20 vgutierrez: restarting varnish-fe safely across esams/text cluster - T215389
12:19 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: SWAT: Use separate DB connection for ID insertions on testwikidatawiki (T215147), Part II (duration: 00m 54s)
12:17 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use separate DB connection for ID insertions on testwikidatawiki (T215147), Part I (duration: 00m 55s)
11:58 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3042.esams.wmnet
11:57 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3042.esams.wmnet
11:56 vgutierrez: restarting varnish-fe in cp3042 - T215389
11:02 _joe_: restarting nginx safely across the appserver fleets in order to be able to run puppet without errors
10:41 marostegui: Revoke access to testreduce from ruthenium on m5 - https://phabricator.wikimedia.org/T214740
10:04 moritzm: reimaging graphite2002 to buster
10:01 akosiaris: restart varnish-frontend on cp3030 T215389
10:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 (duration: 00m 52s)
09:33 marostegui: Remove wikiuser from dbstore1003-dbstore1005 T210478
09:15 godog: swift codfw-prod: more weight for ms-be2047 - T209395 T209921
09:00 marostegui: Create research_role on dbstore1003-1005 on all instances - T214469
08:49 marostegui: Deploy schema change on db1091 - T210713
08:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 (duration: 00m 53s)
08:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 (duration: 00m 53s)
07:51 marostegui: Deploy schema change on db1121 - this will generate lag on s4 labs - also upgrade MySQL on db1121 T210713
07:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 (duration: 00m 54s)
07:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 (duration: 00m 54s)
07:19 marostegui: Deploy schema change on wikitech T210713
07:14 marostegui: Stop 's4' slave on dbstore1002
07:13 marostegui: Deploy schema change on db1103:3314 (db1097:3314 was also done previously) - T210713
07:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 (duration: 00m 53s)
07:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 (duration: 00m 56s)
06:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 (duration: 01m 06s)
04:39 mutante: reloaded icinga service, cant find new check command definition
03:14 twentyafterfour@deploy1001: Finished scap: testwikis wikis to 1.33.0-wmf.16 refs T206670 (duration: 04m 18s)
03:09 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.33.0-wmf.16 refs T206670
03:05 mutante: actinium - gzipping and rotating some access logs
03:01 twentyafterfour@deploy1001: Synchronized scap/plugins/updateinterwikicache.py: (no justification provided) (duration: 00m 55s)
02:47 mutante: actinium - blocking a bad domain and restarting squid3
02:40 twentyafterfour@deploy1001: Finished scap: sync and update localization for 1.33.0-wmf.16 (duration: 15m 50s)
02:32 XioNoX: push firewall rule to pfw3-eqiad - T215364
02:27 mutante: actinium - apt-get clean for 8% more disk space after icinga alert
02:25 twentyafterfour@deploy1001: Started scap: sync and update localization for 1.33.0-wmf.16
02:16 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.14 refs T206670
02:12 eileen: civicrm revision changed from 6042acb363 to 613b388916, config revision is 306b4de48f
02:02 twentyafterfour@deploy1001: scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
01:22 XioNoX: remove peering4/6 prefix-list from routers
01:07 XioNoX: add maintenance and rollback to junos operations class
00:47 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.33.0-wmf.16 refs T206670
00:33 niharika29@deploy1001: Synchronized php-1.33.0-wmf.14/extensions/MobileFrontend/: EditorOverlay: captcha/abusefilter weren't being shown correctly T215101, T202374 (duration: 00m 50s)
00:24 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Demystify Logstash debug level behavior (duration: 00m 51s)
00:23 niharika29@deploy1001: Synchronized wmf-config/logging.php: Demystify Logstash debug level behavior (duration: 00m 46s)
00:18 niharika29@deploy1001: Synchronized wmf-config/logging.php: Add PHP version to MW logs T215350 (duration: 00m 46s)
00:16 niharika29@deploy1001: Synchronized wmf-config/CommonSettings.php: Preserve Composer's include paths - T215126, T215224 (duration: 01m 40s)

2019-02-05

18:56 arlolra@deploy1001: Finished deploy [parsoid/deploy@a4acfa6]: (no justification provided) (duration: 02m 06s)
18:53 arlolra@deploy1001: Started deploy [parsoid/deploy@a4acfa6]: (no justification provided)
18:39 arlolra@deploy1001: Finished deploy [parsoid/deploy@a4acfa6]: Updating Parsoid to fb67a71 (duration: 09m 54s)
18:29 arlolra@deploy1001: Started deploy [parsoid/deploy@a4acfa6]: Updating Parsoid to fb67a71
18:26 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@2959e12]: Update mobileapps to 107c1b1 (T214714) (duration: 04m 43s)
18:21 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@2959e12]: Update mobileapps to 107c1b1 (T214714)
18:17 mutante: contint1001/contint2001 -manually deleting crontab lines unpuppetized in gerrit:488019 (T209361)
18:13 Jeff_Green: authdns-update to deploy 7fee817fd3
17:22 mutante: scandium - restart parsoid-vd service
17:21 mutante: scandium -- copy /srv/visualdiff/testrecude/testrun.ids from ruthenium to the same locatio
15:15 godog: force curator action 'replicas' to set older logstash indices to 1 replica - T213078
14:30 marostegui: Deploy schema change on s4 codfw master with replication, lag will be generated on s4 codfw - T210713
14:26 Jeff_Green: authdns-update for payments dev/testing hostname
14:12 marostegui: Deploy schema change on db1066 (s2 master) - T210713
14:05 marostegui: Delete non used grants from dbstore1002: log, warehouse,project_illustration, cognate\_wiktionary, datasets - T212487 T210478
13:55 godog: swift codfw-prod: add ms-be2047 - T209395 T209921
12:18 addshore: swat done
12:18 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable confirmation prompt on rollback by default T215019 (duration: 00m 47s)
11:35 moritzm: added firmware-enriched buster netboot image (T213546)
11:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 (duration: 00m 46s)
10:43 marostegui: Deploy schema change on db1074 with replication, lag will be generated on s2 - T210713
10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 (duration: 00m 47s)
10:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 (duration: 00m 46s)
09:42 hashar: contint1001: docker image prune -f
09:34 marostegui: Deploy schema change on db1090:3312 - T210713
09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 45s)
09:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1076 (duration: 00m 46s)
09:11 marostegui: Start all slaves on dbstore1002 - T213670
08:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1076 (duration: 00m 45s)
08:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1076 (duration: 00m 46s)
08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1076 (duration: 00m 45s)
07:56 marostegui: Upgrade MySQL and kernel on db1076
07:44 marostegui: Deploy schema change on db1076 - T210713
07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 T210713 (duration: 00m 47s)
07:13 marostegui: Taking mysqldump from dbstore1002.staging - T210478
07:05 marostegui: Reboot mysql on db1117:3323 (this will make the dbproxies complain) T214248
02:24 XioNoX: remove BGP session to as6412 on cr2-eqiad (gone from IX)
02:21 XioNoX: delete 2nd as9121 router on cr2-esams
00:47 XioNoX: add BGP sessions to AS64050 on cr1-eqsin
00:24 maxsem@deploy1001: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/486405/ (duration: 00m 46s)
00:11 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 46s)

2019-02-04

22:05 mutante: scandium - systemctl start parsoid-vd (T201366)
20:01 herron: manually ran puppet on mc1023
19:50 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Clean-up: Stop setting wgParsoidWikiPrefix, unused since the Parsoid extension (duration: 00m 45s)
19:45 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Clean-up: Stop setting wgFlowEventLogging, unread (duration: 00m 45s)
19:39 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Clean-up: Stop setting values for wgEcho*FooterNotice*, unread (duration: 00m 46s)
19:32 James_F: Manually purged atjwiki*.png logos for T215122.
19:28 jforrester@deploy1001: Synchronized static/images/project-logos/atjwiki.png: SWAT: Milestone lobo for atjwiki T215122, 1x (duration: 00m 46s)
19:27 jforrester@deploy1001: Synchronized static/images/project-logos/atjwiki-1.5x.png: SWAT: Milestone lobo for atjwiki T215122, 1.5x (duration: 00m 45s)
19:26 jforrester@deploy1001: Synchronized static/images/project-logos/atjwiki-2x.png: SWAT: Milestone lobo for atjwiki T215122, 2x (duration: 00m 44s)
19:22 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T191039 Enable wgAbuseFilterRuntimeProfile on all wikis (duration: 00m 47s)
19:19 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@8b2f078]: Weekly GUI deploy (duration: 09m 47s)
19:09 smalyshev@deploy1001: Started deploy [wdqs/wdqs@8b2f078]: Weekly GUI deploy
18:31 XioNoX: adding Papaul to root@wiki
18:22 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Clean-up: Drop reading for wgEcho*FooterNotice*, unread (duration: 00m 46s)
18:18 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Clean-up: Stop setting wgEchoConfig, unused since 2016 (duration: 00m 48s)
18:11 jforrester@deploy1001: Synchronized dblists/: T213504: Finally, drop the wikidatarepo dblist (duration: 00m 45s)
18:09 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T213504: Stop telling CommonsSettings about the wikidatarepo dblist (duration: 00m 45s)
18:06 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T213504: Unconfigure the wikidatarepo dblist (duration: 00m 46s)
18:05 XioNoX: manually rotate log file wtmp on csw2-esams
18:02 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T213504: Configure wikibaserepo dblist just like the wikidatarepo one (duration: 00m 46s)
17:58 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T213504: Tell CommonSettings about the new wikibaserepo dblist (duration: 00m 47s)
17:56 jforrester@deploy1001: Synchronized dblists/wikibaserepo.dblist: T213504: Create the new wikibaserepo dblist (duration: 00m 47s)
17:25 papaul: powering down thumbor2002 for disk replacement
17:10 XioNoX: revert ospf metrics to normal values on esams-eqiad Level3 link
16:50 Lucas_WMDE: deployed patch for T212118
12:41 Lucas_WMDE: EU SWAT done
12:40 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix Wikidata base URI in client config (T198946) (duration: 00m 46s)
12:34 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Populate wmgWikibaseRepoSpecialSiteLinkGroups for commonswiki (T213975) (duration: 00m 51s)
11:04 moritzm: installing ghostscript security updates
08:48 jynus: fixing dbstore1002 x1 replication
07:56 vgutierrez: uploaded certcentral 0.8 to apt.wikimedia.org (stretch) - T209980 T213820 T213301

2019-02-03

20:25 elukey: powercycle mw1272 - no ssh, no tty available via com2 - DIMM correctable errors + OEM errors registered in getsel
18:56 elukey: started a tmux session on dbstore1002 to migrate all the tokudb tables of mediawikiwiki to InnoDB - (s3 replication broken)
17:53 elukey: start all slaves on dbstore1002 (After a crash + recovery) + moved mediawikiwiki.revision_actor_temp to Innodb to unblock s3 slave replication (still broken though)
04:55 legoktm@deploy1001: Synchronized wmf-config/extension-list: Remove WikibaseQuality from extensions-list (T208499) (duration: 00m 51s)
01:10 elukey: powercycle mw1299 - can't ssh nor get a tty via console - racadm getsel shows "An OEM diagnostic event occurred."

2019-02-02

20:42 chaomodus: restarted pdfrender on scb1003
20:41 chaomodus: restarted pdfrender on scb1004
20:06 chaomodus: parsoid was failed on scandium and alerting, the service parsoid-vd was restarted and appears to have come back
05:44 jforrester@deploy1001: Synchronized php-1.33.0-wmf.14/extensions/VisualEditor/lib/ve/src/ui/dialogs/ve.ui.FindAndReplaceDialog.js: b/src/ui/dialogs/ve.ui.FindAndReplaceDialog.js T214963 Hot-deploy VE fix to stop hitting user pref writes without debounce (duration: 01m 02s)

2019-02-01

23:16 vgutierrez: restart pdfrender on scb1004
21:57 ejegg: updated payments-wiki-staging from 7767c7027e to 52a271e681
21:25 ejegg: updated payments-wiki-staging to fundraising/REL1_31 branch
07:13 bawolff_: reset 2FA on wikitech for User:Cicalese

Server Admin Log

2019-02-19

2019-02-18

2019-02-17

2019-02-16

2019-02-15

2019-02-14

2019-02-13

2019-02-12

2019-02-11

2019-02-10

2019-02-09

2019-02-08

2019-02-07

2019-02-06

2019-02-05

2019-02-04

2019-02-03

2019-02-02

2019-02-01

Archives

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

Cloud VPS & Toolforge

Tools

Print/export