Server Admin Log

From Wikitech
(Redirected from Server admin log)
Jump to navigation Jump to search

2019-02-20

  • 15:55 bblack: authdns2001: upgrade gdnsd to 3.0.0-1~wmf1
  • 15:37 fsero: restarting docker-registry service on systemd
  • 15:35 moritzm: temporarily stop prometheus instances on prometheus1004 for systemd upgrade/journald restart
  • 14:43 gehel@cumin2001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)
  • 14:35 gehel@cumin2001: START - Cookbook sre.elasticsearch.rolling-upgrade
  • 14:35 volans: upgraded spicerack to 0.0.18 on cumin[12]001
  • 14:34 volans: uploaded spicerack_0.0.18-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
  • 14:00 gehel@cumin2001: END (ERROR) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=97)
  • 14:00 gehel@cumin2001: START - Cookbook sre.elasticsearch.rolling-upgrade
  • 13:59 gehel: rolling upgrade of elasticsearch / cirrus / codfw to 5.6.14 - T215931
  • 13:51 godog: prometheus on prometheus2004 crashed/exited after journald upgrade -- starting up again now
  • 13:00 jbond42: rolling restarts for hhvm in eqiad
  • 12:28 volans: upgraded spicerack to 0.0.17 on cumin[12]001
  • 12:25 volans: uploaded spicerack_0.0.17-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
  • 12:08 moritzm: restarted ircecho on kraz.wikimedia.org
  • 11:46 jbond42: rolling restarts for hhvm in codfw
  • 11:28 akosiaris: rebuild and re-upload rsyslog_8.38.0-1~bpo9+1wmf1_amd64.changes to apt.wikimedia.org/stretch-wikimedia to have mmkubernetes package
  • 10:36 marostegui: Deploy schema change on db1095:3313 - T210713
  • 10:04 marostegui: Deploy schema change on dbstore1004:3313 - T210713
  • 09:57 moritzm: installing systemd security updates on jessie hosts
  • 09:33 marostegui: Deploy schema change on db2043 (s3 codfw master), lag will be generated on s3 codfw - T210713
  • 09:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1109 (duration: 00m 52s)
  • 08:48 moritzm: powercycling rdb1001 for a test
  • 07:45 moritzm: installing gnupg2 updates on stretch
  • 07:14 marostegui: Deploy schema change on s1 primary master (db1067) - T210713
  • 07:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 T210713 (duration: 00m 52s)
  • 07:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1109 after kernel upgrade (duration: 00m 52s)
  • 06:54 oblivian@deploy1001: Synchronized wmf-config/profiler.php: Fix the tideways setup (duration: 00m 52s)
  • 06:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1109 after kernel upgrade (duration: 00m 52s)
  • 06:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 T210713 (duration: 00m 51s)
  • 06:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 T210713 (duration: 00m 51s)
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1109 after kernel upgrade (duration: 00m 52s)
  • 06:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1109 after kernel upgrade (duration: 00m 52s)
  • 06:18 marostegui: Stop MySQL on db1109 for kernel and mysql upgrade
  • 06:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 for kernel and mysql upgrade (duration: 00m 52s)
  • 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 T210713 (duration: 01m 05s)
  • 04:45 XioNoX: add avoid-paths WIRESTAR-OPTICALTEL to cr2-eqdfw
  • 02:15 mobrovac@deploy1001: Finished deploy [restbase/deploy@751dc5c]: Temporarily collect VE lrequest ogs for T215956 (duration: 22m 37s)
  • 01:52 mobrovac@deploy1001: Started deploy [restbase/deploy@751dc5c]: Temporarily collect VE lrequest ogs for T215956
  • 00:24 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.17/skins/MinervaNeue/resources/skins.minerva.content.styles/lists.less: Revert switch to outside list style from ordered lists (duration: 00m 52s)
  • 00:23 ebernhardson@deploy1001: Synchronized php-1.33.0-wmf.18/skins/MinervaNeue/resources/skins.minerva.content.styles/lists.less: Revert switch to outside list style from ordered lists (duration: 00m 59s)
  • 00:05 ebernhardson@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: SWAT T215969 Return cirrussearch master timeout back to the default value (duration: 00m 57s)

2019-02-19

  • 23:51 ebernhardson: restarted ferm on relforge1001
  • 23:50 ebernhardson: temporarly stop ferm on relforge1001 to test where a connection is being blocked
  • 20:49 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.18
  • 20:34 thcipriani@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.18 and rebuild l10n cache (duration: 30m 31s)
  • 20:07 gehel@cumin1001: END (PASS) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=0)
  • 20:04 thcipriani@deploy1001: Started scap: testwiki to php-1.33.0-wmf.18 and rebuild l10n cache
  • 20:01 gehel@cumin1001: START - Cookbook sre.elasticsearch.rolling-upgrade
  • 19:57 thcipriani: restarting ci-jenkins for plugin update
  • 19:49 thcipriani@deploy1001: Pruned MediaWiki: 1.33.0-wmf.13 (duration: 11m 52s)
  • 19:39 gtirloni: re-pooled labsdb1011 T216481
  • 19:09 andrewbogott: rebooting cloudvirt1009 to poke around in the bios
  • 18:20 thcipriani: starting branch-cut for 1.33.0-wmf.18
  • 17:55 herron: temporarily increased eqiad logstash elasticsearch low disk watermark to 87% (will restore to 85% when eqiad expansion hosts are fully online)
  • 17:52 jijiki: Restarting memcache on mc1027 - T208844
  • 17:00 hashar: Offlined compiler1002.puppet-diffs.eqiad.wmflabs from Jenkins. Its disk is corrupt | T216513
  • 16:39 gtirloni: depooled labsdb1011 T216481
  • 16:33 moritzm: installing libssh update from stretch point release
  • 16:28 jforrester@deploy1001: Synchronized php-1.33.0-wmf.17/includes/specials/pagers/ActiveUsersPager.php: T216200 Hot deploy variable name fix for ActiveUsersPager query (duration: 00m 48s)
  • 16:26 herron: enabling elasticsearch on new eqiad hosts logstash101[0-2]
  • 16:18 gtirloni: re-pooled labsdb1010 T216481
  • 16:07 gehel@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)
  • 16:04 gehel@cumin1001: START - Cookbook sre.elasticsearch.rolling-upgrade
  • 15:47 jijiki: Reimaging thumbor2002 to stretch - T214597
  • 15:38 gehel@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)
  • 15:32 hashar: apt-get upgrade on compiler1001 and compiler1002.puppet-diffs.eqiad.wmflabs
  • 15:27 gehel@cumin1001: START - Cookbook sre.elasticsearch.rolling-upgrade
  • 15:25 hashar: Started instance compiler1002.puppet-diffs.eqiad.wmflabs via Horizon. It was in shutoff state | T216513
  • 15:10 _joe_: uploading tideways-xhprof_5.0.0~beta3 to reprepro T176916
  • 15:09 gtirloni: depooled labsdb1010 T216481
  • 14:53 jynus: stopping db2089 for hw maintenance T216240
  • 14:41 gehel@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)
  • 14:40 gehel@cumin1001: START - Cookbook sre.elasticsearch.rolling-upgrade
  • 14:36 gehel@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)
  • 14:35 gehel@cumin1001: START - Cookbook sre.elasticsearch.rolling-upgrade
  • 14:31 gehel@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)
  • 14:30 gehel@cumin1001: START - Cookbook sre.elasticsearch.rolling-upgrade
  • 14:30 gehel: rolling upgrade of elasticsearch on relforge - T215931
  • 14:16 jynus: stop db2090 for reboot testing T216240
  • 14:04 gtirloni: running `maintain-views --all-databases --replace-all --clean --debug` on labsdb1010 (T216481)
  • 13:44 Amir1: mwscript maintenance/createAndPromote.php --wiki=testwikidatawiki --force --interface-admin Ladsgroup
  • 13:43 Amir1: ladsgroup@mwmaint1002:~$ mwscript maintenance/createAndPromote.php --wiki=testwikidatawiki --force --sysop Ladsgroup (T215919)
  • 13:31 moritzm: installing rssh update for jessie
  • 13:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1118 T210713 (duration: 00m 46s)
  • 13:23 gtirloni: running `maintain-views --all-databases --replace-all --clean --debug` on labsdb1009 (T216481)
  • 12:57 zeljkof: EU SWAT finished
  • 12:56 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add new throttle rule for Kickstarter Edit-a-thon (T215839) (duration: 00m 43s)
  • 12:50 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgArticleCountMethod = any on fiwikinews (T216333) (duration: 00m 45s)
  • 12:37 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add namespace Додатак on srwiktionary (T216343) (duration: 00m 46s)
  • 12:29 _joe_: creating gerrit repo operations/debs/tideways-xhprof T176916
  • 12:28 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add new throttle rule for WikiProject Women in red, enwiki (T215295) (duration: 00m 47s)
  • 12:19 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Modifying configuration about Chinese Wikiversity (T212919) (duration: 00m 48s)
  • 11:59 marostegui: Deploy schema change on db1118 - T210713
  • 11:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1118 T210713 (duration: 00m 46s)
  • 11:53 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1064 (duration: 00m 46s)
  • 11:49 moritzm: installing ruby-rack security updates
  • 11:26 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T187299 Launch performance perception survey on eswiki (duration: 00m 46s)
  • 11:22 jynus: stop and restart db1064
  • 11:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 T210713 (duration: 00m 46s)
  • 11:10 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1064 (duration: 00m 46s)
  • 11:05 marostegui: Deploy schema change on dbstore1002
  • 10:25 marostegui: Deploy schema change on db1083 - T210713
  • 10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 T210713 (duration: 00m 46s)
  • 10:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1093 after kernel upgrade (duration: 00m 46s)
  • 09:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: More traffic to db1093 after kernel upgrade (duration: 00m 46s)
  • 09:42 mforns@deploy1001: Finished deploy [analytics/refinery@0d7ec19]: deploying refinery to update EL sanitization whitelist (duration: 07m 49s)
  • 09:34 mforns@deploy1001: Started deploy [analytics/refinery@0d7ec19]: deploying refinery to update EL sanitization whitelist
  • 09:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 T210713 (duration: 00m 45s)
  • 09:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 on API after kernel upgrade (duration: 00m 46s)
  • 09:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1093 after kernel upgrade (duration: 00m 46s)
  • 08:56 _joe_: experimenting with php-fpm configuration on mwdebug1001 for T176916
  • 08:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 for kernel upgrade (duration: 00m 45s)
  • 08:55 hashar: Cleaning contint1001 / partition
  • 08:50 marostegui: Deploy schema change on db1089 - T210713
  • 08:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 T210713 (duration: 00m 46s)
  • 08:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 T210713 (duration: 00m 49s)
  • 07:51 marostegui: Drop ep_* tables on s1 - T174802
  • 07:50 moritzm: installing systemd security updates on stretch
  • 07:46 marostegui: Reboot db1106 for kernel upgrade (and remove debug from kernel) T216240 T216273
  • 07:21 marostegui: Drop ep_* tables on s3 - T174802
  • 06:56 marostegui: Deploy schema change on db1106 - this will generate lag on labsdb:s1 T210713
  • 06:56 marostegui: Deploy schema change on db1106 - T210713
  • 06:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 T210713 (duration: 00m 52s)
  • 05:31 XioNoX: delete local pref for peering sessions in ulsfo - T204281
  • 05:17 XioNoX: deleted previously deactivated BGP_community_actions terms - T204281
  • 00:01 XioNoX: disable BGP to Zayo on cr2-codfw for intrusive testing - T215193

2019-02-18

  • 20:19 gtirloni: icinga2001 ran puppet ahead of schedule (enable tools-checker-toolsdb monitor)
  • 18:26 jynus: setting clouddb1001 in read_write mode
  • 18:14 volans: upgraded to spicerack 0.0.16-1 cumin[12]001
  • 18:12 volans: uploaded spicerack_0.0.16-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
  • 18:08 jynus: killing mysql on labsdb1005
  • 18:08 jynus: disabled puppet and edited my.cnf on labsdb1005
  • 17:56 jynus: restarting labsdb1004
  • 17:53 jynus: set clouddb1001 in read_only=1
  • 17:50 jijiki: Reimaging thumbor1004 to stretch - T214597
  • 17:49 jijiki: Reimaging thumbor1004 to stretc - T214597
  • 15:41 jynus: performing es2 & es3 backups into es2002
  • 15:21 jynus: move logical backups to subdirectory T210292
  • 14:29 moritzm: rebooting mw2167 for kernel tests
  • 13:59 marostegui: Drop ep_* tables from s7 - T174802
  • 13:25 jijiki: Depooling thumbor1004 to check if the rest of our hosts can handle the load without it - T214597
  • 12:34 moritzm: installing brltty bugfix update from stretch point release
  • 12:31 moritzm: installing upgrading stat1005 to buster
  • 12:28 XioNoX: update clouddb_return term from cloud-in4 on cr1/2-eqiad - T216353
  • 11:53 moritzm: installing hdparm bugfix update from stretch point release
  • 11:36 moritzm: installing uriparser security updates
  • 11:11 moritzm: installing c3p0 security updates
  • 10:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T210713 (duration: 00m 46s)
  • 10:54 jijiki: Reimaging thumbor2002 to stretch - T214597
  • 10:40 marostegui: Drop tables ep_* from s2 (cswiki nlwiki ptwiki svwiki) T174802
  • 09:50 marostegui: Deploy schema change on db1105:3311 T210713
  • 09:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 T210713 (duration: 00m 46s)
  • 09:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 T210713 (duration: 00m 46s)
  • 09:28 marostegui: Drop ep_* from s6 (ruwiki) - T174802
  • 09:16 marostegui: Deploy schema change on db1099:3311 - T210713
  • 09:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 T210713 (duration: 00m 48s)
  • 09:08 marostegui: Deploy schema change on dbstore1003:3311 and dbstore1001:3311 - T210713
  • 08:27 marostegui: Drop ep_* tables from s5 (srwiki) - T174802
  • 08:23 marostegui: Deploy schema change on s1 codfw master (db2048), lag will be generated on s1 codfw - T210713
  • 07:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1119 after mysql upgrade (duration: 00m 46s)
  • 06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1119 into API service after mysql upgrade (duration: 00m 46s)
  • 06:49 marostegui: Reboot db2085 to disable debug mode on kernel T216273
  • 06:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1119 after mysql upgrade (duration: 00m 46s)
  • 06:29 marostegui: Stop MySQL on db1119 for mysql and kernel upgrade
  • 06:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 for mysql upgrade (duration: 01m 01s)
  • 05:55 marostegui: Deploy schema change on s8 primary master (db1071) - T210713
  • 05:52 marostegui: Set dbstore1002 on read only to start the migration T210478 T215589

2019-02-17

  • 21:20 bstorm_: The slave of labsdb1005.eqiad.wmnet is now clouddb1001.clouddb-services.eqiad.wmflabs
  • 13:14 XioNoX: add term labsdb_return to cloud-in4 - T216353

2019-02-16

  • 16:26 ariel@deploy1001: Finished deploy [dumps/dumps@8f83eea]: fix up multistream index file recombines for large files; better errors for misc dumps failures (duration: 00m 03s)
  • 16:25 ariel@deploy1001: Started deploy [dumps/dumps@8f83eea]: fix up multistream index file recombines for large files; better errors for misc dumps failures
  • 14:21 arturo: T194855 cloudvirt1020 is poweroff, waiting for disk setup before installing
  • 00:20 XioNoX: add port 22 in cloud-in4 term labsdb

2019-02-15

  • 20:40 andrewbogott: enabled virtualization (all three settings) on cloudvirt1019
  • 19:41 arturo: T193264 reimaging cloudvirt1019 to get mitaka/stretch
  • 18:51 arturo: T193264 icinga downtime cloudvirt1019 for 1 week
  • 18:44 bstorm_: stopped replication and then mariadb on labsdb1004
  • 16:52 cdanis: correction, needed to increment version; adding backported rasdaemon 0.6.0-1.2+deb8u2 to jessie-wikimedia
  • 16:48 cdanis: adding backported rasdaemon 0.6.0-1.2+deb8u1 to jessie-wikimedia
  • 16:29 bblack: reprepro: uploaded gdnsd-3.0.0-1~wmf1 to stretch-wikimedia
  • 15:45 moritzm: rebooting auth1001 for kernel security update
  • 14:50 moritzm: installing unbound update from stretch point release
  • 14:45 moritzm: removed labvirt1012 from debmonitor (got renamed to cloudvirt1012) (T216190)
  • 14:06 moritzm: rebooting mwlog1001 for kernel security update
  • 13:54 moritzm: rebooting mwlog2001 for kernel security update
  • 13:46 jbond42: install tar security updates
  • 13:19 moritzm: rolling reboot of mwdebug servers in eqiad to pick up SSBD-enabled qemu
  • 13:12 gtirloni: reboot cloudvirt1020
  • 13:11 arturo: T216239 labvirt1019 has been drained of any workload
  • 13:06 moritzm: installing NSS security updates
  • 12:42 moritzm: installing squid3 security updates
  • 12:30 jynus: stop db2089 mysql instances for reboot testing T216240
  • 12:30 arturo: T216239 schedule 1week of icinga downtime for labvirt1019
  • 10:48 akosiaris: upgrade docker on contint2001 to 18.06.2 T216236
  • 10:42 akosiaris: upgrade docker on contint1001 to 18.06.2 T216236
  • 10:35 gtirloni: reboot cloudvirt1019
  • 09:44 gehel: repool maps100[12]
  • 09:33 moritzm: imported php-defaults debs to thirdparty/php72
  • 08:42 akosiaris: restart gerrit to pick up https://gerrit.wikimedia.org/r/490640 T177868
  • 08:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 46s)
  • 08:28 moritzm: rolling restart of apertium to pick up Python 3.4 security update
  • 07:55 godog: bounce prometheus@ops on prometheus2004 to take a snapshot
  • 06:41 marostegui: Stop puppet on labsdb1005 to leave "max_user_connections" on my.cnf - T216170 T216208
  • 06:39 marostegui: Restart labsdb1005 with max_user_connections = 20 T216208
  • 06:17 marostegui: Deploy schema change on db1109 - T210713
  • 06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 49s)
  • 06:13 marostegui: Reload haproxy on dbproxy11 to repool labsdb1009
  • 00:39 mutante: puppetmaster1001: sudo puppet node clean bast3003.wikimedia.org ; sudo puppet node deactivate bast3003.wikimedia.org (T216199)
  • 00:15 jynus: setting labsdb1005 back into read-write

2019-02-14

  • 23:47 jynus: restarting labsdb1005 mysql in read only mode
  • 23:37 niharika29@deploy1001: Finished deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302 (duration: 00m 02s)
  • 23:37 niharika29@deploy1001: Started deploy [scholarships/scholarships@25ea138]: Update app with updated dependencies to mitigate PHPMailer error T215302
  • 22:07 andrewbogott: rebuilding labvirt1012 as cloudvirt1012, T216190
  • 20:38 bstorm_: Restarted mariadb on labsdb1005 for https://wikitech.wikimedia.org/wiki/Incident_documentation/20190214-labsdb1005
  • 20:09 ejegg: updated fundraising CiviCRM from 02ea871b88 to 165fbf5894
  • 19:42 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.17/extensions/GrowthExperiments/modules/help: SWAT: Help Panel: Fix IME broken in help panel search T216131 (duration: 00m 54s)
  • 19:14 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Stop NavPopups gadget conflict with PagePreviews on Wikivoyage T214878 (duration: 00m 54s)
  • 19:01 mutante: scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev
  • 18:52 mutante: scandium - deleting parsoid clone dir and running puppet one more time, to fix permissions to allow wikidev
  • 18:12 mutante: scandium - deleting parsoid clone dir and running puppet
  • 18:03 fsero: upgrading tiller to 2.12.2 on eqiad
  • 17:34 godog: bounce rsyslog on wezen/lithium, tls listener timeout in icinga
  • 16:59 moritzm: restarting apertium-apy on scb1001 to pick up Python security update
  • 16:39 marostegui: Depool labsdb1009 - T210713
  • 16:26 fsero: upgrading tiller on codfw
  • 16:11 fsero: updating tiller version on staging cluster
  • 16:10 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2085 - T214840 (duration: 00m 52s)
  • 15:50 fsero: building and publishing new tiller docker image on boron
  • 15:50 END: (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0) (volans@cumin1001)
  • 15:43 START: - Cookbook sre.hosts.upgrade-and-reboot (volans@cumin1001)
  • 15:28 volans: upgraded spicerack to v0.0.15 on cumin[12]001
  • 15:26 volans: uploaded spicerack_0.0.15-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
  • 15:12 marostegui: Clear idrac logs from db2085 - T214840
  • 14:45 godog: depool and stop logstash1009 for stretch reimage - T213898
  • 14:20 marostegui: Stop MySQL on db2085 for on-site maintenance - T214840
  • 14:12 jijiki: Enabling puppet on thumbor* servers - T214597
  • 13:39 arturo: T215892 icinga downtime cloudvirt1024 for 2 weeks
  • 12:22 zeljkof: EU SWAT finished
  • 12:21 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.17/extensions/ExternalGuidance/: SWAT: Fix the eventlogging schema definition as per manifest_version=2 (duration: 00m 55s)
  • 11:43 _joe_: restarting hhvm on mw1338, hot tc exhausted T216084
  • 11:04 _joe_: upgrading python3-etcd on stretch T209136
  • 11:03 jbond42: rolling security updates for curl
  • 11:02 jijiki: Disabling puppet on thumbor* servers - T214597
  • 10:59 moritzm: installing python3.4 security updates
  • 10:53 godog: bounce prometheus instances on prometheus2004 to take a snapshot
  • 08:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 T214840 (duration: 00m 52s)
  • 07:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 T210713 (duration: 00m 54s)
  • 07:36 marostegui: Stop MySQL on db1106 for reboot - T214840
  • 06:10 marostegui: Deploy schema change on db1087 with replication, lag will be generated on labsdb:s8 T210713
  • 06:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 T210713 (duration: 00m 55s)
  • 01:52 mutante: scandium - removing parsoid deploy dir and letting puppet re-clone it after merging gerrit fix 484602 - replace manual clone with proper puppetization (T201366)
  • 01:52 mutante: scandium - removing parsoid deploy dir and letting puppet re-clone it after merging gerrit fix 484602 - replace manual hack with proper puppet
  • 01:15 mutante: phab1001 - phabricator mail config converted to cluster.mailers to adjust to upstream change (T212989)
  • 00:36 bd808@deploy1001: Finished deploy [scholarships/scholarships@1d89fe2]: Live hack PHPMailer namespace T215302 (duration: 00m 02s)
  • 00:36 bd808@deploy1001: Started deploy [scholarships/scholarships@1d89fe2]: Live hack PHPMailer namespace T215302
  • 00:32 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES (damaging only) on itwiki (T211032) (duration: 00m 53s)
  • 00:24 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable help panel search on cswiki and kowiki (T209301) (duration: 00m 55s)

2019-02-13

  • 23:42 niharika29@deploy1001: Finished deploy [scholarships/scholarships@1d89fe2]: Update scholarships app for 2019 cycle T215302 (duration: 00m 02s)
  • 23:42 niharika29@deploy1001: Started deploy [scholarships/scholarships@1d89fe2]: Update scholarships app for 2019 cycle T215302
  • 21:31 jijiki: Restarting nutcracker on scb100*.eqiad.wmnet
  • 20:54 mutante: ruthenium - shell access for parsoid-testers revoked by puppet, please use scandium.eqiad.wmnet (T201366)
  • 20:44 otto@deploy1001: Started restart [eventstreams/deploy@07033d4]: bouncing eventstreams to apply page-links-change stream config
  • 20:43 mutante: ms-be2021 - powercycling
  • 20:09 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.17 (duration: 00m 53s)
  • 20:08 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.17
  • 19:55 mforns@deploy1001: Finished deploy [analytics/refinery@5f1461e]: Deploying analytics refinery with refinery-source v0.0.85 jars (duration: 07m 36s)
  • 19:48 mforns@deploy1001: Started deploy [analytics/refinery@5f1461e]: Deploying analytics refinery with refinery-source v0.0.85 jars
  • 18:13 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1014 (duration: 00m 52s)
  • 18:06 godog: reimage prometheus2003 - T187987
  • 18:01 krinkle@deploy1001: Synchronized php-1.33.0-wmf.17/includes/libs/rdbms/loadbalancer/LoadBalancer.php: Id70fdfa62ef / T215611 (duration: 00m 55s)
  • 17:49 marostegui: Stop MYSQL on db1114 for onsite maintenance - T214720
  • 17:25 jijiki: Pooling mw1299 back - T215569
  • 17:06 cmjohnson1: db1106, troubleshooting idrac issue and updating f/w
  • 16:58 otto@deploy1001: scap-helm eventgate-analytics finished
  • 16:58 otto@deploy1001: scap-helm eventgate-analytics cluster staging completed
  • 16:58 otto@deploy1001: scap-helm eventgate-analytics upgrade staging stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
  • 16:30 elukey: reimage stat1005 to Debian Buster (again)
  • 16:22 otto@deploy1001: scap-helm list finished
  • 16:22 otto@deploy1001: scap-helm list cluster staging completed
  • 16:22 otto@deploy1001: scap-helm list [namespace: list, clusters: staging]
  • 16:13 otto@deploy1001: scap-helm eventgate-analytics finished
  • 16:13 otto@deploy1001: scap-helm eventgate-analytics cluster staging completed
  • 16:13 otto@deploy1001: scap-helm eventgate-analytics upgrade staging stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
  • 15:46 marostegui: Stop MySQL on db1106 for onsite maintenance - this will generate lag on s1 labs - T214840
  • 15:28 jynus: stop and upgrade es1014
  • 15:27 otto@deploy1001: scap-helm eventgate-analytics finished
  • 15:27 otto@deploy1001: scap-helm eventgate-analytics cluster staging completed
  • 15:27 otto@deploy1001: scap-helm eventgate-analytics upgrade staging stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
  • 15:27 otto@deploy1001: scap-helm eventgate-analytics upgrade staging stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad,codfw]
  • 15:17 akosiaris@deploy1001: scap-helm eventgate-analytics finished
  • 15:17 akosiaris@deploy1001: scap-helm eventgate-analytics cluster staging completed
  • 15:17 akosiaris@deploy1001: scap-helm eventgate-analytics install -f /srv/scap-helm/eventgate/eventgate-analytics-staging-values.yaml --set service.port=31193 ../ [namespace: eventgate-analytics, clusters: staging]
  • 15:16 moritzm: updated thirdparty/php72 component to PHP 7.2.15
  • 15:10 akosiaris@deploy1001: scap-helm eventgate-analytics finished
  • 15:10 akosiaris@deploy1001: scap-helm eventgate-analytics cluster staging completed
  • 15:10 akosiaris@deploy1001: scap-helm eventgate-analytics install -f /srv/scap-helm/eventgate/eventgate-analytics-staging-values.yaml --set service.port=31193 ../ [namespace: eventgate-analytics, clusters: staging]
  • 15:09 akosiaris@deploy1001: scap-helm eventgate-analytics install -f /srv/scap-helm/eventgate/eventgate-analytics-staging-values.yaml ../ [namespace: eventgate-analytics, clusters: staging]
  • 15:08 akosiaris@deploy1001: scap-helm eventgate-analytics finished
  • 15:08 akosiaris@deploy1001: scap-helm eventgate-analytics cluster staging completed
  • 15:08 akosiaris@deploy1001: scap-helm eventgate-analytics install --dry-run --debug -f /srv/scap-helm/eventgate/eventgate-analytics-staging-values.yaml ../ [namespace: eventgate-analytics, clusters: staging]
  • 15:05 akosiaris@deploy1001: scap-helm eventgate-analytics finished
  • 15:05 akosiaris@deploy1001: scap-helm eventgate-analytics cluster staging completed
  • 15:05 akosiaris@deploy1001: scap-helm eventgate-analytics install --dry-run --debug -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
  • 15:05 akosiaris@deploy1001: scap-helm eventgate-analytics install -n staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics --dry-run --debug [namespace: eventgate-analytics, clusters: staging]
  • 14:53 otto@deploy1001: scap-helm eventgate-analytics finished
  • 14:53 otto@deploy1001: scap-helm eventgate-analytics cluster staging completed
  • 14:53 otto@deploy1001: scap-helm eventgate-analytics install -n staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
  • 14:25 elukey: reimage stat1005 back to stretch to test GPU drivers
  • 14:06 godog: cancel https://integration.wikimedia.org/ci/job/operations-mw-config-composer-test-docker/12236 to unblock test-prio zuul queue
  • 14:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1120, depool es1014 (duration: 00m 52s)
  • 12:34 arturo: T216030 icinga downtime cloudvirt1018 for 2 weeks
  • 12:32 arturo: T216030 T216004 rebooting cloudvirt1018
  • 11:55 moritzm: installing avahi security updates
  • 11:49 jynus: stop and upgrade db1120
  • 11:43 moritzm: installing golang updates on jessie
  • 11:41 volans: upgraded spicerack on cumin[12]001 to v0.0.14
  • 11:38 volans: uploaded spicerack_0.0.14-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
  • 11:33 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1120 (duration: 00m 53s)
  • 11:11 moritzm: installing postgis security updates
  • 09:46 moritzm: installing golang security updates
  • 09:33 gtirloni: labsdb1005 rebooted server
  • 09:26 gtirloni: labsdb1005 stopped mysql
  • 09:22 marostegui: Stop MySQL on db1106 - T214840
  • 09:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 T214840 (duration: 00m 53s)
  • 08:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 (duration: 00m 53s)
  • 06:46 vgutierrez: uploaded acme-chief 0.10 to apt.wikimedia.org (buster) - T215925
  • 06:18 marostegui: Deploy schema change on db1104 - T210713
  • 06:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 (duration: 01m 07s)
  • 06:12 marostegui: Stop MySQL on db2085 to keep debugging kernel issues - T214840
  • 01:31 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Add ExternalGuidance extension T213076 (part 3) (duration: 00m 53s)
  • 01:30 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add ExternalGuidance extension T213076 (part 2) (duration: 00m 53s)
  • 01:15 thcipriani@deploy1001: Finished scap: SWAT: Add ExternalGuidance extension T213076 (part I: build l10n and sync code) (duration: 27m 51s)
  • 00:47 thcipriani@deploy1001: Started scap: SWAT: Add ExternalGuidance extension T213076 (part I: build l10n and sync code)
  • 00:41 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.17/extensions/Thanks/modules/ext.thanks.mobilediff.css: SWAT: Follow ups to I807f729c1b1a9e9b5952685bb18f540f81d70f47 (duration: 00m 55s)
  • 00:27 XioNoX: merge VRRP Icinga Check

2019-02-12

  • 23:14 jforrester@deploy1001: Finished scap: Another full scap, hoping to find the new i18n in RL for T214482 T215471 T215472 (duration: 06m 01s)
  • 23:09 foks: removed 4 files for legal compliance
  • 23:08 jforrester@deploy1001: Started scap: Another full scap, hoping to find the new i18n in RL for T214482 T215471 T215472
  • 22:47 jforrester@deploy1001: Finished scap: Full scap for new i18n and code for T214482 T215471 T215472 (duration: 18m 03s)
  • 22:29 jforrester@deploy1001: Started scap: Full scap for new i18n and code for T214482 T215471 T215472
  • 21:38 robh: icinga1001 in hardware testing, dont mess with it T214760
  • 21:10 robh: working on troubleshooting icinga1001 via T214760
  • 20:58 jforrester@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/Wikibase/view/resources/resources.php: Hot-deploy I74f6389ae for other code, file 2 (duration: 00m 52s)
  • 20:57 jforrester@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/Wikibase/view/lib/resources.php: Hot-deploy I74f6389ae for other code, file 1 (duration: 00m 51s)
  • 20:52 jforrester@deploy1001: Synchronized php-1.33.0-wmf.16/resources/Resources.php: Hot-deploy If0d7b687e for other code (duration: 00m 54s)
  • 20:06 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Group0 to 1.33.0-wmf.17
  • 19:59 thcipriani@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.17 and rebuild l10n (duration: 18m 54s)
  • 19:40 thcipriani@deploy1001: Started scap: testwiki to php-1.33.0-wmf.17 and rebuild l10n
  • 19:37 thcipriani@deploy1001: Pruned MediaWiki: 1.33.0-wmf.12 (duration: 03m 10s)
  • 19:32 thcipriani@deploy1001: Pruned MediaWiki: 1.33.0-wmf.9 (duration: 10m 05s)
  • 18:48 thcipriani: make-wmf-branch 1.33.0-wmf.17
  • 17:54 chaomodus: notebook1003 - restarted nagios-nrpe-server T212824
  • 17:04 marostegui: Start MySQL again on db2085 for s1 and s8 - T214840
  • 16:18 akosiaris: refresh kubernetes default egress policy T211247
  • 15:58 akosiaris@deploy1001: scap-helm eventgate-analytics finished
  • 15:58 akosiaris@deploy1001: scap-helm eventgate-analytics cluster codfw completed
  • 15:58 akosiaris@deploy1001: scap-helm eventgate-analytics cluster eqiad completed
  • 15:58 akosiaris@deploy1001: scap-helm eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad,codfw]
  • 15:46 akosiaris: create namespaces for eventgate-analytics on eqiad/codfw/staging cluster T211247 T213194
  • 15:45 moritzm: rebooting db2085 for some tests
  • 15:38 marostegui: Stop MySQL on db2085 - T214840
  • 15:38 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2085 - T214840 (duration: 00m 47s)
  • 15:30 otto@deploy1001: scap-helm --help finished
  • 15:30 otto@deploy1001: scap-helm --help cluster codfw completed
  • 15:30 otto@deploy1001: scap-helm --help cluster eqiad completed
  • 15:30 otto@deploy1001: scap-helm --help [namespace: --help, clusters: eqiad,codfw]
  • 15:03 ejegg: updated fundraising CiviCRM from a541a83cb2 to 02ea871b88
  • 14:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1092 (duration: 00m 46s)
  • 14:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: More api traffic to db1092 (duration: 00m 44s)
  • 14:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: More traffic to db1092 (duration: 00m 46s)
  • 13:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Give some api traffic to db1092 (duration: 00m 46s)
  • 13:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1092 (duration: 00m 47s)
  • 13:39 vgutierrez: uploaded acme-chief 0.9 to apt.wikimedia.org (stretch) - T207389 T213737
  • 12:57 moritzm: installing openssl1.0 security updates
  • 12:30 zeljkof: EU SWAT finished
  • 12:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add https://polona.pl/ to $wgCopyUploadsDomains (T215501) (duration: 00m 46s)
  • 12:19 moritzm: install ghostscript security updates on scb*
  • 12:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create extendedconfirmed user group for viwiki (T215493) (duration: 00m 47s)
  • 12:10 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Rollbackers User Group Right on azwiki (T215200) (duration: 00m 47s)
  • 12:03 marostegui: Stop MySQL on db1092 to upgrade mysql and kernel
  • 11:27 moritzm: rebooting stat1005
  • 11:20 moritzm: installing ghostscript security updates on remaining thumbor hosts
  • 10:25 marostegui: Deploy schema change on db1092 T210713
  • 10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 46s)
  • 10:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 46s)
  • 10:00 moritzm: installing ghostscript security updates on thumbor1001
  • 09:36 moritzm: reimaging stat1005 to buster
  • 08:20 marostegui: Deploy schema change on db1101:3318 - T210713
  • 08:20 marostegui: Depool db1101:3318 - T210713
  • 08:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 00m 46s)
  • 08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 (duration: 00m 49s)
  • 07:49 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@125354e]: maintain symlink for old venv path with new virtualenv deploy script (duration: 03m 55s)
  • 07:46 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@125354e]: maintain symlink for old venv path with new virtualenv deploy script
  • 07:40 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@125354e]: testing simplified virtualenv deploy (take 2) (duration: 04m 14s)
  • 07:35 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@125354e]: testing simplified virtualenv deploy (take 2)
  • 07:31 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@125354e]: testing simplified virtualenv deploy (duration: 01m 07s)
  • 07:30 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@125354e]: testing simplified virtualenv deploy
  • 07:26 elukey: update analytics-in4 term mysql-dbstore on cr1/cr2 eqiad
  • 07:09 marostegui: Rename ep_* tables on db1089 (s1) - T174802
  • 06:33 kart_: Finished fourth manual run of unpublished draft purge script (T203059)
  • 06:14 marostegui: Deploy schema change on db1099:3318 T210713
  • 06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 (duration: 00m 52s)
  • 06:04 kart_: Fourth manual run of unpublished draft purge script (T203059)
  • 02:18 thcipriani: restarting gerrit due to high load
  • 00:49 ebernhardson@deploy1001: Finished scap: SWAT: full sync for gerrit:489309 i18n (duration: 18m 20s)
  • 00:30 ebernhardson@deploy1001: Started scap: SWAT: full sync for gerrit:489309 i18n
  • 00:28 ebernhardson@deploy1001: Synchronized wmf-config/WikibaseSearchSettings.php: gerrit:489780 T214515 Promote new wbsearchentities profiles to default in de, fr, es (duration: 00m 46s)
  • 00:13 jforrester@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/CentralNotice/: SWAT Merge branch 'master' into wmf_deploy I8e52d222eb (duration: 00m 49s)
  • 00:05 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Stop setting wgSessionsInObjectCache, it's being removed from MW I2946b5b9a (duration: 00m 47s)

2019-02-11

  • 23:22 cdanis: T214760 icinga2001% sudo killall nsca
  • 22:53 cdanis: icinga.w.o-->icinga2001 DNS change deployed T214760
  • 22:40 cdanis: icinga1001 now passive T214760
  • 22:34 cdanis: failing over icinga to icinga2001
  • 21:33 arlolra: Updated Parsoid to b4b9603 (T208901, T215537, T213468, T215638)
  • 21:24 arlolra@deploy1001: Finished deploy [parsoid/deploy@4e9b142]: Updating Parsoid to b4b9603 (duration: 09m 33s)
  • 21:22 otto@deploy1001: Synchronized wmf-config/CommonSettings.php: Use newer RCFeed config for EventBus based recentchange event - T215834 (duration: 00m 47s)
  • 21:20 ottomata: deploying mediawiki-config change for update to EventBus RCFeed config (no-op)
  • 21:16 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@c6a6285]: Weekly GUI deploy (duration: 11m 54s)
  • 21:14 arlolra@deploy1001: Started deploy [parsoid/deploy@4e9b142]: Updating Parsoid to b4b9603
  • 21:13 mobrovac@deploy1001: Finished deploy [citoid/deploy@0b91bea]: Use Zotero for DOIs and pass it the A-L header - T214766 T210806 T215755 (duration: 03m 47s)
  • 21:09 mobrovac@deploy1001: Started deploy [citoid/deploy@0b91bea]: Use Zotero for DOIs and pass it the A-L header - T214766 T210806 T215755
  • 21:04 smalyshev@deploy1001: Started deploy [wdqs/wdqs@c6a6285]: Weekly GUI deploy
  • 20:08 ppchelko@deploy1001: Finished deploy [changeprop/deploy@bdb4740]: Update dependencies, minor refactor, safer deduplication, T207329 (duration: 01m 37s)
  • 20:07 ppchelko@deploy1001: Started deploy [changeprop/deploy@bdb4740]: Update dependencies, minor refactor, safer deduplication, T207329
  • 19:42 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106, db1118 with full weight (duration: 00m 46s)
  • 19:34 catrope@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: Remove main page special casing from lawiki (T215709) (duration: 00m 46s)
  • 19:28 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgRestrictionLevels on Serbian projects (T215653) (duration: 00m 46s)
  • 19:16 catrope@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/GrowthExperiments/: Help panel search instrumentation (T211166) (duration: 00m 47s)
  • 19:08 catrope@deploy1001: Synchronized wmf-config/throttle.php: Lift account creation cap for edit-a-thon (T215069) (duration: 00m 47s)
  • 19:08 jijiki: Repooled thumbor1004 - T215411
  • 18:50 robh: thumbor1004 rebooted and updated firmware T215411
  • 18:50 robh: thumbor1004 rebooted and updated firmware
  • 16:49 jynus: stop, upgrade and restart db1106
  • 16:36 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1011
  • 16:31 marostegui: Reverse password for globaldev user on dbstore1002 - T200801
  • 16:29 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 (duration: 00m 52s)
  • 15:49 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1118 (duration: 00m 48s)
  • 15:24 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY (duration: 00m 47s)
  • 15:23 marostegui: Relohad haproxy on dbproxy1010 to depool labsdb1011 - https://phabricator.wikimedia.org/T212308
  • 15:21 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Wikibase.php, add conditional setting of useEntitySourceBasedFederation (duration: 00m 47s)
  • 15:20 marostegui: Repool labsdb1010 - T212308
  • 15:19 jynus: add missing grants to db1118
  • 15:07 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Revert, second try (duration: 00m 47s)
  • 15:00 addshore@deploy1001: sync-file aborted: Wikibase.php, add conditional setting of useEntitySourceBasedFederation (duration: 00m 01s)
  • 14:55 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Revert (duration: 00m 45s)
  • 14:53 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool db1118 for the first time (duration: 00m 47s)
  • 14:51 mbsantos@deploy1001: Finished deploy [tilerator/deploy@d546183] (stretch): Updating maps2004 tilerator for the stretch migration work (duration: 00m 39s)
  • 14:50 mbsantos@deploy1001: Started deploy [tilerator/deploy@d546183] (stretch): Updating maps2004 tilerator for the stretch migration work
  • 14:48 mbsantos@deploy1001: Finished deploy [kartotherian/deploy@173adbe] (stretch): Updating maps2004 kartotherian for the stretch migration work (duration: 00m 21s)
  • 14:48 mbsantos@deploy1001: Started deploy [kartotherian/deploy@173adbe] (stretch): Updating maps2004 kartotherian for the stretch migration work
  • 14:47 moritzm: installing curl security updates on trusty
  • 14:21 marostegui: Remove staging from dbstore1003 - T210478
  • 14:16 godog: depool and take a snapshot of prometheus data for all instances on prometheus2003 - T187987
  • 14:09 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - T212308
  • 14:08 marostegui: Deploy schema change on db1116:3318 - T210713
  • 12:21 godog: bounce rsyslogd on lithium / wezen, syslog tls listener stuck
  • 12:19 zeljkof: EU SWAT finished
  • 12:18 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Senior Citizens Write Wikipedia course (T215618) (duration: 00m 48s)
  • 12:14 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Clean expired throttle rules (duration: 00m 48s)
  • 10:47 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 10:46 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 48s)
  • 10:41 jynus: upgrading mariadb client on cumin* hosts
  • 10:27 mvolz@deploy1001: scap-helm zotero finished
  • 10:27 mvolz@deploy1001: scap-helm zotero cluster codfw completed
  • 10:27 mvolz@deploy1001: scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 10:24 mvolz@deploy1001: scap-helm zotero finished
  • 10:24 mvolz@deploy1001: scap-helm zotero cluster eqiad completed
  • 10:24 mvolz@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 10:19 marostegui: Add dbstore1005:3350 to tendril and zarcillo - T210478
  • 10:17 mvolz@deploy1001: scap-helm zotero finished
  • 10:17 mvolz@deploy1001: scap-helm zotero cluster staging completed
  • 10:17 mvolz@deploy1001: scap-helm zotero upgrade staging -f zotero-values-staging.yaml --version=0.0.1 stable/zotero [namespace: zotero, clusters: staging]
  • 10:17 jynus: restart db1114
  • 09:38 marostegui: Stop all mysql instances on dbstore1005 for reboot
  • 09:11 marostegui: Stop all mysql instances on dbstore1003 for reboot
  • 08:17 moritzm: removed cloudcontrol2001-dev.codfw.wmnet from debmonitor (actual hostname in use is cloudcontrol2001-dev.wikimedia.org)
  • 08:07 marostegui: Deploy schema change on s8 codfw master (db2045) - this will generate lag on codfw T210713
  • 07:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1100 (duration: 00m 46s)
  • 07:39 marostegui: Deploy schema change on s7 primary master (db1062) - T210713
  • 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Give api traffic to db1100 (duration: 00m 46s)
  • 07:18 marostegui: Stop all mysql instances on dbstore1004 for a reboot
  • 07:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 with low weight (duration: 00m 46s)
  • 07:06 marostegui: Upgrade MySQL on db1100
  • 07:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for mysql upgrade (duration: 00m 47s)
  • 07:00 marostegui: Restart icinga on icinga1001 - checks went awol
  • 06:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 48s)
  • 06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 48s)
  • 06:14 marostegui@deploy1001: sync-file aborted: Depool db0179 (duration: 00m 01s)
  • 04:23 TimStarling: on mwmaint1002: running normalizeThrottleParameters.php --dry-run on all wikis (T209565)
  • 04:19 tstarling@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: maintenance script update for new dry run (duration: 00m 47s)
  • 04:19 tstarling@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/WikimediaEvents/tests/phpunit/PageViewsTest.php: test-only undeployed change (duration: 00m 46s)
  • 04:18 tstarling@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/NavigationTiming/tests/ext.navigationTiming.test.js: test-only undeployed change (duration: 00m 51s)
  • 04:10 tstarling@deploy1001: sync-file aborted: test-only undeployed change (duration: 00m 12s)
  • 03:05 kartik@deploy1001: Finished deploy [cxserver/deploy@ee4a15a]: Update cxserver to 8928852 (T213256) (duration: 04m 08s)
  • 03:01 kartik@deploy1001: Started deploy [cxserver/deploy@ee4a15a]: Update cxserver to 8928852 (T213256)

2019-02-10

  • off: force rebooting mw1299, stuck again - T215569
  • off: forcing reboot of icinga1001 because it's stuck again (no ping, no ssh, CPU stuck messages on console) - T214760
  • 09:25 marostegui: Disable notifications for lag checks on dbstore1002 - T210478

2019-02-09

  • 21:42 Reedy: running `foreachwiki refreshImageMetadata.php --mediatype BITMAP --mime image/vnd.djvu --force` on mwmaint1002 T215635
  • 21:41 Reedy: refreshImageMetadata.php for commonswiki done T215635
  • 16:51 Jeff_Green: restarted icinga process on icinga1001 because of passive check alert-storm

2019-02-08

  • 23:23 Reedy: running `refreshImageMetadata.php --mediatype BITMAP --mime image/vnd.djvu --force` against commonswiki on mwmaint1002 T215635 (this time we mean it)
  • 22:56 Reedy: running `refreshImageMetadata.php --mediatype BITMAP --mime image/vnd.djvu` against commonswiki on mwmaint1002 T215635
  • 21:25 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: Move variable (duration: 00m 49s)
  • 19:50 krinkle@deploy1001: Synchronized w/touch.php: Ia1e610a5f (duration: 00m 46s)
  • 19:49 krinkle@deploy1001: Synchronized w/robots.php: Ia1e610a5f (duration: 00m 46s)
  • 19:48 krinkle@deploy1001: Synchronized w/favicon.php: Ia1e610a5f (duration: 00m 46s)
  • 19:47 krinkle@deploy1001: Synchronized w/extract2.php: Ia1e610a5f (duration: 00m 48s)
  • 18:14 gtirloni: T213527 graphite2002 disabled puppet and commented prometheus_puppet_agent_stats cronjob due to cronspam
  • 18:08 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for s1 rc slaves (duration: 00m 49s)
  • 17:55 mutante: phab1001 - restart aphlict service
  • 17:52 mutante: phab1001 - restarting phd service
  • 17:49 arturo: T215605 add prometheus-openstack-exporter 0.0.8-4 to stretch-wikimedia
  • 17:47 mutante: phab1001 - restarting apache2 service for library upgrade
  • 17:42 mutante: graceful reload of apache on phabricator prod server (phab1001)
  • 17:27 XioNoX: merge Icinga: add ping check for ulsfo PDUs
  • 16:50 ejegg: updated payments-wiki-staging from 52a271e681 to 31647bc97e
  • 16:09 jynus: stopping s1 replication on dbstore1001 to speed up cloning T214720
  • 16:08 moritzm: imported git-fat 0.1.3-2+deb10u1 to buster-wikimedia (T213527)
  • 15:46 marostegui: Repool labsdb1009 - T212308
  • 15:33 _joe_: apt-get upgrade on mwmaint2001 to fix the php installation T215376
  • 15:31 moritzm: imported debmonitor 0.1.5-1+deb10u1 to buster-wikimedia (T213527)
  • 15:31 _joe_: upgraded all php extensions to php 7.2 compatible versions on mwmaint1002
  • 15:10 jijiki: Upgrading php-redis 4.1.1 to mwmaint1002 - T215376
  • 14:51 marostegui: Reload haproxy on dbproxy1011 to depool labsdb1009 - https://phabricator.wikimedia.org/T212308
  • 13:56 moritzm: updated firmware-enriched buster netboot image to 20190208 daily build, the alpha5 image no longer works as Linux 4.19.16-1 bumped the ABI and migrated to testing yesterday
  • 13:45 jynus: racadm serveraction powercycle db1114
  • 13:39 onimisionipe: starting osm-initial-import for maps2004 which is the newly migrated to stretch master - T198622
  • 13:37 elukey: roll restart of aqs on aqs1* to pick up new druid backend changes
  • 13:05 arturo: T209029 reimaging cloudelastic1004
  • 12:54 ejegg: updated fundraising CiviCRM from 3a1bb82373 to a541a83cb2
  • 12:51 jynus: disabling notifications on db1114
  • 12:44 elukey@deploy1001: Synchronized wmf-config/db-eqiad.php: depooling db1114, host down (duration: 00m 47s)
  • 11:36 moritzm: reimage graphite2002 to buster
  • 11:08 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 fully (duration: 00m 47s)
  • 10:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099 (duration: 00m 47s)
  • 10:27 jijiki: Restarting memcached on mc1026 to apply '-R 200' - T208844
  • 10:23 godog: swift codfw-prod: more weight to ms-be2047 - T209395 T209921
  • 10:15 jynus: stop and upgrade db1099
  • 10:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 47s)
  • 09:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 46s)
  • 09:28 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099 (duration: 00m 46s)
  • 09:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1086 (duration: 00m 46s)
  • 09:16 moritzm: installing rssh security updates
  • 09:06 moritzm: installing libarchive security updates
  • 09:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: More traffic to db1086 (duration: 00m 47s)
  • 08:53 moritzm: reimage graphite2002 to buster
  • 08:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 with low load (duration: 00m 46s)
  • 08:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1086 (duration: 00m 47s)
  • 08:24 jynus: stop and upgrade db1083
  • 08:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 00m 47s)
  • 08:15 marostegui: Upgrade MySQL on db1086
  • 08:05 marostegui: Upgrade MySQL on db1086 and deploy schema change
  • 08:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 46s)
  • 07:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Full repool db1094 (duration: 00m 47s)
  • 07:45 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw1299.eqiad.wmnet
  • 07:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: More traffic to db1094 (duration: 02m 55s)
  • 07:27 marostegui@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1299.eqiad.wmnet
  • 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1094 (duration: 02m 56s)
  • 07:12 marostegui: Upgrade mysql and kernel on db1094
  • 06:58 marostegui: Deploy schema change on db1094 T210713
  • 06:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 46s)
  • 06:54 marostegui: Take a mysqldump from staging on dbstore1003 from dbstore1002 - T210478
  • 06:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 (duration: 00m 49s)
  • 06:29 marostegui: powercycle mw1299 - T215569
  • 06:21 marostegui: Deploy schema change on db1098:3317
  • 06:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 (duration: 02m 58s)
  • 06:07 marostegui: Drop staging.mep_word_persistence from dbstore1002 T215450 T213706
  • 02:34 ejegg: updated fundraising CiviCRM from 08be00e87f to 3a1bb82373
  • 01:37 dzahn@puppetmaster1001: conftool action : set/pooled=no; selector: name=mw1299.eqiad.wmnet
  • 01:10 mutante: mw1299 has been down about 8 hours, does it need deployment.. depooling
  • 01:08 mutante: powercycle crashed mw1299 via mgmt (garbled console output) (T215569)
  • 00:22 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:488588 phab:T214515 Turn off wikidata wbsearchentities ab test in de, fr, es (duration: 02m 55s)
  • 00:16 ebernhardson: scap sync timed out on mw1299.eqiad.wmnet
  • 00:15 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:483044 T209873 Give protect right to centralnoticeadmin on Meta (duration: 02m 56s)

2019-02-07

  • 23:29 XioNoX: restart ps1-22-ulsfo
  • 23:23 reedy@deploy1001: Synchronized tests/dblistTest.php: Sync test (duration: 02m 55s)
  • 23:18 reedy@deploy1001: Synchronized README: must be up to date (duration: 02m 54s)
  • 22:48 reedy@deploy1001: Synchronized dblists/: alphasort dblists (duration: 02m 56s)
  • 21:43 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group2 wikis to 1.33.0-wmf.16 refs T206670
  • 21:38 robh: updating firmware on ps1-23-ulsfo via T209101 ps1-22-ulsfo update completed
  • 21:22 robh: updating firmware on ps1-22-ulsfo via T209101
  • 20:55 twentyafterfour: train status: deploying 1.33.0-wmf.16 to group2
  • 20:19 sbisson@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/WikibaseLexeme/src/DataAccess/Search/LexemeFulltextResult.php: SWAT: Fix fatal error - EmptySet does not exist anymore (duration: 03m 03s)
  • 19:45 sbisson@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/GrowthExperiments/: SWAT: Help Panel: Fix iOS scroll bug (duration: 03m 02s)
  • 19:28 sbisson@deploy1001: sync-file aborted: SWAT: GrowthExperiments: Enable search for help panel on testwiki (duration: 02m 22s)
  • 19:25 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: GrowthExperiments: Enable search for help panel on testwiki (duration: 03m 04s)
  • 18:32 mutante: LDAP - adding raz-shuty to group nda (T214488)
  • 17:06 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 (duration: 03m 03s)
  • 16:03 jynus: restart db1085, temporary s6 lag on wikireplicas
  • 15:55 gehel: starting reimage of maps2004 - T198622
  • 15:51 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 (duration: 00m 58s)
  • 15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on wikitech for T215464. This may cause lag in codfw.
  • 15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 8 wikis for T215464. This may cause lag in codfw.
  • 15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 7 wikis for T215464. This may cause lag in codfw.
  • 15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 6 wikis for T215464. This may cause lag in codfw.
  • 15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 5 wikis for T215464. This may cause lag in codfw.
  • 15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 4 wikis for T215464. This may cause lag in codfw.
  • 15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on remaining section 3 wikis for T215464. This may cause lag in codfw.
  • 15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 2 wikis for T215464. This may cause lag in codfw.
  • 15:16 anomie@mwmaint1002: Fixing log_search after migrateActors.php on section 1 wikis for T215464. This may cause lag in codfw.
  • 15:07 anomie@mwmaint1002: Fixing log_search after migrateActors.php on test wikis and mediawikiwiki for T215464. This may cause lag in codfw.
  • 15:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1101 (duration: 00m 55s)
  • 14:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1101 after alter and mysql upgrade (duration: 00m 55s)
  • 14:34 jbond42: deploying security updates for libgd3
  • 12:42 Amir1: EU SWAT is done
  • 12:42 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: SWAT: Set EntityUsageTable addUsage batch size to 300, Part II (duration: 00m 54s)
  • 12:42 marostegui: Set dbstore1002 as IDEMPOTENT - T213670
  • 12:39 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set EntityUsageTable addUsage batch size to 300 (T215146), Part I (duration: 00m 55s)
  • 12:34 marostegui: Powercycle mw1299 as it is down and not responding
  • 12:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1101 after alter and mysql upgrade (duration: 03m 02s)
  • 12:26 ladsgroup@deploy1001: Synchronized wmf-config/interwiki.php: SWAT: Update interwiki cache to have yuewiktionary instead of zh-yue (T214400) (duration: 03m 04s)
  • 12:06 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4026.ulsfo.wmnet
  • 12:03 arturo: T214448 reimaging again cloudvirt200[1-3]-dev.codfw.wmnet
  • 11:55 marostegui: Stop MySQL on db1101:3317 and db1101:3318 for mysql upgrade
  • 11:37 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2055 (duration: 03m 02s)
  • 11:17 fsero: upgrade helm to 2.12.2 on deploy{1001,2001} and contint{1001,2001} T215244
  • 11:16 fsero: upgrade helm to 2.12.2 on deploy{1001,2001} and contint{1001,2001}
  • 10:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101 for alter and mysql upgrade (duration: 00m 56s)
  • 10:43 marostegui: Run mysqldump from dbstore1003 to dump dbstore1002:staging.mep_word_persistence - T215450
  • 09:49 marostegui: Deploy schema change on db1116 - T210713
  • 09:41 akosiaris: reboot mwdebug1001, mwdebug1002, mwdebug2001, mwdebug2002 for VCPU upgrade. T212955
  • 09:23 jynus: running alter table on db2055 for perforamance testing T212092
  • 09:15 fsero: uploading helm and tiller 2.12.2 deb package to stretch and jessie
  • 08:53 marostegui: Deploy schema change on s7 codfw master (db2047), this will generate lag on s7 codfw - T210713
  • 08:34 godog: swift codfw-prod: more weight to ms-be2047 - T209395 T209921
  • 08:14 marostegui: Deploy schema change on s4 primary master (db1068) - T210713
  • 08:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 (duration: 00m 54s)
  • 07:50 marostegui: Deploy schema change on db1081
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 (duration: 00m 53s)
  • 07:48 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 20s)
  • 07:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 00m 53s)
  • 07:42 reedy@deploy1001: Synchronized dblists/: Wikimania T215486 (duration: 00m 54s)
  • 07:03 marostegui: Deploy schema change on db1084 - T210713
  • 07:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 55s)
  • 06:48 marostegui: Restore consistency options on db2051
  • 06:14 marostegui: Ease consistency options on db2051 (s4 master) to let it catch up on replication
  • 04:35 tstarling@deploy1001: Synchronized wmf-config/set-time-limit.php: (no justification provided) (duration: 00m 54s)
  • 04:00 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable EP namespaces on wikis with no EP pages (duration: 00m 57s)
  • 01:31 eileen: civicrm revision changed from c5aec3ae76 to 08be00e87f, config revision is 306b4de48f
  • 01:24 eileen: civicrm revision changed from 6161a021c0 to c5aec3ae76, config revision is 306b4de48f
  • 01:05 twentyafterfour: US Evening SWAT is complete
  • 01:04 twentyafterfour: no phabricator deployment tonight
  • 01:04 eileen: civicrm revision changed from 613b388916 to 6161a021c0, config revision is 306b4de48f
  • 00:57 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT config change for Bug: T214003 (duration: 00m 53s)
  • 00:53 twentyafterfour@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/VisualEditor/: SWAT f89e12f to fix bug: T209610 (duration: 00m 55s)
  • 00:48 twentyafterfour@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/MobileFrontend/: SWAT dd8654a (duration: 01m 00s)
  • 00:47 twentyafterfour: syncing commit dd8654a for Bug: T209052
  • 00:24 twentyafterfour: running `mwscript migrateUserGroup.php commonswiki extended-uploader autopatrolled` on deploy1001

2019-02-06

  • 23:58 mutante: restarting icinga on icinga1001 to pick up new check command ?
  • 22:22 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.16 refs T206670 (duration: 00m 53s)
  • 22:22 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.16 refs T206670
  • 21:45 mutante: LDAP - adding brennen to wmf, releng, ciadmin - Welcome Brennen Bearnes, Software Engineer in Release Engineering (T215365 T214556)
  • 21:05 arlolra@deploy1001: Finished deploy [parsoid/deploy@a4acfa6]: Updating Parsoid to fb67a71 (duration: 03m 43s)
  • 21:04 Krinkle: krinkle@webperf1002 Kill xenon-log (pid 449). It seems its Redis TCP socket to mwlog1001 has been stuck since Dec 13, causing the process to indefinitely hang on listen()/socket.recv()
  • 21:01 arlolra@deploy1001: Started deploy [parsoid/deploy@a4acfa6]: Updating Parsoid to fb67a71
  • 20:49 mutante: LDAP - adding h78na to wmf - welcome Hana Worku, developer on the multimedia team (T215352)
  • 20:40 mutante: LDAP - adding egardner to wmf - welcome Eric Gardner , software engineer in Audiences (T214654)
  • 20:35 twentyafterfour: 1.33.0-wmf.16 has a significantly higher rate of "entire web request took longer than 60 seconds and timed out"
  • 20:03 twentyafterfour: Resuming the MediaWiki train for version 1.33.0-wmf.16. Will deploy Group0 wikis first and then catch up to group1 after a few minutes monitoring logs for stability.
  • 19:50 robh: updated firmware on cp4026 and re-seated (already well seated) dimm b3. errors have cleared for now T214516
  • 19:24 milimetric@deploy1001: Finished deploy [analytics/refinery@cd413dd]: Small bug fix for history checker (duration: 12m 45s)
  • 19:13 robh: taking cp4026 offline to flash firmware and reseat dimm for testing on T214516
  • 19:12 milimetric@deploy1001: Started deploy [analytics/refinery@cd413dd]: Small bug fix for history checker
  • 19:11 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@3272a46]: Add healthcheck plugin (no restart) cobalt T214326 (duration: 00m 09s)
  • 19:11 thcipriani@deploy1001: Started deploy [gerrit/gerrit@3272a46]: Add healthcheck plugin (no restart) cobalt T214326
  • 19:09 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@3272a46]: Add healthcheck plugin (no restart) gerrit2001 first (duration: 00m 10s)
  • 19:09 mutante: LDAP - adding afandian2 and toddleroux to nda (T214727)
  • 19:09 thcipriani@deploy1001: Started deploy [gerrit/gerrit@3272a46]: Add healthcheck plugin (no restart) gerrit2001 first
  • 19:04 jforrester@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/Flow/includes/Conversion/Utils.php: I405dd193 Update Parsoid Accept header to 2.0.0 so service can deploy (duration: 00m 54s)
  • 19:03 jforrester@deploy1001: Synchronized php-1.33.0-wmf.14/extensions/Flow/includes/Conversion/Utils.php: I405dd193 Update Parsoid Accept header to 2.0.0 so service can deploy (duration: 00m 56s)
  • 18:03 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Removed namespace Коментар, added namespace Портал on srwikinews T214561 T214563 (duration: 00m 53s)
  • 18:01 mutante: LDAP - adding alaasarhan to wmde (T215066)
  • 17:57 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Changed wgImportSources for srwikinews T214562 (duration: 00m 53s)
  • 17:53 thcipriani@deploy1001: Synchronized dblists/s3.dblist: SWAT: dblists/s3.dblist: Fix sorting of list of wikis per alphabetical order (duration: 00m 54s)
  • 17:49 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/MobileFrontend: SWAT: VE: Load HTML in parallel with modules T209052 (duration: 00m 57s)
  • 17:40 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.16/extensions/MobileFrontend: SWAT: EditorOverlay: Pass constructor of itself to VisualEditorOverlay, not instance T215408 (duration: 00m 57s)
  • 17:10 jynus: setting db1111 in read-write mode
  • 16:24 moritzm: reimaging graphite2002 to buster
  • 16:19 jynus: running alter table on db2055 T93564
  • 16:14 gehel@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad
  • 15:44 papaul: powering down thumbor2002 for disk replacement
  • 15:42 moritzm: installing spice security updates
  • 15:41 andrewbogott: rebooting cloudvirt1015 to make sure that nothing drastic changes once libguestfs is installed T215423
  • 15:11 moritzm: installing libav security updates
  • 15:08 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2055 for performance testing T93564 (duration: 00m 55s)
  • 14:50 moritzm: draining restbase1018 for eventual reboot for kernel security update (bundled with Java update)
  • 14:36 moritzm: draining restbase1017 for eventual reboot for kernel security update (bundled with Java update)
  • 14:29 elukey: add term mysql-dbstore to analytics-in4/6 on cr1/2-eqiad to allow tcp connections to dbstore100[3-5] - T210478
  • 12:30 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3032.esams.wmnet
  • 12:29 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3032.esams.wmnet
  • 12:28 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3033.esams.wmnet
  • 12:26 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3033.esams.wmnet
  • 12:25 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3040.esams.wmnet
  • 12:24 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3040.esams.wmnet
  • 12:22 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3041.esams.wmnet
  • 12:22 Amir1: EU SWAT is done
  • 12:21 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3041.esams.wmnet
  • 12:20 vgutierrez: restarting varnish-fe safely across esams/text cluster - T215389
  • 12:19 ladsgroup@deploy1001: Synchronized wmf-config/Wikibase.php: SWAT: Use separate DB connection for ID insertions on testwikidatawiki (T215147), Part II (duration: 00m 54s)
  • 12:17 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use separate DB connection for ID insertions on testwikidatawiki (T215147), Part I (duration: 00m 55s)
  • 11:58 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3042.esams.wmnet
  • 11:57 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3042.esams.wmnet
  • 11:56 vgutierrez: restarting varnish-fe in cp3042 - T215389
  • 11:02 _joe_: restarting nginx safely across the appserver fleets in order to be able to run puppet without errors
  • 10:41 marostegui: Revoke access to testreduce from ruthenium on m5 - https://phabricator.wikimedia.org/T214740
  • 10:04 moritzm: reimaging graphite2002 to buster
  • 10:01 akosiaris: restart varnish-frontend on cp3030 T215389
  • 10:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 (duration: 00m 52s)
  • 09:33 marostegui: Remove wikiuser from dbstore1003-dbstore1005 T210478
  • 09:15 godog: swift codfw-prod: more weight for ms-be2047 - T209395 T209921
  • 09:00 marostegui: Create research_role on dbstore1003-1005 on all instances - T214469
  • 08:49 marostegui: Deploy schema change on db1091 - T210713
  • 08:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 (duration: 00m 53s)
  • 08:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 (duration: 00m 53s)
  • 07:51 marostegui: Deploy schema change on db1121 - this will generate lag on s4 labs - also upgrade MySQL on db1121 T210713
  • 07:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 (duration: 00m 54s)
  • 07:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 (duration: 00m 54s)
  • 07:19 marostegui: Deploy schema change on wikitech T210713
  • 07:14 marostegui: Stop 's4' slave on dbstore1002
  • 07:13 marostegui: Deploy schema change on db1103:3314 (db1097:3314 was also done previously) - T210713
  • 07:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 (duration: 00m 53s)
  • 07:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 (duration: 00m 56s)
  • 06:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 (duration: 01m 06s)
  • 04:39 mutante: reloaded icinga service, cant find new check command definition
  • 03:14 twentyafterfour@deploy1001: Finished scap: testwikis wikis to 1.33.0-wmf.16 refs T206670 (duration: 04m 18s)
  • 03:09 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.33.0-wmf.16 refs T206670
  • 03:05 mutante: actinium - gzipping and rotating some access logs
  • 03:01 twentyafterfour@deploy1001: Synchronized scap/plugins/updateinterwikicache.py: (no justification provided) (duration: 00m 55s)
  • 02:47 mutante: actinium - blocking a bad domain and restarting squid3
  • 02:40 twentyafterfour@deploy1001: Finished scap: sync and update localization for 1.33.0-wmf.16 (duration: 15m 50s)
  • 02:32 XioNoX: push firewall rule to pfw3-eqiad - T215364
  • 02:27 mutante: actinium - apt-get clean for 8% more disk space after icinga alert
  • 02:25 twentyafterfour@deploy1001: Started scap: sync and update localization for 1.33.0-wmf.16
  • 02:16 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.14 refs T206670
  • 02:12 eileen: civicrm revision changed from 6042acb363 to 613b388916, config revision is 306b4de48f
  • 02:02 twentyafterfour@deploy1001: scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
  • 01:22 XioNoX: remove peering4/6 prefix-list from routers
  • 01:07 XioNoX: add maintenance and rollback to junos operations class
  • 00:47 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.33.0-wmf.16 refs T206670
  • 00:33 niharika29@deploy1001: Synchronized php-1.33.0-wmf.14/extensions/MobileFrontend/: EditorOverlay: captcha/abusefilter weren't being shown correctly T215101, T202374 (duration: 00m 50s)
  • 00:24 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Demystify Logstash debug level behavior (duration: 00m 51s)
  • 00:23 niharika29@deploy1001: Synchronized wmf-config/logging.php: Demystify Logstash debug level behavior (duration: 00m 46s)
  • 00:18 niharika29@deploy1001: Synchronized wmf-config/logging.php: Add PHP version to MW logs T215350 (duration: 00m 46s)
  • 00:16 niharika29@deploy1001: Synchronized wmf-config/CommonSettings.php: Preserve Composer's include paths - T215126, T215224 (duration: 01m 40s)

2019-02-05

  • 18:56 arlolra@deploy1001: Finished deploy [parsoid/deploy@a4acfa6]: (no justification provided) (duration: 02m 06s)
  • 18:53 arlolra@deploy1001: Started deploy [parsoid/deploy@a4acfa6]: (no justification provided)
  • 18:39 arlolra@deploy1001: Finished deploy [parsoid/deploy@a4acfa6]: Updating Parsoid to fb67a71 (duration: 09m 54s)
  • 18:29 arlolra@deploy1001: Started deploy [parsoid/deploy@a4acfa6]: Updating Parsoid to fb67a71
  • 18:26 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@2959e12]: Update mobileapps to 107c1b1 (T214714) (duration: 04m 43s)
  • 18:21 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@2959e12]: Update mobileapps to 107c1b1 (T214714)
  • 18:17 mutante: contint1001/contint2001 -manually deleting crontab lines unpuppetized in gerrit:488019 (T209361)
  • 18:13 Jeff_Green: authdns-update to deploy 7fee817fd3
  • 17:22 mutante: scandium - restart parsoid-vd service
  • 17:21 mutante: scandium -- copy /srv/visualdiff/testrecude/testrun.ids from ruthenium to the same locatio
  • 15:15 godog: force curator action 'replicas' to set older logstash indices to 1 replica - T213078
  • 14:30 marostegui: Deploy schema change on s4 codfw master with replication, lag will be generated on s4 codfw - T210713
  • 14:26 Jeff_Green: authdns-update for payments dev/testing hostname
  • 14:12 marostegui: Deploy schema change on db1066 (s2 master) - T210713
  • 14:05 marostegui: Delete non used grants from dbstore1002: log, warehouse,project_illustration, cognate\_wiktionary, datasets - T212487 T210478
  • 13:55 godog: swift codfw-prod: add ms-be2047 - T209395 T209921
  • 12:18 addshore: swat done
  • 12:18 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable confirmation prompt on rollback by default T215019 (duration: 00m 47s)
  • 11:35 moritzm: added firmware-enriched buster netboot image (T213546)
  • 11:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 (duration: 00m 46s)
  • 10:43 marostegui: Deploy schema change on db1074 with replication, lag will be generated on s2 - T210713
  • 10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 (duration: 00m 47s)
  • 10:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 (duration: 00m 46s)
  • 09:42 hashar: contint1001: docker image prune -f
  • 09:34 marostegui: Deploy schema change on db1090:3312 - T210713
  • 09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 45s)
  • 09:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1076 (duration: 00m 46s)
  • 09:11 marostegui: Start all slaves on dbstore1002 - T213670
  • 08:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1076 (duration: 00m 45s)
  • 08:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1076 (duration: 00m 46s)
  • 08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1076 (duration: 00m 45s)
  • 07:56 marostegui: Upgrade MySQL and kernel on db1076
  • 07:44 marostegui: Deploy schema change on db1076 - T210713
  • 07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 T210713 (duration: 00m 47s)
  • 07:13 marostegui: Taking mysqldump from dbstore1002.staging - T210478
  • 07:05 marostegui: Reboot mysql on db1117:3323 (this will make the dbproxies complain) T214248
  • 02:24 XioNoX: remove BGP session to as6412 on cr2-eqiad (gone from IX)
  • 02:21 XioNoX: delete 2nd as9121 router on cr2-esams
  • 00:47 XioNoX: add BGP sessions to AS64050 on cr1-eqsin
  • 00:24 maxsem@deploy1001: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/486405/ (duration: 00m 46s)
  • 00:11 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 46s)

2019-02-04

  • 22:05 mutante: scandium - systemctl start parsoid-vd (T201366)
  • 20:01 herron: manually ran puppet on mc1023
  • 19:50 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Clean-up: Stop setting wgParsoidWikiPrefix, unused since the Parsoid extension (duration: 00m 45s)
  • 19:45 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Clean-up: Stop setting wgFlowEventLogging, unread (duration: 00m 45s)
  • 19:39 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Clean-up: Stop setting values for wgEcho*FooterNotice*, unread (duration: 00m 46s)
  • 19:32 James_F: Manually purged atjwiki*.png logos for T215122.
  • 19:28 jforrester@deploy1001: Synchronized static/images/project-logos/atjwiki.png: SWAT: Milestone lobo for atjwiki T215122, 1x (duration: 00m 46s)
  • 19:27 jforrester@deploy1001: Synchronized static/images/project-logos/atjwiki-1.5x.png: SWAT: Milestone lobo for atjwiki T215122, 1.5x (duration: 00m 45s)
  • 19:26 jforrester@deploy1001: Synchronized static/images/project-logos/atjwiki-2x.png: SWAT: Milestone lobo for atjwiki T215122, 2x (duration: 00m 44s)
  • 19:22 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T191039 Enable wgAbuseFilterRuntimeProfile on all wikis (duration: 00m 47s)
  • 19:19 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@8b2f078]: Weekly GUI deploy (duration: 09m 47s)
  • 19:09 smalyshev@deploy1001: Started deploy [wdqs/wdqs@8b2f078]: Weekly GUI deploy
  • 18:31 XioNoX: adding Papaul to root@wiki
  • 18:22 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Clean-up: Drop reading for wgEcho*FooterNotice*, unread (duration: 00m 46s)
  • 18:18 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Clean-up: Stop setting wgEchoConfig, unused since 2016 (duration: 00m 48s)
  • 18:11 jforrester@deploy1001: Synchronized dblists/: T213504: Finally, drop the wikidatarepo dblist (duration: 00m 45s)
  • 18:09 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T213504: Stop telling CommonsSettings about the wikidatarepo dblist (duration: 00m 45s)
  • 18:06 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T213504: Unconfigure the wikidatarepo dblist (duration: 00m 46s)
  • 18:05 XioNoX: manually rotate log file wtmp on csw2-esams
  • 18:02 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T213504: Configure wikibaserepo dblist just like the wikidatarepo one (duration: 00m 46s)
  • 17:58 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T213504: Tell CommonSettings about the new wikibaserepo dblist (duration: 00m 47s)
  • 17:56 jforrester@deploy1001: Synchronized dblists/wikibaserepo.dblist: T213504: Create the new wikibaserepo dblist (duration: 00m 47s)
  • 17:25 papaul: powering down thumbor2002 for disk replacement
  • 17:10 XioNoX: revert ospf metrics to normal values on esams-eqiad Level3 link
  • 16:50 Lucas_WMDE: deployed patch for T212118
  • 12:41 Lucas_WMDE: EU SWAT done
  • 12:40 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix Wikidata base URI in client config (T198946) (duration: 00m 46s)
  • 12:34 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Populate wmgWikibaseRepoSpecialSiteLinkGroups for commonswiki (T213975) (duration: 00m 51s)
  • 11:04 moritzm: installing ghostscript security updates
  • 08:48 jynus: fixing dbstore1002 x1 replication
  • 07:56 vgutierrez: uploaded certcentral 0.8 to apt.wikimedia.org (stretch) - T209980 T213820 T213301

2019-02-03

  • 20:25 elukey: powercycle mw1272 - no ssh, no tty available via com2 - DIMM correctable errors + OEM errors registered in getsel
  • 18:56 elukey: started a tmux session on dbstore1002 to migrate all the tokudb tables of mediawikiwiki to InnoDB - (s3 replication broken)
  • 17:53 elukey: start all slaves on dbstore1002 (After a crash + recovery) + moved mediawikiwiki.revision_actor_temp to Innodb to unblock s3 slave replication (still broken though)
  • 04:55 legoktm@deploy1001: Synchronized wmf-config/extension-list: Remove WikibaseQuality from extensions-list (T208499) (duration: 00m 51s)
  • 01:10 elukey: powercycle mw1299 - can't ssh nor get a tty via console - racadm getsel shows "An OEM diagnostic event occurred."

2019-02-02

  • 20:42 chaomodus: restarted pdfrender on scb1003
  • 20:41 chaomodus: restarted pdfrender on scb1004
  • 20:06 chaomodus: parsoid was failed on scandium and alerting, the service parsoid-vd was restarted and appears to have come back
  • 05:44 jforrester@deploy1001: Synchronized php-1.33.0-wmf.14/extensions/VisualEditor/lib/ve/src/ui/dialogs/ve.ui.FindAndReplaceDialog.js: b/src/ui/dialogs/ve.ui.FindAndReplaceDialog.js T214963 Hot-deploy VE fix to stop hitting user pref writes without debounce (duration: 01m 02s)

2019-02-01

  • 23:16 vgutierrez: restart pdfrender on scb1004
  • 21:57 ejegg: updated payments-wiki-staging from 7767c7027e to 52a271e681
  • 21:25 ejegg: updated payments-wiki-staging to fundraising/REL1_31 branch
  • 07:13 bawolff_: reset 2FA on wikitech for User:Cicalese


Archives

See Server admin log/Archives.