Server Admin Log

From Wikitech
(Redirected from Server admin log)
Jump to navigation Jump to search

2018-08-21

  • 21:00 moritzm: rebooting mw2232 for some tests
  • 20:17 herron: re-enabling puppet agents after puppetmaster apache update
  • 20:11 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Group0 to 1.32.0-wmf.18
  • 20:07 herron: temporarily disabling puppet agents during puppetmaster apache updates
  • 19:49 thcipriani@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.18 and rebuild l10n cache (duration: 34m 54s)
  • 19:14 thcipriani@deploy1001: Started scap: testwiki to php-1.32.0-wmf.18 and rebuild l10n cache
  • 19:11 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.13 (duration: 01m 41s)
  • 19:08 mutante: deploy1001 - chmod -R g+w /srv/mediawiki-staging/php-1.32.0-wmf.13/.git
  • 18:58 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.12 (duration: 03m 15s)
  • 18:52 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.10 (duration: 07m 37s)
  • 18:17 volans: restarting ircecho on einsteinium - T202314
  • 17:58 volans: restarting ircecho on einsteinium - T202314
  • 17:13 thcipriani: starting branch cut for 1.32.0-wmf.18
  • 16:32 XioNoX: enabling ae1 on asw-a-eqiad then ae1 on cr1-eqiad - T202075
  • 16:27 godog: upgrade hp raid firmware on ms-be2020 - T141756
  • 16:21 XioNoX: disable ae1 on asw-a-eqiad - T202075
  • 16:18 XioNoX: disable ae1 on cr1-eqiad - T202075
  • 16:17 papaul: replacing PEM2 on cr2-codfw
  • 15:38 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096 (duration: 00m 50s)
  • 15:24 ejegg: restarted CiviCRM deduplication jobs
  • 15:10 moritzm: installing apache updates from stretch 9.5 point release
  • 14:58 moritzm: rolling restart of proton* to pick up nodejs security update
  • 13:44 jynus: stop db1096 (both s5 an s6) for upgrade
  • 13:36 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096 (duration: 00m 49s)
  • 13:27 jynus: stop db2090 for upgrade
  • 12:38 ema: power-cycle ms-be2020: multiple alerts, no ssh access, I/O errors in console
  • 12:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 (duration: 00m 50s)
  • 12:31 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 (duration: 00m 50s)
  • 12:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 49s)
  • 12:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 (duration: 00m 49s)
  • 12:13 moritzm: rolling restart of services in scb/eqiad to pick up the nodejs security update
  • 12:05 jynus: stop es1015 for upgrade
  • 11:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1015 (duration: 00m 50s)
  • 11:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 (duration: 00m 50s)
  • 11:16 zeljkof: EU SWAT finished
  • 11:16 zfilipin@deploy1001: Synchronized wmf-config/Wikibase-production.php: SWAT: Update several Wikidata-related configs (duration: 00m 50s)
  • 11:05 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add IP ranges for ptWiki editathon (T202354) (duration: 00m 52s)
  • 09:58 moritzm: installing libxcursor security updates
  • 09:46 moritzm: installing libxml2 security updates on trusty
  • 09:27 moritzm: installing mutt security updates
  • 09:16 moritzm: installing jetty9 security updates
  • 08:37 marostegui: Rename blob_tracking and blob_orphans on db1089 - T59186
  • 07:55 marostegui: Slowly reload haproxies on dbproxy10XX - T201021
  • 06:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 (duration: 00m 50s)
  • 05:12 marostegui: Stop puppet on dbproxy1006 to do some haproxy logging tests - https://phabricator.wikimedia.org/T201021
  • 05:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 (duration: 00m 51s)
  • 04:54 marostegui: Set max_connections back from 800 to 500 on db1073 - T188589
  • 02:47 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: rm unused Id5f5295d (duration: 00m 50s)
  • 02:43 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: rm unused I741b16452 (duration: 00m 56s)
  • 02:39 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 21 02:39:12 UTC 2018 (duration 10m 18s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 08m 32s)
  • 01:35 aaron@deploy1001: Synchronized wmf-config/mc.php: Enable broadcasted mcrouter operations for all wikis (duration: 00m 51s)
  • 00:07 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@9fbf9fa]: Don't clean up the queue size counters (duration: 00m 46s)
  • 00:07 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@9fbf9fa]: Don't clean up the queue size counters

2018-08-20

  • 23:09 twentyafterfour: Finished evening SWAT
  • 23:08 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/454083/ Phab Task: T202065 (duration: 00m 51s)
  • 23:06 twentyafterfour: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/454083/ deployed to mwdebug1001
  • 22:53 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@c947567]: Use timing for queue sizes as a workaround to grafana not plotting gauges (duration: 00m 45s)
  • 22:52 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@c947567]: Use timing for queue sizes as a workaround to grafana not plotting gauges
  • 22:29 eileen: civicrm revision changed from 3f6dd6c9a6 to 8910c34502, config revision is d787269882
  • 22:27 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@541932a]: Make the queue sized report stats once a second T202107 (duration: 00m 46s)
  • 22:26 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@541932a]: Make the queue sized report stats once a second T202107
  • 22:13 mutante: ganeti1001 - creating 3 new VMs for analytics-tools
  • 21:08 eileen: civicrm revision is 3f6dd6c9a6, config revision is d787269882
  • 21:07 fdans@deploy1001: Finished deploy [analytics/refinery@f59ce0c]: deploying changes to virtualpageview spam site filtering (duration: 10m 17s)
  • 20:56 fdans@deploy1001: Started deploy [analytics/refinery@f59ce0c]: deploying changes to virtualpageview spam site filtering
  • 20:43 aaron@deploy1001: Synchronized wmf-config/mc.php: Set "cluster" parameter for mcrouter broadcast routing prefixes (duration: 00m 50s)
  • 20:36 arlolra: Updated Parsoid to 129d71f (T130224, T199926)
  • 20:35 mbsantos@deploy1001: Finished deploy [mobileapps/deploy@cae24fe]: Update mobileapps to 95e976d (T202105) (duration: 06m 19s)
  • 20:29 mbsantos@deploy1001: Started deploy [mobileapps/deploy@cae24fe]: Update mobileapps to 95e976d (T202105)
  • 20:26 arlolra@deploy1001: Finished deploy [parsoid/deploy@44aa5e8]: Updating Parsoid to 129d71f (duration: 11m 29s)
  • 20:14 arlolra@deploy1001: Started deploy [parsoid/deploy@44aa5e8]: Updating Parsoid to 129d71f
  • 19:35 jforrester@deploy1001: Finished scap: i18n sync for WikimediaMessages clean-up following T202095 (duration: 31m 12s)
  • 19:04 jforrester@deploy1001: Started scap: i18n sync for WikimediaMessages clean-up following T202095
  • 19:00 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/WikimediaMessages/: SWAT T202095 Reinstate "Rename global OTRS-member group to otrs-member" (will need scap sync later) (duration: 00m 49s)
  • 18:56 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/TimedMediaHandler/: SWAT T202208Fix regression: double-reg of file types and incorrect mp4 (duration: 00m 52s)
  • 18:44 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T201783 T202014 Add Contact link to footers of fr,ruwiki (duration: 00m 50s)
  • 18:36 gehel@deploy1001: Finished deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater (duration: 10m 25s)
  • 18:32 jforrester@deploy1001: Synchronized wmf-config/throttle.php: SWAT T202003 Add throttle exemption for 24 August (duration: 00m 51s)
  • 18:31 volans: restarted ircecho on einsteinium
  • 18:26 gehel@deploy1001: Started deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater
  • 18:25 gehel@deploy1001: Finished deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 32s)
  • 18:24 gehel@deploy1001: Started deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater (wdqs1009 only)
  • 18:05 volans: restarting icinga, re-occurrence of T196336
  • 17:44 chasemp: install iotop on stat1004
  • 17:08 XenoRyet: updated payments-wiki from 360bea8331 to 268539c2bd
  • 16:59 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@4989231]: Create metrics to track the actual concurrent job executions T202107 (duration: 00m 55s)
  • 16:58 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@4989231]: Create metrics to track the actual concurrent job executions T202107
  • 14:14 moritzm: rolling upgrade of scb in codfw to latest nodejs security update
  • 13:46 moritzm: installing jetty9 security updates
  • 13:00 marostegui: Increase max_connections from 500 to 800 on db1073 to triage issues - T188589
  • 12:51 jynus: reload dbproxy1005
  • 12:29 moritzm: upgrading scb2006 to latest nodejs along with rolling restarts of node-based services
  • 12:02 zeljkof: EU SWAT finished
  • 12:01 arturo: T202261 extend icinga downtime 1D for cloudcontrol1003.wikimedia.org, cloudcontrol1004.wikimedia.org, clounet1003.eqiad.wmnet, cloudnet1004.eqiad.wmnet neutron not properly syncing with agents
  • 11:56 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Adding Chinese Wikiversitys logos (T202127) (duration: 00m 50s)
  • 11:50 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos for lfnwiki (T202228) (duration: 00m 50s)
  • 11:41 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for lfnwiki (T202228) (duration: 00m 50s)
  • 11:36 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Rollbacker User Group at ru.wikiquote (T200201) (duration: 00m 53s)
  • 11:25 arturo: T202261 icinga downtime 1h for cloudcontrol1003.wikimedia.org, cloudcontrol1004.wikimedia.org, clounet1003.eqiad.wmnet, cloudnet1004.eqiad.wmnet previous to patch merge
  • 11:07 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow all bureaucrats to remove interface-admin (duration: 00m 54s)
  • 11:01 arturo: T202261 disabled puppet in cloudcontrol1003.wikimedia.org, cloudcontrol1004.wikimedia.org, clounet1003.eqiad.wmnet, cloudnet1004.eqiad.wmnet
  • 10:31 moritzm: rebooting mw1261 for kernel update/some tests
  • 10:23 moritzm: rebooting mw2234 for some kernel tests
  • 10:07 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf2 to apt.wikimedia.org
  • 09:31 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf2+jessie to apt.wikimedia.org
  • 08:50 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request (duration: 06m 29s)
  • 08:43 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request
  • 08:43 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request - T201974 (duration: 18m 17s)
  • 08:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 (duration: 00m 50s)
  • 08:31 moritzm: installing systemd updates from stretch 9.5 point release
  • 08:25 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request - T201974
  • 08:24 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request (duration: 06m 02s)
  • 08:18 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request
  • 08:16 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request - T201974 (duration: 04m 16s)
  • 08:15 marostegui: Deploy schema change on db1100 to check for regressions
  • 08:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 50s)
  • 08:12 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request - T201974
  • 08:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db1100 (duration: 01m 02s)
  • 07:29 gehel: resetting management card on elastic1022
  • 06:57 volans: reset-failed debmonitor failed session on ms-be2024 ^^^^
  • 02:47 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 20 02:47:17 UTC 2018 (duration 10m 22s)
  • 02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 14m 23s)

2018-08-17

  • 16:24 gehel: disabling systemd state check for elastic eqiad until T202120 is fixed
  • 14:38 moritzm: rebooting elnath for some kernel tests
  • 13:02 jynus: stopping db2084 for upgrade
  • 12:49 imarlier@deploy1001: Finished deploy [performance/coal@aff3793]: (no justification provided) (duration: 01m 06s)
  • 12:48 imarlier@deploy1001: Started deploy [performance/coal@aff3793]: (no justification provided)
  • 12:46 gehel: rebooting relforge for JVM and kernel upgrade
  • 12:36 moritzm: installing openjdk-8 security updates on elastic* in codfw (eqiad already has it via the recent reimage to stretch)
  • 12:30 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097 (duration: 00m 53s)
  • 12:07 gehel: repooling wdqs2003, catched up on updates
  • 10:58 moritzm: rebooting mw2152-mw2162 for kernel security update (also bundling wikidiff and apache updates)
  • 10:35 moritzm: powercycling mw2286
  • 09:47 moritzm: rebooting mw2243-mw2290 for kernel security update (also bundling wikidiff and apache updates)
  • 09:03 gehel: depool wdqs2003 to catch up on updates
  • 08:29 jynus: stopping db2073 for upgrade
  • 08:13 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097 (duration: 00m 54s)
  • 08:06 gehel: silencing systemd check as mjolnir is flapping
  • 07:59 gehel: restarting mjolnir on all elastic / cirrus nodes
  • 07:42 jynus: stopping db2085 for upgrade
  • 07:41 moritzm: rebooting mw2220-mw2242 for kernel security update (also bundling wikidiff and apache updates)
  • 00:42 krinkle@deploy1001: Synchronized wmf-config/: rm StartProfiler.php / Ia83751 / T201782 (duration: 00m 50s)
  • 00:39 krinkle@deploy1001: Synchronized docroot/noc/: Ia83751 (duration: 00m 50s)

2018-08-16

  • 23:41 ejegg: re-activated CiviCRM scheduled jobs
  • 23:34 catrope@deploy1001: Synchronized php-1.32.0-wmf.16/skins/MinervaNeue/resources/skins.minerva.content.styles.images/magnifying-glass.svg: Correct MinervaNeue search icon (T199000) (duration: 00m 51s)
  • 23:18 ejegg: disabled CiviCRM scheduled jobs for version upgrade
  • 23:15 eileen: civicrm revision changed from b60ceb3d2f to 3f6dd6c9a6, config revision is 74ae910b0e
  • 22:32 XioNoX: re-activating BGP sessions between cr1/2-ulsfo and the office's router2
  • 22:25 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@e70f9d5]: retry git fat init fail (duration: 00m 14s)
  • 22:24 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@e70f9d5]: retry git fat init fail
  • 22:24 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@e70f9d5]: dont override the spark sharelib (duration: 00m 11s)
  • 22:24 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@e70f9d5]: dont override the spark sharelib
  • 21:56 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@651904b]: use spark 2.3.0, oozie still doesnt like 2.3.1 (duration: 00m 16s)
  • 21:56 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@651904b]: use spark 2.3.0, oozie still doesnt like 2.3.1
  • 21:56 ebernhardson@deploy1001: deploy aborted: try again after git-fat init fail (duration: 00m 01s)
  • 21:55 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: try again after git-fat init fail
  • 21:51 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: try again after git-fat init fail (duration: 00m 18s)
  • 21:50 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: try again after git-fat init fail
  • 21:46 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0 (duration: 04m 56s)
  • 21:44 mutante: rebooting cobalt (gerrit.wikimedia.org) for kernel upgrade
  • 21:41 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0
  • 21:39 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0 (duration: 00m 14s)
  • 21:39 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0
  • 21:29 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.1 (duration: 00m 18s)
  • 21:28 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.1
  • 21:23 thcipriani: contint1001 - installing jenkins upgrade
  • 21:08 mutante: contint2001 - installing jenkins upgrade
  • 21:05 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail (duration: 00m 16s)
  • 21:04 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail
  • 21:01 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail (duration: 00m 06s)
  • 21:01 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail
  • 21:00 mutante: releases1001 - installing package upgrade like on releases2001 before, scheduling downtime, reboot
  • 20:53 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: (no justification provided) (duration: 00m 02s)
  • 20:53 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: (no justification provided)
  • 20:50 mutante: releases2001 - rebooting
  • 20:48 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x (duration: 00m 46s)
  • 20:48 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x
  • 20:47 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x (duration: 01m 33s)
  • 20:46 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x
  • 20:38 mutante: releases2001: upgrading openjdk, systemd, jenkins
  • 20:33 mutante: releases1001/2001: upgrading apache2 packages
  • 20:26 mutante: gerrit2001 - scheduled downtime, rebooting
  • 19:54 mutante: phab1002 closing idle root screen that was used for rsyncing repos
  • 19:15 thcipriani: clearing gerrit accounts cache
  • 19:11 twentyafterfour: twentyafterfour and thcipriani performing online maintenance on gerrit All-Users repo
  • 19:10 legoktm: T201314 mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'EricEnfermero' 'Larry Hockett' --ignorestatus
  • 19:06 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@9fb53c4]: (no justification provided) (duration: 00m 17s)
  • 19:05 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@9fb53c4]: (no justification provided)
  • 18:57 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@040690e]: coordinator.properties should reference a coordinator, not bundle (duration: 00m 18s)
  • 18:56 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@040690e]: coordinator.properties should reference a coordinator, not bundle
  • 18:52 arlolra: Updated Parsoid to dbbad6a (T201115)
  • 18:47 arlolra@deploy1001: Finished deploy [parsoid/deploy@59f6585]: Updating Parsoid to dbbad6a (duration: 09m 51s)
  • 18:45 gehel: reimage of elasticsearch eqiad completed - T198391 / T193649
  • 18:37 arlolra@deploy1001: Started deploy [parsoid/deploy@59f6585]: Updating Parsoid to dbbad6a
  • 18:25 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5d87cc0]: now without the shebang (duration: 00m 17s)
  • 18:25 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5d87cc0]: now without the shebang
  • 18:22 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES filters in PageTriage on testwiki (duration: 00m 50s)
  • 18:16 catrope@deploy1001: Synchronized wmf-config/throttle.php: Throttle exemptions for cswiki (T202038) (duration: 00m 53s)
  • 18:14 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@0a704b6]: push new python dependency handling (duration: 03m 49s)
  • 18:10 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@0a704b6]: push new python dependency handling
  • 18:06 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@d994cb9]: push new python dependency handling (duration: 00m 05s)
  • 18:06 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@d994cb9]: push new python dependency handling
  • 17:59 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@122080c]: push new python dependency handling (duration: 00m 20s)
  • 17:59 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@122080c]: push new python dependency handling
  • 17:57 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@fec00bc]: Push updated transfer-to-es oozie job (duration: 00m 08s)
  • 17:57 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@fec00bc]: Push updated transfer-to-es oozie job
  • 17:36 gehel: reimaging elastic1029
  • 16:12 gehel: banning, depooling and shutting down elastic1029 for memory replacement - T201991
  • 15:55 jynus: stopping db2034 for upgrade
  • 15:53 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2019 (duration: 00m 51s)
  • 15:34 jynus: stopping es2019 for upgrade
  • 15:29 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2018, depool es2019 (duration: 00m 50s)
  • 15:16 bblack: restarting pybal on lvs1016 - T201694
  • 15:13 cmjohnson1: lvs1016 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 15:09 bblack: stopping pybal on lvs1016 to fail traffic to lvs1006 for T201694
  • 15:07 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp107[78]\.eqiad\.wmnet
  • 15:05 cmjohnson1: cp107[78] moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 15:05 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp107[78]\.eqiad\.wmnet
  • 15:04 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp107[56]\.eqiad\.wmnet
  • 15:03 cmjohnson1: cp1076 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 15:02 cmjohnson1: cp1075 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 15:01 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp107[56]\.eqiad\.wmnet
  • 15:01 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp107[56]\.eqiad\.wmnet
  • 15:00 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp107[56]\.eqiad\.wmnet
  • 15:00 jynus: stopping es2018 for upgrade
  • 14:58 cmjohnson1: torrelay1001 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:57 cmjohnson1: ms-be1040 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:56 cmjohnson1: dbproxy1013 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:54 cmjohnson1: labstore1009 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:51 papaul: shutting down mw2184 for maintenance
  • 14:50 cmjohnson1: db1066 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:49 cmjohnson1: db1118 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:46 cmjohnson1: db1116 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:44 cmjohnson1: labstore1008 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:43 bblack@neodymium: conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org
  • 14:43 cmjohnson1: dbproxy1012 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:41 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@166eafa]: Update mobileapps to a808c9d (T201979) (duration: 06m 03s)
  • 14:40 cmjohnson1: dns1001 moving uplink from asw2-a eqiad to asw-a-eqiad
  • 14:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=dns1001.wikimedia.org
  • 14:38 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2018 (duration: 00m 55s)
  • 14:34 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@166eafa]: Update mobileapps to a808c9d (T201979)
  • 14:33 cmjohnson1: cloudelastic1001 moving uplink from asw2-a eqiad to asw2-a2
  • 14:27 cmjohnson1: lvs1015 moving cross connect from asw2-a2 to asw2-a5 T201694
  • 14:25 XioNoX: starting moving asw2-a-eqiad servers' uplinks for T201694
  • 14:22 gehel: repooling wdqs[12]003
  • 14:11 moritzm: rebooting labtestweb2001 for kernel security update
  • 13:54 moritzm: rebooting labtestvirt2003 for kernel security update
  • 13:46 moritzm: rebooting labtestservices2002/2003 for kernel security update
  • 13:31 moritzm: rebooting seaborgium for kernel security update
  • 13:15 moritzm: rebooting serpens for kernel security update
  • 13:05 gehel: restarting blazegraph on wdqs[12]003
  • 12:44 moritzm: upgrading wikidiff to 1.7.2 on labweb* (HHVM bytecode cache is pruned during update)
  • 12:08 arturo: T201473 install a new version of `prometheus-pdns-exporter` (0.3) into jessie-wikimedia, due to errors in the postinst script
  • 12:06 gehel: depooling wdqs[12]003 to catchup on updates
  • 12:00 moritzm: installing ruby2.3 security updates
  • 11:37 arturo: T201473 copy `prometheus-pdns-exporter` from trusty-wikimedia to jessie-wikimedia in reprepro
  • 11:34 Amir1: EU mid-day SWAT is done
  • 11:31 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.16/maintenance/populateChangeTagDef.php: SWAT: Add option to populateChangeTagDef not to update the count (duration: 00m 53s)
  • 11:12 jynus: stopping db2042 for maintenance T202051
  • 11:09 moritzm: upgrading wikidiff to 1.7.2 on mw1339-mw1348 (HHVM bytecode cache is pruned during update)
  • 11:03 arturo: manually delete glance rsync image cronjob from the glancesync user in labcontrol1001.wikimedia.org (leftover after glance merge in main/eqiad1)
  • 10:43 moritzm: upgrading wikidiff to 1.7.2 on mw1285-mw1290 and mw1312-mw1317 (HHVM bytecode cache is pruned during update)
  • 10:30 _joe_: restarting changeprop on scb1002, not listening on its tcp port
  • 10:27 _joe_: restarting cpjobqueue on scb1002, not listening on its tcp port
  • 10:00 volans: add jiji to the 'ops' LDAP group - T201849
  • 09:46 gehel: all elasticsearch nodes reimaged (except elastic1029, waiting on memory issue) - T198391 / T193649 / T201991
  • 09:16 gehel: reimaging elastic1022
  • 08:40 moritzm: upgrading wikidiff to 1.7.2 on mw1319-mw1333 (HHVM bytecode cache is pruned during update)
  • 08:37 ema: reboot cp2009 for kernel upgrade
  • 08:35 vgutierrez: Reimaging radon as spare system - T202040
  • 08:32 moritzm: uploaded jenkins 2.121.3 to apt.wikimedia.org (for jessie and stretch)
  • 08:16 moritzm: upgrading wikidiff to 1.7.2 on mw1334-mw1338/mw1307/mw1318/ (HHVM bytecode cache is pruned during update)
  • 07:49 gehel: reimaging elastic10(23|24)
  • 07:45 volans@deploy1001: Finished deploy [debmonitor/deploy@1f01fd1]: Release v0.1.8 (duration: 00m 31s)
  • 07:45 volans@deploy1001: Started deploy [debmonitor/deploy@1f01fd1]: Release v0.1.8
  • 07:27 moritzm: rebooting install1002 for kernel security update
  • 07:24 moritzm: rebooting install2002 for kernel security update
  • 05:12 _joe_: moving away corrupted commitlog file on aqs1007 cassandra-a instance, trying to restart it
  • 02:38 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Aug 16 02:38:46 UTC 2018 (duration 10m 17s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 08m 38s)

2018-08-15

  • 22:43 krinkle@deploy1001: Finished scap: rm StartProfiler.php - T201782 (duration: 31m 46s)
  • 22:33 XioNoX: update old eqiad recdns IPs in cr1/2-eqiad bgp group Anycast4 with dns1001/2
  • 22:11 krinkle@deploy1001: Started scap: rm StartProfiler.php - T201782
  • 21:57 krinkle@deploy1001: Synchronized php-1.32.0-wmf.10: rm StartProfiler.php - T201782 (duration: 03m 01s)
  • 21:48 krinkle@deploy1001: Synchronized wmf-config/: I173a02910c (duration: 00m 50s)
  • 21:46 krinkle@deploy1001: Synchronized scap/plugins/: I173a02910c (duration: 00m 50s)
  • 20:49 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I207421 (duration: 00m 57s)
  • 19:07 gehel: reimaging elastic10(18|19|20)
  • 17:20 XioNoX: Rollback: redirect ns2 to authdns1001 - T201608
  • 17:16 moritzm: reboot eeden for kernel security update
  • 17:07 XioNoX: redirect ns2 to authdns1001 - T201608
  • 17:00 moritzm: reboot radon for kernel security update
  • 16:50 XioNoX: redirect ns0 to authdns1001 - T201608
  • 16:42 XioNoX: rollback: redirect ns1 to radon - T201608
  • 16:35 moritzm: reboot authdns2001 for kernel security update
  • 16:27 XioNoX: redirect ns1 to radon - T201608
  • 14:58 gehel: reimaging elastic1028
  • 14:50 ejegg: disabled CiviCRM dedupe jobs
  • 14:47 ejegg: updated SmashPig free-standing deployment from 2736e41045 to 3603e191a3
  • 14:13 gehel: reimaging elastic102[567]
  • 13:47 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2184.codfw.wmnet
  • 13:35 moritzm: installing samba security updates
  • 12:51 Trey314159: reindexing Polish wikis on elastic@eqiad and elastic@codfw complete (T200037)
  • 12:22 moritzm: rebooting mw2190-mw2219 for kernel security update (also bundling wikidiff and apache updates)
  • 12:19 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2184.codfw.wmnet
  • 12:04 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.16/includes/changetags/ChangeTags.php: Swap SET and WHERE statements in ChangeTags::undefineTag (T201934) (duration: 00m 55s)
  • 10:52 moritzm: rebooting mw2163-mw2189 for kernel security update (also bundling wikidiff and apache updates)
  • 09:39 gehel: reimaging elastic103[012], this will trigger master re-election
  • 09:17 moritzm: rebooting mw2135-mw2147 for kernel security update (also bundling wikidiff and apache updates)
  • 08:59 moritzm: rebooting deployment-mediawiki-09
  • 08:14 gehel: masking cassandra-a instance on aqs1007 since it is flapping - T201986
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 15 02:45:47 UTC 2018 (duration 10m 24s)
  • 02:15 ejegg: updated payments-wiki from 6002d9edc0 to 46ed86f389
  • 00:12 catrope@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/PageTriage/: SWAT: PageTriage fixes (T199357, T201812, T201560, T201373, T201253) (duration: 00m 51s)

2018-08-14

  • 23:55 TimStarling: restarted populateContentTables.php on s2
  • 23:48 tzatziki: deleting three images for legal compliance
  • 23:39 tzatziki: Correction: Change email for User:Textorus
  • 23:38 tzatziki: Change password for User:Textorus
  • 23:20 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Enable CentralNotice EventLogging at a low sample rate (0.01) (duration: 00m 50s)
  • 23:20 fdans@deploy1001: Finished deploy [analytics/refinery@21e07ae]: Deploying revert to prevent partition dropping jobs from failing (duration: 08m 27s)
  • 23:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable wgLegacyJavaScriptGlobals on all group0 wikis (T35837) (duration: 00m 54s)
  • 23:11 fdans@deploy1001: Started deploy [analytics/refinery@21e07ae]: Deploying revert to prevent partition dropping jobs from failing
  • 22:51 ejegg: updated payments-wiki from 7efaea7953 to 6002d9edc0
  • 21:19 jgleeson: payments-wiki changed from 6393e83ce0 to 7efaea7953, config revision is a5bce0fc89
  • 19:35 krinkle@deploy1001: Synchronized php-1.32.0-wmf.16/includes/cache/MessageCache.php: I6093113a / T201893 (duration: 00m 52s)
  • 19:11 mforns@deploy1001: Finished deploy [analytics/refinery@a4d1d99]: adding hashing to EL whitelist (duration: 13m 04s)
  • 19:06 filippo@neodymium: conftool action : set/pooled=no; selector: name=logstash1008,service=gelf
  • 18:58 mforns@deploy1001: Started deploy [analytics/refinery@a4d1d99]: adding hashing to EL whitelist
  • 18:44 XioNoX: delete peer 4589 on cr2-esams (no more direct peering)
  • 18:27 XioNoX: renumber v4 IP of 8560 on cr1-eqord
  • 18:20 mforns@deploy1001: Finished deploy [analytics/refinery@cb57843]: corresponding to refinery-source v0.0.70 (duration: 14m 27s)
  • 18:15 XioNoX: bump PCCW max accepted prefixes on cr2-eqiad
  • 18:13 XioNoX: bump PCCW max accepted prefixes on cr2-esams
  • 18:10 XioNoX: re-activate peer 13285 on cr2-ulsfo
  • 18:07 XioNoX: update NTP servers on pfw
  • 18:05 mforns@deploy1001: Started deploy [analytics/refinery@cb57843]: corresponding to refinery-source v0.0.70
  • 18:02 ejegg: re-enabled CiviCRM Omnimail import jobs
  • 18:00 volans@deploy1001: Finished deploy [netbox/deploy@eae2c9d]: Fix broken dependency (duration: 00m 33s)
  • 17:59 Trey314159: reindexing Polish wikis on elastic@eqiad and elastic@codfw (T200037)
  • 17:59 volans@deploy1001: Started deploy [netbox/deploy@eae2c9d]: Fix broken dependency
  • 17:22 XioNoX: configuring eqiad A switch ports for T201694
  • 17:19 gehel: restarting elasticsearch on elastic1050 (high load)
  • 16:57 gehel: reimage of elastic103[345]
  • 16:18 volans@deploy1001: Finished deploy [netbox/deploy@e2fd41d]: Security upgrade of dependency (duration: 00m 34s)
  • 16:17 volans@deploy1001: Started deploy [netbox/deploy@e2fd41d]: Security upgrade of dependency
  • 16:01 volans@deploy1001: Finished deploy [netbox/deploy@792d4d5]: Security upgrade of dependency (duration: 01m 23s)
  • 15:59 volans@deploy1001: Started deploy [netbox/deploy@792d4d5]: Security upgrade of dependency
  • 15:39 volans@deploy1001: Finished deploy [netbox/deploy@792d4d5]: Security upgrade of dependency (duration: 01m 03s)
  • 15:38 volans@deploy1001: Started deploy [netbox/deploy@792d4d5]: Security upgrade of dependency
  • 15:01 moritzm: rebooting meitnerium/archiva.wikimedia.org for kernel security update
  • 14:59 ema: cache_text eqiad: restart varnish-be without transient storage caps
  • 14:53 moritzm: rebooting cloudelastic* for kernel update
  • 14:50 gehel: reimage of elastic103[678]
  • 14:41 bstorm@deploy1001: Finished deploy [striker/deploy@13da520]: (no justification provided) (duration: 01m 13s)
  • 14:40 bstorm@deploy1001: Started deploy [striker/deploy@13da520]: (no justification provided)
  • 14:35 Krinkle: Copying xenon/logs/daily/2018-*{all,load,index,api,RunSingleJob}.log from mwlog1001 to webperfX002 hosts
  • 14:26 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1102 with full weight (duration: 00m 52s)
  • 14:21 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1089.eqiad.wmnet,service=varnish-be
  • 14:05 ema: restart varnish-be on cp108[79], fetch failures after child crashes
  • 13:47 gehel: restarting elasticsearch on elastic1051 (overloaded)
  • 13:21 gehel: restarting elasticsearch on elastic1043 (overloaded)
  • 13:11 moritzm: upgrading wikidiff to 1.7.2 on mw1308-mw1311/mw1293-mw1296 (HHVM bytecode cache is pruned during update)
  • 12:57 gehel: restarting tilerator on maps eqiad
  • 12:54 Trey314159: reindexing Indonesian wikis on elastic@eqiad and elastic@codfw complete (T200204)
  • 12:49 moritzm: upgrading wikidiff to 1.7.2 on snapshot hosts
  • 11:58 moritzm: upgrading wikidiff to 1.7.2 on mw1276-mw1283 (HHVM bytecode cache is pruned during update)
  • 11:40 TimStarling: restarted populateContentTables.php on s2 (T183488)
  • 11:20 zeljkof: EU SWAT finished
  • 11:19 moritzm: upgrading wikidiff to 1.7.2 on mw1299-mw1306 (HHVM bytecode cache is pruned during update)
  • 11:19 zfilipin@deploy1001: Synchronized vendor/perftools/xhgui-collector/external/header.php: SWAT: Fix "the the" typo in vendor/perftools/xhgui-collector/external/header.php (T201491) (duration: 00m 49s)
  • 11:17 zfilipin@deploy1001: Synchronized wmf-config/CirrusSearch-common.php: SWAT: Fix the the typo in wmf-config/CirrusSearch-common.php (T201491) (duration: 00m 51s)
  • 11:14 moritzm: repooled mw1233 (debugging completed)
  • 10:45 moritzm: repooled mw1227 (was probably overlooked to repool after previous debugging)
  • 10:41 volans: upgraded cumin on labpuppetmaster* to fix cumin with the new openstack region - T201881
  • 10:16 moritzm: upgrading wikidiff to 1.7.2 on mw1221-mw1235 (HHVM bytecode cache is pruned during update)
  • 10:12 volans: uploaded cumin_3.0.2-2_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 09:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1102 with low load (duration: 00m 50s)
  • 09:32 jynus: stop and restart db1122 for maintenance
  • 09:27 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1102 (duration: 00m 52s)
  • 09:23 addshore: for i in {10000..12500}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
  • 09:20 moritzm: upgrading wikidiff to 1.7.2 on mw1238-mw1258 (HHVM bytecode cache is pruned during update)
  • 09:17 addshore: for i in {6000..10000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
  • 09:13 addshore: for i in {3000..6000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
  • 09:08 addshore: for i in {1000..3000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
  • 09:07 addshore: for i in {1..1000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
  • 08:51 addshore: addshore@labweb1001:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki GoranSMilovanovic # T201122
  • 08:19 moritzm: upgrading wikidiff to 1.7.2 on mw1266-mw1275 (HHVM bytecode cache is pruned during update)
  • 07:56 moritzm: installing ghostscript security updates
  • 07:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:s7 and db1101:s8 (duration: 00m 52s)
  • 06:45 _joe_: rolling restart of hhvm in eqiad api, high load
  • 06:37 _joe_: depooling mw1233 from live traffic for debugging
  • 05:13 TimStarling: killed populateContentTables.php for s2 at jcrespo's request
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 14 02:45:46 UTC 2018 (duration 10m 25s)
  • 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 13m 57s)
  • 00:35 aaron@deploy1001: Synchronized wmf-config/mc.php: Enable broadcasted mcrouter cache operations for test wikis and mw.org (duration: 00m 49s)
  • 00:15 mutante: graphite2002 - stopping carbon-local-relay, running puppet to start it again to confirm no issues with gerrit:448779
  • 00:09 aaron@deploy1001: Synchronized wmf-config/mc.php: Only do cache writes to mcrouter for all wikis (duration: 00m 52s)

2018-08-13

  • 23:26 urandom: Restart Cassandra in RESTBase dev env -- T201863
  • 23:24 godog: restart carbon-c-relay on graphite1001 to mirror traffic to graphite1004 - T196484
  • 23:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable wp10 and draftquality ORES models on testwiki (T198997) (duration: 00m 51s)
  • 22:32 bawolff@deploy1001: Synchronized php-1.32.0-wmf.16/resources/src/startup/mediawiki.js: backport I3cd2d7 (CSP fixes) (duration: 00m 50s)
  • 22:31 Trey314159: reindexing Indonesian wikis on elastic@eqiad and elastic@codfw (T200204)
  • 22:30 bawolff@deploy1001: Synchronized php-1.32.0-wmf.16/includes/OutputPage.php: backport I3cd2d7 (CSP fixes) (duration: 00m 50s)
  • 22:29 bawolff@deploy1001: Synchronized php-1.32.0-wmf.16/includes/ContentSecurityPolicy.php: backport I3cd2d7 (CSP fixes) (duration: 00m 51s)
  • 22:29 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw complete (T200204)
  • 21:30 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Deploy adjust CSP settings I74a79dc7defaa (duration: 00m 52s)
  • 21:29 mutante: elastic1049 - started ferm
  • 19:32 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw (T200204)
  • 18:47 otto@deploy1001: Finished deploy [analytics/refinery@c7f68b7]: camus - Add --check-java-opts and --check-emails-to option - T198908 (duration: 11m 10s)
  • 18:35 otto@deploy1001: Started deploy [analytics/refinery@c7f68b7]: camus - Add --check-java-opts and --check-emails-to option - T198908
  • 17:16 gehel@deploy1001: Finished deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI (duration: 10m 54s)
  • 17:15 thcipriani@deploy1001: Synchronized wmf-config/wikitech.php: wikitech.php: update keystone URLs (duration: 00m 53s)
  • 17:05 gehel@deploy1001: Started deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI
  • 17:02 gehel@deploy1001: Finished deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 30s)
  • 17:01 gehel@deploy1001: Started deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI (wdqs1009 only)
  • 15:41 otto@deploy1001: Finished deploy [analytics/refinery@a051125]: fix for T198908 (duration: 08m 06s)
  • 15:33 papaul: shutting down lvs2009 to disable on board NICs
  • 15:33 otto@deploy1001: Started deploy [analytics/refinery@a051125]: fix for T198908
  • 15:10 otto@deploy1001: Finished deploy [analytics/refinery@9006a4e]: refinery changes for T198908 (duration: 10m 30s)
  • 15:00 otto@deploy1001: Started deploy [analytics/refinery@9006a4e]: refinery changes for T198908
  • 14:51 moritzm: installing ghostscript security updates
  • 14:47 cmjohnson1: disabling puppet and shutting down db1052 for final decommissioning
  • 14:25 moritzm: installing dpkg updates from stretch point release
  • 14:10 arturo: T201504 disable keystone in main and eqiad1 deployments, all has been downtimed in icinga
  • 14:09 andrewbogott: stopping nodepool, downtiming horizon for T201504
  • 14:05 moritzm: upgrading mw1262-1265 to wikidiff 1.7.2 (T199801)
  • 12:56 gehel: start reimaging of elasticsearch / cirrus / eqiad cluster (RAID0 / Stretch) - T193649 / T198391
  • 12:56 gehel: reimaging of elasticsearch / cirrus / codfw cluster (RAID0 / Stretch) completed - T193649 / T198391
  • 12:53 moritzm: upgrading mw1261 servers to wikidiff 1.7.2 (T199801)
  • 12:28 moritzm: upgrading mwdebug* servers to wikidiff 1.7.2 (T199801)
  • 12:28 moritzm: upgrading mwdebug* servers to wikidiff 1.7.2
  • 12:24 moritzm: uploaded hhvm-wikidiff2 1.7.2 to apt.wikimedia.org (source package name is still php-wikidiff2 for historical reasons) (T199801)
  • 12:06 zeljkof: EU SWAT finished
  • 12:00 moritzm: rebooting cloudcontrol* for kernel security update
  • 11:58 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set mobile wordmark for pswikivoyage (T200152) (duration: 00m 52s)
  • 11:47 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Translate extension on oldwikisource (T201814) (duration: 00m 50s)
  • 11:42 moritzm: rebooting dbmonitor* hosts for kernel security update
  • 11:29 zfilipin@deploy1001: Synchronized static/images/mobile/copyright/: SWAT: Add missing PNG files in mobile logo folder (T200152) (duration: 00m 49s)
  • 11:05 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 11:04 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
  • 11:01 moritzm: rebooting kraz/irc.wikimedia.org for kernel security update
  • 10:29 jynus: upgrade and restart db2036
  • 08:42 jynus: upgrade and restart db1101
  • 08:39 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:s7 and db1101:s8 (duration: 00m 51s)
  • 07:16 legoktm@deploy1001: Synchronized php-1.32.0-wmf.16/maintenance/dumpTextPass.php: Add the 'full' option explicitly to dumpTextPass.php (T201803) (duration: 00m 58s)
  • 06:53 ema: finish up cache reboots for kernel updates: cp5001.eqsin.wmnet,cp3035.esams.wmnet,cp[4021-4022].ulsfo.wmnet
  • 06:46 oblivian@neodymium: conftool action : set/pooled=inactive; selector: name=restbase2003.codfw.wmnet
  • 06:34 _joe_: powercycling restbase2003, in kernel panic
  • 02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 13 02:46:41 UTC 2018 (duration 10m 28s)
  • 02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 14m 21s)
  • 01:45 TimStarling: on mwmaint1001 running populateContentTables.php on all wikis using ~tstarling/pct-list T183488

2018-08-12

  • 15:43 gehel: full cache invalidation of maps tiles - T201772
  • 06:15 ejegg: stopped CiviCRM omnimail jobs

2018-08-10

  • 21:50 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/UploadWizard: T201708 UBN fix (e498c7d) (duration: 00m 56s)
  • 18:45 bblack: restart varnish-be on cp1087
  • 18:44 bblack: restart varnish-be on cp1079
  • 18:44 bblack: restart varnish-be on cp1075
  • 18:41 bblack: restart varnish-be on cp1081
  • 17:49 jgleeson: Released config update containing Google API Key (37a1a28dfb)
  • 17:34 bblack: cp1085 + cp1089: restart varnish with wiped storage, repool
  • 17:04 ema: restart varnish-be on cp1083
  • 16:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1089.eqiad.wmnet
  • 16:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1085.eqiad.wmnet
  • 16:36 bblack: depool cp1089, cp1085
  • 16:12 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw abandoned (T200204)
  • 16:02 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw (T200204)
  • 14:48 ema: powercycle cp5005, stuck rebooting
  • 12:27 gehel: resetting management card on elastic2017 - T201671
  • 12:18 gehel: restarting icinga on einsteinium - T196336
  • 11:43 moritzm: installing busybox security updates
  • 11:38 moritzm: installing jansson security updates for trusty
  • 11:17 moritzm: installing ant security updates
  • 11:06 moritzm: installing libxcursor security updates
  • 09:35 mobrovac@deploy1001: Finished deploy [citoid/deploy@983d80c]: Remove the bibtex spec.yaml x-ample - T197242 (duration: 03m 04s)
  • 09:32 mobrovac@deploy1001: Started deploy [citoid/deploy@983d80c]: Remove the bibtex spec.yaml x-ample - T197242
  • 09:27 jynus: upgrade and restart db1125
  • 09:19 moritzm: rearmed keyholder on labpuppetmaster*
  • 09:12 moritzm: rebooting labpuppetmaster1002 for kernel security update/microcode update
  • 09:03 jynus: upgrade and restart db1124
  • 08:58 moritzm: rebooting labpuppetmaster1001 for kernel security update/microcode update
  • 08:29 jynus: upgrade and restart db2095
  • 08:28 arturo: rebuild python-pykube for stretch-wikimedia and add it to apt.wikimedia.org T200660
  • 08:03 jynus: upgrade and restart db2094
  • 07:57 ema: powercycle cp3040, kernel crash
  • 07:03 moritzm: installing mutt security updates

2018-08-09

  • 23:40 ejegg: updated payments-wiki from d78c98cf0f to 6393e83ce0
  • 23:33 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/451810/ (duration: 00m 51s)
  • 23:27 maxsem@deploy1001: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/451792/ (duration: 00m 51s)
  • 23:12 maxsem@deploy1001: Synchronized static/images/mobile/copyright/wikivoyage-wordmark-ps.svg: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/451791/ (duration: 00m 52s)
  • 22:31 XioNoX: disable fpc1-fpc3 - T201145
  • 22:09 bblack: re-enabling pybal on lvs1016
  • 21:31 XioNoX: disable fpc3-fpc4 - T201145
  • 21:21 XioNoX: disable fpc5-fpc6 - T201145
  • 21:21 bblack: stop pybal on lvs1016 (should move services to lvs1006)
  • 21:06 XioNoX: disable fpc4-fpc6 - T201145
  • 21:01 XioNoX: enable fpc5-fpc6 - T201145
  • 20:54 XioNoX: enable fpc3-fpc4 - T201145
  • 20:49 XioNoX: enable fpc1-fpc3 - T201145
  • 20:44 XioNoX: disable fpc3-fpc5 and fpc5-fpc7 - T201145
  • 20:31 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@8c6a7f8]: add python snappy library (duration: 07m 25s)
  • 20:23 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@8c6a7f8]: add python snappy library
  • 20:18 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1080.eqiad.wmnet
  • 19:16 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.16 refs T191062
  • 19:09 twentyafterfour: Preparing to deploy 1.32.0-wmf.16 to group2 wikis.
  • 19:02 herron: logstash1008:~# systemctl restart logstash T200960
  • 19:02 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add temporary debug logging to ThumbnailRenderJob T201305 (duration: 00m 57s)
  • 18:58 niharika29@deploy1001: Synchronized php-1.32.0-wmf.16/includes/jobqueue/jobs/ThumbnailRenderJob.php: Add temporary debug logging T201305 (duration: 01m 11s)
  • 18:49 bblack: remaining cache_misc sites will move to cache_text shortly - https://gerrit.wikimedia.org/r/451659
  • 18:29 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@39be980]: Repair daemons failing to load logging config (duration: 03m 33s)
  • 18:26 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@39be980]: Repair daemons failing to load logging config
  • 18:23 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@22a50af]: temp fix while waiting for jenkins (duration: 03m 35s)
  • 18:19 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@22a50af]: temp fix while waiting for jenkins
  • 18:17 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add media.farsnews.com to wgCopyUploadDomains T200872 (duration: 01m 00s)
  • 18:15 niharika29@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 01s)
  • 17:59 XioNoX: connecting asw2-a5-eqiad to asw2-a6-eqiad - T201145
  • 17:54 moritzm: installing gnupg security updates on trusty (Debian already fixed)
  • 17:52 awight@deploy1001: Finished deploy [ores/deploy@1712122]: fawiki wp10; ORES update: T201518 (duration: 31m 58s)
  • 17:35 mforns@deploy1001: Finished deploy [analytics/refinery@9f4267c]: deploy refinery together with refinery-source v0.0.68 (duration: 08m 22s)
  • 17:26 mforns@deploy1001: Started deploy [analytics/refinery@9f4267c]: deploy refinery together with refinery-source v0.0.68
  • 17:21 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@6581b28]: Update mobileapps to 616ffef (T191640) (duration: 06m 35s)
  • 17:20 awight@deploy1001: Started deploy [ores/deploy@1712122]: fawiki wp10; ORES update: T201518
  • 17:14 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@6581b28]: Update mobileapps to 616ffef (T191640)
  • 15:14 mobrovac@deploy1001: Finished deploy [citoid/deploy@92e5071]: Record format stats (duration: 17m 07s)
  • 14:57 mobrovac@deploy1001: Started deploy [citoid/deploy@92e5071]: Record format stats
  • 14:52 moritzm: rebooting dns2002 for kernel security update/enabling microcode updates
  • 14:50 bblack: all cpNNNN: max tcp per-flow rate reducing 1Gbps -> 256Mbps - https://gerrit.wikimedia.org/r/c/operations/puppet/+/451535
  • 14:44 moritzm: rebooting dns2001 for kernel security update/enabling microcode updates
  • 14:34 moritzm: rebooting maerlant for kernel security update/enabling microcode updates
  • 14:28 herron: rolling reboot of mx servers for kernel updates
  • 14:26 moritzm: rebooting nescio for kernel security update/enabling microcode updates
  • 14:18 herron: rebooting fermium (lists) to switch out of -rt kernel variant
  • 14:12 moritzm: rebooting dns4002 for kernel security update/enabling microcode updates
  • 13:55 moritzm: rebooting dns4001 for kernel security update/enabling microcode updates
  • 13:48 moritzm: rebooting dns5002 for kernel security update/enabling microcode updates
  • 13:28 moritzm: rebooting dns5001 for kernel security update/enabling microcode updates
  • 12:22 moritzm: rebooting ununpentium for kernel security update
  • 12:14 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Do not leak local $wgWBShared… variables to the global scope (duration: 00m 56s)
  • 12:13 moritzm: rebooting netmon1003 (servermon.wikimedia.org) for kernel security update
  • 12:10 moritzm: rearmed keyholder on netmon1002
  • 12:09 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable RDF export for lexicographical data (T201153) (duration: 00m 56s)
  • 12:06 moritzm: rebooting netmon1002 for kernel security update
  • 12:05 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Enable RDF export for lexicographical data (T201153) (duration: 00m 58s)
  • 12:05 moritzm: rearmed keyholder on netmon2001
  • 11:57 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos in hewikibooks (T201562) (duration: 00m 55s)
  • 11:50 moritzm: rebooting netmon2001 for kernel security update
  • 11:42 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Update logo for hewikibooks (T201562) (duration: 01m 00s)
  • 11:30 ema: resume rolling reboots of caches for numa_networking T193865
  • 11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Transwiki import in zhwikiversity (T201328) (duration: 00m 56s)
  • 11:19 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove noratelimit from epcoordinator group on cswiki (T201010) (duration: 00m 58s)
  • 11:07 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateStyles everywhere (T199909) (duration: 00m 58s)
  • 10:57 mobrovac: truncating commons and others mobile tables - T201103
  • 10:42 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 and db1123 fully (duration: 00m 59s)
  • 10:38 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2069 due to crash (duration: 01m 00s)
  • 10:38 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #5 - T201103 (duration: 06m 27s)
  • 10:31 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #5 - T201103
  • 10:31 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #4 - T201103 (duration: 12m 11s)
  • 10:31 moritzm: rebooting archiva1001 for kernel security update
  • 10:25 jynus: disabling alerts for db2069
  • 10:21 jynus: handling db2069 crash, no user impact
  • 10:19 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #4 - T201103
  • 10:19 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #3 - T201103 (duration: 04m 20s)
  • 10:14 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #3 - T201103
  • 10:14 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #2 - T201103 (duration: 08m 38s)
  • 10:05 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #2 - T201103
  • 10:03 moritzm: rebooting url downloaders for kernel security update (alsafi, actinium, aluminium, alcyone)
  • 10:02 mobrovac@deploy1001: Finished deploy [restbase/deploy@cb6b4b4]: Drop mobile-sections, feed and media end points from non-WPs - T201103 (duration: 04m 26s)
  • 09:57 mobrovac@deploy1001: Started deploy [restbase/deploy@cb6b4b4]: Drop mobile-sections, feed and media end points from non-WPs - T201103
  • 09:57 moritzm: rebooting dubnium for kernel security update
  • 09:56 mobrovac@deploy1001: deploy aborted: Drop mobile-sections, feed and media end points from non-WPs - T201103 (duration: 06m 14s)
  • 09:50 mobrovac@deploy1001: Started deploy [restbase/deploy@cb6b4b4]: Drop mobile-sections, feed and media end points from non-WPs - T201103
  • 09:42 moritzm: rebooting pollux for kernel security update
  • 09:24 moritzm: rebooting iron for kernel security update
  • 09:23 moritzm: reset racadm on iron, com2 was unresponsive
  • 09:08 moritzm: rebooting bast5001 for kernel security update
  • 09:02 moritzm: rebooting bast4002 for kernel security update
  • 08:57 moritzm: rebooting bast2001 for kernel security update
  • 08:37 moritzm: rebooting bast2002 for kernel security update
  • 07:26 ema: cp3032 mbox lagged: reboot w/ numa_networking
  • 05:52 kartik@deploy1001: Finished deploy [cxserver/deploy@957ff6a]: Update cxserver to 27813b6 (T201085) (duration: 03m 59s)
  • 05:48 kartik@deploy1001: Started deploy [cxserver/deploy@957ff6a]: Update cxserver to 27813b6 (T201085)
  • 05:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool pc2004 and pc2005 after BIOS upgrade - T201387 (duration: 01m 00s)
  • 03:23 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Aug 9 03:23:32 UTC 2018 (duration 10m 34s)
  • 03:12 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 15m 23s)
  • 02:38 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 16m 05s)
  • 02:06 twentyafterfour: restarted apache on phab1001 to hotfix antivandalism bug
  • 01:39 mutante: baham - installing BIOS upgrade (2.4.2 for Dell R320) - server is on role(spare) and the last that did not get the upgrade on T162850
  • 00:14 reedy@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/VisualEditor/: Bug fix (duration: 00m 58s)
  • 00:05 twentyafterfour: finished phabricator upgrade
  • 00:02 twentyafterfour: deploying phabricator release/2018-08-08/1

2018-08-08

  • 23:55 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: satwiki sitename (duration: 01m 05s)
  • 23:48 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: sq* collation (duration: 00m 56s)
  • 23:40 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: extension registration config for GWToolset (duration: 00m 56s)
  • 23:39 reedy@deploy1001: Synchronized wmf-config/extension-list: GWToolset to extension.json (duration: 00m 57s)
  • 23:27 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove old or same image config (duration: 00m 56s)
  • 23:22 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgVariantArticlePath for zhwikiversity (duration: 01m 05s)
  • 21:04 twentyafterfour: 1.32.0-wmf.16 appears to be stable, no noticeable increase in errors logged.
  • 20:59 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.16 refs T191062 (duration: 00m 57s)
  • 20:58 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.16 refs T191062
  • 20:49 James_F: Updated mobileapps to 162ebcf in Production.
  • 20:48 jforrester@deploy1001: Finished deploy [mobileapps/deploy@4ef02e1]: Update mobileapps to 162ebcf (duration: 07m 36s)
  • 20:40 jforrester@deploy1001: Started deploy [mobileapps/deploy@4ef02e1]: Update mobileapps to 162ebcf
  • 19:57 _joe_: powercycling cp1077, network card problems
  • 19:54 bblack: reboot cp1088 for network card
  • 19:53 bblack: reboot cp1090 for network card
  • 19:53 _joe_: powercycling cp1079, network card problems
  • 19:46 bblack: rebooting cp1084 to debug eth hardware
  • 19:35 XioNoX: fix interface description on asw-a-eqiad uplinks
  • 19:19 volans: completed forced puppet run where it was failed
  • 19:18 bblack: depooling front-edge traffic from eqiad in DNS - https://gerrit.wikimedia.org/r/c/operations/dns/+/451401
  • 19:11 volans: force puppet run where it's still failed
  • 19:01 twentyafterfour: Waiting to deploy the train until after I've established that network issues are resolved.
  • 18:23 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f0246f7]: Only rerender mobile-sections for wikipedia T201103 (duration: 01m 29s)
  • 18:22 ppchelko@deploy1001: Started deploy [changeprop/deploy@f0246f7]: Only rerender mobile-sections for wikipedia T201103
  • 17:55 herron: performing rolling restart of logstash instances T200960
  • 17:01 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/includes/logging/LogFormatter.php: SWAT T185049 Unbreak views of invalid title log entries, wmf.16 (duration: 00m 58s)
  • 17:00 jforrester@deploy1001: Synchronized php-1.32.0-wmf.15/includes/logging/LogFormatter.php: SWAT T185049 Unbreak views of invalid title log entries, wmf.15 (duration: 01m 01s)
  • 16:53 bblack: reimagine cp1071-4, cp1099
  • 16:20 bblack: reimaging cp1060, cp1062-8
  • 15:58 bblack: reimaging cp1050.eqiad.wmnet cp1052.eqiad.wmnet cp1053.eqiad.wmnet cp1054.eqiad.wmnet cp1055.eqiad.wmnet cp1059.eqiad.wmnet
  • 15:54 bblack: downtimed all ipsec checks :P
  • 15:45 bblack: reimaging cp1046-9
  • 14:55 jynus: switching over, stopping, upgrading and restarting labsdb1011
  • 13:34 papaul: BIOS update in progress on pc2005
  • 13:10 papaul: BIOS update in progress on pc2004
  • 12:53 jynus: switching over, stopping, upgrading and restarting labsdb1010
  • 12:00 jynus: testing new read only check on labsdb wikireplicas
  • 11:21 jynus: switching over, stopping, upgrading and restarting labsdb1009
  • 11:14 marostegui: Stop MySQL on pc2005 for BIOS upgrade - T201387
  • 11:14 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2005 - T201387 (duration: 00m 59s)
  • 11:02 ariel@deploy1001: Finished deploy [dumps/dumps@d6bd774]: fix up getConfiguration invocation (duration: 00m 04s)
  • 11:02 ariel@deploy1001: Started deploy [dumps/dumps@d6bd774]: fix up getConfiguration invocation
  • 10:27 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 and db1123 with low load after maint (duration: 00m 57s)
  • 10:11 ema: power-cycle cp2022, stuck rebooting
  • 09:08 ema: begin rolling reboots of caches for numa_networking T193865
  • 08:58 marostegui: Drop unused grants from 208.80.153.48
  • 08:44 marostegui: Drop unused grants from 208.80.154.12
  • 08:41 marostegui: Drop unused grants from 208.80.154.136
  • 08:03 jynus: stop db1123 for provisioning and upgrade
  • 07:55 jynus: stop db1100 for provisioning and upgrade
  • 07:47 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 and db1123 (duration: 00m 58s)
  • 07:27 marostegui: Stop MySQL on pc2004 - T201387
  • 06:17 kartik@deploy1001: Finished deploy [cxserver/deploy@6a0cab1]: Update cxserver to 951fdba (T199308, T199512, T199320, T200665, T200453, T106437) (duration: 03m 32s)
  • 06:13 kartik@deploy1001: Started deploy [cxserver/deploy@6a0cab1]: Update cxserver to 951fdba (T199308, T199512, T199320, T200665, T200453, T106437)
  • 05:23 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2004 - T201387 (duration: 00m 56s)
  • 05:12 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 01m 06s)
  • 03:12 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 8 03:12:43 UTC 2018 (duration 10m 26s)
  • 03:02 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 16m 06s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 08m 40s)
  • 00:11 ejegg: updated payments-wiki from 8cb7c86b12 to d78c98cf0f

2018-08-07

  • 23:46 ejegg: updated payments-wiki from 0c9c24d30c to 8cb7c86b12
  • 23:28 TimStarling: restarted populateContentTables.php for commonswiki, it died at rev_id 60164000
  • 22:14 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1048.eqiad.wmnet
  • 21:58 ejegg: updated CiviCRM from d626907f2c to b60ceb3d2f
  • 21:17 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.16 refs T191062
  • 21:07 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1049.eqiad.wmnet
  • 20:45 twentyafterfour@deploy1001: Finished scap: testwikis wikis to 1.32.0-wmf.16 refs T191062 (duration: 68m 20s)
  • 20:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1050.eqiad.wmnet
  • 20:34 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1062.eqiad.wmnet
  • 19:45 herron: upload prometheus-logstash-exporter_0.1.2-1 to apt.wikimedia.org/jessie-wikimedia/main
  • 19:45 herron: upload prometheus-logstash-exporter_0.1.2-1 to apt.wikimedia.org/stretch-wikimedia/main
  • 19:36 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.32.0-wmf.16 refs T191062
  • 19:30 andrewbogott: shutting down labnodepool1002 in advance of a rename. T201439
  • 19:09 otto@deploy1001: Finished deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (all nodes) (duration: 01m 51s)
  • 19:07 otto@deploy1001: Started deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (all nodes)
  • 19:07 otto@deploy1001: Finished deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (scb2001 only) (duration: 00m 20s)
  • 19:06 otto@deploy1001: Started deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (scb2001 only)
  • 19:03 twentyafterfour: Branching 1.32.0-wmf.16
  • 18:24 mutante: netbox - restored database from dump file - backed up and back-up (T190184)
  • 18:13 mutante: netbox - temp dropped databae to test restoring from dump
  • 18:02 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1063.eqiad.wmnet
  • 18:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet
  • 16:51 hoo: Deployed patch for T201418
  • 16:50 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@22a50af]: bump to master, prep for deploy to cirrus servers, take two (duration: 01m 51s)
  • 16:48 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@22a50af]: bump to master, prep for deploy to cirrus servers, take two
  • 16:47 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1052.eqiad.wmnet
  • 16:47 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet
  • 16:44 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@65e6bb9]: bump to master, prep for deploy to cirrus servers (duration: 02m 54s)
  • 16:41 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@65e6bb9]: bump to master, prep for deploy to cirrus servers
  • 15:55 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1019, db1122 and db1081 with full weight (duration: 00m 51s)
  • 15:26 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1053.eqiad.wmnet
  • 15:26 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1077.eqiad.wmnet
  • 14:58 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1054.eqiad.wmnet
  • 14:57 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1079.eqiad.wmnet
  • 14:57 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1064.eqiad.wmnet
  • 14:57 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1078.eqiad.wmnet
  • 14:26 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low load (duration: 00m 47s)
  • 14:18 herron: double again logstash jvm heap size to 1g and rolling restart logstash instances
  • 14:18 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1019 and db1102 with low load after maintenance (duration: 00m 52s)
  • 14:11 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1055.eqiad.wmnet,service=varnish-be
  • 14:11 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=varnish-be
  • 14:11 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1071.eqiad.wmnet,service=varnish-be
  • 14:11 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet,service=varnish-be
  • 13:42 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1055.eqiad.wmnet,service=varnish-fe
  • 13:42 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=varnish-fe
  • 13:41 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1071.eqiad.wmnet,service=varnish-fe
  • 13:41 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet,service=varnish-fe
  • 13:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1072.eqiad.wmnet
  • 13:39 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet
  • 13:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1065.eqiad.wmnet
  • 13:39 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet
  • 13:37 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1055.eqiad.wmnet,service=nginx
  • 13:37 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=nginx
  • 13:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1071.eqiad.wmnet,service=nginx
  • 13:36 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet,service=nginx
  • 13:10 otto@deploy1001: Started restart [analytics/aqs/deploy@6fafc63]: Bouncing AQS for mediawiki_history index update
  • 12:58 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=hydrogen.wikimedia.org
  • 12:58 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=chromium.wikimedia.org
  • 12:52 bblack: rebooting authdns1001
  • 12:44 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns1002.wikimedia.org
  • 12:41 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org
  • 11:55 marostegui: Drop unused grants repl@208.80.155.117
  • 11:19 zeljkof: EU SWAT finished
  • 11:18 mobrovac@deploy1001: Synchronized rpc/RunSingleJob.php: RunSingleJob: remove unneeded base64 decoding (duration: 00m 49s)
  • 11:17 marostegui: Remove unused repl grants for 10.64.0%
  • 11:07 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove hewiki interface-editor group (T200698) (duration: 00m 49s)
  • 10:51 jynus: shutdown and upgrade of db1081 (last message was wrong)
  • 10:51 jynus: shutdown and upgrade of db1082
  • 10:45 jynus: shutdown and upgrade of db1122
  • 10:44 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 and db1081 (duration: 00m 49s)
  • 10:33 godog: bounce logstash on logstash1007 for tests
  • 10:19 jynus: shutting down es1019 for hw maintenance T201132
  • 10:18 ema: reboot lvs-eqiad primaries for kernel upgrade
  • 10:04 godog: reboot einsteinium for kernel upgrade
  • 09:57 marostegui: Deploy schema change on db2075 - T67448 T114117 T5119
  • 09:55 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 00m 48s)
  • 09:43 ema: reboot lvs-codfw primaries for kernel upgrade
  • 09:15 _joe_: rolling restart of HHVM in the api-eqiad cluster
  • 09:13 _joe_: restarting hhvm on mw1231, high load
  • 09:10 _joe_: restarting hhvm on mw1226, then repooling it
  • 09:07 ema: reboot lvs3001 for kernel upgrade
  • 09:02 _joe_: depool mw1226 for investigation
  • 08:50 ema: reboot lvs-eqsin primaries for kernel upgrade
  • 08:29 gehel: start reimaging of elasticsearch / cirrus / codfw cluster (RAID0 / Stretch) - T193649 / T198391
  • 08:28 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 49s)
  • 08:28 ema: reboot lvs-ulsfo primaries for kernel upgrade
  • 08:13 godog: reboot tegmen with a new kernel
  • 07:37 jynus: restarted es1014 prometheus exporter (last message was wrong)
  • 07:37 jynus: restarted es1019 prometheus exporter
  • 07:29 volans: migrated puppetboard and debmonitor to cache_text - T164609
  • 07:05 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool pc2006 T200641 (duration: 00m 49s)
  • 06:58 marostegui: Deploy schema change on db1075 (s3 master) T144010 T51190 T199368
  • 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 (duration: 00m 50s)
  • 06:53 ema: reboot lvs secondaries for kernel upgrade
  • 05:21 marostegui: Deploy schema change on db1123 T144010 T51190 T199368
  • 05:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 (duration: 00m 50s)
  • 03:46 TimStarling: on mwmaint1001 running populateContentTables.php concurrently on wikidatawiki and commonswiki (T183488)
  • 02:34 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 7 02:34:24 UTC 2018 (duration 10m 31s)
  • 02:23 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 08m 46s)
  • 01:06 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1065.eqiad.wmnet,service=nginx
  • 01:06 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet,service=nginx
  • 01:05 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1072.eqiad.wmnet,service=nginx
  • 01:05 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet,service=nginx
  • 00:29 aaron@deploy1001: Synchronized wmf-config/mc.php: Use mcrouter for cache reads on all wikis (duration: 00m 49s)

2018-08-06

  • 23:42 bblack@neodymium: conftool action : set/weight=1; selector: dc=esams,cluster=cache_text,service=nginx
  • 23:42 James_F: SWAT complete.
  • 23:42 bblack@neodymium: conftool action : set/weight=1; selector: dc=esams,cluster=cache_text,service=varnish-fe
  • 23:41 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Revert PageImages: Make it possible to add extra namespaces, part II, fatalmonitor unhappy (duration: 00m 49s)
  • 23:38 jforrester@deploy1001: sync-file aborted: SWAT PageImages: Add NS_CATEGORY for Commons T198716 (duration: 00m 04s)
  • 23:35 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT PageImages: Make it possible to add extra namespaces, part II - I3f225d051 (duration: 00m 48s)
  • 23:35 bblack@neodymium: conftool action : set/weight=1; selector: dc=eqiad,cluster=cache_upload,service=nginx
  • 23:33 bblack@neodymium: conftool action : set/weight=1; selector: dc=eqiad,cluster=cache_upload,service=varnish-fe
  • 23:32 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT PageImages: Make it possible to add extra namespaces, part I - I6c51c1fe1 (duration: 00m 49s)
  • 23:30 jforrester@deploy1001: Synchronized wmf-config/extension-list: SWAT Load TimedMediaHandler i18n via static extension registration (duration: 00m 48s)
  • 23:27 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Load TimedMediaHandler via static extension registration T140852 (duration: 00m 48s)
  • 23:21 jforrester@deploy1001: Synchronized multiversion/: SWAT Delete multiversion/submodules.json (duration: 00m 48s)
  • 23:15 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Add new wikimania wiki to CORS origins I493cadb871 (duration: 00m 49s)
  • 23:11 ejegg: updated CiviCRM from 03012506b9 to d626907f2c
  • 23:07 James_F: jforrester@mwmaint1001 SWAT Updating sewiki collation via updateCollation.php T182431
  • 23:06 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Update sewiki collation config T182431 (duration: 00m 51s)
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1090.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1088.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1086.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1084.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1082.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1078.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1076.eqiad.wmnet,service=varnish-fe
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1084.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1082.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1078.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1076.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1086.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1088.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1090.eqiad.wmnet,service=nginx
  • 22:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1066.eqiad.wmnet
  • 22:09 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1085.eqiad.wmnet
  • 22:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1073.eqiad.wmnet
  • 22:08 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1086.eqiad.wmnet
  • 20:39 arlolra: Updated Parsoid to e02124b (T199849, T198400, T199577, T199509, T200403, T201054)
  • 20:31 arlolra@deploy1001: Finished deploy [parsoid/deploy@6b16b57]: Updating Parsoid to e02124b (duration: 11m 20s)
  • 20:19 arlolra@deploy1001: Started deploy [parsoid/deploy@6b16b57]: Updating Parsoid to e02124b
  • 20:00 gehel@deploy1001: Finished deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater (duration: 10m 55s)
  • 19:50 gehel@deploy1001: Started deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater
  • 19:46 gehel@deploy1001: Finished deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 22s)
  • 19:46 gehel@deploy1001: Started deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater (wdqs1009 only)
  • 19:14 mutante: bast1002 - rebooting
  • 19:04 mutante: bast1002 - the last log line is about bast1002, not bast1001
  • 19:03 mutante: bast1001 - installing package updates, incl systemd,kernel
  • 18:56 herron: rebooting fermium (lists) for security updates
  • 18:54 herron: rolling reboot of mx servers for security updates
  • 18:16 moritzm: rebooting radium for kernel security update
  • 18:15 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable Special:Block Feedback Request (duration: 00m 52s)
  • 18:09 mutante: rebooting bast3002 for kernel upgrade
  • 18:08 mutante: bast2002 - installing package upgrades (future bastion, to be setup)
  • 18:04 herron: double logstash jvm heap size from 256m to 512m and rolling restart of logstash instances T200960
  • 17:57 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1067.eqiad.wmnet
  • 17:57 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet
  • 17:57 mutante: rebooting bast4001
  • 17:56 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1099.eqiad.wmnet
  • 17:56 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1088.eqiad.wmnet
  • 17:37 mobrovac@deploy1001: Finished deploy [citoid/deploy@4f5ba15]: Use WorldCat for open search as well - T162357 (duration: 03m 29s)
  • 17:36 moritzm: uploaded linux 4.9.110+deb9u1~wmf1 for jessie-wikimedia to apt.wikimedia.org
  • 17:34 mobrovac@deploy1001: Started deploy [citoid/deploy@4f5ba15]: Use WorldCat for open search as well - T162357
  • 17:04 gehel@deploy1001: Finished deploy [wdqs/wdqs@0d3c6a6]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 33s)
  • 17:04 gehel@deploy1001: Started deploy [wdqs/wdqs@0d3c6a6]: new version of wdqs GUI and updater (wdqs1009 only)
  • 16:53 andrewbogott: power down labservices1001 for thermal paste fix, T196252
  • 16:53 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1068.eqiad.wmnet
  • 16:52 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1089.eqiad.wmnet
  • 15:54 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1074.eqiad.wmnet
  • 15:50 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1090.eqiad.wmnet
  • 15:50 bblack: pooling cp1090 (upload@eqiad, first of new hardware)
  • 15:05 papaul: shutting down pc2006 for maintenance
  • 15:02 bblack: rebooting cp1075-90
  • 14:56 marostegui: Stop MySQL for onsite maintenance - T200641
  • 14:30 godog: roll-restart thumbor after adding new private containers - T201187
  • 14:07 marostegui: Reload haproxy to apply new configuration
  • 11:32 zeljkof: EU SWAT finished
  • 11:31 zfilipin@deploy1001: Synchronized wmf-config/LabsServices.php: SWAT: deployment-prep: Change urldownloader host (T201160) (duration: 00m 50s)
  • 11:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add suppressredirect permission to rollbacker and patroller at zhwiki (T201160) (duration: 00m 49s)
  • 11:13 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 48s)
  • 11:13 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
  • 11:08 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Give hewiki interface-admins the rights interface-editors have (T200698) (duration: 00m 50s)
  • 10:54 godog: repair sde on ms-be2040
  • 10:38 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 04m 38s)
  • 10:33 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
  • 10:33 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 04m 44s)
  • 10:29 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
  • 10:28 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 10m 51s)
  • 10:18 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
  • 10:17 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 11m 12s)
  • 10:06 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
  • 09:33 mobrovac@deploy1001: Finished deploy [cpjobqueue/deploy@62716a5]: CirrusSearch jobs: Increase checker concurrency to 10 - T198462 (duration: 00m 44s)
  • 09:32 mobrovac@deploy1001: Started deploy [cpjobqueue/deploy@62716a5]: CirrusSearch jobs: Increase checker concurrency to 10 - T198462
  • 09:31 mobrovac@deploy1001: Finished deploy [cpjobqueue/deploy@62716a5]: (no justification provided) (duration: 00m 27s)
  • 09:31 mobrovac@deploy1001: Started deploy [cpjobqueue/deploy@62716a5]: (no justification provided)
  • 08:24 marostegui: Remove unused mysql users allowed to connect from 208.80.152.226
  • 08:23 _joe_: restarting logstash on logstash1007, losing packets as well
  • 08:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 (duration: 00m 51s)
  • 08:03 _joe_: restarting logstash on logstash1009, losing packets again
  • 07:51 _joe_: restarting logstash on logstash1008, losing packets again
  • 06:49 marostegui: Deploy schema change on db1077 with replication, this will generate lag on labsdb:s3 T144010 T51190 T199368
  • 06:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 (duration: 00m 53s)
  • 04:29 TimStarling: on mwmaint1001 running populateContentTables.php on metawiki T183488
  • 04:09 TimStarling: on mwmaint1001 running populateContentTables.php on mediawikiwiki T183488
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 6 02:45:42 UTC 2018 (duration 10m 28s)
  • 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 13m 40s)

2018-08-05

  • 02:41 andrewbogott: rebooting labservices1001 from mgmt

2018-08-04

  • 16:40 krinkle@deploy1001: Synchronized php-1.32.0-wmf.15/includes/resourceloader/ResourceLoaderWikiModule.php: I5cb93c173 (duration: 00m 52s)
  • 16:37 krinkle@deploy1001: Synchronized php-1.32.0-wmf.15/includes/Storage/DerivedPageDataUpdater.php: I5cb93c173 [1/2] (duration: 01m 03s)
  • 14:50 godog: roll-restart logstash because of increasing packet loss

2018-08-03

  • 23:07 XioNoX: - restart asw2-b5-eqiad into loader - T201145
  • 18:07 marxarelli: restarting jenkins on contint1001
  • 18:05 thcipriani: restarting releases-jenkins
  • 15:45 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1239.eqiad.wmnet
  • 14:13 jynus: stopping haproxy on dbproxy1006
  • 14:11 reedy@deploy1001: Synchronized php-1.32.0-wmf.15/includes/Storage/DerivedPageDataUpdater.php: Fix article counting logic in DerivedPageDataUpdater (duration: 00m 50s)
  • 13:06 ema: reboot cp3030 to SSBD-enabled microcode/kernel
  • 12:51 ema: trafficserver 7.1.3+ds-4wm2 uploaded to stretch-wikimedia T200178
  • 12:05 gehel: reimage relforge to stretch - T193649
  • 11:27 jynus: setting dbstore1001:s1 mariadb as read_only=1
  • 10:54 jynus: setting dbstore2002 instances as read only
  • 10:39 hoo: Running populateSitesTable.php on all Wikidata clients for T201003
  • 10:31 jynus: setting read only=1 on all instances on dbstore2001
  • 09:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 (duration: 00m 49s)
  • 09:50 jynus: testing new monitoring on dbstore2001
  • 09:36 paravoid: asw2-b-eqiad: set disable for xe-2/0/5 (cloudvirt1023), xe-7/0/22 (cloudvirt1024), ge-8/0/23 (rdb1009)
  • 09:36 jynus: disabling puppet on mariadb multiinstance hosts
  • 08:24 filippo@neodymium: conftool action : set/pooled=no; selector: name=logstash1007.eqiad.wmnet
  • 08:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 00m 50s)
  • 02:35 ejegg: updated CiviCRM from a0a7ea846a to 03012506b9
  • 00:57 mutante: restarted postgresql on netmon2001 as part of debugging issue with failed icinga replication check

2018-08-02

  • 23:26 maxsem@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikimediaEvents/+/450080/ (duration: 00m 49s)
  • 23:03 XioNoX: adding package anycast-healthchecker to stretch-wikimedia
  • 22:44 XioNoX: replacing package python3-jsonlogger with python3-json-logger in stretch-wikimedia
  • 22:21 mutante: webperf2002 - systemctl start proc-sys-fs-binfmt_misc.automount to fix "Check systemd degraded" Icinga check
  • 22:11 XioNoX: importing python3-jsonlogger_0.1.9 into stretch-wikimedia
  • 20:30 bd808@deploy1001: Finished deploy [striker/deploy@2329901]: Update Striker to 2329901 (T177407, T198076, T190543) (duration: 01m 26s)
  • 20:28 bd808@deploy1001: Started deploy [striker/deploy@2329901]: Update Striker to 2329901 (T177407, T198076, T190543)
  • 20:23 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/VisualEditor/includes/ApiVisualEditorEdit.php: sync https://gerrit.wikimedia.org/r/450090/ to unbreak prod refs T201083 T191061 (duration: 00m 49s)
  • 19:10 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.15 refs T191061
  • 19:07 twentyafterfour: T191061 is free of blockers, proceeding with the train deployment: group2 wikis to 1.32.0-wmf.15
  • 18:21 Amir1: ladsgroup@mwmaint1001:~$ mwscript extensions/ORES/maintenance/PurgeScoreCache.php --wiki=wikidatawiki and --old (T200680)
  • 18:20 Amir1: Morning SWAT is done
  • 18:20 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/ORES/maintenance/PurgeScoreCache.php: SWAT: Join decomposition on maintenance/PurgeScoreCache.php (T200680) (duration: 00m 57s)
  • 18:15 gehel: un-banning and repooling elastic1030 - T201039
  • 18:10 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix a typo in ORES models config (duration: 00m 57s)
  • 16:00 milimetric@deploy1001: Finished deploy [analytics/refinery@5e5b5a9]: Quick fix for sqoop script (duration: 07m 52s)
  • 15:53 milimetric@deploy1001: Started deploy [analytics/refinery@5e5b5a9]: Quick fix for sqoop script
  • 14:27 dcausse: restarting elastic1030 to trigger a master election
  • 13:45 ema: bounce traffic back from lvs3004 to lvs3002
  • 13:38 ema: reboot lvs3002 to SSBD-enabled microcode/kernel
  • 13:30 reedy@deploy1001: Synchronized dblists/: Update size dblists (duration: 00m 55s)
  • 13:18 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 fully (duration: 00m 56s)
  • 13:12 ema: bounce traffic from lvs3002 to lvs3004 (SSBD-enabled)
  • 13:01 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: fix wikimaniawiki wgserver and wgcanonicalserver (duration: 00m 56s)
  • 12:58 marostegui: Sanitize zhwikiversity satwiki T199599 T198401
  • 12:57 ema: reboot lvs3004 to SSBD-enabled microcode/kernel
  • 12:51 dcausse: banning elastic1030 (master struggling)
  • 12:48 marostegui: Sanitize wikimaniawiki - T201001
  • 12:45 dcausse: depooling elastic1030 (master struggling)
  • 12:35 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: update foundation url (duration: 00m 56s)
  • 12:34 reedy@deploy1001: Synchronized wmf-config/missing.php: Update foundation urls (duration: 00m 55s)
  • 12:21 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 25s)
  • 12:19 Reedy: Wikis created T196748 T198401 T199599
  • 12:18 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: fix id_internal (duration: 00m 55s)
  • 12:15 reedy@deploy1001: rebuilt and synchronized wikiversions files: new wikis!
  • 12:11 reedy@deploy1001: Synchronized wmf-config/: New wikis (duration: 00m 55s)
  • 12:10 reedy@deploy1001: Synchronized dblists/: new wikis (duration: 00m 55s)
  • 12:08 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: add id_internal (duration: 00m 55s)
  • 12:07 reedy@deploy1001: Synchronized langlist: Add sat (duration: 00m 55s)
  • 12:06 reedy@deploy1001: Synchronized static/images/project-logos/: New logos! (duration: 00m 57s)
  • 11:22 godog: temporarily remove multiline filter from logstash100[789] and bump pipeline workers to 4 - T200960
  • 11:21 moritzm: installing jansson security updates on trusty
  • 11:17 godog: temporarily remove multiline filter from logstash to allow using multiple workers - T200960
  • 11:16 moritzm: installing tomcat8 security updates
  • 11:12 zeljkof: EU SWAT finished
  • 11:11 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Enable moved paragraph detection for inline diffs on beta cluster (T200975) (duration: 00m 58s)
  • 11:03 godog: comment "pipeline.workers: 1" from logstash1007 - T200960
  • 10:47 godog: test bumping heap to 512mb on logstash1007 - T200960
  • 10:35 volans@deploy1001: Finished deploy [netbox/deploy@ac54feb]: Security upgrade of dependency (duration: 00m 05s)
  • 10:35 volans@deploy1001: Started deploy [netbox/deploy@ac54feb]: Security upgrade of dependency
  • 10:30 volans@deploy1001: Finished deploy [debmonitor/deploy@73b640d]: Release v0.1.7 (duration: 01m 01s)
  • 10:29 volans@deploy1001: Started deploy [debmonitor/deploy@73b640d]: Release v0.1.7
  • 10:25 volans@deploy1001: Finished deploy [debmonitor/deploy@691d2f8]: Release v0.1.7 (duration: 00m 51s)
  • 10:24 volans@deploy1001: Started deploy [debmonitor/deploy@691d2f8]: Release v0.1.7
  • 10:19 godog: test bumping rmem_default to 4MB on logstash1007 - T200960
  • 10:11 reedy@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/LabeledSectionTransclusion/: Consistency (duration: 00m 55s)
  • 10:10 reedy@deploy1001: Synchronized php-1.32.0-wmf.14/languages/Language.php: Revert out deprecation warning of hook (duration: 00m 57s)
  • 09:18 legoktm@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/LabeledSectionTransclusion/: revert again (duration: 00m 55s)
  • 09:14 legoktm@deploy1001: scap failed: RuntimeError Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org) (duration: 22m 08s)
  • 09:14 legoktm@deploy1001: Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org)
  • 08:52 legoktm@deploy1001: Started scap: try once more
  • 08:40 legoktm@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/LabeledSectionTransclusion/: revert (duration: 00m 56s)
  • 08:36 legoktm@deploy1001: scap failed: RuntimeError Scap failed!: 11/11 canaries failed their endpoint checks(http://en.wikipedia.org) (duration: 24m 33s)
  • 08:36 legoktm@deploy1001: Scap failed!: 11/11 canaries failed their endpoint checks(http://en.wikipedia.org)
  • 08:14 marostegui: Deploy schema change on db2043 (s3 codfw master) this will generate lag on codfw:s3 T144010 T51190 T199368
  • 08:11 legoktm@deploy1001: Started scap: LST: Use modern i18n mechanisms for localization (T198173, T200960)
  • 07:34 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 with low load (duration: 00m 56s)
  • 07:20 TimStarling: restarting logstash on logstash1008 and logstash1009
  • 07:06 TimStarling: on logstash1007 increased net.core.rmem_default from 212992 to 851968 in soft state and restarted logstash
  • 06:59 TimStarling: on logstash1007 restarting logstash
  • 06:58 jynus: stop db1092 for reimage
  • 06:53 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 55s)
  • 06:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 (duration: 00m 55s)
  • 06:18 marostegui: Deploy schema change on db1106 with replication, this will generate lag on labsdb:s1 T144010 T51190 T199368
  • 06:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 (duration: 00m 54s)
  • 06:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 (duration: 00m 55s)
  • 06:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 (duration: 00m 57s)
  • 05:30 TimStarling: on mwmaint1001 running populateContentTables.php as described in T183488
  • 05:14 TimStarling: restarted stashbot
  • 05:07 TimStarling: ran patch-slot-origin.sql on labswiki and labtestwiki to fix editing on labswiki (some SAL entries missing)
  • 02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 42s)

2018-08-01

  • 23:39 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscodeJob.php: SWAT: Work around transcode failures in newer ffmpeg (duration: 00m 56s)
  • 23:37 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscodeJob.php: SWAT: Work around transcode failures in newer ffmpeg (duration: 00m 57s)
  • 23:17 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Re-enable VP8 video transcodes to fix playback regression (duration: 00m 56s)
  • 22:58 krinkle@deploy1001: Synchronized wmf-config: I4db5d03f8af7 (step 2) (duration: 00m 57s)
  • 22:57 krinkle@deploy1001: Synchronized wmf-config: I4db5d03f8af7 (step 1) (duration: 00m 57s)
  • 20:36 twentyafterfour: T191061 Finished deploying 1.32.0-wmf.15 to group1 wikis. Fatalmonitor is quiet and everything appears to be stable. See you tomorrow for group2.
  • 20:24 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.15 refs T191061 (duration: 00m 55s)
  • 20:23 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.15 refs T191061
  • 20:17 ejegg: updated payments-wiki from 626ff1cb72 to 0c9c24d30c
  • 20:10 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@c2448e0]: Update mobileapps to 282f368 (T200464 T200459) (duration: 05m 42s)
  • 20:04 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@c2448e0]: Update mobileapps to 282f368 (T200464 T200459)
  • 19:53 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/Collection/: Sync Change: https://gerrit.wikimedia.org/r/449796 Bug: T189636 unblocks: T191061 (duration: 00m 57s)
  • 19:07 twentyafterfour: preparing to deploy 1.32.0-wmf.15 to group1 wikis. Gonna merge a couple of patches first to fix some logspam.
  • food: updated DjangoBannerStats from 73d74f1b5e to 02be6cbb74
  • 18:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #4 (duration: 03m 21s)
  • 18:10 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #4
  • 18:09 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #3 (duration: 06m 27s)
  • 18:03 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #3
  • 18:03 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #2 (duration: 05m 08s)
  • 17:58 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #2
  • 17:57 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens (duration: 12m 23s)
  • 17:47 ejegg: updated DjangoBannerStats from 43e47f98a2 to 73d74f1b5e
  • 17:45 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens
  • 17:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c] (dev-cluster): Multi-content bucket: delete old revisions when a new render happens (duration: 02m 52s)
  • 17:42 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c] (dev-cluster): Multi-content bucket: delete old revisions when a new render happens
  • 17:22 aaron@deploy1001: Synchronized wmf-config/mc.php: Use mcrouter for cache reads for test wikis (duration: 00m 55s)
  • 17:19 XioNoX: enable puppet on all cp2* servers after merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/449760 - T195365
  • 16:51 mforns: Finished deploying refinery using scap, then refinery-deploy-to-hdfs
  • 16:41 vgutierrez: Turn off AES128-SHA support - T192555 && T147202
  • 16:38 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/PageTriage/modules/ext.pageTriage.views.list/ext.pageTriage.listControlNav.js: SWAT: Fix namespaces shown in NPP filters T200826 T200821 (duration: 00m 55s)
  • 16:36 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/PageTriage/modules/ext.pageTriage.views.list/ext.pageTriage.listControlNav.js: SWAT: Fix namespaces shown in NPP filters T200826 T200821 (duration: 00m 58s)
  • 16:34 mforns@deploy1001: Finished deploy [analytics/refinery@7fba0fc]: fixing EL sanitization whitelist (duration: 07m 54s)
  • 16:26 mforns@deploy1001: Started deploy [analytics/refinery@7fba0fc]: fixing EL sanitization whitelist
  • 16:24 mforns: deploying analytics-refinery using scap
  • 16:08 moritzm: upgrading proton* to chromium 68.0.3440.75-1~deb9u1 (new release tested successfully in deployment-prep)
  • 15:50 moritzm: installing libarchive-zip-perl security updates
  • 15:43 akosiaris: upload mapnik_3.0.20+ds-1 to apt.wikimedia.org/stretch-wikimedia/main T200284
  • 15:16 XioNoX: disable puppet on all codfw cp* servers - T195365
  • 14:52 elukey: created user 'research' on db1108 (eventlogging slave - only select grant for the log database)
  • 13:49 moritzm: upgrade application servers in beta to wikidiff 1.7,2 (T199801)
  • 13:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 (duration: 00m 55s)
  • 13:29 marostegui: Deploy schema change on db1083 T144010 T51190 T199368
  • 13:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 00m 55s)
  • 13:02 volans: restarting ircecho on einsteinium to unblock icinga IRC bot
  • 12:59 moritzm: installing tomcat7 security updates on jessie
  • 12:35 marostegui: Deploy schema change on db1105:3311 T144010 T51190 T199368
  • 12:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 (duration: 00m 55s)
  • 12:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 (duration: 00m 55s)
  • 12:16 moritzm: installing fuse security updates
  • 12:16 marostegui: Deploy schema change on db1099:3311 T144010 T51190 T199368
  • 12:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 (duration: 00m 56s)
  • 11:56 zeljkof: EU SWAT finished
  • 11:54 zfilipin@deploy1001: Synchronized wmf-config/: SWAT: CleanupParent for draftquality model when PageTriage is used (T199357) (duration: 00m 56s)
  • 11:45 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.15/skins/MinervaNeue/: SWAT: Restore page issues (T200867) (duration: 00m 57s)
  • 11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable SandboxLink on the isiZulu Wikipedia (T200522) (duration: 00m 56s)
  • 11:20 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Fix config for $wgLocalisationUpdateRepositories (T148965) (duration: 00m 57s)
  • 11:17 moritzm: installing openjpeg2 security updates on jessie
  • 10:44 ema: reboot cp1045 for kernel update
  • 10:40 marostegui: Reload haproxy on dbproxy1008 and dbproxy1003
  • 09:50 moritzm: installing wireshark security updates
  • 09:50 marostegui: Deploy schema change on db2048 (s1 codfw master) with replication, this will generate lag on codfw s1 T144010 T51190 T199368
  • 09:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 55s)
  • 09:41 ema: cp3044 (upload) repooled after upgrade to stretch T200445
  • 09:39 marostegui: Run community_metrics.sh on phab1001
  • 09:36 moritzm: installing clamav security updates
  • 09:19 marostegui: Deploy schema change on db1092 T144010 T51190 T199368
  • 09:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 55s)
  • 09:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 56s)
  • 08:38 elukey: restart hadoop-yarn-nodemanager on analytics10[31-77] to apply the new memory settings
  • 08:37 marostegui: Deploy schema change on db1109:3318 T144010 T51190 T199368
  • 08:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 55s)
  • 08:22 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s1 populateChangeTagDef.php --sleep 2 (T193873)
  • 08:21 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s8 populateChangeTagDef.php --sleep 3 (T193873)
  • 08:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 56s)
  • 08:16 marostegui: Stop MySQL on db2078
  • 07:59 elukey: restart hadoop-yarn-nodemanager on analytics10[28-30] to test new memory settings
  • 07:12 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db1052 from config as it will be decommissioned - T199861 (duration: 00m 55s)
  • 07:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db1052 from config as it will be decommissioned - T199861 (duration: 00m 56s)
  • 07:09 marostegui: Remove db1052 from tendril - T199861
  • 06:59 elukey: restart eventlogging on eventlog1002 to pick up new logging settings
  • 06:58 elukey@deploy1001: Finished deploy [eventlogging/analytics@762ca2b]: Deploy https://gerrit.wikimedia.org/r/#/c/eventlogging/+/449422/ (duration: 00m 07s)
  • 06:58 elukey@deploy1001: Started deploy [eventlogging/analytics@762ca2b]: Deploy https://gerrit.wikimedia.org/r/#/c/eventlogging/+/449422/
  • 06:47 marostegui: Deploy schema change on db1087 with replication, this will generate lag on labs:s8 T144010 T51190 T199368
  • 06:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 (duration: 00m 55s)
  • 06:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic db1089 (duration: 00m 55s)
  • 06:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 55s)
  • 06:22 brion: restarting transcode batch job (errors believed caused by hhvm config restarts)
  • 06:11 brion: stopping requeueTranscodes.php for now
  • 06:11 brion: seeing spike in errors on video scalers
  • 05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1089 (duration: 00m 57s)
  • 05:02 marostegui: Stop MySQL on db1089 for a binlog format change (and also upgrade kernel)
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 (duration: 00m 57s)
  • 04:47 marostegui: Stop MySQL on db1052 to copy its content to dbstore1001 - https://phabricator.wikimedia.org/T199861
  • 04:46 marostegui: Deploy schema change on db1101:3318 T144010 T51190 T199368
  • 04:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 01m 06s)
  • 03:17 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 1 03:17:12 UTC 2018 (duration 10m 24s)
  • 03:06 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 14m 45s)
  • 02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 14m 25s)
  • 00:15 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php: SWAT: Workaround for job queue reporting 0 length (T200813) (duration: 00m 56s)
  • 00:09 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/ParserFunctions/includes/ExtParserFunctions.php: SWAT: Remove & from $mwDefault variable assignment T200772 (duration: 00m 57s)

2018-07-31

  • 23:55 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php: SWAT: Workaround for job queue reporting 0 length (T200813) (duration: 00m 57s)
  • 23:41 ejegg: updated payments-wiki from 6afaca3de3 to 626ff1cb72
  • 23:32 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable $wgCiteResponsiveReferences for Meta-Wiki T200707 (duration: 00m 56s)
  • 23:21 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateStyles on Meta-Wiki T200613 (duration: 00m 57s)
  • 22:56 eileen: update CiviCRM civicrm revision changed from 24b497072d to a0a7ea846a, config revision is 35a258d015
  • 21:37 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.15
  • 21:05 twentyafterfour@deploy1001: Finished scap: testwikis wikis to 1.32.0-wmf.15 (duration: 44m 49s)
  • 20:20 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.32.0-wmf.15
  • 19:00 urandom: enabling `unchecked_tombstone_compaction` on enwiki_T_mobile__ng_remaining -- T192689
  • 18:20 XioNoX: stopping puppet across the fleet for puppetmaster1001 uplink move
  • 18:04 XioNoX: stopping pybal on lvs1002
  • 18:01 XioNoX: repool lvs1001
  • 17:53 XioNoX: stopping pybal on lvs1001
  • 17:34 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=dns,service=pdns_recursor,name=chromium.wikimedia.org
  • 17:19 twentyafterfour: branching 1.32.0-wmf.15 refs T191061
  • 17:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all hosts in row B - T183585 (duration: 00m 51s)
  • 16:40 elukey: repool druid1005 after network maintenance
  • 16:26 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=dns,service=pdns_recursor,name=chromium.wikimedia.org
  • 16:25 godog: repool thumbor100[12] - T183585
  • 16:25 elukey: pool eventbus on kafka1002 after network maintenance
  • 16:24 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=aqs,service=cassandra,name=aqs1004.eqiad.wmnet
  • 16:24 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=aqs,service=aqs,name=aqs1004.eqiad.wmnet
  • 16:16 elukey: precautionary restart of eventbus on kafka1002 after network downtime (DNS name res errors, Kafka broker conn issues, etc..)
  • 16:07 marostegui: Reload dbproxy1003 and dbproxy1008
  • 16:06 ema: reboot cp3044 for kernel update
  • 16:01 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=prometheus,service=prometheus,name=prometheus1004.eqiad.wmnet
  • 15:51 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=druid-public,service=druid-public-broker,name=druid1005.eqiad.wmnet
  • 15:50 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=aqs,service=cassandra,name=aqs1004.eqiad.wmnet
  • 15:50 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=aqs,service=aqs,name=aqs1004.eqiad.wmnet
  • 15:46 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=eventbus,service=eventbus,name=kafka1002.eqiad.wmnet
  • 15:45 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=prometheus,service=prometheus,name=prometheus1004.eqiad.wmnet
  • 15:36 ema: power-cycle cp3031, stuck rebooting
  • 15:29 _joe_: restarting apache on phab1001
  • 15:20 godog: depool thumbor100[12] ahead of switch move - T183585
  • 15:02 XioNoX: starting the eqiad row B servers move - T183585
  • 13:51 moritzm: installing ffmpeg 3.2.12 security updates on video scalers
  • 13:34 akosiaris: repool mathoid eqiad cluster in discovery
  • 13:18 akosiaris: upgrade kubernetes1004 to 1.9.9
  • 13:14 akosiaris: upgrade kubernetes1002 to 1.9.9
  • 13:14 akosiaris: upgrade kubernetes1003 to 1.9.9
  • 13:03 akosiaris: upgrade kubernetes1001 to 1.9.9
  • 12:58 akosiaris: upgrade chlorine.eqiad.wmnet to kubernetes 1.9.9
  • 12:54 akosiaris: upgrade argon.eqiad.wmnet to kubernetes 1.9.9
  • 12:45 akosiaris: depool eqiad mathoid in discovery
  • 12:29 gehel@puppetmaster1001: conftool action : set/weight=15; selector: dc=eqiad,cluster=wdqs,name=wdqs1005.eqiad.wmnet
  • 12:29 gehel@puppetmaster1001: conftool action : set/weight=15; selector: dc=eqiad,cluster=wdqs,name=wdqs1004.eqiad.wmnet
  • 12:29 gehel: rebalance LVS weights to send less traffic to wdqs1003 - T200563
  • 12:13 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove logging now in mw core (duration: 00m 48s)
  • 12:11 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmg - for Sentry (duration: 00m 48s)
  • 12:09 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove wmg - for Sentry (duration: 00m 48s)
  • 12:06 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Load Sentry with wfLoadExtension (duration: 00m 48s)
  • 12:05 reedy@deploy1001: Synchronized wmf-config/liquidthreads.php: Remove lqt cruft, load with wfLoadExtension (duration: 00m 48s)
  • 11:44 gehel: restarting blazegraph on wdqs1003 - JVM unresponsive
  • 11:38 zeljkof: EU SWAT finished
  • 11:37 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Introduce autopatrolled on bnwikisource (T199475) (duration: 00m 48s)
  • 11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos for hewiktionary (T200470) (duration: 00m 48s)
  • 11:24 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for hewiktionary (T200470) (duration: 00m 48s)
  • 11:16 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Milestone logo for atjwiki (T200713) (duration: 00m 49s)
  • 11:10 zfilipin@deploy1001: Synchronized static/images/project-logos/hewikiquote.png: SWAT: Update hewikiquote logo (T200296) (duration: 00m 50s)
  • 10:55 _joe_: test log
  • 10:40 akosiaris@deploy1001: scap-helm mathoid finished
  • 10:40 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
  • 10:39 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
  • 10:39 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw]
  • 09:46 ema: reboot cp1045 (jessie) for kernel updates
  • 09:29 ema: upload varnishkafka 1.0.13-1 to stretch-wikimedia T200445 T186250
  • 08:44 jynus: stop and reimage db1104 to stretch
  • 08:17 akosiaris: repool mathoid codfw in discovery, the kubernetes upgrade is done
  • 08:08 akosiaris: upgrade kubernetes2004 to 1.9.9
  • 08:05 akosiaris: upgrade kubernetes2003 to 1.9.9
  • 08:00 akosiaris: upgrade kubernetes2002 to 1.9.9
  • 07:55 ema: reboot cp2018 for kernel updates
  • 07:54 moritzm: reboot cp1008 for kernel tests
  • 07:53 akosiaris: upgrade kubernetes2001 to 1.9.9
  • 07:50 ema: reboot cp2025 for kernel updates
  • 07:47 akosiaris: upgrade acrux.codfw.wmnet to kubernetes 1.9.9
  • 07:44 jynus: stopping db1118 mariadb and starting mysql there
  • 07:36 moritzm: reboot multatuli for kernel tests
  • 07:35 akosiaris: upgrade acrab.codfw.wmnet to kubernetes 1.9.9
  • 07:32 moritzm: run decomission_appserver on terbium
  • 07:27 akosiaris: backup kubetcd2001 etcd data before kubernetes upgrade
  • 07:24 akosiaris: depool codfw mathoid from discovery for kubernetes upgrade
  • 07:10 moritzm: installing mutt security updates
  • 07:05 marostegui: Remove unused and old grants from codfw hosts - T146149#4456082
  • 06:49 jynus: dropping unnecesary tendril web users from tendril db backends
  • 06:45 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool row B es hosts, reduce db1089 load (duration: 00m 49s)
  • 06:11 moritzm: installing ant security updates
  • 05:01 marostegui: Deploy schema change on db1099:3318 T144010 T51190 T199368
  • 04:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool all hosts in row B - T183585 (duration: 00m 50s)
  • 04:47 marostegui: Stop change_tag_def populate maintenance script for wikidata and enwiki - T193873
  • 02:37 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 31 02:37:40 UTC 2018 (duration 10m 20s)
  • 02:27 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 08m 12s)
  • 01:20 ejegg: updated payments-wiki from 344233cff8 to 6afaca3de3
  • 00:44 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Switch in WebM VP9/Opus video transcodes to replace WebM VP8/Vorbis T63805 (duration: 00m 48s)
  • 00:39 thcipriani@deploy1001: Finished scap: SWAT: Convert to extension.json T87981 Use maps for wgEnabledTranscodeSet T118080 Adjust VP9 encoding for speed, quality More conservative max bitrate for VP9 video transcodes (duration: 33m 44s)
  • 00:07 ejegg: updated payments-wiki from 8c4d6aaa13 to 344233cff8
  • 00:05 thcipriani@deploy1001: Started scap: SWAT: Convert to extension.json T87981 Use maps for wgEnabledTranscodeSet T118080 Adjust VP9 encoding for speed, quality More conservative max bitrate for VP9 video transcodes

2018-07-30

  • 23:35 XioNoX: re-enabling puppet on cp40* - T195365
  • 23:27 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/RevisionSlider/modules: SWAT: RevisionSlider: Fix missing pin icon T200263 (duration: 00m 49s)
  • 23:15 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:Block Feedback Request (2) T199919 (duration: 00m 49s)
  • 22:35 XioNoX: applying static route + fixed MTU to cp4025 - T195365
  • 22:29 XioNoX: - puppet disabled on cp40* hosts - T195365
  • 22:10 thcipriani: restarting jenkins after plugin updates
  • 21:55 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (cobalt) (duration: 00m 10s)
  • 21:55 thcipriani@deploy1001: Started deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (cobalt)
  • 21:51 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (gerrit2001 only) (duration: 00m 09s)
  • 21:51 thcipriani@deploy1001: Started deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (gerrit2001 only)
  • 21:01 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@debb5b0]: Update spark.yaml with latest training configuration (duration: 04m 12s)
  • 20:56 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@debb5b0]: Update spark.yaml with latest training configuration
  • 20:41 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@69ac0d5]: fix wrong yaml import in cli entry point (duration: 02m 30s)
  • 20:39 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@69ac0d5]: fix wrong yaml import in cli entry point
  • 20:14 ejegg: updated CiviCRM from d76e3997e6 to 24b497072d
  • 20:10 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@61c5f47]: ship missing bc alias for deployed daemon (duration: 01m 35s)
  • 20:09 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@61c5f47]: ship missing bc alias for deployed daemon
  • 19:53 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 3 (duration: 06m 25s)
  • 19:47 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 3
  • 19:46 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 2 (duration: 05m 06s)
  • 19:40 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 2
  • 19:40 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 (duration: 14m 31s)
  • 19:26 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465
  • 19:24 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b] (dev-cluster): Language variants for summaries T198465 (duration: 03m 05s)
  • 19:22 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@b1e18ca]: (no justification provided) (duration: 01m 28s)
  • 19:21 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@b1e18ca]: (no justification provided)
  • 19:20 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b] (dev-cluster): Language variants for summaries T198465
  • 19:19 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 01m 32s)
  • 19:17 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
  • 18:57 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/WikimediaMessages/WikimediaMessages.hooks.php: SWAT: Escape Special:Block Feedback Request Message T194301 (duration: 00m 49s)
  • 18:25 thcipriani@deploy1001: Synchronized wmf-config/mc.php: SWAT: Make all wikis write to both nutcracker and mcrouter (3) T198239 (duration: 00m 48s)
  • 18:10 krinkle@deploy1001: Finished deploy [performance/navtiming@f79b313]: (no justification provided) (duration: 00m 05s)
  • 18:10 krinkle@deploy1001: Started deploy [performance/navtiming@f79b313]: (no justification provided)
  • 17:12 XioNoX: repool ulsfo
  • 17:11 gehel@deploy1001: Finished deploy [wdqs/wdqs@137780f]: new version of wdqs GUI (duration: 09m 17s)
  • 17:02 gehel@deploy1001: Started deploy [wdqs/wdqs@137780f]: new version of wdqs GUI
  • 17:01 gehel@deploy1001: Finished deploy [wdqs/wdqs@137780f]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 30s)
  • 17:00 gehel@deploy1001: Started deploy [wdqs/wdqs@137780f]: new version of wdqs GUI (wdqs1009 only)
  • 15:57 ema: pool cp2012, upgraded to stretch T200445
  • 15:50 ema: reboot cp2012 for kernel update
  • 15:27 jynus: restart and upgrade mariadb at db1118
  • 15:14 bblack: reboot cp1075
  • 15:08 godog: upgrade hp raid firmware on ms-be1028 - T141756
  • 15:05 Amir1: running ladsgroup@mwmaint1001:~$ mwscript extensions/ORES/maintenance/PurgeScoreCache.php on all wikis
  • 13:58 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.14
  • 13:39 ema: reboot cp2006 for kernel update
  • 12:59 zeljkof: EU SWAT finished
  • 12:58 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/AbuseFilter/: SWAT: Fix jQuery selector when editing filters (T200604) (duration: 00m 55s)
  • 12:55 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/FileImporter/: SWAT: Fix flipped array indexes in template removal code (T200406) (duration: 00m 57s)
  • 12:45 godog: fix group write on deploy1001 /srv/mediawiki-staging/php-1.32.0-wmf.14/.git/objects
  • 12:40 godog: reboot ms-be2040 with page_poisoning=1 - T199198
  • 12:33 tgr@deploy1001: Finished scap: T190015 Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default (duration: 60m 10s)
  • 11:39 ema: pool cp2006, upgraded to stretch T200445
  • 11:39 moritzm: upgrading intel-microcode to 20180703 on the servers which have microcode updates enabled (reboots to pick up the new version will be coordinated separately)
  • 11:33 tgr@deploy1001: Started scap: T190015 Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default
  • 11:12 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reading from change_tag_def everywhere (T199334) (duration: 00m 55s)
  • 11:10 volans: upgrading cumin to 3.0.2 on the remaining cumin masters (neodymium, labpuppetmaster*)
  • 11:01 volans: upgrading cumin to 3.0.2 on sarin
  • 10:50 volans: uploaded cumin_3.0.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 10:25 godog: run xfs_repair on sdc1 on ms-be2040 - T199198
  • 10:04 akosiaris@deploy1001: scap-helm help finished
  • 10:04 akosiaris@deploy1001: scap-helm help cluster codfw completed
  • 10:04 akosiaris@deploy1001: scap-helm help cluster eqiad completed
  • 10:04 akosiaris@deploy1001: scap-helm help [namespace: help, clusters: eqiad,codfw]
  • 10:03 akosiaris: upgrade kubernetes staging cluster to 1.9.9
  • 10:01 akosiaris: reboot kubestage1001 for kubernetes upgrade
  • 09:59 akosiaris: reboot kubestage1002 for kubernetes upgrade
  • 09:31 moritzm: uploaded intel-microcode 20180703 for jessie-wikimedia/stretch-wikimedia to apt.wikimedia.org (tested successfully on a number of canary hosts for approx two weeks)
  • 09:30 jynus: migrate dbtree to dbmonitor1001
  • 09:25 akosiaris@deploy1001: scap-helm mathoid finished
  • 09:14 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
  • 09:13 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
  • 09:09 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw]
  • 09:08 akosiaris: upgrade mathoid helm chart from 0.0.5 to 0.0.9
  • 09:07 akosiaris@deploy1001: scap-helm mathoid finished
  • 09:07 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
  • 09:07 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
  • 09:07 akosiaris@deploy1001: scap-helm mathoid upgrade -h [namespace: mathoid, clusters: eqiad,codfw]
  • 08:49 marostegui: Deploy schema change on dbstore1002:s8 T144010 T51190 T199368
  • 08:07 elukey@deploy1001: Finished deploy [eventlogging/analytics@54d43e4]: Band aid for T200630 (duration: 00m 05s)
  • 08:07 elukey@deploy1001: Started deploy [eventlogging/analytics@54d43e4]: Band aid for T200630
  • 07:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Give some traffic to db1120 to on x1 (duration: 00m 53s)
  • 07:43 moritzm: installing opencv security updates
  • 07:39 moritzm: installing mercurial security updates
  • 07:32 marostegui: Deploy schema change on db2045 (s8 codfw master) this will generate lag on s8 codfw T144010 T51190 T199368
  • 07:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Add db1120 to x1 T196376 (duration: 00m 54s)
  • 07:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Add db1120 to x1 T196376 (duration: 00m 59s)
  • 05:28 marostegui: Deploy schema change on db1068 (commonswiki master) - T51190
  • 05:00 marostegui: Deploy schema change on db1062 (s7 primary master) T144010 T51190 T199368
  • 03:13 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 30 03:13:42 UTC 2018 (duration 10m 29s)
  • 03:03 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 33s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 11m 41s)

2018-07-29

  • 09:56 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2006 T200641 (duration: 00m 57s)
  • 02:35 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Neuter EducationProgram T188410 (duration: 00m 58s)

2018-07-28

  • 17:49 Krinkle: Navigation Timing alerts indicate we're losing a significant about of traffic
  • 17:35 elukey: restart eventlogging on eventlog1002 after tons of kafka disconnects (still not clear what happened)
  • 08:16 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 fully (duration: 00m 57s)

2018-07-27

  • 22:05 ejegg: updated payments-wiki from f0138ab066 to 8c4d6aaa13
  • 21:39 ejegg: updated payments-wiki from 5e01f50983 to f0138ab066
  • 20:40 SMalyshev: re-pooled wdqs1003
  • 19:04 mutante: terbium is being removed from ferm rules on elasticsearch/relforge, logstash/collector, mariadb/labtestwikitech and mw-maintenance itself (T192092)
  • 18:57 SMalyshev: temporarily depooled wdq1003 to let it catch up with lag
  • 18:15 mutante: syncing home dirs from mwmaint1001 to mwmaint2001 (once, manually, not currently set to auto-sync, but to keep another copy of former terbium homes in case we failover) (T192092)
  • 18:12 mutante: terbium: deleting rsyncd config fragment, stopping rsyncd, not used anymore, decom prep
  • 17:06 gehel: repooling wdqs1003
  • 16:43 gehel: depooling wdqs1003 to let it catch up on update lag
  • 15:24 joal@deploy1001: Finished deploy [analytics/aqs/deploy@6fafc63]: Update AQS new-pages metric (duration: 04m 12s)
  • 15:20 joal@deploy1001: Started deploy [analytics/aqs/deploy@6fafc63]: Update AQS new-pages metric
  • 15:06 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 with low load (duration: 00m 55s)
  • 14:36 mutante: terbium - removing accountcheck cron job (duplicate mail from the same thing now on mwmaint1001)
  • 14:23 jynus: stop and reimage db1094
  • 14:18 elukey: execute echo 'https://wikimania.wikimedia.org' | mwscript purgeList.php on mwmain1001
  • 13:27 ema: libvmod-re2 1.3.1-2 uploaded to stretch-wikimedia T200445
  • 13:15 ema: libvmod-netmapper 1.7-2 uploaded to stretch-wikimedia T200445
  • 13:09 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 for reimage (duration: 00m 54s)
  • 11:33 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@941e3cf]: Make the main processing loop async - T199813 (duration: 02m 02s)
  • 11:31 mobrovac@deploy1001: Started deploy [eventstreams/deploy@941e3cf]: Make the main processing loop async - T199813
  • 09:37 ema: vhtcpd 0.1.1-2 uploaded to stretch-wikimedia T200445
  • 09:00 godog: adjust aggregation to 'sum' for MediaWiki.edit sum metrics - T199968
  • 08:36 Amir1: ladsgroup@mwmaint1001:~$ foreachwikiindblist s3 populateChangeTagDef.php --sleep 2 (T193873)
  • 05:59 akosiaris: reboot webperf1002, webperf2002 for new disk to appear T199853
  • 05:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 (duration: 00m 55s)
  • 04:59 marostegui: Deploy schema change on db1094 T144010 T51190 T199368
  • 04:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 57s)
  • 01:19 aaron@deploy1001: Synchronized php-1.32.0-wmf.14/includes/libs/objectcache/BagOStuff.php: f208a431f912 (duration: 00m 56s)

2018-07-26

  • 23:04 XioNoX: depool ulsfo, outages on both physical links to the site
  • 21:57 ejegg: updated payments-wiki from 6390e50abd to 5e01f50983
  • 21:33 tgr@deploy1001: Synchronized php-1.32.0-wmf.14/includes/Title.php: SWAT T200456: Handle $title === null in Title::newFromText (duration: 00m 57s)
  • 21:22 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T200346 Update custom domain for thumbnail rendering (duration: 00m 54s)
  • 21:21 tgr@deploy1001: Synchronized wmf-config/LabsServices.php: SWAT: T200346 Update custom domain for thumbnail rendering (duration: 00m 54s)
  • 21:19 tgr@deploy1001: Synchronized wmf-config/ProductionServices.php: SWAT: T200346 Update custom domain for thumbnail rendering (duration: 00m 55s)
  • 21:12 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T190015 gerrit 421122 + 421123 prepare for interface-admin group (duration: 00m 54s)
  • 21:11 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: T190015 gerrit 421122 + 421123 prepare for interface-admin group (duration: 00m 55s)
  • 20:55 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Do not set deprecated value for $wgExternalDiffEngine (duration: 00m 54s)
  • 20:39 addshore@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/Wikibase/repo/includes/Store/Sql/SqlChangeDispatchCoordinator.php: Use getClientLockName value for releaseClientLock when dispatching T200420 (duration: 00m 57s)
  • 20:12 ebernhardson: set merge.policy.reclaim_deletes_weight back to 2.0 for wikidatawiki_content
  • 20:11 thcipriani: gerrit upgrade to 2.15.3 complete
  • 20:06 thcipriani: restart gerrit for notedb config update
  • 20:03 thcipriani: running puppet on cobalt for gerrit notedb changes
  • 19:59 mobrovac@deploy1001: Finished deploy [proton/deploy@883cacd]: Use a more secure MW API template - T198461 (duration: 00m 33s)
  • 19:58 mobrovac@deploy1001: Started deploy [proton/deploy@883cacd]: Use a more secure MW API template - T198461
  • 19:40 thcipriani: restarting gerrit
  • 19:39 thcipriani: running puppet on cobalt for gerrit.config update
  • 19:38 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@b1f577d]: Provide better stream and error handling and stop using compression - T199813 (duration: 02m 13s)
  • 19:36 mobrovac@deploy1001: Started deploy [eventstreams/deploy@b1f577d]: Provide better stream and error handling and stop using compression - T199813
  • 19:31 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@b69b468]: upgrade to gerrit 2.15.3 (duration: 00m 08s)
  • 19:31 thcipriani@deploy1001: Started deploy [gerrit/gerrit@b69b468]: upgrade to gerrit 2.15.3
  • 19:22 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@b69b468]: deploy gerrit 2.15.3 to cold-replica (duration: 00m 10s)
  • 19:22 thcipriani@deploy1001: Started deploy [gerrit/gerrit@b69b468]: deploy gerrit 2.15.3 to cold-replica
  • 18:19 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: Revert "all wikis to 1.32.0-wmf.14"
  • 18:17 ebernhardson: set merge.policy.reclaim_deletes_weight=12.0 for wikidatawiki_content in eqiad to attempt to reduce the deleted documents count
  • 18:13 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.14
  • 18:08 arlolra: Updated Parsoid to cdf8ace (T194806, T110004, T156099, T188478, T194083)
  • 17:59 arlolra@deploy1001: Finished deploy [parsoid/deploy@1dce68c]: Updating Parsoid to cdf8ace (duration: 11m 22s)
  • 17:58 XioNoX: moving row C vrrp master back to cr2 - T187962
  • 17:54 mobrovac@deploy1001: Finished deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test, take #2 - T199491 (duration: 06m 16s)
  • 17:47 arlolra@deploy1001: Started deploy [parsoid/deploy@1dce68c]: Updating Parsoid to cdf8ace
  • 17:47 mobrovac@deploy1001: Started deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test, take #2 - T199491
  • 17:46 mobrovac@deploy1001: Finished deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test - T199491 (duration: 19m 36s)
  • 17:45 mutante: created new archiva user tyler for Tyler Cipriani and gave him role of global repo manager (for gerrit uploads instead of using archiva-deploy user which can cause password sync issues, so personalized users )
  • 17:27 mobrovac@deploy1001: Started deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test - T199491
  • 17:01 XioNoX: moving row C vrrp master to cr1 - T187962
  • 16:08 gehel: rolling restart of elasticsearch / cirrus / eqiad completed - T156137
  • 15:47 mobrovac@deploy1001: Finished deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491 (duration: 01m 04s)
  • 15:46 mobrovac@deploy1001: Started deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491
  • 15:46 mobrovac@deploy1001: Finished deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491 (duration: 14m 56s)
  • 15:31 mobrovac@deploy1001: Started deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491
  • 15:30 mobrovac@deploy1001: Finished deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491 (duration: 37m 37s)
  • 15:27 addshore@deploy1001: Synchronized wmf-config: Remove unused wikibaseDispatchRedisLockManager logging T200420 (duration: 00m 56s)
  • 15:21 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add LockManager logging T200420 (duration: 00m 55s)
  • 14:52 mobrovac@deploy1001: Started deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491
  • 14:38 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add wikibaseDispatchRedisLockManager to wmgMonologChannels T200420 (duration: 00m 54s)
  • 14:35 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Add a logger to wikibaseDispatchRedisLockManager T200420 (duration: 00m 56s)
  • 14:34 addshore@deploy1001: sync-file aborted: Add a logger to wikibaseDispatchRedisLockManager T200420 (duration: 00m 02s)
  • 14:24 akosiaris: sudo gnt-instance modify --disk add:size=150G webperf1002 T199853
  • 14:24 akosiaris: sudo gnt-instance modify --disk add:size=150G webperf2002 T199853
  • 13:44 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new wikibase dispatch lock manager on wikidatawiki T200420 T178652 (duration: 00m 55s)
  • 12:44 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: T200420 - Wikidata, dispatch, select 20 instead of 15 wikis (duration: 00m 55s)
  • 12:38 reedy@deploy1001: rebuilt and synchronized wikiversions files: wikidatawiki back to .13 T200420
  • 11:57 zeljkof: EU SWAT finished
  • 11:56 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos in hewikiquote (T200296) (duration: 00m 52s)
  • 11:45 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Upload HD logos for hewikiquote (T200296) (duration: 00m 54s)
  • 11:37 zfilipin@deploy1001: Synchronized tests/InitialiseSettingsTest.php: SWAT: Test spaces in ExtraNamespaces (T199162) (duration: 00m 55s)
  • 11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create TemplateEditor group on enwikivoyage (T198056) (duration: 00m 55s)
  • 11:20 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix unexpected space in pswikis ExtraNamespaces definition (T199480) (duration: 00m 55s)
  • 11:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Dont assign sysop-level privileges to bureaucrats explicitly (T197095) (duration: 00m 56s)
  • 11:02 Amir1: ladsgroup@mwmaint1001:~$ foreachwikiindblist wikitech populateChangeTagDef.php (T193873)
  • 11:01 Amir1: ladsgroup@mwmaint1001:~$ mwscript sql.php --wiki=labtestwiki /srv/mediawiki-staging/php-1.32.0-wmf.14/maintenance/archives/patch-change_tag_def.sql
  • 10:46 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.14 (duration: 00m 54s)
  • 10:45 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.14
  • 10:33 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES on test2wiki (T200412) (duration: 00m 55s)
  • 10:26 Amir1: ladsgroup@mwmaint1001:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=test2wiki ores (T200412)
  • 10:24 ema: cache-text: upgrade to varnish 5.1.3-1wm9 and apply alternate domains patch T164609
  • 10:08 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.32.0-wmf.14"
  • 10:07 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s7 populateChangeTagDef.php --sleep 2 (T193873)
  • 09:50 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.14 (duration: 00m 53s)
  • 09:49 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.14
  • 09:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 55s)
  • 09:40 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 fully (duration: 00m 55s)
  • 09:27 gehel: rolling restart of elasticsearch / cirrus / eqiad to disable G1 - T156137
  • 09:22 marostegui: Deploy schema change on db1079 with replication, this will generate lag on labsdb:s7 T144010 T51190 T199368
  • 09:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 55s)
  • 09:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 55s)
  • 09:00 jynus: restarting hhvm at mw1226, mw1276, mw1281
  • 08:54 jynus: restarting hhvm at mw1230
  • 08:26 joal@deploy1001: Finished deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts - try 2 (duration: 29m 02s)
  • 08:20 marostegui: Deploy schema change on db1090:3317 T144010 T51190 T199368
  • 08:18 mobrovac@deploy1001: Synchronized wmf-config/CommonSettings.php: Set readOnlyReason to false everywhere for JobQueueEventBus - T199594 (duration: 00m 55s)
  • 08:10 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 with low load after reimage (duration: 00m 55s)
  • 07:57 joal@deploy1001: Started deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts - try 2
  • 07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 54s)
  • 07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 (duration: 00m 54s)
  • 07:24 jynus: stop and reimage es1015
  • 07:19 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1015 (duration: 00m 55s)
  • 06:47 marostegui: Deploy schema change on db1086 T144010 T51190 T199368
  • 06:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 55s)
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 (duration: 00m 55s)
  • 05:02 marostegui: Deploy schema change on db1101:3317 T144010 T51190 T199368
  • 05:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 (duration: 01m 05s)
  • 04:13 kartik@deploy1001: Finished deploy [cxserver/deploy@3c99775]: Update cxserver to e98c81vb (T200283) (duration: 05m 12s)
  • 04:08 kartik@deploy1001: Started deploy [cxserver/deploy@3c99775]: Update cxserver to e98c81vb (T200283)
  • 03:16 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 26 03:16:04 UTC 2018 (duration 10m 24s)
  • 03:06 twentyafterfour: stopped phabricator rebuild-identities migration as it's suspected of causing database connection exhaustion and replag
  • 03:05 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 13s)
  • 02:50 Krinkle: phabricator.wikimedia.org is spewing HTTP 500 ("Can Not Connect to MySQL")
  • 02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 40s)
  • 01:42 mutante: phab1002 - starting rsync of repo data from phab1001 in a screen session after gerrit:447949 T190568
  • 01:32 mutante: phab1001 - rm /usr/local/sbin/sync-srv-repos that has a reference to non-existing server iridium.eqiad.wmnet (formerly phab) (T190568)
  • 00:16 twentyafterfour: phabricator update complete with only momentary downtime
  • 00:04 twentyafterfour: deploying phabricator release/2018-07-25/1, scheduled downtime for phab1001 in icinga
  • 00:01 mutante: temp. disable puppet on netmon1002,netmon2001 before applying puppet ferm change out of abundance of caution. applied gerrit:439554 without issues on netmon1003

2018-07-25

  • 23:35 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWInlineImageNode.js: SWAT: ve.dm.MWInlineImageNode: Fix undefined data-mw in toDomElements output T198941 T200214 (duration: 00m 57s)
  • 23:05 mutante: added dchen to LDAP group "wmf" (was already in admins as shell user, so didn't have to be added in puppet repo) (T200366)
  • 20:23 bblack: pooling cp5006 - T187157
  • 20:05 krinkle@deploy1001: Synchronized wmf-config/: Remove wgUploadThumbnailRenderHttpCustomDomain override - T200346 (duration: 00m 57s)
  • 18:59 bstorm_: Added new disk shelf to labstore1007
  • 17:53 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 54s)
  • 17:45 krinkle@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/Cite/includes/Cite.php: (no justification provided) (duration: 00m 55s)
  • 17:07 ejegg: Updated SmashPig settings: extended Ingenico Connect API timeout to 12 seconds
  • 16:49 Amir1: morning SWAT is done
  • 16:49 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reading from the new backend for Special:Tags in several large wikis (T199334) (duration: 00m 56s)
  • 16:40 akosiaris: remove ORES abuser blocking T200338, let's reevaluate
  • 16:02 joal@deploy1001: Finished deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts (duration: 17m 51s)
  • 15:49 gehel: elasticsearch cluster restart on codfw completed - T156137
  • 15:44 joal@deploy1001: Started deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts
  • 15:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 (duration: 00m 55s)
  • 15:31 anomie: running populateContentTables.php on testwiki for T183488
  • 15:26 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 55s)
  • 15:21 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgMultiContentRevisionSchemaMigrationStage to write-both read-old on testwiki (T197817) (duration: 00m 55s)
  • 14:45 marostegui: Deploy schema change on db1098:3317 T144010 T51190 T199368
  • 14:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 (duration: 00m 54s)
  • 14:39 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: (no justification provided)
  • 14:29 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1014 fully (duration: 00m 55s)
  • 14:00 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.14 (duration: 00m 55s)
  • 13:59 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.14
  • 13:54 jynus: running compare.py on all es3 databases
  • 13:48 ema: text-eqiad: test alternate domains patch with varnish 5.1.3-1wm9 T164609
  • 13:44 Amir1: restarting ores celery workers on codfw
  • 13:34 ema: repool cp1067 w/ alternate domains patch and varnish 5.1.3-1wm9 T164609
  • 13:27 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.14
  • 13:26 ema: depool cp1067 and test alternate domains patch with varnish 5.1.3-1wm9 T164609
  • 13:00 gehel: resetting postgres data on maps1002 after failing replication - T200228
  • 12:21 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s5 populateChangeTagDef.php --sleep 2 (T193873)
  • 12:19 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache (duration: 59m 27s)
  • 11:20 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache
  • 11:16 dcausse: EU SWAT done
  • 11:14 dcausse@deploy1001: Synchronized ./wmf-config/CirrusSearch-common.php: [cirrus] allow term_freq and remove deprecated settings (duration: 00m 48s)
  • 11:01 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.10 [keeping static files] (duration: 05m 13s)
  • 10:21 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1014 with low load after maintenance (duration: 00m 47s)
  • 10:07 marostegui: Deploy schema change on db2040 (s7 codfw master) with replication, this will generate lag on s7 codfw T144010 T51190 T199368
  • 10:05 ema: upgrade varnish to 5.1.3-1wm9 on text-eqiad T164609
  • 09:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 (duration: 00m 47s)
  • 09:53 godog: bounce grafana on krypton
  • 09:31 ema: upload varnish 5.1.3-1wm9 to apt.w.o (fixing POST requests w/ separate VCL) T164609
  • 09:20 gehel: restarting elasticsearch cluster restart on codfw - T156137
  • 08:41 jynus: stop es1014 for reimage
  • 08:14 marostegui: Deploy schema change on db1121 with replication, this will generate lag on labsdb hosts for s4 T144010 T51190 T199368
  • 08:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 (duration: 00m 47s)
  • 08:10 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s4 populateChangeTagDef.php --sleep 2 (T193873)
  • 08:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 db1091 (duration: 00m 47s)
  • 08:02 gehel: pausing elasticsearch cluster restart on codfw - T156137
  • 07:58 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1014 (duration: 00m 48s)
  • 07:35 gehel: rolling restart of elasticsearch / cirrus / codfw to disable G1 - T156137
  • 07:12 marostegui: Deploy schema change on db1091 T144010 T51190 T199368
  • 06:53 gehel: resetting postgres data on maps1004 after failing replication - T200228
  • 06:38 marostegui: Stop replication in sync on db1091 and db1097:3314
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 db1091 (duration: 00m 48s)
  • 06:33 jynus: finished es1014 -> es1017 switch T197073
  • 06:27 jynus: enabling semi-sync master on es1017, disabling it as client
  • 06:21 jynus: deploy es3-master dns change
  • 06:02 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Switchover es3 master eqiad from es1014 to es1017 (duration: 00m 24s)
  • 06:01 jynus: switchover es3 eqiad master from es1014 to es1017
  • 04:35 tstarling@deploy1001: Synchronized php-1.32.0-wmf.13/includes/api/ApiMain.php: record all API requests in statsd (duration: 00m 49s)
  • 02:43 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 25 02:43:05 UTC 2018 (duration 10m 19s)
  • 02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 28s)

2018-07-24

  • 21:59 XioNoX: re-pooling eqsin
  • 21:15 XioNoX: re1 is master routing engine on cr1-eqsin, triggering a re switch
  • 21:10 XioNoX: starting to see recoveries from cr1-eqsin upgrade
  • 21:06 XioNoX: Install done, cr1-eqsin re-rebooting
  • 21:00 XioNoX: restarting cr1-eqsin for software upgrade
  • 20:32 XioNoX: depooling eqsin for cr1-eqsin software upgrade
  • 19:09 gehel: resetting postgres data on maps1003 after failing replication - T200228
  • 18:34 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@690fdad]: Wait for the client to consume the meesage being sent before consuming the next one - T199813 (duration: 02m 18s)
  • 18:32 mobrovac@deploy1001: Started deploy [eventstreams/deploy@690fdad]: Wait for the client to consume the meesage being sent before consuming the next one - T199813
  • 17:40 ema: re-enable puppet on all cache nodes with alternate domains disabled T164609
  • 17:33 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.13
  • 17:18 thcipriani: train window running long, services deploy delayed
  • 17:18 ema: restart varnish-fe on cp1068 to clear "child restarted" alert T164609
  • 17:17 elukey: restart eventstreams on scb2* nodes (hopefully last time before deploying the fix) to avoid mem leaks issues during the EU night
  • 17:06 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.14/includes/page/PageArchive.php: PageArchive: Pass correct overrides to newRevisionFromArchiveRow() (T200072) (duration: 01m 01s)
  • 16:58 jynus: finishing test on es3 hosts T199224
  • 16:42 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.13/includes/page/PageArchive.php: PageArchive: Pass correct overrides to newRevisionFromArchiveRow() (T200072) (duration: 01m 03s)
  • 16:07 jynus: test switchover from es2018 to es2017
  • 16:02 dcausse: T156137: unbanning elastic1031
  • 15:59 dcausse: T156137: restarting elasticsearch on elastic1031 to disable G1GC
  • 15:55 jynus: test switchover from es2017 to es2018
  • 15:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 (duration: 01m 02s)
  • 15:38 jynus: stopping puppet on es2017, es2018; changing mysql configuration for production testing
  • 15:29 gehel: restart postgres on maps1001 - T200228
  • 14:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 01m 02s)
  • 14:47 dcausse: T156137: banning elastic1031 due to high load (same "getEntryAfterMiss" symptoms)
  • 14:09 marostegui: Deploy schema change on db1103:3314 T144010 T51190 T199368
  • 13:45 ema: apply alternate domains patch to text-eqiad T164609
  • 13:43 marostegui: Deploy schema change on db1084 T144010 T51190 T199368
  • 13:42 marostegui: Stop replication in sync db1084 and db1103:3314
  • 13:40 marostegui: Deploy schema change on db1081 T144010 T51190 T199368
  • 13:38 marostegui: Stop replication in sync db1081 and db1103:3314
  • 13:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 01m 59s)
  • 13:07 ema: repool cp1067 with alternate domains support T164609
  • 12:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 (duration: 00m 55s)
  • 12:12 gehel: vacuum full of postgres on maps1001 to try to reclaim space - T200228
  • 12:07 ema: depool cp1067 to test alternate domains patch T164609
  • 11:58 zfilipin@deploy1001: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4179557944" --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 02m 50s)
  • 11:55 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache
  • 11:45 zfilipin@deploy1001: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2212739269" --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 03m 42s)
  • 11:42 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache
  • 11:35 ema: disable puppet on cp-text hosts to merge alternate domains patch T164609
  • 11:21 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s1 populateChangeTagDef.php (T193873)
  • 11:19 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s2 populateChangeTagDef.php --sleep 2 (T193873)
  • 11:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 00m 55s)
  • 11:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 (duration: 01m 06s)
  • 10:44 marostegui: Stop replication in sync on db1084 and db1097:3314
  • 10:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 55s)
  • 09:17 marostegui: Deploy schema change on db1097:3314 T144010 T51190 T199368
  • 09:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 (duration: 00m 54s)
  • 08:38 ema: restart varnish-fe on cache_text instances with cold, labeled VCL T200207
  • 08:21 elukey: rolling restart of kafka jumbo/main-(eqiad|codfw) clusters to pick up the new max open files limit (infinity -> 128k)
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 (duration: 00m 55s)
  • 06:32 marostegui: Deploy schema change on dbstore1002:s4 T144010 T51190 T199368
  • 05:33 kartik@deploy1001: Finished deploy [cxserver/deploy@d378d27]: Update cxserver to d3c9d15 (T198941) (duration: 04m 01s)
  • 05:29 kartik@deploy1001: Started deploy [cxserver/deploy@d378d27]: Update cxserver to d3c9d15 (T198941)
  • 05:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 (duration: 00m 54s)
  • 05:05 marostegui: Deploy schema change on db1081 T144010 T51190 T199368
  • 04:54 marostegui: Stop replication in sync on db1081 and db1121
  • 04:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084, db1121 (duration: 00m 56s)
  • 04:44 marostegui: Deploy schema change on db1066 (s2 primary master) T144010 T51190 T199368
  • 03:03 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 24 03:03:08 UTC 2018 (duration 10m 22s)
  • 02:52 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 09m 19s)
  • 02:23 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 09m 44s)

2018-07-23

  • 22:53 mobrovac@deploy1001: Started restart [eventstreams/deploy@5ed03e0]: A ggodnight restart - T199813
  • 22:27 thcipriani@deploy1001: Finished scap: Revert "Accept BCP 47 codes as aliases for nonstandard variants" T199941 Revert "Ensure LanguageCode::bcp47() returns a valid BCP 47 language code" T199941 (duration: 33m 21s)
  • 21:53 thcipriani@deploy1001: Started scap: Revert "Accept BCP 47 codes as aliases for nonstandard variants" T199941 Revert "Ensure LanguageCode::bcp47() returns a valid BCP 47 language code" T199941
  • lunch: T156137: restarting elastic1035 to disable G1GC
  • lunch: T156137: restarting elastic1027 to disable G1GC
  • 20:27 arlolra: Updated Parsoid to a1e851c (T199808, T194083)
  • 20:22 arlolra@deploy1001: Finished deploy [parsoid/deploy@80384a5]: Updating Parsoid to a1e851c (duration: 09m 45s)
  • 20:13 arlolra@deploy1001: Started deploy [parsoid/deploy@80384a5]: Updating Parsoid to a1e851c
  • 20:09 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@565b41a]: Update mobileapps to 254cef5 (duration: 05m 56s)
  • 20:03 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@565b41a]: Update mobileapps to 254cef5
  • 19:49 gehel@deploy1001: Finished deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (duration: 11m 35s)
  • 19:38 gehel@deploy1001: Started deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI
  • 19:37 gehel@deploy1001: Finished deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 22s)
  • 19:36 gehel@deploy1001: Started deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (wdqs1009 only)
  • 17:56 gehel@deploy1001: Finished deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 25s)
  • 17:56 gehel@deploy1001: Started deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only)
  • 17:53 ejegg: updated payments-wiki from d8d8293b87 to 6390e50abd
  • 17:47 gehel@deploy1001: Finished deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 03s)
  • 17:47 gehel@deploy1001: Started deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only)
  • 17:29 XenoRyet: updated payments-wiki from 63372154d0 to d8d8293b87
  • 17:16 gehel: canceling deployment of new WDQS GUI to all servers, looks like there is a JS issue. Will reschedule once the issue is fixed.
  • 17:07 gehel@deploy1001: Finished deploy [wdqs/wdqs@d7e8292]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 30s)
  • 15:25 gehel: decrease gc_grace_seconds to 4 days on cassandra / maps
  • 15:02 mobrovac@deploy1001: Finished deploy [changeprop/deploy@5cfdf26]: Increase concurrency back to normal levels (from 20 to 50) (duration: 01m 38s)
  • 15:00 mobrovac@deploy1001: Started deploy [changeprop/deploy@5cfdf26]: Increase concurrency back to normal levels (from 20 to 50)
  • 14:57 mobrovac@deploy1001: Started restart [eventstreams/deploy@5ed03e0]: Fresh start of the service after alleviating 404s
  • 14:53 elukey: delete empty/not-used/wrongly-created topics in Kafka main-eqiad - T199510
  • 14:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081, db1097:3314 (duration: 00m 53s)
  • 14:17 marostegui: Stop replication in sync on db1081 and db2051
  • 14:00 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@5ed03e0]: Update node-rdkafka to v2.3.4 - T199813 (duration: 02m 29s)
  • 13:57 mobrovac@deploy1001: Started deploy [eventstreams/deploy@5ed03e0]: Update node-rdkafka to v2.3.4 - T199813
  • 13:53 marostegui: Stop replication in sync on db1081 and db1097:3317
  • 13:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081, db1097:3314 (duration: 00m 54s)
  • 13:44 ema: !log trafficserver 7.1.3+ds-4wm1 uploaded to stretch-wikimedia T200178
  • 13:36 marostegui: Deploy schema change on s4 codfw master (db2051), this will generate lag on s4 codfw T144010 T51190 T199368
  • 13:23 marostegui: Deploy schema change on labswiki and labstestwiki T144010 T51190 T199368
  • 13:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074, db1105:3312 (duration: 00m 55s)
  • 11:34 zeljkof: EU SWAT finished
  • 11:32 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Exempt Template and Module namespace on zhwiktionary (T187783) (duration: 00m 55s)
  • 11:14 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Change zhwikiquote logo (T199863) (duration: 00m 55s)
  • 10:10 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 54s)
  • 10:09 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 55s)
  • 09:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074, db1105:3312 (duration: 00m 53s)
  • 09:42 marostegui: Stop replication in sync on db1105:3312 db1074
  • 09:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122, db1103:3312 (duration: 00m 54s)
  • 09:00 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s8 populateChangeTagDef.php (T193873)
  • 08:47 marostegui: Stop replication in sync on db1103:3312 db2035
  • 08:42 godog: enable deprecation page for status.w.o - T199816
  • 08:23 marostegui: Stop replication in sync on db1122 and db1103:3312
  • 08:21 marostegui: Apply schema change T192926 to labswiki and labstestwiki T200140
  • 08:15 marostegui: Apply schema change T160415 to labswiki and labstestwiki T200140
  • 08:11 marostegui: Apply schema change  T191316 to labswiki and labstestwiki T200140
  • 08:07 marostegui: Apply schema change  T191519 to labswiki and labstestwiki T200140
  • 08:05 marostegui: Apply schema change  T190148 to labswiki and labstestwiki T200140
  • 08:04 marostegui: Apply schema change  T197891 to labswiki and labstestwiki T200140
  • 07:54 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s6 populateChangeTagDef.php (T193873)
  • 07:54 marostegui: Stop replication in sync on db1122 and db2035
  • 07:33 marostegui: Deploy schema change on db1122 T144010 T51190 T199368
  • 07:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122, db1103:3312 (duration: 00m 55s)
  • 06:37 jynus: stop db1102 to clone to db1118
  • 04:42 marostegui: Deploy schema change on db1061 (s6 primary master) T144010 T51190 T199368
  • 03:08 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 23 03:08:23 UTC 2018 (duration 10m 37s)
  • 02:57 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 51s)
  • 02:24 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 09m 42s)

2018-07-22

  • 16:15 elukey: rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix)
  • 12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.76.159
  • 12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.76.150
  • 12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.68.116
  • 12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 196.208.95.30
  • 12:03 addshore@deploy1001: Synchronized wmf-config/throttle.php: T198288 More Wikimania throttle rules (duration: 01m 18s)

2018-07-21

  • 22:29 Reedy: Added ct_tag_id to labswiki and labtestwiki T200139
  • 17:35 elukey: rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix)

2018-07-20

  • 21:50 bawolff: deployed patch T200104
  • 17:16 XioNoX: enable bgp on eqdfw-knams link (GTT)
  • 17:08 XioNoX: enable ospf on eqdfw-knams link (GTT)
  • 15:33 elukey: rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix)
  • 15:19 elukey: powercycle ms-be1016 - RAID errors, no ssh available, I/O errors in com2 console
  • 14:15 reedy@deploy1001: rebuilt and synchronized wikiversions files: wikidatawiki back to .12 T199983
  • 14:11 Reedy: T199983 wikidata wiki abck to .12 on mwdebug1001
  • 13:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085, db1113:3316 (duration: 00m 55s)
  • 13:13 marostegui: Stop replication in sync db1085 and db1113:3316
  • 13:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316, depool db1113:3316 (duration: 00m 54s)
  • 13:06 legoktm@deploy1001: Synchronized wmf-config/throttle.php: Update Wikimania IP address - T198288 (duration: 00m 54s)
  • 12:53 marostegui: Stop replication in sync db1085 and db1096:3318
  • 12:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316, depool db1098:3316 (duration: 00m 54s)
  • 12:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 and db1096:3316 (duration: 00m 56s)
  • 12:32 marostegui: Stop replication in sync db1085 and db1096:3316
  • 12:31 marostegui@deploy1001: sync-file aborted: Depool db1085 and db1096:33156 (duration: 00m 01s)
  • 12:00 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@01fac88]: Lower librdkafka settings related to fetching messages - T199813 (duration: 02m 11s)
  • 11:58 mobrovac@deploy1001: Started deploy [eventstreams/deploy@01fac88]: Lower librdkafka settings related to fetching messages - T199813
  • 10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312, db1105:3312 (duration: 00m 53s)
  • 10:30 marostegui: Stop replication in sync on db1090:3312 and db1105:3312 - T200061
  • 10:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312, depool db1105:3312 (duration: 00m 54s)
  • 10:17 marostegui: Stop replication in sync on db1090:3312 and db2035 - T200061
  • 10:14 marostegui: Stop replication in sync on db1090:3312 and db1103:3312 - T200061
  • 10:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 and db1103:3312 (duration: 00m 55s)
  • 09:46 marostegui: Stop replication on db2063 to fix db2095 - T200061
  • 08:50 marostegui: Stop replication on s7 codfw master to check arwiki.change_tag - T200061
  • 08:47 Amir1: stopping populateChangeTagDef.php (T200061)
  • 08:41 foks: reset email for User:Nomden
  • 08:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 55s)
  • 07:46 marostegui: Stop replicaiton on s8 codfw master - T200061
  • 07:43 jynus: stopping db1095 mariadb and cloning it to db1102
  • 07:14 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099, es1019 fully (duration: 00m 54s)
  • 07:11 marostegui: Stop replication in sync on db1074 and db1125:3312
  • 06:13 marostegui: Deploy schema change on db1074 with replication, this will generate lag on labsdb:s2 T144010 T51190 T199368
  • 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 53s)
  • 06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 after alter table (duration: 00m 54s)
  • 05:49 marostegui: Deploy schema change on db1076 T144010 T51190 T199368
  • 05:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 for alter table (duration: 00m 56s)
  • 05:08 marostegui: Start to remove some unused files on db1067 - T200039
  • 01:38 krinkle@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/CentralAuth/includes/CentralAuthUser.php: T170971 (duration: 00m 55s)
  • 00:31 krinkle@deploy1001: Synchronized php-1.32.0-wmf.13/includes/filerepo/: I58706b5610 and I40f6ad2a3d - T200026 (duration: 00m 56s)
  • 00:30 ejegg: updated CiviCRM from 22a8c47668 to d76e3997e6
  • 00:12 bblack: disabled puppet temporary on cp* and kafka-jumbo* for ipsec unconfiguration

2018-07-19

  • 23:43 ejegg: updated CiviCRM from 08680be68e to 22a8c47668
  • 23:31 ejegg: updated CiviCRM from 2bb0d0f9d0 to 08680be68e
  • 23:11 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable Special:Block Feedback Request" (duration: 00m 55s)
  • 22:51 XioNoX: disable ospf on eqdfw-knams link (GTT)
  • 22:18 XioNoX: enable ospf on eqdfw-knams link (GTT)
  • 21:52 hashar: CI Jenkins has been upgraded successfully!
  • 21:21 hashar: restarting CI on contint1001 and actually upgrading it
  • 21:19 hashar: starting CI jenkins on contint1001
  • 21:16 hashar: stopping the CI jenkins on contint1001
  • 20:20 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099 with low load (duration: 00m 54s)
  • 19:25 ariel@deploy1001: Finished deploy [dumps/dumps@f0c4a70]: handle empty tables properly; script to show runtimes of dump steps (duration: 00m 03s)
  • 19:25 ariel@deploy1001: Started deploy [dumps/dumps@f0c4a70]: handle empty tables properly; script to show runtimes of dump steps
  • 19:21 ejegg: updated CiviCRM from 6f311f48d9 to 2bb0d0f9d0
  • 19:18 XioNoX: setting mtu 9192 to all codfw interface ranges
  • 19:04 elukey: roll restart eventstreams on scb2* hosts to prevent OOM issues over the EU night - T199813
  • 19:00 Amir1: Morning SWAT is done
  • 18:57 ladsgroup@deploy1001: Synchronized wmf-config/abusefilter.php: SWAT: Add abusefilter-modify-restricted right to sysops on itwiktionary (T199783) (duration: 00m 53s)
  • 18:50 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/WikibaseQualityConstraints: SWAT: Report SPARQL errors as TODO, not VIOLATION (T199788) (duration: 00m 56s)
  • 18:27 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/WikibaseQualityConstraints: SWAT: Report SPARQL errors as TODO, not VIOLATION (T199788) (duration: 00m 56s)
  • 18:18 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist all populateChangeTagDef.php (T193873) It will take a couple of days
  • 18:14 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set to write the new change tag backend everywhere, enable reading in frwiki (T194165) (duration: 00m 55s)
  • 16:28 jynus: stop, clone away and upgrade db1099 two mysql instances
  • 16:16 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully depool db1099, including from s8 (duration: 00m 55s)
  • 16:00 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 15:53 hashar: CI: npmjs is slow. Bumping Quibble jobs timeout from 30 to 45 minutes | https://gerrit.wikimedia.org/r/#/c/446863 | T198348
  • 15:50 jynus: droping undocumented grants on labsdb1005 using replication T199186
  • 15:21 volans: reimaging mw2226 to test py3 reimage with new cumin, conftool and apache test from neodymium - T187773
  • 15:12 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2225.codfw.wmnet
  • 15:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 after alter table (duration: 00m 54s)
  • 15:00 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099 (duration: 00m 54s)
  • 14:59 moritzm: rebooting multatuli for kernel test
  • 14:30 marostegui: Deploy schema change on db1105:3312 T144010 T51190 T199368
  • 14:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 for alter table (duration: 00m 54s)
  • 14:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 after alter table (duration: 00m 54s)
  • 14:17 elukey: roll restart kafka on kafka-jumbo* and kafka main-codfw (kafka2*) to pick up new Xmx/Xms settings (1g -> 2g)
  • 14:15 volans: upgrading cumin/debdeploy/reimage and other tools on neodymium - T187773
  • 13:57 elukey: restart kafka on kafka-jumbo1001 to raise Xmx/Xms jvm settings (1g -> 2g)
  • 13:43 marostegui: Deploy schema change on db1103:3312 T144010 T51190 T199368
  • 13:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for alter table (duration: 00m 54s)
  • 13:38 volans: reimaging mw2225 to test py3 reimage with new cumin, conftool and apache test - T187773
  • 13:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1103:3314 (duration: 00m 53s)
  • 13:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
  • 13:22 moritzm: installing python-pysaml2 security updates
  • 13:05 moritzm: installing ffmpeg security updates
  • 12:48 moritzm: installing xerces-c security updates
  • 12:38 moritzm: uploaded debdeploy 0.0.99.5 to apt.wikimedia.org
  • 11:49 zeljkof: EU SWAT finished
  • 11:46 zfilipin@deploy1001: Synchronized dblists/categories-rdf.dblist: SWAT: Remove labs wikis from the categories-rdf list, dont need them (T198356) (duration: 00m 55s)
  • 11:44 moritzm: installing reportbug update from stretch point release to test py3-based debdeploy
  • 11:40 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:Block Feedback Request (T199919) (duration: 00m 55s)
  • 11:33 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Structured Discussions beta feature on orwiki (T199971) (duration: 00m 55s)
  • 11:22 volans: testing reimage of californium (spare host) with the upgraded cumin
  • 11:20 chasemp: maintain-views --all-databases --replace-all --table ores_classification --debug (labsdb)
  • 11:19 chasemp: maintain-views --all-databases --replace-all --table ores_model --debug (labsdb)
  • 11:18 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: enable TemplateStyles on enwiki, frwiki, zhwiki T197603 T191452 T189022 (duration: 00m 55s)
  • 11:00 volans: upgrading cumin on sarin - T187773
  • 10:52 legoktm: made myself a crat on test2wiki
  • 10:06 moritzm: installing file/libmagic security updates
  • 09:55 moritzm: installing check-postgres update from stretch 9.5 point release
  • 09:49 aaron@deploy1001: Synchronized php-1.32.0-wmf.13/includes/libs/objectcache/MultiWriteBagOStuff.php: 2cee0ab (duration: 00m 55s)
  • 09:35 volans: cumin upgrade on labpuppetmaster* hosts completed - T187773
  • 09:33 moritzm: installing openssl security updates
  • 09:33 volans: removed leftover cumin installation from labtestpuppetmaster2001 - T187773
  • 09:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
  • 09:27 godog: run xfs_repair on filesystems reporting negative space available on ms-be1040 - T199198
  • 09:24 moritzm: installing tomcat7 security updates
  • 09:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 54s)
  • 09:21 moritzm: installing ant security updates
  • 09:14 moritzm: uploaded jenkins 2.121.2 to apt.wikimedia.org (T199448)
  • 08:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
  • 08:39 volans: start upgrade of cumin to 3.0.1-1 on labpuppetmaster* - T187773
  • 08:32 marostegui: Deploy schema change on dbstore1002:s2 T144010 T51190 T199368
  • 08:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
  • 08:08 elukey: restart eventstreams on scb2* hosts to pick up new Kafka settings (pointing it to main-codfw) - T199813
  • 08:04 kartik@deploy1001: Finished deploy [cxserver/deploy@fe0d521]: Update cxserver to 92e3d2e (T199380, T199885) (duration: 03m 58s)
  • 08:00 kartik@deploy1001: Started deploy [cxserver/deploy@fe0d521]: Update cxserver to 92e3d2e (T199380, T199885)
  • 07:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
  • 07:55 marostegui: Deploy schema change on s2 codfw masters (db2035) this will generate lag on s2 codfw T144010 T51190 T199368
  • 07:50 jynus: stop es1019 and reimage it
  • 07:48 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 54s)
  • 07:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3312 after MySQL upgrade (duration: 00m 55s)
  • 07:26 moritzm: removing cdrom and sr_mod (related module) kernel modules across the fleet (following the blacklisting of the kernel module via puppet)
  • 06:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3312 after MySQL upgrade (duration: 00m 54s)
  • 05:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3312 after MySQL upgrade (duration: 00m 55s)
  • 05:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1103:3312 after MySQL upgrade (duration: 00m 54s)
  • 05:14 marostegui: Stop MySQL on db1103 to upgrade MySQL
  • 05:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for MySQL upgrade (duration: 00m 56s)
  • 04:12 kart_: Finished ContentTranslation draft purge script on mwmaint1001 (T199658)
  • 04:05 kart_: Starting ContentTranslation draft purge script on mwmaint1001, dry run first and then --really
  • 03:57 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I978f3eda8 - Farewell temporary hack from 2013 (duration: 00m 56s)
  • 03:20 ejegg: updated CiviCRM from b21d783e7f to 6f311f48d9
  • 03:09 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 19 03:09:19 UTC 2018 (duration 10m 35s)
  • 02:58 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 15m 05s)
  • 02:24 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 10m 41s)

2018-07-18

  • 23:35 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.13/includes/logging/LogEventsList.php: SWAT: LogEventsList: Use GET in HTMLForm T199856 (duration: 00m 54s)
  • 23:33 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWMediaDialog.js: SWAT: ve.ui.MWMediaDialog: Fix confusion between #getSetupProcess and #getReadyProcess T185944 T199841 (duration: 00m 56s)
  • 22:16 ejegg: updated SmashPig payments listener from 2b4186715e to 2736e41045
  • 19:24 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Unset wgVipsThumbnailHost T199937 (duration: 00m 55s)
  • 16:28 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable new backend for Special:Tags on fawikisource (T199334) (duration: 00m 54s)
  • 16:14 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES on testwiki (T199913) (duration: 00m 55s)
  • 15:25 volans: installing conftool 1.0.1 on jessie and stretch
  • 14:50 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@09f0efe]: Set the maximum rdkafka receive buffer size to 64MB - T199813 (duration: 02m 28s)
  • 14:48 mobrovac@deploy1001: Started deploy [eventstreams/deploy@09f0efe]: Set the maximum rdkafka receive buffer size to 64MB - T199813
  • 14:47 marostegui: Partition commonswiki.logging table on db1103:3314 - T199790
  • 14:46 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2224.codfw.wmnet
  • 14:43 volans: uploaded conftool (with python-conftool and python3-conftool) version 1.0.1 for jessie and stretch to apt.w.o
  • 14:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table - T199790 (duration: 00m 52s)
  • 14:37 moritzm: rebooting poolcounter2001 for some tests
  • 14:22 moritzm: rebooting vega for some tests
  • 13:59 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 54s)
  • 13:53 moritzm: installing ruby2.1 security updates for jessie
  • 13:17 marostegui: Deploy schema change on db1052 - T187089
  • 13:13 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.13 (duration: 00m 53s)
  • 13:13 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.13
  • 12:58 marostegui: Stop MySQL on db1052 to upgrade socket and MySQL
  • 12:57 elukey: restart eventstreams on scb200[5,6] as precautionary after mem consumption too high
  • 12:57 elukey: restart eventstreams on scb2003 as precautionary after mem consumption too high
  • 12:56 elukey: restart eventstreams on scb2001 as precautionary after mem consumption too high (still investigating a fix)
  • 12:54 dcausse: T156137: unbanning elastic1030
  • 12:52 dcausse: T156137: restarting elastic1030 to disable G1GC
  • 12:51 moritzm: installing PHP5 security updates (remaining hosts which only use -cli)
  • 12:48 moritzm: installing PHP security updates on tungsten
  • 12:36 moritzm: installing PHP security updates on icinga.wikimedia.org
  • 12:24 elukey: manually added queued.max.messages.kbytes: 65535 to eventstreams on scb2002 as test for T199813
  • 12:16 moritzm: installing PHP security updates on yubiauth servers
  • 12:02 moritzm: installing PHP security update on bohrium
  • 11:35 moritzm: installing imagemagick security updates
  • 11:34 zeljkof: EU SWAT finished
  • 11:32 zfilipin@deploy1001: Synchronized wmf-config/abusefilter.php: SWAT: Enable AbuseFilter block option on itwiktionary (T199783) (duration: 00m 54s)
  • 11:25 moritzm: installing policykit security updates on trusty
  • 11:20 Amir1: making ores tables on euwiki
  • 11:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Publisher namespace in Bengali Wikisource (T199028) (duration: 00m 54s)
  • 11:05 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ores extension wp10 storage in Basque Wikipedia (T198358) (duration: 00m 54s)
  • 10:30 moritzm: ran puppet clean/deactivate for lawrencium, long-standing source of errors in package deployments (T191360)
  • 10:24 moritzm: installing cups/libcups security updates for jessie
  • 10:10 dcausse: T156137: banning elastic1030 from the cluster (high load)
  • 10:02 dcausse: clearing internal cache on elastic1030
  • 09:58 hashar: contint1001 dropping unused docker containers
  • 09:36 moritzm: draining restbase1007 for eventual reboot for tests with SSBD microcode
  • 08:51 godog: run xfs_repair on filesystems reporting negative space available on ms-be1043 - T199198
  • 08:50 moritzm: reboot ms-be1040 for tests with SSBD microcode
  • 08:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 53s)
  • 08:38 twentyafterfour: restart apache on phab1001
  • 08:37 chasemp: deploy antivandalism stuff to phab https://gerrit.wikimedia.org/r/c/operations/puppet/+/445329 (needs herald to fully enable)
  • 08:31 elukey: drain + reboot analytics1030 for kernel updates
  • 08:29 marostegui: Deploy schema change on db1085 with replication, this will generate lag on labsdb hosts for s6 T144010 T51190 T199368
  • 08:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 01m 01s)
  • 08:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 after alter table (duration: 00m 53s)
  • 08:06 marostegui: Deploy schema change on db1088 T144010 T51190 T199368
  • 08:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 53s)
  • 07:41 jforrester@deploy1001: Synchronized wmf-config/missing.php: Fix typo in missing.php Ia3aae912d9m (duration: 00m 54s)
  • 07:37 moritzm: upgrading tor on radium to 0.3.3.9
  • 07:25 marostegui: Deploy schema change on db1052 - https://phabricator.wikimedia.org/T191316 https://phabricator.wikimedia.org/T192926 https://phabricator.wikimedia.org/T89737 https://phabricator.wikimedia.org/T195193
  • 07:22 moritzm: updated tor on apt.wikimedia.org to 0.3.3.9
  • 06:45 jynus: enable semi-sync on db1099:3311 and db1105:3311
  • 06:08 marostegui: s1 failover finished T197069
  • 06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: read only OFF after failover T197069 (duration: 00m 53s)
  • 06:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Set db1067 as master T197069 (duration: 00m 53s)
  • 06:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Set s1 on ready only for maintenance T197069 (duration: 01m 08s)
  • 06:00 marostegui: Starting s1 failover from db1052 to db1067 - T197069
  • 05:11 Krinkle: mwscript deleteEqualMessages.php --wiki dewiki (1 page) - T45917
  • 05:11 Krinkle: mwscript deleteEqualMessages.php --wiki ruwiki (4 pages) - T45917
  • 05:11 Krinkle: mwscript deleteEqualMessages.php --wiki frwiki (5 pages) - T45917
  • 05:09 Krinkle: mwscript deleteEqualMessages.php --wiki metawiki (deleted 1 page) - T45917
  • 05:09 Krinkle: mwscript deleteEqualMessages.php --wiki test2wiki (deleted 2 pages) - T45917
  • 04:56 marostegui: Starting s1 failover pre steps - https://phabricator.wikimedia.org/T197069
  • 03:05 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 18 03:05:46 UTC 2018 (duration 10m 27s)
  • 02:55 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 09m 43s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 08m 35s)
  • 00:00 reedy@deploy1001: Finished scap: log event fixes (duration: 33m 18s)

2018-07-17

  • 23:26 reedy@deploy1001: Started scap: log event fixes
  • 22:43 reedy@deploy1001: Synchronized wmf-config/interwiki-labs.php: (no justification provided) (duration: 00m 53s)
  • 22:24 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove override for foundationwiki in wgMobileUrlTemplate (duration: 00m 53s)
  • 22:21 reedy@deploy1001: Synchronized tests/urls.txt: testding! (duration: 00m 53s)
  • 20:57 reedy@deploy1001: Synchronized wmf-config/: make foundation.wikimedia.org writeable T188776 (duration: 00m 53s)
  • 20:50 reedy@deploy1001: Finished scap: updating l10n cache for foundationwiki url changes (duration: 58m 32s)
  • 20:17 ebernhardson: un-ban elastic1030 from eqiad search cluster
  • 19:52 reedy@deploy1001: Started scap: updating l10n cache for foundationwiki url changes
  • 19:35 reedy@deploy1001: Synchronized portals: T199808 (duration: 00m 54s)
  • 19:34 reedy@deploy1001: Synchronized portals/wikipedia.org/assets: T199808 (duration: 00m 53s)
  • 18:46 bblack: re-enabling puppet on mw* for more foundationwiki work - T188776
  • 18:39 ebernhardson: ban elastic1030 from eqiad search cluster for latency issues. Likely related to T156137
  • 18:38 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 53s)
  • 18:36 bblack: disabling puppet on mw* for more foundationwiki work - T188776
  • 18:33 reedy@deploy1001: Synchronized wmf-config/: foundationwiki (duration: 00m 54s)
  • 18:33 elukey: rolling restart eventstreams on scb2* nodes to avoid OOMs during the EU night
  • 18:32 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: foundationwiki (duration: 00m 53s)
  • 18:31 reedy@deploy1001: Synchronized tests/: foundationwiki (duration: 00m 53s)
  • 18:29 reedy@deploy1001: Synchronized docroot/: update foundationwiki urls (duration: 00m 54s)
  • 18:28 reedy@deploy1001: Synchronized errorpages/: update foundationwiki urls (duration: 00m 54s)
  • 18:17 reedy@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/ConfirmEdit: Make a function public again (duration: 00m 56s)
  • 17:55 bblack: re-enabling puppet on mw* for foundationwiki apache config change T188776
  • 17:00 bblack: disabling puppet on most of mw* for foundationwiki apache config deploy - T188776
  • 16:52 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Make foundationwiki readonly (duration: 00m 54s)
  • 16:49 moritzm: installing faad2 security updates
  • 15:51 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/tests/phpunit/includes/libs/objectcache/BagOStuffTest.php: (no justification provided) (duration: 00m 54s)
  • 15:49 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/includes/libs/objectcache/BagOStuff.php: 2d919ad (duration: 00m 56s)
  • 15:42 XioNoX: switching default analytics-in6 term to reject+log - T198623
  • 15:40 moritzm: rebooting ms-fe1006 with SSBD-enabled microcode
  • 15:18 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2224.codfw.wmnet
  • 14:58 volans: repooled mw2224
  • 14:50 moritzm: rebooting ms-fe1005 with SSBD-enabled microcode
  • 14:47 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.13
  • 14:26 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache (duration: 06m 14s)
  • 14:19 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache
  • 14:18 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.999 (duration: 03m 01s)
  • 14:12 moritzm: rebooting wtp1025 with SSBD-enabled microcode
  • 14:07 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.8 (duration: 03m 11s)
  • 14:03 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.7 (duration: 03m 39s)
  • 13:54 moritzm: rebooting mw1261 with SSBD-enabled microcode
  • 13:35 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.6 (duration: 06m 18s)
  • 13:27 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache (duration: 62m 17s)
  • 13:25 moritzm: pruned obsolete hosts from debmonitor
  • 13:06 marostegui: Drop unused grants on db1073, db1117:3323, db1117:3325
  • 12:25 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache
  • 12:03 zeljkof: EU SWAT finished
  • 11:58 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Thesaurus NS at thwikt (T198585) Fix problem with namespaces at patch 446005 (T198585) (duration: 00m 51s)
  • 11:46 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/ContentTranslation: SWAT: Fix colors on CX1 (T199503) (duration: 00m 55s)
  • 11:22 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ULS webfonts by default at Burmese Wikipedia (mywiki) (T196219) (duration: 01m 50s)
  • 11:15 mobrovac@deploy1001: Finished deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #4 (duration: 02m 05s)
  • 11:12 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #4
  • 11:12 mobrovac@deploy1001: Finished deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #3 (duration: 07m 29s)
  • 11:04 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #3
  • 11:03 mobrovac@deploy1001: Finished deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #2 - T199458 T195390 (duration: 19m 43s)
  • 10:43 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #2 - T199458 T195390
  • 10:31 moritzm: restarting archiva to pick up OpenJDK security update
  • 10:31 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20 - T199458 T195390
  • 10:14 marostegui: Deploy schema change on db1093 T144010 T51190 T199368
  • 10:08 volans: reimage mw2224 to test the new stretch netboot and wmf-auto-reimage changes
  • 10:07 moritzm: restarting zookeeper in codfw to pick up Java security updates
  • 09:41 marostegui: Deploy schema change on db1113:3316 T144010 T51190 T199368
  • 09:38 marostegui: Drop unused grants from db2037, db2042, db2078:3323, db2078:3325
  • 09:11 marostegui: Deploy schema change on db1098:3316 T144010 T51190 T199368
  • 08:51 moritzm: installing libipc-run-perl update from stretch 9.5 point release
  • 08:31 marostegui: Deploy schema change on db1096:3316 T144010 T51190 T199368
  • 07:50 volans@deploy1001: Finished deploy [debmonitor/deploy@691d2f8]: Release v0.1.6 (duration: 00m 41s)
  • 07:50 volans@deploy1001: Started deploy [debmonitor/deploy@691d2f8]: Release v0.1.6
  • 07:29 godog: un xfs_repair on filesystems reporting negative space available on ms-be1042 - T199198
  • 07:24 moritzm: installing xapian-core security updates
  • 07:17 marostegui: Drop unused grants on es eqiad hosts
  • 07:03 moritzm: updated stretch netinst image for 9.5
  • 06:42 moritzm: installing patch security updates
  • 06:39 moritzm: installing subversion security updates
  • 06:32 marostegui: Drop unused grants on es codfw hosts
  • 06:30 moritzm: installing postgresql-9.6 updates from stretch point release
  • 05:56 marostegui: Deploy schema change on db2039 (s6 codfw master) with replication, this will generate lag on codfw T144010 T51190 T199368
  • 05:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 00m 49s)
  • 05:34 marostegui: Deploy schema change on db1070 (s5 primary master) T144010 T51190 T199368
  • 05:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 00m 49s)
  • 05:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 after alter table (duration: 00m 49s)
  • 05:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 00m 49s)
  • 05:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 00m 49s)
  • 05:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 00m 50s)
  • 04:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 49s)
  • 04:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 53s)
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 17 02:45:01 UTC 2018 (duration 10m 21s)
  • 02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 13m 51s)

2018-07-16

  • 20:57 ejegg: re-enabled payments listener failmail stream
  • 20:11 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@fcae441]: Update mobileapps to bed7b29 (T174809) (duration: 09m 01s)
  • 20:02 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@fcae441]: Update mobileapps to bed7b29 (T174809)
  • 18:35 mepps: updated payments-wiki config to 4e0c6c3cfd
  • 18:10 niharika29@deploy1001: Synchronized wmf-config/: Rollout Watchlist Structured Filters to all wikis T181193 (duration: 00m 51s)
  • 14:54 moritzm: installing openssh updates from stretch point release
  • 14:52 marostegui: Change expire_log_days on db1067 - https://phabricator.wikimedia.org/T197069
  • 14:49 marostegui: Drop unused ceilometer database from db1073
  • 14:47 marostegui: Remove unused grants from db1073
  • 14:39 marostegui: Remove unused grants from labsdb1004 and labsdb1005
  • 14:37 marostegui: Remove unused grants from es2019
  • 14:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2076 (duration: 00m 49s)
  • 14:24 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 49s)
  • 14:22 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Explicitly set wgMultiContentRevisionSchemaMigrationStage to current default (T174044) (duration: 00m 50s)
  • 13:59 marostegui: Stop replication on db2076 (db2095's master)
  • 13:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2076 for maintenance (duration: 00m 50s)
  • 12:31 moritzm: rebooting mw2136 for microcode tests
  • 12:14 moritzm: rebooting multatuli for microcode tests
  • 11:56 zeljkof: EU SWAT finished
  • 11:55 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Reconstruction NS at frwikt (T199631) (duration: 00m 49s)
  • 11:38 vgutierrez: power cycle cp3033 - T199677
  • 11:36 pmiazga@deploy1001: Synchronized php-1.32.0-wmf.12/includes/WebResponse.php: SWAT: WebReponse: Use values altered in WebResponseSetCookie hook (T198525) (duration: 00m 54s)
  • 11:20 zfilipin@deploy1001: Synchronized robots.txt: SWAT: Remove frwiki outdated entries in robots.txt (T199496) (duration: 00m 50s)
  • 11:18 zfilipin@deploy1001: Synchronized robots.txt: SWAT: Remove frwiki outdated entries in robots.txt (T199496) (duration: 00m 49s)
  • 11:14 mobrovac@deploy1001: Finished deploy [changeprop/deploy@ab8f7e9]: Bug fix: Remove anchors in blacklisting URIs and decode event URIs - T198386 (duration: 01m 33s)
  • 11:12 mobrovac@deploy1001: Started deploy [changeprop/deploy@ab8f7e9]: Bug fix: Remove anchors in blacklisting URIs and decode event URIs - T198386
  • 11:06 Amir1: start of ladsgroup@mwmaint1001:~$ mwscript populateChangeTagDef.php --wiki=frwiki
  • 11:05 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp5001.eqsin.wmnet
  • 11:04 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Write to the new change tag backend in frwiki (T194165) (duration: 00m 50s)
  • 11:02 vgutierrez: Power cycling cp5001 to attempt recovering it
  • 10:13 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 10:12 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 08:29 godog: put back ms-be1036 to full weight - T196873
  • 08:27 marostegui: Drop unused grants on db1108 - this might get dbproxy1009 to complain
  • 08:19 godog: run xfs_repair on filesystems reporting negative space available on ms-be1041 - T199198
  • 08:05 marostegui: Drop unused grants on eqiad hosts
  • 07:33 marostegui: Drop unused grants on codfw hosts
  • 07:19 marostegui: Deploy schema change on dbstore1002:s5 T144010 T51190 T199368
  • 07:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 50s)
  • 07:11 marostegui: Deploy schema change on db1110 T144010 T51190 T199368
  • 07:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 50s)
  • 05:24 marostegui: Deploy schema change on db2059 T144010 T51190 T199368
  • 05:10 marostegui: Deploy schema change on db2075 T144010 T51190 T199368
  • 02:43 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 16 02:43:43 UTC 2018 (duration 10m 19s)
  • 02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 13m 36s)

2018-07-14

  • 03:41 krinkle@deploy1001: Synchronized wmf-config/: I2bff4e - beta only (duration: 00m 51s)
  • 03:39 krinkle@deploy1001: Synchronized wmf-config/ProductionServices.php: Ib079ec90ae515 - clean up (duration: 00m 49s)

2018-07-13

  • 18:48 Trey314159: reindexing Mirandese wikis on elastic@eqiad (T197890)
  • 18:42 Trey314159: reindexing Mirandese wikis on elastic@codfw (T197890)
  • 18:39 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 fully (duration: 00m 50s)
  • 18:33 jynus: droping undocumented grants on m5 T199518
  • 14:34 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 with low load (duration: 00m 50s)
  • 14:03 jynus: stop and reimage db1079
  • 13:54 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 fully, depool db1079 (duration: 00m 50s)
  • 11:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 with low load (duration: 00m 50s)
  • 10:12 moritzm: installing cups/libcups security updates on stretch/trusty
  • 09:45 moritzm: installing libpng security updates on trusty (Debian already updated)
  • 09:36 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/includes/libs/objectcache: 5efa9f6 (duration: 00m 51s)
  • 09:35 aaron@deploy1001: sync-dir aborted: (no justification provided) (duration: 00m 01s)
  • 09:33 moritzm: installing ruby-sprockets security updates on jessie
  • 09:27 jynus: stop and reimage db1078
  • 09:05 jynus: apply updated grants to m5 hosts
  • 08:47 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 00m 50s)
  • 07:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 51s)
  • 06:50 elukey: powercycle ms-be1041 after diagnostic tests
  • 06:42 elukey: unblocked stuck dpkg processes on an107[2,5] that broke puppet
  • 05:59 jynus: stop and reimage db1087
  • 05:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2068 after maintenance (duration: 00m 49s)
  • 05:21 marostegui: Stop MySQL on db2068 for upgrade and check unused grants
  • 05:20 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2068 for grants testing (duration: 00m 50s)
  • 05:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 50s)
  • 05:06 marostegui: Rename wbc_entity_usage.eu_touched column on db1110 - T144010
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 52s)

2018-07-12

  • 23:32 jforrester@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/VisualEditor: SWAT VisualEditor: Add missing i18n keys to manifest T198064 (duration: 00m 52s)
  • 23:22 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Timeless config simplification, part II Ifc02818b6e (duration: 00m 49s)
  • 23:20 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Timeless config simplification, part I Ifc02818b6e (duration: 00m 50s)
  • 21:12 XioNoX: advertising 185.15.56.0/24 to the DFZ - T193496
  • 18:40 niharika29@deploy1001: Synchronized wmf-config/Wikibase.php: Revert - Wikidata dispatch, disable dispatching for testwikidatawiki https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/430925/ (duration: 00m 49s)
  • 18:32 niharika29@deploy1001: Synchronized wmf-config/: Wikidata dispatch, Use a LockManager with short TTL for testwikidata T178652 (duration: 00m 51s)
  • 18:18 niharika29@deploy1001: Synchronized wmf-config/Wikibase.php: Disable dispatching for testwikidatawiki https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/430924/ (duration: 00m 50s)
  • 16:56 chasemp: labcontrol1003:~# neutron net-update 7425e328-560c-4f00-8e99-706f3fb90bb4 --port_security_enabled=true
  • 16:45 moritzm: restarting smokeping on netmon1002 to pick up new config after wasat rename
  • 15:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 (duration: 00m 50s)
  • 15:49 akosiaris: rolling restart apertium-apy on scb nodes
  • 15:49 akosiaris: downgrade apertium-fra-cat to apertium-fra-cat_1.2.0~r78602-1+wmf2
  • 15:46 XioNoX: progressively pushing new (tighter) mgmt firewall policies
  • 15:07 chasemp: cloudvirt1021:~# /sbin/reboot
  • 14:37 addshore@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/Wikibase/repo: T199379 Fix missing label for unit items when displayed on entity page gerrit:445413 (duration: 01m 02s)
  • 14:07 marostegui: Drop unused grants from db2068
  • 13:15 akosiaris: upload apertium-fra-cat_1.3.0~r84327-1+wmf2 to apt.wikimedia.org/jessie-wikimedia/main
  • 13:14 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.12
  • 13:14 moritzm: installing openssh updates from stretch 9.4 point release
  • 13:05 godog: run xfs_repair /dev/sdd1 on ms-be1043 - T199198
  • 12:57 marostegui: Drop unused grants from db1073
  • 11:32 dereckson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix bewikibooks logo (T189218) and bnwikisource namespace (T199161) (duration: 00m 57s)
  • 11:14 dereckson@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/WikimediaEvents/extension.json: Add event logging for WMDE fundraising banners (duration: 00m 58s)
  • 11:08 godog: swift eqiad-prod: more weight to ms-be1036 after hw repair
  • 09:46 godog: shut down ms-be1041 for hardware diagnostics - T199198
  • 09:17 moritzm: ran puppet node clean/deactivate on db2064, hardware is broken for good and caused ongoing connection failures in cumin/debdeploy (T195228)
  • 08:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1083 (duration: 00m 56s)
  • 08:44 ppchelko@deploy1001: Started restart [cpjobqueue/deploy@ba672a3]: It logs disconnected consumers after many rebalances during the outage
  • 08:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1083 (duration: 00m 55s)
  • 08:30 oblivian@puppetmaster1001: conftool action : set/ttl=300; selector: dnsdisc=eventbus,name=.*
  • 08:28 _joe_: forcing puppet run on scb* to pick up the eventstreams change
  • 08:25 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=eventbus,name=.*
  • 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1083 (duration: 00m 56s)
  • 08:20 oblivian@puppetmaster1001: conftool action : set/ttl=10; selector: dnsdisc=eventbus,name=.*
  • 08:11 _joe_: reeabling and running puppet on scb1*
  • 08:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1083 (duration: 00m 56s)
  • 07:37 volans: reimaging spare system californium for testing reimage script changes
  • 07:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 with low weight (duration: 00m 57s)
  • 07:18 marostegui: Stop MySQL on db1083 to pick up new binlog format and MySQL upgrade
  • 07:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 for maintenance (duration: 00m 58s)
  • 07:01 marostegui: Drop unused grants from dbstore2002:3320 dbstore1001:3311 db2037 db2034
  • 06:38 marostegui: Drop unused grants from db1061 db1096:3316 db1113:3316
  • 06:26 elukey: restart rsyslog on wezen - T199406
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 after alter table (duration: 00m 57s)
  • 05:01 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db1075 (s3 primary master) - T187521
  • 05:00 marostegui: Deploy schema change on db1075 (s3 primary master) T146591 T197891 T196379
  • 04:48 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db1123 - T187521
  • 04:48 marostegui: Deploy schema change on db1123 T146591 T197891 T196379
  • 04:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 for alter table (duration: 00m 58s)
  • 03:18 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 12 03:18:58 UTC 2018 (duration 10m 25s)
  • 03:08 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 14m 31s)
  • 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 14m 44s)
  • 00:15 ejegg: updated CiviCRM from d24c2bc08b to b21d783e7f

2018-07-11

  • 23:32 thcipriani@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 39s)
  • 23:26 thcipriani@deploy1001: Synchronized wmf-config/throttle.php: SWAT: throttle: lift limits for Kaqchikel edit-a-thon; clear expired rules T199040 (duration: 00m 56s)
  • 23:17 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable anonymous cookie blocking T192017 (duration: 00m 58s)
  • 22:25 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d (T195325 T189830 T177619 T196523) (duration: 06m 44s)
  • 22:23 Reedy: created wikilove_log on ckbwiki T199336
  • 22:19 ejegg: activated SmashPig recurring donation job
  • 22:18 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d (T195325 T189830 T177619 T196523)
  • 21:53 elukey: restart rsyslog on lithium - in:imtcp stuck in EAGAIN (Resource temporarily unavailable) due to a old socket to tegmen.wikimedia.org
  • 21:47 twentyafterfour: disabled phabricator throttling
  • 21:47 elukey: re-enable kafka mirror maker on kafka100[1-3]
  • 21:41 ppchelko@deploy1001: Finished deploy [changeprop/deploy@45c3807]: Temporary decrease concurrency (duration: 01m 17s)
  • 21:40 ppchelko@deploy1001: Started deploy [changeprop/deploy@45c3807]: Temporary decrease concurrency
  • 21:16 elukey: starting kafka on kafka100[1-3] after zk cleanup
  • 21:14 _joe_: repooling mw1280
  • 20:57 krinkle@deploy1001: Synchronized wmf-config/mc.php: Ifa659de6453 - Revert multi-write mcrouter for most wikis - T198239 (duration: 00m 58s)
  • 20:49 _joe_: depooling mw1280 for debugging
  • 20:48 _joe_: repooled mw1223 after investigation
  • 20:39 _joe_: depooling mw1223 for debugging
  • 20:20 bearND: rolled back mobileapps deploy
  • 20:20 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d (T195325 T189830 T177619 T196523) (duration: 03m 30s)
  • 20:16 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d (T195325 T189830 T177619 T196523)
  • 20:03 volans: rolling restarting mediawiki API in eqiad with the highest load
  • 19:45 gehel: note: updater was not and will not be using kafka
  • 19:45 gehel: restarting wdqs-updater on wdqs1010 (still not using Kafka)
  • 19:31 elukey: cleaned up *change-prop.retry.change-prop.retry* in /srv/kafka/data on kafka100[1-3]
  • 19:22 akosiaris: stop changeprop on all scb hosts
  • 19:16 _joe_: restarting a few hhvm appservers with high load
  • 18:44 akosiaris: ok, change merged, running puppet on scb hosts
  • 18:34 elukey: restarted topic nuke script for kafka main
  • 18:14 elukey: start kafka on kafka1002
  • 18:14 elukey: stop mirror makers on kafka100[1-3]
  • 18:10 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 fully (duration: 00m 57s)
  • 17:36 _joe_: restarted cpjobqueue in codfw
  • 17:31 elukey: restart kafka on kafka1001 (oom registered)
  • 17:12 elukey: restart kafka on kafka1003 with 2G heap settings
  • 17:10 elukey: restart kafka on kafka1002 with 2G heap settings
  • 17:06 elukey: restarted kafka on kafka1001 with Xmx 2G and Xms 2F
  • 17:05 _joe_: restarting cpjobqueue on scb2001
  • 17:00 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=eventbus,name=eqiad
  • 16:50 elukey: stop topics cleaner script
  • 16:40 _joe_: masking and stopping cpjobqueue, changeprop everywhere
  • 16:36 elukey: start topic clean procedure on kafka1001 (tmux root session)
  • 16:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 with low load (duration: 00m 58s)
  • 16:19 elukey: restart kafka on kafka1003
  • 16:19 Pchelolo: restart cpjobqueue
  • 16:02 ejegg: disabled failmail logstream for SmashPig
  • 15:40 _joe_: restarting eventbus on kafka-main in eqiad
  • 15:26 chasemp: install dtach on labnet1003
  • 15:18 _joe_: restarting kafka on kafka1002
  • 15:16 gehel: killing wdqs-updater on wdqs10(09|10) to diminish load on kafka
  • 15:11 elukey: restart again kafka on kafka100[1,2] - failed for OOM
  • 15:08 Pchelolo: stop cpjobqueue in eqiad
  • 15:03 elukey: restart kafka on kafka1003
  • 14:58 papaul: shutting down furud to disconnect disk array shelves
  • 14:57 elukey: rolling restart of eventbus on kafka100[1-3]
  • 14:53 elukey: restart kafka on kafka1002
  • 14:52 elukey: restart kafka on kafka1001
  • 14:18 ppchelko@deploy1001: Finished deploy [changeprop/deploy@2e56855]: Move static blacklisting to change-prop T198386 (duration: 01m 22s)
  • 14:16 ppchelko@deploy1001: Started deploy [changeprop/deploy@2e56855]: Move static blacklisting to change-prop T198386
  • 14:15 volans: reset iLO on hpiLO on conf1005 to see if it fixes the issues with the mgmt interface
  • 13:41 thcipriani@deploy1001: Synchronized README: noop: test scap 3.8.4-1 (duration: 00m 56s)
  • 13:25 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.12 (duration: 00m 57s)
  • 13:24 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.12
  • 13:16 akosiaris: restart apertium-apy for apertium-fra-cat upgrade on scb boxes to 1.3.0~r84327-1+wmf1, T189076
  • 13:14 elukey: roll restart of aqs on aqs* to pick up the new Druid config
  • 13:13 akosiaris: upgrade apertium-fra-cat on scb boxes to 1.3.0~r84327-1+wmf1, T189076
  • 12:59 marostegui: Remove unused grants from db1098:3316
  • 12:43 reedy@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/ContactPage: New config! (duration: 00m 57s)
  • 12:41 reedy@deploy1001: Synchronized wmf-config/MetaContactPages.php: ContactPage new config (duration: 00m 56s)
  • 12:40 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: ContactPage new config (duration: 00m 57s)
  • 12:24 raynor: EU SWAT finished
  • 12:23 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleanup: Remove unsued MFForceSecureLogin (duration: 00m 56s)
  • 12:19 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleanup: Remove unused MinervaDownloadIcon (duration: 00m 57s)
  • 12:11 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleanup: Remove unsued VectorExperimentalPrintStyles (duration: 00m 56s)
  • 12:05 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove ambox images on mobile wikis (T191303) (duration: 00m 57s)
  • 11:53 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable page previews for all new editors (T197719) (duration: 00m 56s)
  • 11:38 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/EventBus/: SWAT: Properly handle null content format (duration: 00m 59s)
  • 11:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable FileExporter for sourceswiki (T198594) (duration: 00m 56s)
  • 11:12 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist wikisource populateChangeTagDef.php
  • 11:10 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgChangeTagsSchemaMigrationStage to write both for Wikisource (T194165) (duration: 00m 58s)
  • 11:09 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere.Take 2, feeds timed out. T169009 (duration: 02m 56s)
  • 11:08 arturo: re-enable puppet in install1002
  • 11:06 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere.Take 2, feeds timed out. T169009
  • 11:06 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere. T169009 (duration: 08m 07s)
  • 11:05 jynus: stop db1086 for reimage
  • 11:03 akosiaris: restart HHVM on mw1282
  • 10:58 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere. T169009
  • 10:45 arturo: disabled puppet in install1002 for a debian installer live hack (T199202)
  • 10:35 moritzm: installing tiff security updates on jessie hosts
  • 10:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully depool db1086 (duration: 00m 56s)
  • 10:11 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/includes/libs/rdbms/ChronologyProtector.php: ee89660 (duration: 00m 57s)
  • 10:09 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/includes/libs/rdbms/ChronologyProtector.php: 552dbdb (duration: 00m 59s)
  • 10:08 vgutierrez: reimage baham as spare system - T199247
  • 09:55 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. Take 2, check timed out. T169009 (duration: 11m 14s)
  • 09:44 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. Take 2, check timed out. T169009
  • 09:41 ppchelko@deploy1001: deploy aborted: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. T169009 (duration: 03m 13s)
  • 09:38 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. T169009
  • 09:34 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3] (dev-cluster): Upgrade cassandra driver to 3.5.0 T169009 (duration: 04m 20s)
  • 09:33 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 57s)
  • 09:30 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3] (dev-cluster): Upgrade cassandra driver to 3.5.0 T169009
  • 09:07 moritzm: installing remaining php7 security updates for stretch
  • 08:37 godog: reboot ms-be1040 to run hardware diagnostics - T199198
  • 08:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 after alter table (duration: 00m 56s)
  • 08:14 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db1077 with replication, this will generate lag on s3 labs hosts - T187521
  • 08:13 marostegui: Deploy schema change on db1077 with replication, this will generate lag on s3 labs hosts T146591 T197891 T196379
  • 08:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 for alter table (duration: 00m 56s)
  • 08:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 after alter table (duration: 00m 57s)
  • 07:57 elukey: roll restart of aqs on aqs* to rollback the druid config
  • 07:51 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on dbstore1002:s3, db1078 - T187521
  • 07:50 marostegui: Deploy schema change on db1078 T146591 T197891 T196379
  • 07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 for alter table (duration: 00m 56s)
  • 07:15 marostegui: Deploy schema change on dbstore1002:s3 T146591 T197891 T196379
  • 06:59 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db2043 (s3 codfw master), this will generate lag on s3 codfw - T187521
  • 06:40 marostegui: Deploy schema change on s3 codfw master (db2043) with replication, this will generate lag on s3 codfw T146591 T197891 T196379
  • 06:30 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1062 (s7 primary master) - T187521
  • 06:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 after alter table (duration: 00m 55s)
  • 06:26 marostegui: Deploy schema change on db1062 (s7 primary master) T146591 T197891 T196379
  • 06:12 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1086 - T187521
  • 06:12 marostegui: Deploy schema change on db1086 T146591 T197891 T196379
  • 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 for alter table (duration: 00m 57s)
  • 06:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 after alter table (duration: 00m 57s)
  • 05:55 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: fix logspam, ParserMigration breakage (duration: 00m 57s)
  • 05:49 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1079 with replication, this will generate lag on s7 labs hosts - T187521
  • 05:31 marostegui: Deploy schema change on db1079 with replication, this will generate lag on s7 labs hosts T146591 T197891 T196379
  • 05:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 for alter table (duration: 00m 56s)
  • 05:23 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1090 - T187521
  • 05:20 marostegui: Deploy schema change on db1090 T146591 T197891 T196379
  • 05:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 after alter table (duration: 00m 56s)
  • 05:06 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1094 - T187521
  • 05:05 marostegui: Deploy schema change on db1094 T146591 T197891 T196379
  • 05:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 for alter table (duration: 01m 07s)
  • 03:15 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 11 03:15:37 UTC 2018 (duration 10m 23s)
  • 03:05 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 15m 35s)
  • 02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 12m 37s)
  • 01:27 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I423f8fac62 - Remove wgGadgetsCacheType setting (duration: 00m 54s)
  • 01:01 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I0c621368d6 - Remove unused tmarray variables (duration: 00m 56s)
  • 00:44 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I3d02da810 - Clean up wgLocaltimezone (duration: 00m 56s)
  • 00:40 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ia017dea257d - Clean up wgLocaltimezone (duration: 00m 56s)
  • 00:35 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I1aad80c3bcb - Remove setting of wgLocalTZOffset (duration: 00m 57s)

2018-07-10

  • 23:50 ejegg: disabled ProcessSmashPigRecurring in Civi scheduled jobs, to be replaced with process-control job
  • 22:27 ejegg: updated payments-wiki from 383f667171 to 63372154d0
  • 21:40 chasemp: reboot labnet100[34]
  • 21:04 XioNoX: re-configure GTT circuit in eqiad/knams
  • 20:50 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/tests/phpunit/includes/libs/objectcache/MultiWriteBagOStuffTest.php: 4fba9f6a032 (duration: 00m 56s)
  • 20:48 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/includes/libs/objectcache/MultiWriteBagOStuff.php: 4fba9f6a032 (duration: 00m 57s)
  • 18:33 aaron@deploy1001: Synchronized wmf-config/mc.php: Make all non-test wikis write to both nutcracker and mcrouter again (duration: 00m 57s)
  • 16:51 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.12
  • 16:03 vgutierrez: replacing baham with authdns2001 - T196664
  • 15:27 ema: merge alternate_domains vcl patch T164609
  • 15:19 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.12 and rebuild l10n cache (duration: 61m 27s)
  • 14:31 marostegui: Set disk #0 offline for replacement - T199056
  • 14:17 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.12 and rebuild l10n cache
  • 14:15 moritzm: installing tiff security updates on jessie
  • 13:53 moritzm: installing ruby-sprockets security updates
  • 13:48 moritzm: installing subversion updates from jessie 9.11 point release
  • 13:40 moritzm: rolling restart of thumbor to pick up new exiv2/openssl
  • 13:29 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.5 (duration: 03m 13s)
  • 13:26 moritzm: installing cups security updates on jessie
  • 13:25 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.4 (duration: 06m 18s)
  • 13:14 thcipriani@deploy1001: Synchronized scap/plugins/clean.py: no op sync for consistancy Scap clean: remove remote cache directory (duration: 00m 51s)
  • 12:53 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/includes/libs/objectcache/ReplicatedBagOStuff.php: 4ad6b70 (duration: 00m 51s)
  • 12:24 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/SpamBlacklist/includes/SpamBlacklist.php: 08a2153f7aa (duration: 00m 51s)
  • 12:20 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/SpamBlacklist/includes/SpamBlacklist.php: 583dc7a92f9b (duration: 00m 51s)
  • 12:06 aaron@deploy1001: Synchronized wmf-config/mc.php: Revert "Make all non-test wikis write to both nutcracker and mcrouter" (duration: 00m 56s)
  • 11:56 ppchelko@deploy1001: Finished deploy [restbase/deploy@2674971]: Roll out cassandra-driver@3.5.0 to restbase2001 (duration: 02m 52s)
  • 11:53 ppchelko@deploy1001: Started deploy [restbase/deploy@2674971]: Roll out cassandra-driver@3.5.0 to restbase2001
  • 11:43 arturo: updated compiler facts `PUPPET_MASTERS=puppetmaster1001.eqiad.wmnet PUPPET_COMPILER=compiler02.puppet3-diffs.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts`
  • 11:32 aaron@deploy1001: Synchronized wmf-config/mc.php: Make all non-test wikis write to both nutcracker and mcrouter (duration: 00m 51s)
  • 11:25 dcausse@deploy1001: Synchronized ./wmf-config/InitialiseSettings.php: [cirrus] cleanup unused config vars 2/2 (duration: 01m 40s)
  • 11:24 moritzm: installing PHP 7 security updates on stretch
  • 11:09 dcausse@deploy1001: Synchronized ./wmf-config/: [cirrus] cleanup unused config vars 1/2 (duration: 00m 53s)
  • 10:27 ppchelko@deploy1001: Finished deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header, take 2, checer timed out (duration: 06m 39s)
  • 10:20 ppchelko@deploy1001: Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header, take 2, checer timed out
  • 10:20 ppchelko@deploy1001: deploy aborted: Fix up the invalid Vary header (duration: 12m 50s)
  • 10:07 ppchelko@deploy1001: Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header
  • 10:07 moritzm: restarting thumbor on thumbor1001 to pick up exiv2 security updates
  • 10:03 elukey: restart analytics100[1,2]'s hadoop resource managers, some I/O socket errors after the ip6 interface change
  • 09:34 elukey: forced umount of /mnt/hdfs on stat1004, several processes hang for it (causing load) and transport not connected
  • 09:25 godog: swift eqiad-prod add ms-be1036 back gradually - T196873
  • 09:05 moritzm: installing libsoup security updates on jessie/stretch
  • 08:38 moritzm: installing libjpeg-turbo security updates on trusty
  • 08:37 godog: drain and restart cassandra-a on restbase2001 to test a restart
  • 08:34 elukey: rolling restart of AQS to apply the new config
  • 08:09 godog: disable puppet on hosts running cassandra before merging 444247 and 443114
  • 08:08 moritzm: installing openslp security updates on trusty
  • 07:58 moritzm: installing ntp security updates on trusty (may trigger some Icinga warnings about clocks, these recover after a while)
  • 07:33 twentyafterfour: deploying fix for phabricator userpage having hard-coded my username
  • 06:53 twentyafterfour: deployed rPHEX03173dd0097451f60faa3a1705abee58a9fe4c5f
  • 06:32 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on s7 codfw master (db2040) with replication, this will generate lag on s7 codfw - T187521
  • 06:23 marostegui: Deploy schema change on codfw s7 master (db2040) with replication, this will generate lag on s7 codfw T146591 T197891 T196379
  • 06:17 marostegui: Deploy schema change on s8 primary master (db1071) T146591 T197891 T196379
  • 06:06 marostegui: Deploy schema change on dbstore1002:s8 T146591 T197891 T196379
  • 06:02 marostegui: Deploy schema change on codfw s8 master (db2045) with replication, this will generate lag on s8 codfw T146591 T197891 T196379
  • 05:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 after alter table (duration: 00m 50s)
  • 05:40 marostegui: Deploy schema change on s4 primary master (db1068) T146591 T197891 T196379
  • 05:36 marostegui: Deploy schema change on db1091 T146591 T197891 T196379
  • 05:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 for alter table (duration: 00m 50s)
  • 05:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 after alter table (duration: 00m 49s)
  • 05:27 marostegui: Deploy schema change on db1081 T146591 T197891 T196379
  • 05:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 for alter table (duration: 00m 50s)
  • 05:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 00m 50s)
  • 05:21 marostegui: Deploy schema change on db1121 with replication, this will generate lag on s4 labs hosts T146591 T197891 T196379
  • 05:17 marostegui: Optimize frwiki.wbc_entity_usage on s6 eqiad hosts T187521
  • 05:16 marostegui: Deploy schema change on db1103:3314 T146591 T197891 T196379
  • 05:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 50s)
  • 05:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 00m 50s)
  • 05:04 marostegui: Deploy schema change on db1097:3314 T146591 T197891 T196379
  • 05:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 for alter table (duration: 00m 51s)
  • 04:59 marostegui: Optimize frwiki.wbc_entity_usage on s6 codfw, this will generate lag on s6 codfw - T187521
  • 04:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 after alter table (duration: 00m 50s)
  • 04:48 marostegui: Deploy schema change on db1084 T146591 T197891 T196379
  • 04:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 for alter table (duration: 00m 52s)
  • 04:36 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1066 (s2 primary master) - T187521
  • 04:28 marostegui: Deploy schema change on s1 primary master (db1052) T146591 T197891 T196379
  • 02:44 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 10 02:44:29 UTC 2018 (duration 10m 18s)
  • 00:21 twentyafterfour: deploying https://phabricator.wikimedia.org/rPHABc6f75c918afa1cc59472c5fe226539e093f6c3ef

2018-07-09

  • 23:58 ejegg: updated payments-wiki from ed40f33c44 to 383f667171
  • 23:58 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop trying to set wgLicenseURL T154069 (duration: 00m 50s)
  • 23:55 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: No need for officewiki-specific upload for MP3s any more I42ffbcd76f6 (duration: 00m 50s)
  • 23:53 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgTmhEnableMp3Uploads Idbf1201813 (duration: 00m 50s)
  • 23:52 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgTmhEnableMp3Uploads I6479dbacd4 (duration: 00m 50s)
  • 23:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgVisualEditorNonAccountEnableProportion I3b8cccc00 (duration: 00m 50s)
  • 23:47 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgVisualEditorNonAccountEnableProportion If2ee24ae (duration: 00m 50s)
  • 23:43 jforrester@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: SWAT Cleanup: Remove Beta Cluster use of wikieditor-preview preference Ib16e6e19b1 (duration: 00m 50s)
  • 23:42 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Remove wgWikiEditorFeatures I7580e94ecf7 (duration: 00m 50s)
  • 23:36 jforrester@deploy1001: Synchronized wmf-config/extension-list: Stop loading the MwEmbedSupport extension, part IV (duration: 00m 50s)
  • 23:34 jforrester@deploy1001: Synchronized multiversion/submodules.json: SWAT MwEmbedSupport extension, part III (duration: 00m 50s)
  • 23:33 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT MwEmbedSupport extension, part II (duration: 00m 51s)
  • 23:26 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT 444650 (duration: 00m 49s)
  • 23:19 jforrester@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/ORES: ORES fix Ia1b09d7711 for RoanKattouw (duration: 00m 50s)
  • 23:18 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Stop loading the MwEmbedSupport extension, part I (duration: 00m 50s)
  • 23:16 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT for bd808 (duration: 00m 51s)
  • 22:54 bstorm_: added disk from new shelf to /srv/dumps on labstore1006
  • 21:48 bawolff: createAndPromote.php --wiki=officewiki BWolff_\(WMF\) --sysop --force
  • 21:45 ejegg: updated payments-wiki from f302c5ffa3 to ed40f33c44 (Outbound cURL logging in SmashPig)
  • 19:27 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: GlobalPrefs everywhere https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/444652/ (duration: 00m 51s)
  • 19:01 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES damaging filter on srwiki (T197012) (duration: 00m 50s)
  • 18:59 ejegg: updated payments-wiki from 198c31a657 to f302c5ffa3 (DI update)
  • 18:52 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES edit quality filters on bswiki (T197010) (duration: 00m 50s)
  • 18:48 XioNoX: apply analytics-in6 firewall filter to cr1/2-eqiad with a default permit+log
  • 18:41 ejegg: updated payments-wiki from 43989ebc96 to 198c31a657 (patches from REL1_27)
  • 18:37 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Avoid losing cached ParserOutput in doEditContent (T198483) (duration: 00m 51s)
  • 18:26 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Resyncing to shut up warnings (duration: 00m 48s)
  • 18:23 catrope@deploy1001: Synchronized wmf-config/: Rollout Watchlist Structured Filters to most wikis (T181193) (duration: 00m 53s)
  • 18:14 XioNoX: updating mr1-eqsin security policies - T199074
  • 18:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable TemplateStyles on huwiktionary (T198725) (duration: 00m 52s)
  • 17:32 XioNoX: refactoring analytics firewall policies
  • 17:24 gehel@deploy1001: Finished deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (duration: 09m 18s)
  • 17:14 gehel@deploy1001: Started deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater
  • 17:12 gehel@deploy1001: Finished deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 32s)
  • 17:12 gehel@deploy1001: Started deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (wdqs1009 only)
  • 16:09 marostegui: Set disk 32:0 offline on db1069 for a replacement - T199056
  • 15:32 elukey: enabled snappy compression for varnishkafka eventlogging
  • 15:09 akosiaris: repool eqiad mathoid discovery RR
  • 14:52 akosiaris: upgrade eqiad kubernetes cluster to 1.8.14
  • 14:25 akosiaris: depool eqiad mathoid discovery RR for kubernetes upgrade
  • 14:16 moritzm: rmmoding floppy kernel module from hosts which had it loaded prior to blacklisting
  • 13:57 akosiaris: repool codfw mathoid discovery
  • 13:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 50s)
  • 13:32 marostegui: Deploy schema change on s4 codfw master (db2051) this will generate lag on s4 codfw T146591 T197891 T196379
  • 13:29 akosiaris: upgrade kubernetes200{1,2,3,4} to kubernetes 1.8.14
  • 13:29 akosiaris: upgrade acrab to kubernetes 1.8.14
  • 13:26 marostegui: Deploy schema change on db1080 T146591 T197891 T196379
  • 13:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 00m 50s)
  • 13:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 00m 50s)
  • 13:11 marostegui: Deploy schema change on db1114 T146591 T197891 T196379
  • 13:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 00m 50s)
  • 13:10 akosiaris: upgrade acrux to kubernetes 1.8.14
  • 13:00 akosiaris: depool codfw for mathoid in preparation for kubernetes cluster update
  • 11:56 zeljkof: EU SWAT finished
  • 11:56 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add importsources to ru.wikinews (T199045) (duration: 00m 50s)
  • 11:51 moritzm: installing chromium 67.0.3396.87-1~deb9u1 security updates on proton* (tested compatability of new release in deployment-prep)
  • 11:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use new wordmark for bnwikivoyage (T196680) (duration: 00m 51s)
  • 11:35 zfilipin@deploy1001: Synchronized static/images/mobile/copyright/wikivoyage-wordmark-bn.svg: SWAT: Upload wordmark for bnwikivoyage (T196680) (duration: 00m 50s)
  • 11:33 moritzm: installing glibc updates from stretch 9.4 point release
  • 11:31 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create group eventparticipant on zhwiki (T198167) (duration: 00m 51s)
  • 10:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 after alter table (duration: 00m 50s)
  • 10:35 marostegui: Deploy schema change on db1083 T146591 T197891 T196379
  • 10:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 for alter table (duration: 00m 50s)
  • 10:31 godog: upgrade hp raid firmware on ms-be1017 - T141756
  • 10:12 godog: powercycle ms-be1017 - can't login on ssh/console
  • 10:07 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 10:06 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
  • 09:22 ppchelko@deploy1001: Finished deploy [restbase/deploy@794d6ee]: Enable language variants for HTML T190689 (duration: 15m 32s)
  • 09:07 ppchelko@deploy1001: Started deploy [restbase/deploy@794d6ee]: Enable language variants for HTML T190689
  • 09:02 vgutierrez: Bump AES128-SHA traffic redirection to 100% - T192555
  • 08:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 00m 49s)
  • 08:35 jynus: updating password for labdb1004/5 for admin users
  • 08:31 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1074 with replication, this will generate lag on s2 on labshosts - T187521
  • 08:26 marostegui: Deploy schema change on db1106 with replication, this will generate lag on labs hosts for s1 T146591 T197891 T196379
  • 08:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 00m 50s)
  • 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 00m 51s)
  • 08:08 marostegui: Deploy schema change on db1105:3311 T146591 T197891 T196379
  • 08:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 00m 49s)
  • 08:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 00m 50s)
  • 07:54 marostegui: Deploy schema change on db1099:3311 T146591 T197891 T196379
  • 07:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 00m 50s)
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 00m 50s)
  • 07:40 marostegui: Deploy schema change on db1067 T146591 T197891 T196379
  • 07:40 ema: reboot lvs canaries for microcode updates: lvs4007 lvs3004 lvs2006 lvs2010 lvs1006 T127825
  • 07:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 00m 50s)
  • 07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 00m 49s)
  • 07:18 marostegui: Deploy schema change on db1119 - T146591 T197891 T196379
  • 07:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 00m 50s)
  • 07:18 elukey: update filter analytics-in4 on cr1/cr2 eqiad
  • 07:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 00m 50s)
  • 06:54 marostegui: Deploy schema change on db1089 - T146591 T197891 T196379
  • 06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 00m 50s)
  • 06:41 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1066 (s2 primary master) T197459
  • 06:30 marostegui: Deploy schema change on dbstore1002:s1 - T146591 T197891 T196379
  • 06:25 elukey: restart hue on thorium to pick up new smtp changes - T196920
  • 06:13 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1076 T197459
  • 06:12 marostegui: Deploy schema change on dbstore1001:s1 - T146591 T197891 T196379
  • 06:03 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on dbstore1002:s2, db1122 and db1105 - T187521
  • 06:01 marostegui: Deploy schema change on s1 codfw master (db2048) with replication, this will generate lag on s1 codfw - T146591 T197891 T196379
  • 05:55 marostegui: Deploy schema change on s2 primary master (db1066) - T146591 T197891 T196379
  • 05:53 marostegui: Deploy schema change on db1076 - T146591 T197891 T196379
  • 05:41 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on s2 codfw master with replication - lag will happen on s2 codfw - T187521
  • 05:31 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1090 T197459
  • 05:31 marostegui: Deploy schema change on db1090 - T146591 T197891 T196379
  • 05:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 51s)
  • 05:12 marostegui: Deploy schema change on db1074 with replication, this will generate lag on s2 on labs hosts - T146591 T197891 T196379
  • 05:04 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1074 with replication, this will generate lag on labs hosts T197459
  • 05:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 52s)
  • 02:44 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 9 02:44:19 UTC 2018 (duration 10m 23s)
  • 02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 33s)

2018-07-08

  • 20:37 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I81d566ba860 (duration: 00m 51s)
  • 20:30 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Id94ca5f55c2c6 (duration: 00m 53s)
  • 19:10 chasemp: disable puppet on labstore1007 to turn down ratelimits for nginx (load is rising and rising)
  • 11:27 elukey: restart rsyslog on lithium - in:imtcp thread stuck at 99% cpu usage

2018-07-07

  • 23:25 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Icbbccf495899 (duration: 00m 50s)
  • 23:23 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I001f68a1b39 (duration: 00m 53s)
  • 23:18 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I1b42c260b5ee3 (duration: 01m 00s)
  • 13:02 _joe_: rebooting lvs2001, panic logs from the ethernet card module

2018-07-06

  • 18:08 foks: reset 2FA for User:Howcheng
  • 16:06 chasemp: labcontrol1003:~# /sbin/reboot T198950
  • 13:36 moritzm: installing openldap security updates on trusty
  • 13:22 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1103:3312 - T197459
  • 13:22 marostegui: Deploy schema change on db1103:3312 - T146591 T197891 T196379
  • 13:05 moritzm: installing bouncycastle security updates
  • 13:00 moritzm: installing mercurial security updates
  • 12:57 hashar: Restarting Zuul
  • 12:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@c42c048]: Update restbase-mod-table-cassandra to v1.1.0 (duration: 15m 32s)
  • 12:38 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1105 and db1122 - T197459
  • 12:33 marostegui: Deploy schema change on db1105:3312 - T146591 T197891 T196379
  • 12:30 marostegui: Deploy schema change on dbstore1002:s2 and db1122 T146591 T197891 T196379
  • 12:29 mobrovac@deploy1001: Started deploy [restbase/deploy@c42c048]: Update restbase-mod-table-cassandra to v1.1.0
  • 12:18 marostegui: Deploy schema change on s2 codfw master (db2035) with replication, this will generate lag on s2 codfw T146591 T197891 T196379
  • 11:51 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on dbstore1002:s2 - T197459
  • 11:15 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db2035 (codfw s2 master), this will generate lag on s2 codfw - T197459
  • 11:09 marostegui: Optimize shwiki.logging on s3 eqiad hosts (one by one) T197459
  • 11:08 marostegui: Optimize shwiki.logging on db2043 (codfw s3 master), this will generate lag on s3 codfw - T197459
  • 11:04 marostegui: Optimize frwiki.logging on db1061 (s6 primary master) - T197459
  • 11:02 marostegui: Deploy schema change on db1061 (s6 primary master) T146591 T197891 T196379
  • 11:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 after alter table (duration: 00m 50s)
  • 10:49 marostegui: Optimize frwiki.logging on db1088 - T197459
  • 10:48 marostegui: Deploy schema change on db1088 T146591 T197891 T196379
  • 10:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 50s)
  • 10:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 after alter table (duration: 00m 51s)
  • 10:02 marostegui: Optimize frwiki.logging on db1093 - T197459
  • 10:01 mobrovac: restbase depool restbase2001 to test the cassandra node driver v3.5.0 - T169009
  • 10:00 marostegui: Deploy schema change on db1093 T146591 T197891 T196379
  • 10:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 for alter table (duration: 00m 51s)
  • 09:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 51s)
  • 09:46 marostegui: Deploy schema change on db1085 with replication, this will generate lag on s6 labsdb T146591 T197891 T196379
  • 09:08 jynus: restart db1115 mariadb instance (this will cause temporary downtime if tendril and dbtree)
  • 08:53 marostegui: Optimize frwiki.logging on db11085 with replication (this will generate lag on s6 on labsdb hosts) - T197459
  • 08:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 00m 51s)
  • 08:18 marostegui: Optimize frwiki.logging on db1113:3316 - T197459
  • 08:15 marostegui: Deploy schema change on db1113:3316 T146591 T197891 T196379
  • 08:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3318 after alter table (duration: 00m 50s)
  • 07:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 for alter table (duration: 00m 50s)
  • 07:47 marostegui: Optimize frwiki.logging on db1098:3316 - T197459
  • 07:46 marostegui@deploy1001: sync-file aborted: Depool db1098:3315 for alter table (duration: 00m 01s)
  • 07:45 marostegui: Deploy schema change on db1098:3316 T146591 T197891 T196379
  • 07:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 50s)
  • 07:22 moritzm: re-enabled puppet on mw2246
  • 07:12 moritzm: upgrading remaining video scalers to vp9-row-mt enabled ffmpeg build (T190333)
  • 07:11 marostegui: Optimize frwiki.logging on db1096:3316 - T197459
  • 07:07 marostegui: Deploy schema change on db1096:3316 T146591 T197891 T196379
  • 07:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 51s)
  • 06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 52s)
  • 06:49 marostegui: Optimize frwiki.logging on dbstore1002 - T197459
  • 06:34 marostegui: Deploy schema change on dbstore1002:s6 T146591 T197891 T196379
  • 06:31 marostegui: Optimize frwiki.logging on db2039 (s6 codfw master), with replication, this will generate lag on s6 codfw - T197459
  • 06:25 marostegui: Deploy schema change on s6 codfw master (db2039) with replication, this will generate lag on s6 codfw T146591 T197891 T196379
  • 06:22 marostegui: Optimize dewiki.logging on db1070 (s5 primary master) - T197459
  • 05:03 kartik@deploy1001: Finished deploy [cxserver/deploy@6f9fcce]: Update cxserver to bfc9c84 (T198830) (duration: 03m 26s)
  • 05:01 marostegui: Optimize dewiki.logging on db1110 - T197459
  • 05:00 kartik@deploy1001: Started deploy [cxserver/deploy@6f9fcce]: Update cxserver to bfc9c84 (T198830)
  • 05:00 marostegui: Deploy schema change on s5 primary master (db1070) T146591 T197891 T196379
  • 04:59 marostegui: Deploy schema change on db1110 T146591 T197891 T196379
  • 04:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 50s)
  • 04:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 after alter table (duration: 00m 51s)
  • 04:42 marostegui: Deploy schema change on db1082 with replication T146591 T197891 T196379
  • 00:08 ejegg: updated CiviCRM from e2c8ddd70e to d24c2bc08b

2018-07-05

  • 23:47 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/Echo/: Handle missing presentation model (T195253) (duration: 00m 52s)
  • 23:39 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Ic87af972e1 - Remove wgRecentEchoInstall (duration: 00m 51s)
  • 23:32 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I8165473c49 - Remove wgVaryOnXFPForAPI (duration: 00m 51s)
  • 23:19 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable static maps on bgwiki (duration: 00m 51s)
  • 23:13 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Use require rather than include_once for $wgInterwikiCache (duration: 00m 52s)
  • 21:55 twentyafterfour: whitelist office IPs in phabricator throttle
  • 21:30 bstorm_: disabled unused P440ar RAID controller on labvirt1020
  • 21:15 bstorm_: disabled unused RAID controller on labvirt1019
  • 19:30 gehel: repool wdqs1003, lag almost completely recovered
  • 19:07 ebernhardson: T194678 un-pause cirrussearch writes to codfw
  • 18:58 gehel: depooling wdqs1003 to help it recover update lag
  • 18:55 ebernhardson: T194678 pause cirrussearch writes to codfw to check how kafka+mirrormaker responds
  • 18:48 gehel: restarting blazegraph on wdqs1003 (updater lag)
  • 18:21 thcipriani@deploy1001: Synchronized wmf-config/Wikibase-production.php: SWAT: Set dispatchLagToMaxLagFactor to 60 for wikidata T194950 (duration: 00m 51s)
  • 18:14 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add "L" as an alias of Lexeme namespace in wikidata T195493 (duration: 00m 59s)
  • 18:10 gehel@deploy1001: Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1010 only) (duration: 00m 28s)
  • 18:09 gehel@deploy1001: Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1010 only)
  • 16:13 jynus: stop and reimage db2045
  • 15:34 mobrovac@deploy1001: Finished scap: Set the Accept-Language header explicitly when making requests to the REST API - T198186 (duration: 30m 50s)
  • 15:26 elukey: upgrade (without jvm restart) prometheus-jmx-exporter on the analytics node listed in debmonitor still not running the last version
  • 15:03 mobrovac@deploy1001: Started scap: Set the Accept-Language header explicitly when making requests to the REST API - T198186
  • 15:02 jynus: stop and reimage db2040
  • 15:00 marostegui: Optimize dewiki.logging on db1082 with replication, this will generate lag on s5 on labsdb hosts/script load irssinotifier - T197459
  • 14:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 00m 50s)
  • 14:50 moritzm: installing php security updates on netmon1002
  • 14:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 00m 50s)
  • 14:15 moritzm: installing dbus updates from stretch point release
  • 14:09 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/EventBus/: SWAT: Dont specify the comment if it is an empty string (duration: 00m 51s)
  • 14:03 jynus: stop and reimage db2052
  • 13:55 moritzm: installing glibc updates from stretch point release
  • 13:55 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Revert "Change logo for eswiki temporarily" (T198846) (duration: 00m 50s)
  • 13:50 moritzm: installing postgresql security updates on maps cluster
  • 13:41 aaron@deploy1001: Synchronized wmf-config/mc.php: Make test wikis just write to both nutcracker and mcrouter (duration: 00m 50s)
  • 13:32 zfilipin@deploy1001: Synchronized wmf-config: SWAT: Enable ORES wp10, draftquality on draft ns (118) for enwiki (T198768) (duration: 00m 51s)
  • 13:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Replace Tidy with RemexHtml everywhere (T175706) (duration: 00m 50s)
  • 13:18 marostegui: Optimize dewiki.logging on db1113:3315 - T197459
  • 13:17 marostegui: Deploy schema change on db1113:3315 T146591 T197891 T196379
  • 13:12 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgChangeTagsSchemaMigrationStage to write both for Wikivoyage, Wikibooks (T194165) (duration: 00m 52s)
  • 11:30 moritzm: installing python-mimeparse updates from jessie point release
  • 11:06 moritzm: rebooting contint1001 for kernel update/enabling microcode
  • 11:05 hashar: Stopping Jenkins CI for kernel upgrade on contint1001
  • 10:31 moritzm: installing ncurses security updates on jessie
  • 09:50 marostegui: Optimize dewiki.logging on db1097:3315 - T197459
  • 09:50 marostegui: Deploy schema change on db1097:3315 T146591 T197891 T196379
  • 09:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 00m 50s)
  • 09:42 moritzm: rebooting ms-be1013/1016/1028/1040 as microcode update canaries
  • 09:40 mobrovac@deploy1001: Finished deploy [zotero/translators@d6702bc]: String.includes() polyfill for Sveriges radio translator - T187023 (duration: 00m 08s)
  • 09:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 52s)
  • 09:40 mobrovac@deploy1001: Started deploy [zotero/translators@d6702bc]: String.includes() polyfill for Sveriges radio translator - T187023
  • 09:25 moritzm: rebooting ms-fe1005 as microcode update canary
  • 09:11 godog: drain restbase200[234] and restart cassandra to pick up updated certificates
  • 08:57 moritzm: draining restbase1007 for reboot to pick up Intel microcode update
  • 08:42 moritzm: draining restbase2010 for reboot to pick up Intel microcode update
  • 08:26 godog: add graphite2003 to carbon-c-relay frontend on graphite1001 - T196483
  • 08:12 moritzm: draining restbase2001 for reboot to pick up Intel microcode update
  • 07:40 marostegui: Optimize dewiki.logging on db1096:3315 - T197459
  • 07:39 marostegui: Deploy schema change on db1096:3315 T146591 T197891 T196379
  • 07:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 50s)
  • 07:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 00m 52s)
  • 07:13 elukey: stop mariadb on analytics1003 to apply https://gerrit.wikimedia.org/r/443893 and enable auth via unix socket
  • 06:39 kartik@deploy1001: Finished deploy [cxserver/deploy@3cb9a21]: Update cxserver to f8c71a1 (T191124, T198779) (duration: 03m 52s)
  • 06:35 kartik@deploy1001: Started deploy [cxserver/deploy@3cb9a21]: Update cxserver to f8c71a1 (T191124, T198779)
  • 06:11 marostegui: Deploy schema change on db1100 T146591 T197891 T196379
  • 06:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 00m 52s)
  • 06:09 marostegui: Optimize dewiki.logging on db1100 and dbstore1002:s5 - T197459
  • 05:50 marostegui: Deploy schema change on dbstore1002:s5 T146591 T197891 T196379
  • 05:29 marostegui: Deploy schema change on s3 primary master (db1075) T191316 T192926 T195193
  • 04:57 marostegui: Deploy schema change on s5 codfw host by host T146591 T197891 T196379
  • 04:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore db1089 original traffic (duration: 00m 51s)
  • 04:42 marostegui: Deploy schema change on s7 primary master (db1062) T191316 T192926 T195193
  • 02:40 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: 370242d13 and 45f4f61c2 (duration: 00m 52s)
  • 02:32 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 5 02:32:29 UTC 2018 (duration 10m 21s)
  • 02:22 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 08m 46s)
  • 00:46 krinkle@deploy1001: Synchronized wmf-config/: Ia3bef874a (duration: 00m 51s)

2018-07-04

  • 22:28 krinkle@deploy1001: Synchronized wmf-config/FeaturedFeedsWMF.php: I004bc9c3e71 (duration: 00m 50s)
  • 22:06 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I5a6e28 (duration: 00m 51s)
  • 21:53 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Iab176f3d205 (duration: 00m 50s)
  • 21:49 krinkle@deploy1001: Synchronized wmf-config/: I6f8cfa8f (duration: 00m 51s)
  • 21:43 krinkle@deploy1001: Synchronized wmf-config/: Ia8deabc5d2625 (duration: 00m 51s)
  • 21:42 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ia8deabc5d2625 (duration: 00m 50s)
  • 21:36 krinkle@deploy1001: Synchronized wmf-config/: If7da2a26bbf (duration: 00m 51s)
  • 21:34 krinkle@deploy1001: Synchronized wmf-config/: I1e78ba4365 (duration: 00m 52s)
  • 21:30 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ie30aeecbe (duration: 00m 52s)
  • 15:55 marostegui: Optimize dewiki.logging on s5 codfw master with replication, this will generate lag on s5 codfw - T197459
  • 15:52 ema: cp300[3-6]: puppet node clean/deactivate T167376
  • 15:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 51s)
  • 14:52 moritzm: installing libipc-run-perl updates from jessie point release
  • 14:36 moritzm: installing perl security updates on trusty (Debian already fixed)
  • 14:25 akosiaris: upgrade kubernetes staging API server to 1.8.14
  • 14:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 50s)
  • 14:10 moritzm: installing file/libmagic security updates on trusty (Debian already fixed)
  • 13:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 50s)
  • 13:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 50s)
  • 13:34 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2038, db2047 (duration: 02m 56s)
  • 12:56 marostegui: Stop MySQL and reboot db1089 to upgrade+change it to statement - T197069
  • 12:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 for maintenance - T197069 (duration: 02m 57s)
  • 12:39 ema: cp3034 repooled after hw maintenance T189305
  • 12:32 volans: shutting down bast3002 for disk replacement
  • 12:04 moritzm: installing ruby 1.9 security updates on trusty
  • 11:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 after alter table (duration: 00m 52s)
  • 11:54 ema: repool cp3043 after hardware maintenance T179953
  • 11:30 ema: shutdown cp3048 and cp3034 (both already depooled) for hardware maintenance T190607 T189305
  • 11:28 moritzm: rolling restart of cassandra on restbase hosts in eqiad completed
  • 11:27 moritzm: resuming rolling restart of cassandra on restbase hosts in eqiad completed
  • 11:17 mark: cp3043: mdadm /dev/md0 --add /dev/sdc1 (sdc is former cp3048:sdb)
  • 11:09 mark: cp3043: mdadm /dev/md0 -- fail /dev/sdb1
  • 11:03 ema: depool cp3043 (cache_upload) for hardware maintenance T179953
  • 10:58 godog: update compiler facts
  • 10:53 jynus: stop db2038 and db2047
  • 10:46 marostegui: Deploy schema change on db1077 with replication, this will generate lag on labs s3 T191316 T192926 T89737 T195193
  • 10:42 marostegui: Stop replication on db1077 to drop triggers on db1124:3313 - T192926
  • 10:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 for alter table (duration: 00m 50s)
  • 10:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 after alter table (duration: 00m 50s)
  • 10:01 moritzm: rolling reboot of sca* for "lazy fpu" kernel updates
  • 09:44 _joe_: stopping all cronjobs via a puppet run on terbium, T192092
  • 09:29 akosiaris: upload kubernetes 1.8.14 to apt.wikimedia.org/stretch-wikimedia/main
  • 09:16 moritzm: uploaded linux-meta 1.18 for jessie-wikimedia to apt.wikimedia.org
  • 09:15 elukey: reimage aqs1009 to Debian Stretch
  • 09:09 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2038, db2047 (duration: 00m 50s)
  • 09:03 marostegui: Optimize recentchanges table on s2 codfw - this will generate lag on codfw s2 - T178290
  • 09:00 akosiaris: manually rebalance the mathoid kubernetes production cluster namespaces pods wise
  • 08:56 moritzm: uploaded linux 4.9.107~wmf1 for jessie-wikimedia to apt.wikimedia.org
  • 08:54 marostegui: Deploy schema change on db1123 T191316 T192926 T89737 T195193
  • 08:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 for alter table (duration: 00m 50s)
  • 08:53 elukey: update analytics-in4 filter rules on cr1/cr2 eqiad - T198623
  • 08:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 after alter table (duration: 00m 50s)
  • 08:19 marostegui: Optimize recentchanges table on s6 codfw - this will generate lag on codfw s6 - T178290
  • 08:09 moritzm: rebooting multatuli for kernel update to 4.9.107~wmf1
  • 08:00 marostegui: Deploy schema change on db1078 T191316 T192926 T89737 T195193
  • 07:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 for alter table (duration: 00m 50s)
  • 07:58 ema: install misc VCL on all text hosts T164609
  • 07:53 volans: reimaging silver (spare host, to-be-decomm'ed) as testing host for the reimage script
  • 07:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 after alter table (duration: 00m 53s)
  • 07:46 ema: install misc VCL on a text host (cp3030) T164609
  • 07:43 moritzm: resuming rolling restart of cassandra on restbase hosts in eqiad to pick up OpenJDK security update
  • 07:38 hashar: rebased /srv/mediawiki-staging on deploy1001 for beta cluster only change https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/443475/
  • 07:22 moritzm: installing imagemagick security updates
  • 07:18 akosiaris: reimage kubernetes100{1,2}.eqiad.wmnet kubernetes200{1,2}.codfw.wmnet without swap
  • 06:43 _joe_: restarting tilerator on maps-test2004, to check if it can recover
  • 06:15 elukey: reimage aqs1008 to Debian Stretch
  • 06:03 marostegui: Optimize recentchanges table on s7 codfw - this will generate lag on codfw s7 - T178290
  • 05:34 marostegui: Optimize recentchanges table on s3 eqiad, host by host T178290
  • 05:28 marostegui: Optimize recentchanges table on s3 codfw - this will generate lag on codfw s3 - T178290
  • 04:59 marostegui: Deploy schema change on db1101:3317 T191316 T192926 T89737 T195193
  • 04:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 for alter table (duration: 00m 52s)
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 4 02:45:51 UTC 2018 (duration 10m 16s)
  • 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 14m 57s)
  • 00:38 Krinkle: Purge https://en.wikipedia.org/static/images/project-logos/eswiki-2x.png
  • 00:38 Krinkle: Purge https://en.wikipedia.org/static/images/project-logos/eswiki-1.5x.png
  • 00:37 Krinkle: Purge https://en.wikipedia.org/static/images/project-logos/eswiki.png
  • 00:36 krinkle@deploy1001: Synchronized static/images/project-logos/: T198761 - Update eswiki logo (duration: 00m 51s)

2018-07-03

  • 23:08 maxsem@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/CodeMirror: SWAT https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CodeMirror/+/443743/ (duration: 00m 52s)
  • 18:21 anomie@deploy1001: Synchronized wmf-config/CommonSettings.php: Fix for T197475 (gerrit:440543) (duration: 00m 52s)
  • 18:09 herron: cobalt:~# systemctl restart gerrit
  • 16:45 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase es1017,db1100 weights (duration: 00m 50s)
  • 16:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 with low weight (duration: 00m 50s)
  • 16:00 addshore: WikibaseLexeme for Wikidata clients, SE01EP05 deploy slot complete
  • 16:00 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY (duration: 00m 51s)
  • 15:46 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: all wikidata clients (duration: 00m 50s)
  • 15:42 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: group1 (duration: 00m 50s)
  • 15:37 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: group0 (duration: 00m 51s)
  • 15:17 godog: reset-failed on ms-be1024 for two user sessions failed when the host was in trouble
  • 15:01 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: testwiki (duration: 00m 51s)
  • 14:59 addshore@deploy1001: sync-file aborted: T197454 EP5 Enable WikibaseLexeme on clients: testwiki (duration: 00m 09s)
  • 14:42 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1017 with low weight (duration: 00m 50s)
  • 14:12 kart_: Finished ContentTranslation draft purge script
  • 14:11 papaul: OS install on graphite2003
  • 14:11 kart_: Running ContentTranslation draft purge script
  • 14:08 moritzm: rolling restart of cassandra on restbase in eqiad to pick up Java security updates
  • 14:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Change es1017 IP and rack T197072 (duration: 00m 50s)
  • 14:06 zeljkof: EU SWAT finished
  • 14:05 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Some wikis bureacurats are able to grant non-grantable groups (T197026) (duration: 00m 51s)
  • 13:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Clean legacy AddGroups/RemoveGroups (T197024) (duration: 00m 50s)
  • 13:38 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow bcts on private&fishbowl wikis advanced privilege manipulation (T197024) (duration: 00m 50s)
  • 13:31 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add namespace alias on pswikivoyage (T197507) (duration: 00m 49s)
  • 13:24 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable SandboxLink on eswikiversity (T198335) (duration: 00m 51s)
  • 13:19 marostegui: Deploy schema change on dbstore1002:s3 T191316 T192926 T89737 T195193
  • 13:14 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Wikimania 2018 (T198288) (duration: 00m 53s)
  • 12:49 jynus: shutting down es1017
  • 11:39 elukey: reimage aqs1007 to Debian Stretch
  • 11:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 after alter table (duration: 00m 52s)
  • 10:46 jynus: stop and reimage to stretch db2051
  • 10:13 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@ba672a3]: Decrease checker job concurrency T198462 (duration: 00m 52s)
  • 10:12 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@ba672a3]: Decrease checker job concurrency T198462
  • 09:59 marostegui: Deploy schema change on s3 codfw primary master (db2043) with replication, this will generate lag on s3 codfw T191316 T192926 T89737 T195193 T197459
  • 09:56 marostegui: Stop replication on db2094:3313 to replace triggers - T192926
  • 09:44 jynus: reimage db1100 for upgrade to stretch
  • 09:13 marostegui: Deploy schema change on db1098:3317 T191316 T192926 T89737 T195193 T197459
  • 09:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 for alter table (duration: 00m 50s)
  • 09:11 elukey: reimage aqs1006 to Debian Stretch
  • 09:09 jynus: deploying sys schema into db1072 (phabricator master)
  • 09:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 after alter table (duration: 00m 50s)
  • 09:04 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 51s)
  • 08:32 jynus: stop es1017 mysql for upgrade and maintenance
  • 08:06 moritzm: resuming rolling restart of cassandra on restbase in codfw to pick up Java security updates
  • 07:38 elukey: reimage aqs1005 to debian stretch
  • 07:33 akosiaris: upload apertium-separable_0.3.1-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main T191404
  • 07:18 moritzm: upgrading ffmpeg on mw1307 (T190333)
  • 07:17 akosiaris: reimage kubestage1001
  • 07:13 marostegui: Deploy schema change on db1090:3317 T191316 T192926 T89737 T195193 T197459
  • 07:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 for alter table (duration: 00m 51s)
  • 07:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 after alter table (duration: 00m 50s)
  • 07:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1017 (duration: 00m 51s)
  • 06:16 marostegui: Stop replication on db1079 to drop triggers on db1125:s7 - T192926
  • 06:14 marostegui: Deploy schema change on db1079 with replication, this will generate lag on s7 on labsdb hosts T191316 T192926 T89737 T195193 T197459
  • 06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 for alter table (duration: 00m 50s)
  • 06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 after alter table (duration: 00m 50s)
  • 05:26 marostegui: Deploy schema change on db1094 T191316 T192926 T89737 T195193 T197459
  • 05:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 for alter table (duration: 00m 49s)
  • 05:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 after alter table (duration: 00m 50s)
  • 05:04 marostegui: Deploy schema change on s8 primary master db1071 T191316 T192926 T195193
  • 04:40 marostegui: Deploy schema change on db1086 T191316 T192926 T89737 T195193 T197459
  • 04:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 for alter table (duration: 00m 52s)
  • 04:27 marostegui: Deploy schema change on s2 primary master db1066 T191316 T192926 T195193
  • 02:43 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 3 02:43:03 UTC 2018 (duration 10m 19s)
  • 02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 07s)

2018-07-02

  • 23:33 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/: Watchlist performance fixes (T198359, T198399) (duration: 00m 51s)
  • 23:09 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Add chapter wikis to CORS domain list (T181165) (duration: 00m 52s)
  • 20:53 urandom: Bringing down Cassandra for hardware testing, restbase2010 - T197477
  • 19:25 urandom: Bringing down Cassandra for hardware testing, restbase2001 - T197477
  • 19:00 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Mark rollbacking revision as patrolled T198449; WikiPage: Do not set undid revision ID for rollbacks T190374 (duration: 00m 50s)
  • 18:47 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/: RCFilters: Hide highlight containers when RCFilters is disabled T198440 (duration: 00m 51s)
  • 18:41 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Fix incorrect arguments to prepareContent() call in WikiPage T198483 (duration: 00m 51s)
  • 18:16 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/PageTriage/: Fix setting of reviewed state based on autopatrol permission T198497 (duration: 00m 52s)
  • 18:11 Niharika: Tried updating interwiki cache but it failed with "<AttributeError> 'Namespace' object has no attribute 'force'"
  • 17:16 gehel@deploy1001: Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (duration: 09m 04s)
  • 17:07 gehel@deploy1001: Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph
  • 17:05 gehel@deploy1001: Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1009 only) (duration: 00m 28s)
  • 17:04 gehel@deploy1001: Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1009 only)
  • 15:55 _joe_: stopping jobrunner, jobchron across all jobrunners T198220
  • 15:38 marostegui: Deploy schema change on dbstore1002:s7 T191316 T192926 T89737 T195193 T197459
  • 15:31 andrewbogott: rebooting labnodepool1002 as part of a paging test
  • 15:06 moritzm: installing file/libmagic security updates
  • 15:06 addshore: Lexeme deploy window done...
  • 15:04 twentyafterfour: restarted apache2 on phab1001: whitelist WMDE
  • 14:29 elukey: copy cassandra-tools-wmf 1.0.2-1 from jessie-wikimedia to stretch-wikimedia
  • 14:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1018 (duration: 00m 52s)
  • 14:16 kart_: Finished: Dry run: ContentTranslation Draft Purge
  • 14:09 kart_: Dry run: ContentTranslation Draft Purge
  • 13:40 moritzm: installing libgcrypt11 updates on trusty
  • 13:34 elukey: reimage aqs1004 to Debian Stretch
  • 13:29 hashar@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/mediawiki.rcfilters/styles/: RCFilters: Fix highlight container selector in Watchlist overrides - T198445 (duration: 00m 51s)
  • 13:28 hashar@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/ContentTranslation: Allow non-Main namespace source titles iff given in URL - T198178 (duration: 00m 52s)
  • 13:21 marostegui: Optimize frwiktionary'.logging on codfw - T197459
  • 13:15 marostegui: Optimize viwiki.logging on codfw - T197459
  • 12:59 ppchelko@deploy1001: Finished deploy [changeprop/deploy@82fd280]: Revert: Move static blacklisting to change-prop T198386 Can't deploy yet due to scap bug T198621 (duration: 01m 26s)
  • 12:57 ppchelko@deploy1001: Started deploy [changeprop/deploy@82fd280]: Revert: Move static blacklisting to change-prop T198386 Can't deploy yet due to scap bug T198621
  • 12:50 marostegui: Stop replication on db2077 to remove triggers from db2095 - T192926
  • 12:49 marostegui: Deploy schema change on s7 codfw master (db2040) with replication, this will generate lag on s7 codfw T191316 T192926 T89737 T195193
  • 12:49 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f20916f]: Move static blacklisting to change-prop T198386 take 2 (duration: 00m 24s)
  • 12:48 ppchelko@deploy1001: Started deploy [changeprop/deploy@f20916f]: Move static blacklisting to change-prop T198386 take 2
  • 12:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 after alter table (duration: 00m 52s)
  • 12:24 ppchelko@deploy1001: Finished deploy [changeprop/deploy@a5a57ff]: Move static blacklisting to change-prop T198386 (duration: 00m 39s)
  • 12:24 ppchelko@deploy1001: Started deploy [changeprop/deploy@a5a57ff]: Move static blacklisting to change-prop T198386
  • 12:10 akosiaris: T197559 upload lttoolbox_3.4.2-2+wmf2 to apt.wikimedia.org/jessie-wikimedia/main
  • 11:28 ema: repool cp3037 T196974
  • 11:02 moritzm: installing libgcrypt20 security updates
  • 10:06 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 10:05 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
  • 09:53 elukey: reboot ms-be1039 (bad disk, spike in I/O and load, not reachable via ssh or mgmt console)
  • 09:37 jynus: SET GLOBAL expire_logs_days = 30 on db1067
  • 09:24 marostegui: Deploy schema change on db1087 with replication, this will generate lag on s8 on labs T191316 T192926 T89737 T195193
  • 09:17 marostegui: Stop replication on db1087 to drop triggers on db1124 - T192926
  • 09:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 for alter table (duration: 00m 51s)
  • 09:00 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@dfdd362]: Disable delayed execution for cirrusSearchCheckerJob T198462 (duration: 00m 50s)
  • 08:59 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@dfdd362]: Disable delayed execution for cirrusSearchCheckerJob T198462
  • 08:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 00m 50s)
  • 08:47 volans@deploy1001: Finished deploy [debmonitor/deploy@15a0776]: GC fix and maintenance command (duration: 00m 24s)
  • 08:46 volans@deploy1001: Started deploy [debmonitor/deploy@15a0776]: GC fix and maintenance command
  • 08:24 moritzm: installing jasper security updates
  • 08:09 jynus: stop and reimage es1018
  • 08:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1018 (duration: 00m 50s)
  • 08:04 gehel: powercycle maps-test2003
  • 04:59 marostegui: Deploy schema change on db1109 T191316 T192926 T89737 T195193
  • 04:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 00m 56s)
  • 04:25 twentyafterfour: Bsadowski1 The throttle is averaged over 5 minutes, you should be unblocked shortly after being blocked
  • 04:24 twentyafterfour: applied security patch on phab1001, source in /srv/patches
  • 02:44 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 2 02:44:40 UTC 2018 (duration 10m 21s)
  • 02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 39s)

2018-07-01

  • 02:44 andrewbogott: forcing reboot of labservices1001 via mgmt

2018-06-30

  • 21:35 ariel@deploy1001: Finished deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063 (duration: 02m 27s)
  • 21:32 ariel@deploy1001: Started deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063
  • 16:28 chasemp: labstore1006:~# service nfs-kernel-server restart

2018-06-29

  • 21:37 chasemp: (late log) of VLAN creations for T184209 cloud-instances2-b-eqiad and cloud-instance-transport1-b-eqiad
  • 19:11 herron: stat1005:~# killall -u ironholds T197895
  • 17:28 XioNoX: Re-activating v6 BGP session to Telia on cr2-eqiad - T198502
  • 17:21 XioNoX: deactivating v6 BGP session to Telia on cr2-eqiad - T198502
  • 14:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 00m 50s)
  • 12:30 moritzm: uploaded ffmpeg 3.2.10-1~deb9u1+wmf1 to apt.wikimedia.org/stretch-wikimedia (component/vp9) (linked against libvpx 1.7 and with backported row-mt support) (T190333)
  • 11:11 moritzm: installing zsh updates from jessie 8.11 point release
  • 11:01 moritzm: installing lame security updates
  • 10:34 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2058 after reimage (duration: 00m 51s)
  • 10:21 mobrovac@deploy1001: Finished deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 - T165105 T197853 (duration: 03m 26s)
  • 10:20 marostegui: Deploy schema change on db1092 T191316 T192926 T89737 T195193
  • 10:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 00m 50s)
  • 10:18 mobrovac@deploy1001: Started deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 - T165105 T197853
  • 10:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 00m 50s)
  • 09:23 marostegui: Stop MySQL on db2058 to reimage it
  • 09:23 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2058 for reimage (duration: 00m 50s)
  • 09:18 moritzm: uploaded libvpx 1.7.0-3+wmf1 to apt.wikimedia.org/stretch-wikimedia (component/vp9) (T190333)
  • 09:08 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2054 after reimage (duration: 00m 51s)
  • 08:39 vgutierrez: Apply new TLS varnishkafka settings in cache::text nodes - T182993
  • 08:27 akosiaris: upgrade php on phab1001 to 5.6.33+dfsg-0+deb8u1
  • 08:03 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1016 (duration: 00m 50s)
  • 07:53 marostegui: Stop MySQL on db2054 for reimage
  • 07:53 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2054 for reimage (duration: 00m 50s)
  • 06:21 jynus: stop and reimage es1016
  • 05:40 elukey: force umount of dumps labstore nfs mountpoints on stat100[56]/notebook100[34] to reduce load (also too many open files)
  • 05:02 marostegui: Deploy schema change on db1104 T191316 T192926 T89737 T195193
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 for alter table (duration: 00m 50s)
  • 04:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 after alter table (duration: 00m 50s)
  • 04:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 (duration: 00m 52s)
  • 01:53 catrope@deploy1001: Finished scap: Full scap to fix i18n issues on testwiki (duration: 32m 42s)
  • 01:20 catrope@deploy1001: Started scap: Full scap to fix i18n issues on testwiki
  • 01:19 krinkle@deploy1001: Synchronized vendor/: I39e592d837 - remove redundant composer packages for xhgui profiling (duration: 00m 52s)

2018-06-28

  • 23:54 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/mediawiki.rcfilters/: Watchlist perf fixes (T198359, T198399) (duration: 00m 52s)
  • 23:18 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Increase Schema:CitationUsage sampling rate to 100% (T191086) (duration: 00m 51s)
  • 21:55 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/PageTriage/: Update extension directory (duration: 00m 51s)
  • 21:52 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Draft namespace and AfC mode for PageTriage on testwiki T198143 (duration: 00m 53s)
  • 20:53 dduvall@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.10
  • 20:46 marxarelli: Rolling 1.32.0-wmf.10 to group2 following fix and successful re-deploy to group1
  • 20:29 dduvall@deploy1001: rebuilt and synchronized wikiversions files: commonswiki to 1.32.0-wmf.10 otra vez
  • 20:07 dduvall@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Syncing table locking fix (T198350) (duration: 00m 57s)
  • 20:04 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Rollback commonswiki to 1.32.0-wmf.8 (resync following ssh hang)
  • 20:02 dduvall@deploy1001: sync-wikiversions aborted: Rollback commonswiki to 1.32.0-wmf.8 (duration: 15m 24s)
  • 19:34 dduvall@deploy1001: rebuilt and synchronized wikiversions files: commonswiki to 1.32.0-wmf.10
  • 19:17 dduvall@deploy1001: Synchronized php: Group1 (less commons) to 1.32.0-wmf.10 (duration: 00m 57s)
  • 19:16 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Group1 (less commons) to 1.32.0-wmf.10
  • 16:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1015 - crashed (duration: 00m 57s)
  • 16:09 moritzm: restarting Cassandra instances on restbase2005 to pick up Java security update
  • 15:01 marostegui: Deploy schema change on db1101:3318 T191316 T192926 T89737 T195193
  • 15:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 for alter table (duration: 00m 57s)
  • 14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 after alter table (duration: 00m 58s)
  • 14:50 raynor: EU SWAT finished
  • 14:49 pmiazga@deploy1001: Synchronized php-1.32.0-wmf.10/skins/Vector/components/watchstar.less: SWAT: Use exactly calculated value to work around a Chrome bug (T196610) (duration: 01m 00s)
  • 14:46 elukey: upgrade piwik 3.2.1 to matomo (new name/package) 3.5.1 - T192298
  • 14:29 moritzm: installing ghostscript security updates
  • 14:26 hashar: CI jobs running npm might suffer from a 10 minutes delay since June 27th | T198348
  • 14:06 gehel: downgrading cassandra to 2.2.6-wmf3 on maps-test2001 (it should never have been upgraded)
  • 13:49 dcausse@deploy1001: Synchronized ./php-1.32.0-wmf.10/includes/htmlform/: Allow overloading of getLabel() with return ' ' (duration: 00m 59s)
  • 13:49 elukey: downgrade cassadra and cassandra-tools from 2.2.6-wmf5 to 2.2.6-wmf3 in jessie-wikimedia component/cassandra22 - T197062
  • 13:25 dcausse@deploy1001: Synchronized ./wmf-config/Wikibase-production.php: Add cirrussearch settings for wikibase (3/3) (duration: 00m 56s)
  • 13:18 dcausse@deploy1001: Synchronized ./wmf-config/: Add cirrussearch settings for wikibase (2/3) (take 2) (duration: 00m 58s)
  • 13:14 vgutierrez: Apply new TLS varnishkafka settings in cache::upload nodes - T182993
  • 13:13 dcausse@deploy1001: Synchronized ./wmf-config/WikibaseSearchSettings.php: Add cirrussearch settings for wikibase (1.5/3) (duration: 00m 56s)
  • 13:01 elukey: upload matomo (new Piwik) 3.5.1-1 to jessie-wikimedia
  • 13:00 moritzm: installing bwm-ng update from jessie 8.11 point release
  • 12:53 moritzm: installing blktrace update from jessie 8.11 point release
  • 12:49 elukey: stop hadoop daemons on analytics1032 + shutdown to swap BBU -T194234
  • 12:37 akosiaris: repool scb1002 T196901
  • 12:37 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,service=.*,cluster=scb,name=scb1002
  • 12:08 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,service=.*,cluster=scb,name=scb1002
  • 12:04 moritzm: installing patch security updates
  • 10:48 twentyafterfour: deployed fix for PhabricatorDataNotAttachedException - https://phabricator.wikimedia.org/rPHEX03971ea8965d3613df69833a766d1502b6d8dabb
  • 10:19 twentyafterfour: hotfixing phabricator DataNotAttached bug
  • 10:18 joal@deploy1001: Finished deploy [analytics/refinery@4fc20a5]: Regular weekly deploy (duration: 08m 21s)
  • 10:11 twentyafterfour: phabricator database migration complete, service restored and appears stable.
  • 10:10 joal@deploy1001: Started deploy [analytics/refinery@4fc20a5]: Regular weekly deploy
  • 10:07 twentyafterfour: running phabricator database migration
  • 10:00 moritzm: installing reportbug update from jessie 8.11 point release
  • 09:58 twentyafterfour: Resuming deployment of phabricator upgrade tagged release/2018-06-27/1 - details: https://phabricator.wikimedia.org/project/profile/3439/ )
  • 09:48 joal@deploy1001: Finished deploy [analytics/aqs/deploy@8eef2a9]: Deploying AQS pageviews-per-country ceiling-value glue code - Corrected (duration: 02m 48s)
  • 09:45 joal@deploy1001: Started deploy [analytics/aqs/deploy@8eef2a9]: Deploying AQS pageviews-per-country ceiling-value glue code - Corrected
  • 09:29 joal@deploy1001: Finished deploy [analytics/aqs/deploy@194ca96]: Deploying AQS pageviews-per-country ceiling-value glue code (duration: 01m 03s)
  • 09:28 joal@deploy1001: Started deploy [analytics/aqs/deploy@194ca96]: Deploying AQS pageviews-per-country ceiling-value glue code
  • 09:20 arturo: T198377 stop nova-spiceproxy daemon in labcontrol1002.wikimedia.org
  • 09:14 mobrovac@deploy1001: Finished deploy [proton/deploy@8a887b5]: Update to dceaf80 - T186748 (duration: 00m 28s)
  • 09:13 mobrovac@deploy1001: Started deploy [proton/deploy@8a887b5]: Update to dceaf80 - T186748
  • 09:10 vgutierrez: Apply new TLS varnishkafka settings in cache::misc nodes - T182993
  • 08:46 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1016 (duration: 01m 04s)
  • 08:46 arturo: aborrero@labtestnet2001:~ 7s 130 $ sudo service nova-spiceproxy stop # daemon in infinite respawning loop
  • 08:15 elukey: restart-hhvm on mw1227 (some threads stuck in jit-related operations, causing high load)
  • 08:09 vgutierrez: updating librdkafka1 && restart varnishkafka instances in cache::text nodes - T182993
  • 07:12 elukey: upload piwik 3.2.1 to jessie-wikimedia
  • 04:46 marostegui: Deploy schema change on db1099:3318 T191316 T192926 T89737 T195193
  • 04:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 for alter table (duration: 00m 59s)
  • 03:03 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jun 28 03:03:34 UTC 2018 (duration 10m 25s)
  • 02:53 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 47s)
  • 02:21 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 07m 54s)
  • 00:33 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/resources: Watchlist perf patches for SWAT, part 2 (T197168, T198140, T198142) (duration: 00m 57s)
  • 00:32 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/includes: Watchlist perf patches for SWAT, part 1 (T197168, T198140, T198142) (duration: 01m 13s)
  • 00:13 twentyafterfour: rolled back and restored service to previous state
  • 00:12 twentyafterfour: phabricator update failed. unable to apply database migrations: mysql access denied
  • 00:05 twentyafterfour: taking apache offline momentarily on phab1001

2018-06-27

  • 23:57 twentyafterfour: phabricator deployment is coming up in just a couple of minutes. There will be downtime while I run database migrations.
  • 23:10 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turning on page creation log for most wikis T196400 (duration: 00m 58s)
  • 21:50 elukey: piwik maintenance on bohrium completed
  • 21:42 XioNoX: setting BFD of the Zayo eqiad-codfw link to standard of 300
  • 20:02 dduvall@deploy1001: Synchronized php: (no justification provided) (duration: 00m 57s)
  • 20:02 SMalyshev: applied fix for T197447 to eqiad wdqs cluster, which involved restart of the services
  • 19:54 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Group1 rolled back to 1.32.0-wmf.8
  • 19:52 marxarelli: Rolling back group1 due to rise in error rate (T198350)
  • 19:22 marxarelli: errors seem due to "INSERT INTO `revision_comment_temp`" statements and lock wait timeout
  • 19:18 marxarelli: seeing rising "Wikimedia\Rdbms\DBQueryError from line 1443 of /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema update..." errors
  • 19:16 dduvall@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.10 (duration: 00m 58s)
  • 19:15 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.10
  • 17:57 XioNoX: updating NTP servers on network devices
  • 17:53 mobrovac@deploy1001: Finished deploy [proton/deploy@cd6ed94]: Update proton to 491e966 - T186748 T197856 (duration: 00m 35s)
  • 17:52 mobrovac@deploy1001: Started deploy [proton/deploy@cd6ed94]: Update proton to 491e966 - T186748 T197856
  • 16:39 marostegui: Deploy schema change on dbstore1002:s8 T191316 T192926 T89737 T195193
  • 15:38 thcipriani@deploy1001: Synchronized README: Scap 3.8.3-1 noop test sync-file (duration: 00m 56s)
  • 15:36 marostegui: Stop replication on db2094:3318 to update triggers on archive table
  • 15:30 godog: upload scap 3.8.3 - T198277
  • 14:51 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1067 (duration: 00m 56s)
  • 13:57 dcausse: EU swat done
  • 13:42 dcausse@deploy1001: Finished scap: wmf-config Add cirrussearch settings for wikibase (1/3) (duration: 05m 41s)
  • 13:40 moritzm: uploaded debmonitor 0.1.5 to apt.wikimedia.org
  • 13:36 dcausse@deploy1001: Started scap: wmf-config Add cirrussearch settings for wikibase (1/3)
  • 13:24 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Change FileImporter config data location (T198050) (duration: 00m 57s)
  • 13:07 elukey: piwik upgraded to 3.2.1 on bohrium + started the db migration procedure (will last 2/3h probably)
  • 12:49 vgutierrez: Upgrade librdkafka1 and restart varnishkafka-webrequest in cache::upload nodes - T182993
  • 11:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 after alter table (duration: 00m 57s)
  • 11:26 marostegui: Deploy schema change on s8 codfw master (db2045) with replication, this will generate lag on s8 codfw T191316 T192926 T89737 T195193
  • 10:38 gehel: removing maps-test2003 from cluster for reimage - T198290
  • 10:36 volans@deploy1001: Finished deploy [debmonitor/deploy@9536ebf]: CSP header hotfix (duration: 00m 22s)
  • 10:36 volans@deploy1001: Started deploy [debmonitor/deploy@9536ebf]: CSP header hotfix
  • 10:24 marostegui: Deploy schema change on db1076 T191316 T192926 T89737 T195193
  • 10:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 for alter table (duration: 00m 56s)
  • 10:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 57s)
  • 10:11 jynus: stopping db1067 and reimage it
  • 09:38 volans@deploy1001: Finished deploy [debmonitor/deploy@052a9ea]: Release v0.1.5 (duration: 00m 24s)
  • 09:38 volans@deploy1001: Started deploy [debmonitor/deploy@052a9ea]: Release v0.1.5
  • 09:14 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1067 (duration: 00m 56s)
  • 08:35 marostegui: Deploy schema change on db1074 with replication, this will generate lag on s2 on labsdb T191316 T192926 T89737 T195193
  • 08:33 marostegui: Stop replication on db1074 to remove triggers from db1125 - T192926
  • 08:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 57s)
  • 08:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 after alter table (duration: 00m 57s)
  • 07:58 vgutierrez: Reinstall acamar & achernar as spare systems
  • 07:50 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns4001.wikimedia.org
  • 07:38 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=dns4001.wikimedia.org
  • 07:37 vgutierrez: Depool dns4001 for server restart - T198215
  • 05:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 for alter table (duration: 01m 06s)
  • 05:08 marostegui: Deploy schema change on db1090:3312 T191316 T192926 T89737 T195193
  • 04:47 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@2207b66]: Update mobileapps to d7221ba (duration: 05m 42s)
  • 04:42 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@2207b66]: Update mobileapps to d7221ba
  • 03:04 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jun 27 03:04:07 UTC 2018 (duration 10m 32s)
  • 02:53 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 15m 53s)
  • 02:20 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 33s)

2018-06-26

  • 21:26 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Group0 to 1.32.0-wmf.10
  • 21:18 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache (duration: 22m 50s)
  • 20:56 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
  • 20:47 marxarelli: removing orphaned /srv/mediawiki/php-1.31.0-wmf.28 dir on deploy1001
  • 20:38 volans: regenerated debian installer for jessie 8.11 point release (released 3 days ago)
  • 20:34 marxarelli: removed /srv/mediawiki/php-1.32.0-wmf.4 on mwdebug2002 to free disk space
  • 20:22 marxarelli: manually cleaning up on mwdebug2002 due to full disk and scap failures
  • 20:21 dduvall@deploy1001: Pruned MediaWiki: 1.32.0-wmf.4 [keeping static files] (duration: 03m 21s)
  • 19:43 marxarelli: cdb-rebuild failed on mwdebug2002 due to lack of disk space
  • 19:41 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache (duration: 39m 11s)
  • 19:02 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
  • 18:39 marxarelli: setting $wmgUseMwEmbedSupport = false in php-1.32.0-wmf.10/LocalSettings.php to extension registry exception (see T197918 and T191056)
  • 18:14 marxarelli: scap sync for testwiki is currently failing on l10update due to MwEmbedSupport's deprecation but no change yet to mediawiki/extensions to remove the extension. blocking train on T197918
  • 18:01 dduvall@deploy1001: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2905568126" --threads=30 --lang en --quiet' returned non-zero exit status 255 (duration: 02m 23s)
  • 17:59 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
  • 17:16 marxarelli: starting branch cut for 1.32-wmf.10
  • 16:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 after alter table (duration: 00m 57s)
  • 14:43 elukey: rm syslog.1.gz puppet.log.1.gz on tegment to fix cronspam
  • 14:19 ottomata: Re-enable job and change-prop Kafka topic mirroring from main-eqiad -> main-codfw
  • 13:37 hashar: European swat completed
  • 13:36 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable zh-my variant on zhwiki - T193983 (duration: 00m 57s)
  • 13:31 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use uploaded HD logos for ruwikiquote - T197508 (duration: 00m 57s)
  • 13:29 hashar: purgeList.php for the various ruwikiquote logos
  • 13:27 hashar@deploy1001: Synchronized static/images/project-logos: Change logo resources for ruwikiquote - T197508 (duration: 00m 56s)
  • 13:25 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable TemplateStyles on ruwikiquote - T197526 (duration: 00m 57s)
  • 13:22 dcausse: elastic@eqiad: forcemerge (only_expunge_deletes=true) on wikidatawiki_content
  • 13:21 dcausse: elastic@eqiad deleting stale index viwiki_general_1525301649
  • 13:19 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Create a few of namespace aliases for ruwiktionary - T197565 (duration: 00m 57s)
  • 13:06 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add some namespace aliases to ruwikiquote - T197058 (duration: 00m 56s)
  • 12:54 chasemp: labcontrol1003 aptitude install ldap-utils python-ldappool
  • 12:15 marostegui: Deploy schema change on db1105:3312 T191316 T192926 T89737 T195193
  • 12:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 for alter table (duration: 00m 57s)
  • 12:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 after alter table (duration: 00m 57s)
  • 11:09 godog: roll-upgrade thumbor in codfw/eqiad
  • 11:08 godog: upload thumbor 2.0-2 to apt.wikimedia.org
  • 10:40 godog: upgrade thumbor to 2.0-1 on thumbor1001
  • 10:10 godog: test upgrade thumbor 2.0 on thumbor2001
  • 10:10 akosiaris: proceed with apertium-apy upgrade on all scb hosts. T194342
  • 10:01 akosiaris: downgrade erroneously upgraded apertium-fra-cat on scb1001, scb1002 from 1.3.0~r84327-1+wmf1 to 1.2.0~r78602-1+wmf2 T194342
  • 09:40 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw
  • 09:39 akosiaris: re-enable icinga-wm that I 've previously silenced
  • 09:37 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=eqiad
  • 09:36 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=appserver
  • 09:33 mobrovac@deploy1001: Finished scap: Switch the last remaining jobs to EventBus, full scap sync with HHVM restarts, take #3 - T190327 (duration: 05m 07s)
  • 09:31 akosiaris: upgrade apertium-apy, apertium-fra-cat, python3-streamparser, enable puppet, run puppet on scb1002 as a last test before full cluster upgrade. T194342
  • 09:31 vgutierrez: updating librdkafka1 and restarting varnishkafka-webrequest on cache::misc nodes - T182993
  • 09:29 dcausse: elastic@eqiad: forcemerge (only_expunge_deletes=true) on commonswiki_file
  • 09:28 mobrovac@deploy1001: Started scap: Switch the last remaining jobs to EventBus, full scap sync with HHVM restarts, take #3 - T190327
  • 09:25 akosiaris: manually install python3-streamparser on scb1001 for testing. T194342
  • 09:19 mobrovac@deploy1001: Finished deploy [changeprop/deploy@5d8eaf1]: Bug fix: Add restore to WD description checks - T197626 (duration: 01m 14s)
  • 09:18 mobrovac@deploy1001: Started deploy [changeprop/deploy@5d8eaf1]: Bug fix: Add restore to WD description checks - T197626
  • 09:11 mobrovac@deploy1001: Finished scap: Switch the last remaining jobs to EventBus, full scap sync, take #2 - T190327 (duration: 08m 34s)
  • 09:09 vgutierrez: upload librdkafka_0.11.3-1~bpo8+1+wikimedia2 to apt.w.o jessie-wikimedia - T182993
  • 09:08 akosiaris: upgrade scb1001 apertium-apy, apertium-fra-cat, enable puppet and run puppet
  • 09:06 akosiaris: T194342. Disable puppet on scb hosts for apertium-apy upgrade
  • 09:04 dcausse: unbanning elastic1042
  • 09:03 mobrovac@deploy1001: Started scap: Switch the last remaining jobs to EventBus, full scap sync, take #2 - T190327
  • 08:52 mobrovac@deploy1001: scap failed: RuntimeError scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) (duration: 04m 27s)
  • 08:52 mobrovac@deploy1001: scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 08:48 mobrovac@deploy1001: Started scap: Switch the last remaining jobs to EventBus, full scap sync for clean-up - T190327
  • 08:47 mobrovac@deploy1001: Synchronized wmf-config/CommonSettings.php: Switch the last remaining jobs to EventBus, only CommonSettings.php now - T190327 (duration: 00m 58s)
  • 08:47 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327 (duration: 00m 58s)
  • 08:46 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327
  • 07:58 vgutierrez: Replaced acamar and achernar with dns200[12] as main lvs name servers in codfw - T196493
  • 07:13 _joe_: restarting ircecho, was not reporting alerts
  • 07:05 marostegui: Deploy schema change on db1103:3312 T191316 T192926 T89737 T195193
  • 07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for alter table (duration: 00m 56s)
  • 06:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 after alter table (duration: 00m 57s)
  • 05:58 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db1054, it is going to be decommissioned T197063 (duration: 00m 55s)
  • 05:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db1054, it is going to be decommissioned T197063 (duration: 00m 57s)
  • 05:52 marostegui: Stop MySQL on db1054 as it is going to be decommissioned - T197063
  • 05:28 SMalyshev: testing fix for T197447 on wdqs1009
  • 05:15 marostegui: Deploy schema change on db1122 T191316 T192926 T89737 T195193
  • 05:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 for alter table (duration: 00m 59s)
  • 03:05 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 12m 59s)
  • 02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 13m 26s)

2018-06-25

  • 23:40 thcipriani: restarting jenkins for updates
  • 23:30 twentyafterfour: After monitoring logstash, everything appears stable. Evening SWAT is complete.
  • 23:23 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440878/ for SWAT: no-op (duration: 00m 56s)
  • 23:10 twentyafterfour@deploy1001: Synchronized wmf-config/Wikibase-production.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440420/ for SWAT (duration: 00m 57s)
  • 21:04 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@770cdb0]: Update mobileapps to 8c76d52 (duration: 05m 41s)
  • 20:58 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@770cdb0]: Update mobileapps to 8c76d52
  • 20:09 mdholloway: deploying mobileapps to canary (scb2001) failed, rolling back to investigate
  • 20:07 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@fb70d4f]: Update mobileapps to 362f2a0 (duration: 01m 34s)
  • 20:06 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@fb70d4f]: Update mobileapps to 362f2a0
  • 20:02 cscott: Completed deploy of Parsoid to version b068bb51 (T197949, T197702, T196799)
  • 19:53 cscott@deploy1001: Finished deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51 (duration: 06m 57s)
  • 19:46 cscott@deploy1001: Started deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51
  • 19:22 cscott@deploy1001: Started deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51
  • 18:33 thcipriani@deploy1001: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 56s)
  • 18:32 thcipriani@deploy1001: Synchronized portals/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 57s)
  • 18:25 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add import sources to ta.wiktionary T196445 (duration: 00m 58s)
  • 18:13 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Enable logging for Schema:CitationUsage T191086 (noop beta-only sync) (duration: 00m 57s)
  • 18:07 gehel@deploy1001: Finished deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (duration: 08m 19s)
  • 17:58 gehel@deploy1001: Started deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater
  • 17:56 gehel@deploy1001: Finished deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 24s)
  • 17:56 gehel@deploy1001: Started deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (wdqs1009 only)
  • 17:36 ottomata: Re-enable job and change-prop kafka topic mirroring from main-codfw -> main-eqiad
  • 16:45 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Beta only gerrit:441909 (duration: 00m 57s)
  • 16:19 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org
  • 16:13 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=acamar.wikimedia.org
  • 16:04 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns2002.wikimedia.org
  • 16:00 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns2001.wikimedia.org
  • 15:21 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: REVERT: Load WikibaseLexeme on testwiki (again again) T197454 (duration: 00m 57s)
  • 15:16 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Load WikibaseLexeme on testwiki (again again) T197454 (duration: 00m 55s)
  • 15:05 addshore: FileImporter slot done, moving onto Lexeme slot
  • 15:03 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable FileExpoter on ar-, de- and fa-wiki T196969 T197066 (duration: 00m 57s)
  • 14:58 marostegui: Deploy schema change on dbstore1002:s2 T191316 T192926 T89737 T195193
  • 14:57 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable FileImporter on Wikimedia Commons T196969 (duration: 00m 56s)
  • 14:40 addshore@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/FileImporter: Ignore namespace string when checking templates and categories (duration: 00m 56s)
  • 14:37 addshore@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/FileImporter: Ignore namespace string when checking templates and categories (duration: 00m 58s)
  • 14:32 addshore@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/FileImporter: Add a temp config to allow interwiki linking T196976 (duration: 00m 57s)
  • 14:30 addshore@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/FileImporter: Add a temp config to allow interwiki linking T196976 (duration: 00m 58s)
  • 14:23 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: Enable license filters for the FileImporter in production T194502 (duration: 00m 55s)
  • 14:19 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: Add ar, de and fa wikipedia to FileImporter interwiki config T196969 T196976 (duration: 00m 56s)
  • 14:17 hashar@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/Echo: Remove masterPos from the job specification - T192945 (duration: 00m 59s)
  • 14:14 elukey: merging jmxtrans and kafkatee's submodules to operations/puppet - part 2 (moving them back from environments/production)
  • 14:13 gehel: banning and depooling elastic1042 (high response times)
  • 14:01 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix ORES config, part II - T197633 (duration: 00m 57s)
  • 13:59 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Whitelist two Indian government websites - T197944 (duration: 00m 57s)
  • 13:57 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Autoconfirmed should require 10 edits&4 days on zhwikivoyage - T198006 (duration: 00m 56s)
  • 13:54 hashar: mwscript namespaceDupes.php --wiki=zhwikivoyage --fix | T198007
  • 13:53 elukey: merging jmxtrans and kafkatee's submodules to operations/puppet - part 1 (moving them under environments/production)
  • 13:51 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Autoconfirmed should require 10 edits&4 days on zhwikivoyage - T198006 (duration: 00m 56s)
  • 13:50 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add namespace aliases to zhwikivoyage - T198007 (duration: 00m 56s)
  • 13:37 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove BN alias for NS_USER on dewikivoyage - T196905 (duration: 00m 57s)
  • 13:34 hashar@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/Echo: Remove masterPos from the job specification - T192945 (duration: 00m 58s)
  • 13:32 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix ORES config - T197633 (duration: 00m 58s)
  • 13:27 hashar@deploy1001: Synchronized dblists/commonsuploads.dblist: Add wikimania2018wiki to commonsupload.dblist | T197714 (duration: 00m 56s)
  • 13:13 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable logging for Schema:CitationUsage at 1% | T191086 (duration: 00m 57s)
  • 13:05 marostegui: Deploy schema change on db2035 (codfw primary master for s2) with replication, this will generate lag on s2 codfw T191316 T192926 T89737 T195193
  • 13:05 godog: refresh cassandra certificates for 'restbase' cluster
  • 13:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 00m 57s)
  • 12:29 kartik@deploy1001: Finished deploy [cxserver/deploy@cc6dc61]: Update cxserver to ece5e7a (T191874, T196354, T195768, T195768) (duration: 03m 33s)
  • 12:26 kartik@deploy1001: Started deploy [cxserver/deploy@cc6dc61]: Update cxserver to ece5e7a (T191874, T196354, T195768, T195768)
  • 12:04 mobrovac: Restbase: Add the "headers" field to Cassandra schemae in production for T197789
  • 12:00 mobrovac@deploy1001: Finished deploy [restbase/deploy@f521e7e] (dev-cluster): Dev cluster: Include pagelanguage in title_revisions (duration: 04m 05s)
  • 12:00 gehel: repooling wdqs1005 it has catched up on updates - T198042
  • 11:56 mobrovac@deploy1001: Started deploy [restbase/deploy@f521e7e] (dev-cluster): Dev cluster: Include pagelanguage in title_revisions
  • 10:32 akosiaris: increase CPU count for proton machines from 2 to 10. T197862
  • 08:22 marostegui: Deploy schema change on db1106 with replication, this will generate lag on s1 labs T191316 T192926 T89737 T195193
  • 08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 00m 56s)
  • 08:16 marostegui: Stop replication on db1106 to change triggers for db1124 and db1095 - T192926
  • 08:11 gehel: rolling restart of wdqs to enable async logging - T198051
  • 08:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 58s)
  • 07:46 gehel: depooling wdqs1005 to allow it to catch up on updates - T198042
  • 07:33 moritzm: slow rollout of most of the remaining debmonitor client installations
  • 05:43 marostegui: Deploy schema change on db1080 T191316 T192926 T89737 T195193
  • 05:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 01m 17s)
  • 02:39 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 07m 37s)
  • 02:21 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 40s)

2018-06-24

  • 22:26 gehel: restarting wdqs1004 - T198042
  • 21:51 gehel: restarting wdqs1005 after taking a few thread dumps - T198042
  • 19:41 gehel: restart blazegraph on wdqs1004
  • 16:09 godog: restart wdqs-updater on wdqs1005
  • 15:40 godog: restart wdqs-blazegraph on wdqs1005
  • 15:40 jynus: restart graphite2001
  • 15:35 godog: systemctl restart wdqs-blazegraph on wdqs1004
  • 15:33 godog: restart-wdqs on wdqs1004

2018-06-22

  • 19:54 jynus: applying transaction manually on dbstore1002 due to weird bug with savepoint on tokudb+image table

2018-06-21

  • 21:38 ebernhardson: banned elastic1036 from search cluster, waited for all load to shift away, and unbanned
  • 16:38 ejegg: updated CiviCRM from 349d43eb8b to e2c8ddd70e
  • 13:28 vgutierrez: Bump AES128-SHA pageview replacement to 4%
  • 12:49 chasemp: remove labvirt1019 canary to start debug of network
  • 11:08 hashar: Refreshed operations-puppet-tests-docker jenkins job to a new Docker container build that includes isc-dhcp-server | https://gerrit.wikimedia.org/r/c/integration/config/+/441367
  • 02:51 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 07m 40s)
  • 02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 14m 12s)

2018-06-20

  • 21:18 ejegg: updated 'domain' settings in CiviCRM so name is now 'Wikimedia Foundation' and description is 'donate.wikimedia.org'
  • 20:08 bearND: rolled back "Update mobileapps to 9d856ec" (was just on canary)
  • 20:07 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@3420e67]: Update mobileapps to 9d856ec (duration: 03m 51s)
  • 20:03 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@3420e67]: Update mobileapps to 9d856ec
  • 19:54 ottomata: removed Kafka MirrorMaker from kafka10(12|13|14)
  • 16:01 sbisson@deploy1001: Finished deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN (duration: 03m 08s)
  • 15:58 sbisson@deploy1001: Started deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN
  • 15:57 sbisson@deploy1001: Finished deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN (duration: 00m 27s)
  • 15:56 sbisson@deploy1001: Started deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN
  • 15:52 sbisson@deploy1001: Finished deploy [kartotherian/deploy@9af9191]: (no justification provided) (duration: 03m 36s)
  • 15:49 sbisson@deploy1001: Started deploy [kartotherian/deploy@9af9191]: (no justification provided)
  • 14:59 dcausse: disk space issue on elastic1020 is due to shard rebalancing (currently receiving 2 enwiki_general shards but removing one wikidatawiki_content)
  • 14:07 imarlier@deploy1001: Finished deploy [performance/navtiming@742edb0]: (no justification provided) (duration: 00m 04s)
  • 14:07 imarlier@deploy1001: Started deploy [performance/navtiming@742edb0]: (no justification provided)
  • 14:03 imarlier@deploy1001: Finished deploy [performance/navtiming@8914e26]: (no justification provided) (duration: 00m 05s)
  • 14:03 imarlier@deploy1001: Started deploy [performance/navtiming@8914e26]: (no justification provided)
  • 13:43 imarlier@deploy1001: Finished deploy [performance/navtiming@995cb0f]: (no justification provided) (duration: 00m 05s)
  • 13:43 imarlier@deploy1001: Started deploy [performance/navtiming@995cb0f]: (no justification provided)
  • 13:38 anomie: Re-running populateExternallinksIndex60.php on plwiki and ptwiki for phab:T59176 (initial run collided with the s2 master switch).
  • 08:57 _joe_: removing user.log as well on tegmen
  • 08:54 _joe_: running logrotate on tegmen
  • 08:54 _joe_: killall nsca on tegmen
  • 08:51 _joe_: restarting nsca daemon on tegmen (gone wild, hundreds of subprocesses
  • 08:47 _joe_: removing user.log.1 and messages.log.1 on tegmen to save some space
  • 07:40 _joe_: powercycled mw1272, down since yesterday
  • 07:29 hashar: deploy1001: rebased php-1.32.0-wmf.8/extensions/Translate to catch up with a non production merged change ( https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/441127 ).
  • 02:20 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 17s)

2018-06-19

  • 16:10 mobrovac@deploy1001: Finished deploy [proton/deploy@43af7d9]: (no justification provided) (duration: 00m 27s)
  • 16:09 mobrovac@deploy1001: Started deploy [proton/deploy@43af7d9]: (no justification provided)
  • 14:13 herron: re-enabling puppet agents
  • 14:11 herron: increased client_max_body_size on puppetdb nginx frontends from 30m to 60m https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441044/
  • 14:06 herron: temporarily disabling puppet agents for deploy of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441044/
  • 12:41 _joe_: hard reboot of ms-be1019 - unable to ssh, console showing i/o errors only
  • 04:56 ebernhardson: unban elastic1035
  • 04:45 ebernhardson: ban elastic1035 from cluster to allow it to recover
  • 02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 06m 53s)
  • 02:19 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 07m 39s)

2018-06-18

  • 19:49 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@bcb2904]: GUI update (duration: 18m 52s)
  • 19:30 smalyshev@deploy1001: Started deploy [wdqs/wdqs@bcb2904]: GUI update
  • 19:30 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: GUI update (duration: 00m 56s)
  • 19:29 smalyshev@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: GUI update
  • 18:41 Trey314159: reindexing Serbian wikis on elastic@eqiad (T196404)
  • 16:39 urandom: DROP unused Cassandra keyspaces - T197080
  • 10:52 _joe_: removing wtp1043 from all pybal configuration until the disk is replaced T196886
  • 10:04 _joe_: initialize namespace "ci" on the kubernetes staging cluster T196654
  • 09:56 joal@deploy1001: Finished deploy [analytics/refinery@e9dbe79]: Regular weekly deploy (duration: 06m 54s)
  • 09:49 joal@deploy1001: Started deploy [analytics/refinery@e9dbe79]: Regular weekly deploy
  • 03:04 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 14m 30s)
  • 02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 13m 12s)
  • 02:27 Trey314159: reindexing Serbian wikis on elastic@codfw (T196404)

2018-06-17

  • 22:24 no_justification: gerrit: started back up, nvm
  • 22:23 no_justification: gerrit: stopping services for a few minutes
  • 13:05 Trey314159: reindexing Serbo-Croatian wikis on elastic@eqiad (T196658)
  • 03:46 Trey314159: reindexing Serbo-Croatian wikis on elastic@codfw (T196658)

2018-06-16

  • 20:42 mdholloway: disabled puppet on maps-test2004 for testing new map styles setup
  • 19:24 Trey314159: reindexing Croatian wikis on elastic@eqiad (T196658)
  • 13:19 Trey314159: reindexing Croatian wikis on elastic@codfw (T196658)
  • 00:15 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ica4cb6644 (duration: 00m 59s)

2018-06-15

  • 18:48 Trey314159: reindexing Bosnian wikis on elastic@eqiad (T196658)
  • 18:10 Trey314159: reindexing Bosnian wikis on elastic@codfw (T196658)
  • 17:15 krinkle@deploy1001: Synchronized wmf-config/mc.php: I619a2ff5db611 (duration: 00m 58s)
  • 15:49 mutante: install2002 - disabling puppet temp, live hackking DHCP config for debugging backup2001 install issue
  • 15:37 mutante: switching noc.wikimedia.org site from terbium to mwamiant1001 backend, running puppet on all cache::misc cp servers (T192092)
  • 15:19 ottomata: bouncing kafka broker on kafka-jumbo1001 to test https://gerrit.wikimedia.org/r/#/c/440520/
  • 15:19 gehel: rolling restart of elasticsearch eqiad for plugin upgrade completed - T194245
  • 14:49 elukey: restart varnishkafka-eventlogging on cp4028, errors logged
  • 14:43 elukey: restart varnishkafka-eventlogging on cp5012 as attempt to clear out the errors (not needed but logging it anyway)
  • 14:12 papaul: OS install on backup2001
  • 13:41 mepps: updated process control to 3b9f51e0fd
  • 12:56 legoktm@deploy1001: Synchronized wmf-config/mc.php: Make sure that mcrouter BagOStuff goes through ObjectCache::newFromParams() - T197450 (duration: 00m 57s)
  • 12:38 jynus: killing refresh counts on commonswiki
  • 12:30 jynus: reenabling db2040 consistency after slaves caught up
  • 12:13 papaul: OS install on bast2002
  • 10:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 00m 58s)
  • 09:48 godog: delete cpjobqueue metrics older than 10d - T196067
  • 09:45 jynus: reenabling db2048 consistency after slaves caught up
  • 09:43 jynus: reducing temp. db2040 consistency to speed up slave lag catch up
  • 09:20 godog: fully remove ms-be1036 from swift due to hw failure - T196873
  • 09:03 bawolff: deploy patch T197279
  • 08:05 gehel: rolling restart of elasticsearch eqiad for plugin upgrade - T194245
  • 05:50 moritzm: slow rollout of debmonitor
  • 05:34 moritzm: installing gnupg security updates on trusty (Debian already fixed)
  • 05:13 marostegui: Deploy schema change on db1119 T191316 T192926 T89737 T195193
  • 05:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 00m 57s)
  • 05:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 01m 07s)

2018-06-14

  • 23:40 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.3 (duration: 04m 38s)
  • 23:31 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.2 (duration: 04m 50s)
  • 23:24 jforrester@deploy1001: Synchronized php-1.32.0-wmf.8/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.less: T195903 SWAT for Roan (duration: 00m 59s)
  • 23:12 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T196400 for SWAT (duration: 01m 00s)
  • 21:56 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/PageTriage/includes/Hooks.php: Fix event presentation class names T197262 (duration: 00m 57s)
  • 21:47 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.999 T196585
  • 21:44 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/PageTriage/includes/Hooks.php: Fix event presentation class names T197262 (duration: 00m 52s)
  • 21:27 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.1 (duration: 06m 54s)
  • 21:18 thcipriani@deploy1001: Finished scap: testwiki to 1.32.0-wmf.999 (multi-content-revisions T196585) and rebuild l10n cache (duration: 65m 50s)
  • 21:12 bblack: re-enable and run puppet on cache_upload - T192555
  • 20:45 bblack: re-enable and run puppet on rest of cache_text (eqiad, eqsin, esams) - T192555
  • 20:21 bblack: re-enable and run puppet on text@ulsfo - T192555
  • 20:12 thcipriani@deploy1001: Started scap: testwiki to 1.32.0-wmf.999 (multi-content-revisions T196585) and rebuild l10n cache
  • 20:11 bblack: re-enable and run puppet on text@codfw - T192555
  • 19:21 vgutierrez: Reenable puppet in cache:misc nodes - T192555
  • 19:13 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.8
  • 18:17 niharika29@deploy1001: Synchronized wmf-config/CommonSettings.php: Bump ExtensionDistributor default to REL1_31 (duration: 00m 57s)
  • 18:14 niharika29@deploy1001: Synchronized langlist-labs: beta: declare beta sr.wikipedia and beta crh.wikipedia to langlist-labs (duration: 00m 58s)
  • 18:10 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on a few more wikis - T195263 (duration: 01m 00s)
  • 17:34 gehel: rolling restart of elasticsearch codfw completed - T194245
  • 17:10 ottomata: bouncing kafka broker on kafka2003 to make sure ACLs are okk
  • 17:08 ottomata: applying ACLs to Kafka main-codfw and main-eqiad - T196081
  • 16:41 vgutierrez: disable puppet on cache nodes before merging gerrit/440114 - T192555
  • 15:39 ottomata: switching EventStreams service to be backed by main-eqiad - T185225
  • 14:48 ejegg: disabled donations and refund queue consumers
  • 14:45 ejegg: updated CiviCRM from 69091d8 to f54f981
  • 14:40 mutante: moving wikidata query dispatcher from terbium to mwmaint1001 - scheduled downtime - check turned into a WARN - disabling puppet on mwmaint1001, removing crons on terbium, waiting a couple minutes for them to finish, re-enabling puppet on mwmaint1001 (T192092)
  • 14:32 moritzm: (slow) initial rollout of debmonitor-client
  • 13:47 zeljkof: EU SWAT finished
  • 13:46 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/Wikibase: SWAT: Revert "Statement transclusion: when entity of unknown type in statement, display ID as string" (T195615) (duration: 01m 21s)
  • 13:35 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set a few of namespace aliases on ruwikisource (T196719) (duration: 00m 59s)
  • 13:11 volans@deploy1001: Finished deploy [debmonitor/deploy@476fd8b]: Release v0.1.4 (duration: 00m 19s)
  • 13:10 volans@deploy1001: Started deploy [debmonitor/deploy@476fd8b]: Release v0.1.4
  • 12:54 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 12:29 ema: cp3037: run update-ocsp-all
  • 12:23 gehel: rolling restart of elasticsearch codfw for plugin upgrade - T194245
  • 11:38 marostegui: Deploy schema change on db1067 T191316 T192926 T89737 T195193
  • 11:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 00m 57s)
  • 11:16 elukey: upgrade cassandra on aqs* to 2.2.6-wmf5
  • 11:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 01m 01s)
  • 10:04 addshore: @terbium: for i in {2001..3000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki // T197222
  • 09:59 addshore: @terbium: for i in {1001..2000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki // T197222
  • 09:57 addshore: @terbium: for i in {1..1000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki // T197222
  • 09:15 elukey: add debmonitor term to analytics-in4 on cr1/cr2 eqiad
  • 08:59 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@2680ba8]: Decrease the checkerJob delays recheck to 10 minutes (duration: 01m 18s)
  • 08:58 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@2680ba8]: Decrease the checkerJob delays recheck to 10 minutes
  • 08:31 elukey: restart hadoop hdfs master nodes to pick up the new journal node settings
  • 08:29 gehel: restart of elasticsearch / relforge for plugin updates
  • 08:08 mutante: switch backend for dbtree.wikimedia.org away from terbium to mwmaint1001 (T192092)
  • 08:07 elukey: roll restart of hadoop journal nodes to pick up the new configuration (two more journal nodes added)
  • 08:03 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETAONLY: senses for beta wikidatas (duration: 00m 58s)
  • 07:59 moritzm: resuming rolling restart of cassandra on restbase2* to pick up OpenJDK security update
  • 07:41 marostegui: Deploy schema change on db1114 T191316 T192926 T89737 T195193
  • 07:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 00m 58s)
  • 07:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 after alter table (duration: 00m 58s)
  • 05:47 mutante: LDAP - added user mepps to wmf group (T192472)
  • 05:15 marostegui: Deploy schema change on db1083 T191316 T192926 T89737 T195193
  • 05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 for alter table (duration: 00m 58s)
  • 05:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 01m 01s)
  • 05:03 marostegui: Deploy schema change on s4 primary master (db1068) T191316 T192926 T195193
  • 03:06 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jun 14 03:06:19 UTC 2018 (duration 10m 30s)
  • 02:55 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 14m 42s)
  • 02:22 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 09m 41s)

2018-06-13

  • 23:15 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/432093/ (duration: 00m 58s)
  • 23:07 maxsem@deploy1001: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440178/ (duration: 01m 00s)
  • 21:20 awight@deploy1001: Finished deploy [ores/deploy@36037b6]: New badwords for ORES in English: T196468 (duration: 72m 56s)
  • 20:07 awight@deploy1001: Started deploy [ores/deploy@36037b6]: New badwords for ORES in English: T196468
  • 19:26 dduvall@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.8 (duration: 00m 57s)
  • 19:25 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.8
  • 18:28 twentyafterfour: finished swat (28m overtime! :P)
  • 18:22 twentyafterfour: ran namespaceDupes for urwiktionary
  • 18:20 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: sync 5b47244 14ca2ba 63bc100m and 2569a77 refs T196488, T196744, T196727, T196763 (duration: 00m 57s)
  • 17:55 twentyafterfour@deploy1001: Synchronized static/images/project-logos/: sync bnwikivoyage logos refs T196803 (duration: 00m 58s)
  • 17:53 twentyafterfour@deploy1001: Synchronized static/images/project-logos/bnwikivoyage-1.5x.png: static/images/project-logos/bnwikivoyage-2x.png static/images/project-logos/bnwikivoyage.png sync bnwikivoyage logos refs T196803 (duration: 00m 58s)
  • 17:49 twentyafterfour@deploy1001: Synchronized wmf-config: Sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/436430/ for SWAT refs T184244 (duration: 01m 00s)
  • 17:08 mutante: terbium - closing unusued screen sessions for all Amir users (2)
  • 16:53 ppchelko@deploy1001: Finished deploy [restbase/deploy@f521e7e]: Add page_language to title_revision table T197082 (duration: 15m 44s)
  • 16:38 ppchelko@deploy1001: Started deploy [restbase/deploy@f521e7e]: Add page_language to title_revision table T197082
  • 16:31 XioNoX: moving mr1-eqiad interfaces to new router
  • 16:14 mutante: rsyncing /home dirs from terbium to mwmaint1001, they will appear later in a subdir "home-terbium" like it was done for tin->deploy1001 (T192092)
  • 16:13 urandom: ALTERing Cassandra schema - T197082
  • 16:11 moritzm: installing plexus-archiver security updates
  • 16:07 moritzm: installing imagemagick security updates on trusty (Debian already fixed)
  • 15:55 elukey: rolling restart of aqs on aqs100[4-9] to pick up the new config changes
  • 15:25 jynus: stopping db1053 and db1059 in preparation for decomm T194634 T196606
  • 15:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1076 (duration: 00m 58s)
  • 14:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1076 (duration: 00m 57s)
  • 14:15 anomie@deploy1001: Synchronized php-1.32.0-wmf.8/includes/Category.php: Backporting fix for T195397 (gerrit:440053) (duration: 01m 00s)
  • 14:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight (duration: 00m 58s)
  • 14:01 zeljkof: EU SWAT finished
  • 14:00 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Implementing Patroller User Rights for azwiki (T196488) (duration: 00m 57s)
  • 13:54 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: English aliases for extra namespaces on urwiktionary (T196614) (duration: 00m 58s)
  • 13:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix wrong language in ur.wiktionary namespace (T196614) (duration: 00m 58s)
  • 13:28 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@160206f]: (no justification provided) (duration: 04m 11s)
  • 13:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make ProofreadPage operate on correct namespaces in pmswikisource (T197033) (duration: 00m 57s)
  • 13:24 elukey@deploy1001: Started deploy [analytics/aqs/deploy@160206f]: (no justification provided)
  • 13:12 zfilipin@deploy1001: Synchronized wmf-config/: SWAT: Enable FileImporter monolog channel in production (T195370) (duration: 01m 00s)
  • 13:02 moritzm: rolling restart of cassandra in codfw to pick up OpenJDK security update
  • 13:00 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@84fab89]: Update AQS for T190213 (duration: 02m 38s)
  • 12:57 elukey@deploy1001: Started deploy [analytics/aqs/deploy@84fab89]: Update AQS for T190213
  • 12:46 elukey: restart mirror maker on kafka1012->1014 to pick up new openjdk-7 upgrades
  • 12:28 elukey: rolling restart of kafka on kafka1012->23 for openjdk-7 upgrades
  • 12:16 arturo: T196633 extend downtime for labcontrol1003
  • 11:21 moritzm: installing perl security updates
  • 10:59 marostegui: Deploy schema change on db1105:3311 T191316 T192926 T89737 T195193
  • 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 00m 58s)
  • 10:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 00m 59s)
  • 10:10 akosiaris: upload apertium-apy_0.11.3-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:00 marostegui: Restart mysql on dbstore1002 for maintenance
  • 08:46 ema: depool cp1053 T165252
  • 08:04 marostegui: Stop MySQL and reboot db1076 - T197063
  • 08:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 for binlog change - T197063 (duration: 00m 57s)
  • 07:59 oblivian@deploy1001: Synchronized wmf-config/ProductionServices.php: Remove unused redis shards from the jobqueue T197003 (duration: 00m 58s)
  • 07:57 Pchelolo: restart cpjobqueue on scb1001 cause, it's lost all it's workers
  • 07:55 marostegui: Deploy schema change on db1089 T191316 T192926 T89737 T195193
  • 07:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 00m 58s)
  • 07:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore s2 default read-only message (duration: 00m 57s)
  • 07:41 marostegui: Stop MySQL on db1054 for socket update and binlog change
  • 07:33 godog: start removing ms-be1036 from swift rings - T196873
  • 06:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 00m 59s)
  • 06:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove read only from s2 - T194870 (duration: 00m 33s)
  • 06:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove read only from s2 - T194870 (duration: 00m 34s)
  • 06:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Set s2 on read-only for primary db master maintnance - T194870 (duration: 01m 08s)
  • 06:00 marostegui: Starting s2 failover from db1054 to db1066 - T194870
  • 05:11 marostegui: Starting topology changes in order to get ready for s2 failover - T194870
  • 05:09 marostegui: Disable gtid on db1066
  • 05:03 marostegui: Deploy schema change on dbstore1001:s1 T191316 T192926 T89737 T195193
  • 03:14 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jun 13 03:14:53 UTC 2018 (duration 10m 19s)
  • 03:04 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 15m 45s)
  • 02:31 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 12m 21s)
  • 01:12 aaron@deploy1001: Synchronized wmf-config/mc.php: Add "memcached-mcrouter" to $wgObjectCaches as default for testwiki (duration: 00m 58s)
  • 01:01 aaron@deploy1001: Synchronized php-1.32.0-wmf.7/tests/phpunit/includes/db/LBFactoryTest.php: (no justification provided) (duration: 00m 57s)
  • 01:00 aaron@deploy1001: Synchronized php-1.32.0-wmf.7/includes/libs/rdbms/lbfactory/LBFactory.php: f83bad6 (duration: 00m 59s)
  • 00:55 aaron@deploy1001: Synchronized php-1.32.0-wmf.8/tests/phpunit/includes/db/LBFactoryTest.php: (no justification provided) (duration: 00m 58s)
  • 00:53 aaron@deploy1001: Synchronized php-1.32.0-wmf.8/includes/libs/rdbms/lbfactory/LBFactory.php: c2df9668d13 (duration: 00m 58s)
  • 00:46 bawolff@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/CentralAuth/AntiSpoof/CentralAuthAntiSpoofHooks.php: Deploy I5f25c5 (prevent users from registering previously renamed users) (duration: 00m 57s)
  • 00:44 bawolff@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/CentralAuth/extension.json: Deploy I5f25c5 (prevent users from registering previously renamed users) (duration: 00m 58s)
  • 00:40 bawolff@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/CentralAuth/AntiSpoof/CentralAuthAntiSpoofHooks.php: Deploy I5f25c5 (prevent users from registering previously renamed users) (duration: 00m 57s)
  • 00:39 bawolff@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/CentralAuth/extension.json: Deploy I5f25c5 (prevent users from registering previously renamed users) (duration: 00m 57s)
  • 00:37 bawolff@deploy1001: Synchronized wmf-config/CommonSettings.php: Deploy I5f25c5 (prevent users from registering previously renamed users) (duration: 00m 59s)

2018-06-12

  • 23:47 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add sites to the wgCopyUploadsDomains whitelist of Wikimedia Commons T195270, T195928 (duration: 00m 59s)
  • 23:05 tzatziki: resetting passwords for compromised accounts (T197046)
  • 23:00 bblack: cp3043 - done, reimaged, in live service for cache_upload
  • 22:59 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp3043.esams.wmnet
  • 22:37 tzatziki: (from yesterday) resetting passwords for compromised accounts (T197046)
  • 22:24 twentyafterfour: phabricator: I scheduled a 24 hour downtime in icinga for the phd service, to give me time to work on this issue. See T196840
  • 22:23 twentyafterfour: phabricator: taking phd offline to relieve the load on the m3 database cluster
  • 21:46 bblack: cp3046 - restart varnish backend for mbox lag
  • 21:40 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp3043.esams.wmnet
  • 21:39 bblack: cp3043 - starting process to move to reimage into cache_upload
  • 19:16 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.8
  • 18:57 herron: restarted icinga service on einsteinium
  • 18:42 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.8 and rebuild l10n cache (duration: 39m 39s)
  • 18:03 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.8 and rebuild l10n cache
  • 17:38 ariel@deploy1001: Finished deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install (duration: 00m 04s)
  • 17:37 ariel@deploy1001: Started deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install
  • 17:37 ariel@deploy1001: Finished deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install (duration: 00m 07s)
  • 17:37 ariel@deploy1001: Started deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install
  • 16:54 marxarelli: starting branch cut for 1.32.0-wmf.8
  • 16:11 volans@deploy1001: Finished deploy [debmonitor/deploy@0eca14a]: Release v0.1.3 (duration: 00m 22s)
  • 16:11 volans@deploy1001: Started deploy [debmonitor/deploy@0eca14a]: Release v0.1.3
  • 15:40 bblack: cp3034 - nevermind, doing different approach later in the day, still pooled in text for now!
  • 15:29 bblack: cp3043 switching from text to upload shortly, downtimed in icinga for 2h - https://gerrit.wikimedia.org/r/c/operations/puppet/+/439936
  • 15:07 ema: cp3039: restart varnish-backend
  • 14:38 addshore: file exporter importer slot done
  • 14:38 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: FileImporter/Exporter Enable FileExporter/Importer on group0 wikis T195370 (duration: 00m 51s)
  • 14:20 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: FileImporter/Exporter Allow setting of export target for FileExporter T195370 (duration: 00m 50s)
  • 14:09 addshore@deploy1001: Finished scap: FileExporter backport - Pre deployment backport (extension not yet deployed) (duration: 30m 37s)
  • 13:38 addshore@deploy1001: Started scap: FileExporter backport - Pre deployment backport (extension not yet deployed)
  • 13:16 moritzm: installing openjdk-8 security updates on restbase-dev along with cassandra restarts
  • 12:38 ema: cp3035: restart varnish-be, mbox lag
  • 12:34 _joe_: repooling mw1230 after reimaging T196881
  • 12:14 marostegui: Deploy schema change on db1099:3311 T191316 T192926 T89737 T195193
  • 12:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 00m 52s)
  • 12:11 marostegui: Deploy schema change on dbstore1002:s1 T191316 T192926 T89737 T195193
  • 12:05 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: dc=.*,service=mathoid,cluster=kubernetes,name=.*
  • 11:47 moritzm: updated component/cassandra311 on apt.wikimedia.org to 3.11.2
  • 10:26 jynus: setting expire_log_days on db1066 as 30
  • 10:21 godog: bounce stuck rsyslog on lithium / wezen - T136312
  • 09:41 vgutierrez: cp3037 has been depooled due to unknown hardware issues T196974
  • 08:48 marostegui: Stop replication on db2094 to change triggers for archive table
  • 08:36 volans: running puppet on failed hosts post small puppet outage and puppetdb reboot
  • 08:35 akosiaris: rebalance ganeti codfw cluster
  • 08:35 ema@neodymium: conftool action : set/pooled=no; selector: name=cp3037.esams.wmnet
  • 08:33 akosiaris: reboot puppetdb1001 for spec-ctrl enable. Bundling it with a minor puppet outage to only have a torrent of harmless puppet failures once
  • 08:15 akosiaris: ganeti2002 reboot for microcode update
  • 08:04 akosiaris: ganeti2006 reboot for microcode update
  • 08:03 marostegui: Deploy schema change on s1 codfw primary master (db2048) with replication, this will generate lag on codfw T191316 T192926 T89737 T195193
  • 07:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 after alter table (duration: 00m 50s)
  • 07:43 akosiaris: ganeti2007 reboot for microcode update
  • 07:41 akosiaris: ganeti2003 reboot for microcode update
  • 07:31 mutante: closing idle screen session on tin (about to be decomed, dont use anymore)
  • 06:37 marostegui: Deploy schema change on db1121 with replication, this will generate lag on labsdb:s4 T191316 T192926 T89737 T195193
  • 06:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 for alter table (duration: 00m 50s)
  • 06:31 marostegui: Stop replication on db1095, db1102, db1125 to change triggers - T192926
  • 06:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 after alter table (duration: 00m 51s)
  • 05:09 marostegui: Deploy schema change on db1091 T191316 T192926 T89737 T195193
  • 05:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 for alter table (duration: 00m 52s)
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jun 12 02:45:53 UTC 2018 (duration 10m 18s)
  • 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 14m 10s)
  • 00:35 legoktm: remove non-deployers from wmf-deployment Gerrit group (T196959)
  • 00:16 ejegg: updated CiviCRM from 69091d8b5f to f54f9810ab

2018-06-11

  • 23:45 ebernhardson@deploy1001: Synchronized wmf-config/CirrusSearch-common.php: SWAT: Lower CirrusSearch delayed job drop timeout (duration: 00m 50s)
  • 23:26 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Tune CirrusSearch slow logging (duration: 00m 48s)
  • 23:18 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Promote Cirrus MLR models from AB test to prod (duration: 00m 51s)
  • 23:02 twentyafterfour: phabricator: restarting apache2 on phab1001 to free up apache workers
  • 22:17 awight@deploy1001: Finished deploy [ores/deploy@6ee8775]: ORES: bswiki, euwiki, srwiki models (duration: 33m 58s)
  • 21:43 awight@deploy1001: Started deploy [ores/deploy@6ee8775]: ORES: bswiki, euwiki, srwiki models
  • 20:57 arlolra: Updated Parsoid to 06b74d2 (T191843)
  • 20:49 arlolra@deploy1001: Finished deploy [parsoid/deploy@97cdab8]: Updating Parsoid to 06b74d2 (duration: 17m 09s)
  • 20:32 arlolra@deploy1001: Started deploy [parsoid/deploy@97cdab8]: Updating Parsoid to 06b74d2
  • 20:27 otto@deploy1001: Finished deploy [eventstreams/deploy@6b013f9]: Enable composite stream and timestamp since param - T196009 , T187418 (duration: 09m 52s)
  • 20:17 otto@deploy1001: Started deploy [eventstreams/deploy@6b013f9]: Enable composite stream and timestamp since param - T196009 , T187418
  • 20:02 ottomata: bouncing varnishkafka-webrequest on cp3039,cp3047,cp2007,cp3010
  • 19:56 ottomata: bouncing varnishkafka on cp3032
  • 19:29 aaron@deploy1001: Synchronized php-1.32.0-wmf.7/includes/libs/rdbms/ChronologyProtector.php: 11e596776f940 - add some logging details (duration: 00m 53s)
  • 18:50 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Restore full config (duration: 00m 16s)
  • 18:49 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Restore full config
  • 18:15 urandom: convert timeline indices to time-windowed compaction - T196024
  • 17:59 twentyafterfour: phabricator: taking phd offline while I clear out the queue backlog (downtime is logged in icinga) see T196840
  • 17:51 twentyafterfour: phabricator: rebuilding git parent caches
  • 17:33 gehel@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (duration: 13m 38s)
  • 17:27 twentyafterfour: phabricator: restarting phd for D1067
  • 17:25 twentyafterfour: Phabricator: deploying hotfix (D1067) refs T196840 T196860 T196855
  • 17:20 gehel@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater
  • 17:19 gehel@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (duration: 00m 03s)
  • 17:19 gehel@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater
  • 17:17 gehel@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 26s)
  • 17:16 gehel@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (wdqs1009 only)
  • 17:00 akosiaris: ganeti2008 reboot for microcode update
  • 16:42 marostegui: Set disk 32:1 offline on db1065 to get a new one - T196806
  • 16:02 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 01m 32s)
  • 16:00 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
  • 15:44 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 00m 33s)
  • 15:43 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
  • 15:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 after alter table (duration: 00m 50s)
  • 15:33 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 00m 22s)
  • 15:33 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
  • 15:31 marostegui: Set offline disk 32:3 on db1063 - T196806
  • 15:31 marostegui: Set offline disk 32:1 on db1065 - T196806
  • 15:02 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 35m 33s)
  • 14:55 marostegui: Deploy schema change on db1084 T191316 T192926 T89737 T195193
  • 14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 for alter table (duration: 00m 50s)
  • 14:53 akosiaris: reboot bohrium for kernel upgrades and spec-ctrl enabling. Manually stopped mysql behorehand
  • 14:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 after alter table (duration: 00m 50s)
  • 14:35 akosiaris: reboot mx1001, poolcounter1001 for kernel upgrades and spec-ctrl enabling
  • 14:26 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
  • 14:25 godog: upload scap 3.8.2-1 - T196710
  • 14:21 hoo: Updated operations/dumps/dcat (536bd5b..559dee3) on snapshot1008
  • 13:58 marostegui: Deploy schema change on db1081 T191316 T192926 T89737 T195193
  • 13:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 for alter table (duration: 00m 48s)
  • 13:56 otto@deploy1001: Finished deploy [eventlogging/analytics@08a1dff]: Producing events with kafka timestamp set to event time - T196407 (duration: 00m 04s)
  • 13:56 otto@deploy1001: Started deploy [eventlogging/analytics@08a1dff]: Producing events with kafka timestamp set to event time - T196407
  • 13:54 otto@deploy1001: Finished deploy [eventlogging/eventbus@08a1dff]: Producing events with kafka timestamp set to event time - T196407 (duration: 01m 55s)
  • 13:52 otto@deploy1001: Started deploy [eventlogging/eventbus@08a1dff]: Producing events with kafka timestamp set to event time - T196407
  • 13:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 00m 50s)
  • 13:47 hashar: European SWAT completed
  • 13:35 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix wgMetaNamespace for pswikivoyage - T196837 (duration: 00m 49s)
  • 13:29 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix wgMetaNamespace for pswikivoyage - T196837 (duration: 00m 50s)
  • 13:21 volans@deploy1001: Finished deploy [debmonitor/deploy@81d7333]: Release v0.1.2 (duration: 00m 56s)
  • 13:20 volans@deploy1001: Started deploy [debmonitor/deploy@81d7333]: Release v0.1.2
  • 13:20 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use uploaded HD logo for bewikiquote - T196134 (duration: 00m 50s)
  • 13:17 hashar@deploy1001: Synchronized static/images/project-logos: Change logo files for bewikiquote - T196134 (duration: 00m 50s)
  • 13:13 hashar@deploy1001: Synchronized static/images/project-logos: Revert "Change bewikiquote logo" - T196134 (duration: 00m 51s)
  • 13:01 volans@deploy1001: Finished deploy [debmonitor/deploy@81d7333]: Release v0.1.2 (duration: 07m 16s)
  • 12:54 volans@deploy1001: Started deploy [debmonitor/deploy@81d7333]: Release v0.1.2
  • 11:45 arturo: T196633 downtime labcontrol100[3,4] due to unexpected puppet errors on installation of keystone
  • 11:38 arturo: T196633 deploy keystone to labcontrol100[3,4].wikimedia.org. Dormant daemon, no DB yet
  • 11:30 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@b5396cd]: Tune cirrus jobs concurrencies (duration: 00m 42s)
  • 11:30 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@b5396cd]: Tune cirrus jobs concurrencies
  • 11:23 marostegui: Deploy schema change on db1103:3314 T191316 T192926 T89737 T195193
  • 11:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 50s)
  • 11:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 00m 51s)
  • 10:52 mutante: phab1002 - editing cached scap config /srv/deployment/phabricator/deployment-cache/.config to replace tin.eqiad with deploy1001.eqiad deployment server, run puppet. other options: run scap with --refresh-config, delet cached .config file (T196019) (T175288)
  • 10:29 _joe_: depooling permantently mw1230 for disk replacement, T196881
  • 10:11 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 10:10 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
  • 09:07 marostegui: Deploy schema change on db1097:3314 T191316 T192926 T89737 T195193
  • 09:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 for alter table (duration: 00m 52s)
  • 08:32 moritzm: installing gnupg2 security updates
  • 08:27 gehel: restart elastic1020 to enable G1 GC - T156137
  • 08:25 moritzm: installing gnupg1 security updates
  • 07:52 moritzm: installing gnupg security updates
  • 07:37 marostegui: Deploy schema change on dbstore1002:s4 T191316 T192926 T89737 T195193
  • 07:29 moritzm: installing openjdk-7 security updates
  • 06:39 elukey: restart pdfrender on scb1002
  • 06:14 marostegui: Deploy schema change on s4 codfw master (db2051) this will generate lag on codfw - T191316 T192926 T89737 T195193
  • 06:10 marostegui: Stop replication on db2095 to update triggers - T192926
  • 05:32 marostegui: Restart mysql on codfw sanitariums (db1095, db1102, db1124, db1125) to pick up new replication filters - T196748
  • 05:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool pc2005 - T196339 (duration: 00m 52s)
  • 05:25 marostegui: Restart mysql on codfw sanitariums (db2094, db2095) to pick up new replication filters - T196748
  • 05:17 marostegui: Stop MySQL and reboot pc2005 for intel-microcode update and final HW check - T196339
  • 05:15 marostegui: Deploy schema change on s6 primary master (db1061) - T191316 T192926 T195193
  • 02:42 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jun 11 02:42:42 UTC 2018 (duration 10m 16s)
  • 02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 13m 06s)

2018-06-10

  • 12:30 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: CSP in report mode for group0 (duration: 00m 55s)
  • 12:01 bawolff: disable some botpasswords (T194204)
  • 11:08 _joe_: restarting gerrit on cobalt as the web interface is unresponsive

2018-06-08

  • 22:41 no_justification: gerrit: restarting
  • 19:36 urandom: upgrade Cassandra to 3.11.2, restbase1015 & restbase1018 - T178905
  • 18:49 urandom: upgrade Cassandra to 3.11.2, restbase1009 & restbase1014 - T178905
  • 18:02 urandom: upgrade Cassandra to 3.11.2, restbase1017 & restbase1013 - T178905
  • 17:33 no_justification: gerrit: up mostly, but will see some errors about "wip" label for a bit until reindexing completes.
  • 17:28 cmjohnson: powering flerovium down to move to a different space in the rack
  • 17:14 demon@deploy1001: Finished deploy [gerrit/gerrit@7324140]: 2.15.2 (duration: 00m 11s)
  • 17:14 demon@deploy1001: Started deploy [gerrit/gerrit@7324140]: 2.15.2
  • 17:12 no_justification: gerrit: taking offline for 2.14 -> 2.15 upgrade
  • 15:35 urandom: upgrade Cassandra to 3.11.2, restbase1012 - T178905
  • 15:26 marostegui: Restart MySQL on codfw sanitarium hosts db2094, db2095 - https://phabricator.wikimedia.org/T196748
  • 15:00 urandom: upgrade Cassandra to 3.11.2, restbase1008 - T178905
  • 14:17 urandom: upgrade Cassandra to 3.11.2, restbase1011 & restbase1016 - T178905
  • 12:14 twentyafterfour: running phabricator public_task_dump script manually to confirm that it's working as expected.
  • 09:31 arturo: merging https://gerrit.wikimedia.org/r/#/c/436337/ for the eqia1 openstack deployment (labcontrol1003/labcontrol1004)
  • 09:19 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Unify and update sanitarium comments - T190704 (duration: 00m 50s)
  • 09:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Unify and update sanitarium comments - T190704 (duration: 00m 50s)
  • 09:06 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
  • 08:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1066 after reboot (duration: 00m 50s)
  • 08:18 marostegui: Stop MySQL and reboot db1066 for intel-microcode install - T194870
  • 08:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1066 for reboot (duration: 00m 50s)
  • 07:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 after alter table (duration: 00m 51s)
  • 07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight (duration: 00m 50s)
  • 07:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1091 (duration: 00m 50s)
  • 07:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increse weight for db1091 (duration: 00m 50s)
  • 06:47 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T196630 Remove unneeded description from performance survey definition (duration: 00m 52s)
  • 06:44 elukey: bounce kafka mirror maker main-eqiad-to-main-codfw (kafka200*) due to errors in the logs (also lag metrics not displaying)
  • 06:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight after reimage (duration: 00m 50s)
  • 05:34 marostegui: Deploy sanitarium events on db1124 - T190704
  • 05:23 marostegui: Stop MySQL on db1091 for reimage
  • 05:22 marostegui: Deploy schema change on db1113:3316 - T191316 T192926 T195193 T89737
  • 05:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3316 for alter table (duration: 00m 52s)
  • 02:12 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c (duration: 00m 42s)
  • 02:11 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c

2018-06-07

  • 23:58 demon@deploy1001: Finished deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state (x2) (duration: 00m 14s)
  • 23:58 demon@deploy1001: Started deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state (x2)
  • 23:57 demon@deploy1001: Finished deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state (duration: 00m 06s)
  • 23:57 demon@deploy1001: Started deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state
  • 21:49 mobrovac@deploy1001: Finished deploy [proton/deploy@97ec4bf]: Initial deploy to production - T186748 (duration: 02m 19s)
  • 21:47 mobrovac@deploy1001: Started deploy [proton/deploy@97ec4bf]: Initial deploy to production - T186748
  • 20:37 herron: ircecho restarted
  • 19:47 urandom: rolling Cassandra restart, restbase2005, restbase2006, restbase2012 -- T178905
  • 19:36 XenoRyet: updated payments-wiki config from 3197c729ee to b11d120362
  • 19:29 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.7
  • 19:16 herron: stopped ircecho
  • 18:28 urandom: rolling Cassandra restart, restbase2004, restbase2008, restbase2011 -- T178905
  • 18:20 urandom: restarting Cassandra, restbase2003-c -- T178905
  • 18:13 subbu: Updated Parsoid (T183706, T192726, T194879, T196357, T196360, T43716)
  • 18:10 ssastry@deploy1001: Finished deploy [parsoid/deploy@2f80639]: Updating Parsoid to 7819c9e7 (duration: 10m 59s)
  • 17:59 ssastry@deploy1001: Started deploy [parsoid/deploy@2f80639]: Updating Parsoid to 7819c9e7
  • 17:21 awight@deploy1001: Finished deploy [ores/deploy@65ce165]: New home page for ORES; T196580 (take 2) (duration: 04m 34s)
  • 17:16 awight@deploy1001: Started deploy [ores/deploy@65ce165]: New home page for ORES; T196580 (take 2)
  • 17:16 awight@deploy1001: Finished deploy [ores/deploy@65ce165]: New home page for ORES; T196580 (duration: 03m 19s)
  • 17:16 reedy@deploy1001: Synchronized wmf-config/interwiki-labs.php: labs! (duration: 00m 57s)
  • 17:13 awight@deploy1001: Started deploy [ores/deploy@65ce165]: New home page for ORES; T196580
  • 16:45 urandom: rolling Cassandra restart, restbase2001, restbase2002, restbase2007 - T178905
  • 16:21 urandom: rolling Cassandra restart, restbase1007 - T178905
  • 15:59 hoo: Finished running "foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https" T196360 T183706 T195014 T196357
  • {{safesubst:SAL entry|1=15:38 urandom: upgrade Cassandra to 3.11.2, restbase1010-{a,b,c} - T178905}}
  • 15:30 hoo: Running "foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https" T196360 T183706 T195014 T196357
  • 15:23 hoo: Emptied out the sites and site_identifiers tables on pswikivoyage, pmswikisource, bnwikivoyage and sahwikiquote for T122520,
  • 15:20 hoo@deploy1001: Synchronized dblists/wikidataclient.dblist: Enable WikidataClient on sahwikiquote and pswikivoyage - T183706, T196360 (duration: 00m 57s)
  • 15:08 marostegui: Sanitize wikis on db1124 (current sanitarium for s3) - T196362 T196358 T196359 T195008 T193187
  • 15:04 reedy@deploy1001: Synchronized wmf-config/interwiki.php: (no justification provided) (duration: 00m 56s)
  • 14:56 ottomata: beginning rolling restarts of all cluster kafka brokers to apply log.message.timestamp.type=CreateTime - T196407
  • 14:53 reedy@deploy1001: rebuilt and synchronized wikiversions files: new wikis
  • 14:53 marostegui: Sanitize wikis on db1095 (old sanitarium) - T196362 T196358 T196359 T195008 T193187
  • 14:53 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: new wikis (duration: 00m 57s)
  • 14:50 reedy@deploy1001: Synchronized static/images/: new wikis (duration: 00m 57s)
  • 14:48 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: idwikimedia (duration: 00m 57s)
  • 14:47 reedy@deploy1001: Synchronized dblists/: 5 new wikis (duration: 00m 55s)
  • 14:45 Reedy: created new wiki databases T183706 T192726 T194879 T196357 T196360
  • 14:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 (duration: 00m 57s)
  • 14:11 reedy@deploy1001: Synchronized wmf-config/jobqueue-labs.php: labs (duration: 00m 56s)
  • 14:10 reedy@deploy1001: Synchronized wmf-config/LabsServices.php: labs (duration: 00m 57s)
  • 14:02 reedy@deploy1001: Synchronized docroot/noc/: Add some more symlinked configs (duration: 00m 57s)
  • 13:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1066 (duration: 00m 57s)
  • 13:31 reedy@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/WikimediaMaintenance/addWiki.php: add all the wikis (duration: 00m 56s)
  • 13:29 reedy@deploy1001: Synchronized php-1.32.0-wmf.6/extensions/WikimediaMaintenance/addWiki.php: add all the wikis (duration: 00m 58s)
  • 13:26 reedy@deploy1001: Synchronized dblists/all-labs.dblist: labs (duration: 00m 54s)
  • 13:24 reedy@deploy1001: Synchronized wikiversions-labs.json: labs (duration: 00m 57s)
  • 13:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1066 (duration: 00m 56s)
  • 13:17 zeljkof: EU SWAT finished
  • 13:16 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: When using the FileExporter set it as BeatFeature by default (T195370) (duration: 00m 56s)
  • 13:10 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add FileExporter to BetaFeaturesWhiteList (T195370) (duration: 00m 57s)
  • 12:59 marostegui: Deploy schema change on db1088 - T191316 T192926 T195193 T89737
  • 12:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 57s)
  • 12:54 jynus: stop, clone and reimage db1091
  • 12:52 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 (duration: 00m 57s)
  • 12:46 mutante: planet1001/puppetmaster: revoke old cert, sign new cert request, initial puppet run, reinstalled, will turn service active-active again once done (T168490)
  • 12:26 mutante: planet1001 - schedule downtime, boot to PXE, reinstall with stretch (ganeti) (T168490)
  • 12:15 moritzm: repooled mw1280 after hardware maintenance (T195734)
  • 11:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 57s)
  • 10:29 moritzm: installing openssl security updates
  • 08:50 marostegui: Deploy schema change on db1085 with replication, this will generate lag on labsdb for s6 section - T191316 T192926 T195193 T89737
  • 08:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 00m 56s)
  • 08:49 akosiaris: slow reboot of all ganeti eqiad VMs (except bohrium, puppetdb1001, poolcounter1001, mx1001) for kernel upgrades and picking up spec-ctrl cpu flag
  • 08:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 after alter table (duration: 00m 55s)
  • 08:34 marostegui: Stop replication on db1102:3316 and db1125:3316 to update triggers for archive table - T192926
  • 08:15 jynus: running ANALYZE on db2091 T196526
  • 07:46 marostegui: Deploy schema change on db1093 - T191316 T192926 T195193 T89737
  • 07:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 for alter table (duration: 00m 56s)
  • 07:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316 after alter table (duration: 00m 57s)
  • 07:09 moritzm: installing git security updates on trusty (Debian already fixed)
  • 06:55 jynus: relaxing write consistency on db2048 due to ongoing maintenance (sync_binlog,flush_log)
  • 06:26 marostegui: Deploy sanitarium events on db1125 - T190704
  • 06:24 jynus: phabricator maintenance finished
  • 06:10 jynus: start database maintenance on phabricator- brief interruptions could happen
  • 05:15 marostegui: Deploy event_sanitarium on codfw sanitariums - T190704
  • 05:12 marostegui: Deploy schema change on db1098:3316 - T191316 T192926 T195193 T89737
  • 05:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 for alter table (duration: 00m 56s)
  • 05:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 after alter table (duration: 00m 58s)
  • 04:01 eileen: civicrm revision changed from 1aaa798e9b to 69091d8b5f, config revision is c60375be8d
  • 03:05 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jun 7 03:05:05 UTC 2018 (duration 10m 19s)
  • 02:54 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 07m 02s)
  • 02:30 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 11m 24s)
  • 02:25 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@a07af40]: log
  • 02:00 paravoid: starting exim4 and reenabling puppet on mx1001, due to T196598
  • 01:12 ejegg: disabled fundraising mailing statistics fetch jobs
  • 01:00 ejegg: updated CiviCRM from 7a06a6a387 to 1aaa798e9b
  • 00:51 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@a07af40]: (no justification provided) (duration: 01m 02s)
  • 00:50 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@a07af40]: (no justification provided)
  • 00:33 ejegg: restarted fundraising jobs
  • 00:25 legoktm@deploy1001: Finished scap: Preference for responsive MonoBook, plus set mobile width cutoff to 550px (gerrit:437875, gerrit:437814) (duration: 64m 01s)

2018-06-06

  • {{safesubst:SAL entry|1=23:51 urandom: upgrade Cassandra to 3.11.2, restbase2012-{a,b,c} - T178905}}
  • 23:49 eileen: civicrm revision changed from ce960c0642 to 7a06a6a387, config revision is b7bc5dc31f
  • 23:43 eileen: civicrm revision changed from ac13914d03 to ce960c0642, config revision is b7bc5dc31f
  • {{safesubst:SAL entry|1=23:23 urandom: upgrade Cassandra to 3.11.2, restbase2009-{a,b,c} - T178905}}
  • 23:21 legoktm@deploy1001: Started scap: Preference for responsive MonoBook, plus set mobile width cutoff to 550px (gerrit:437875, gerrit:437814)
  • 23:19 eileen: civicrm revision changed from 0b97f1f5b2 to ac13914d03
  • 23:07 cwd: disabled process-control for civi update
  • 22:22 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c (duration: 05m 33s)
  • 22:16 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c
  • 20:47 ebernhardson: sighup logstash on logstash100[789] to reload config for gerrit.wikimedia.org/r/437657
  • {{safesubst:SAL entry|1=20:42 urandom: upgrade Cassandra to 3.11.2, restbase2005-{a,b,c} - T178905}}
  • 20:19 bearND: rolled back mobileapps deploy
  • 20:18 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@a07af40]: Update mobileapps to 3bf9be5 (T196402 T195948) (duration: 08m 37s)
  • {{safesubst:SAL entry|1=20:17 urandom: upgrade Cassandra to 3.11.2, restbase2011-{a,b,c} - T178905}}
  • 20:10 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@a07af40]: Update mobileapps to 3bf9be5 (T196402 T195948)
  • {{safesubst:SAL entry|1=19:38 urandom: upgrade Cassandra to 3.11.2, restbase2008-{a,b,c} - T178905}}
  • 19:19 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.7 (duration: 00m 56s)
  • 19:18 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.7
  • 18:55 herron: stopped exim on mx1001 in prep for upgrade to stretch
  • {{safesubst:SAL entry|1=18:54 urandom: upgrade Cassandra to 3.11.2, restbase2004-{a,b,c} - T178905}}
  • 18:38 andrewbogott: rebooting labvirt1019, 1020
  • 18:36 andrewbogott: rebooting labvirt1018, 1021, 1022
  • 18:29 andrewbogott: rebooting labvirt1017
  • 18:23 andrewbogott: rebooting labvirt1016
  • 18:21 andrewbogott: rebooting labvirt1015
  • {{safesubst:SAL entry|1=18:19 urandom: upgrade Cassandra to 3.11.2, restbase2003-{a,b,c} - T178905}}
  • 18:14 andrewbogott: rebooting labvirt1013
  • 18:06 andrewbogott: rebooting labvirt1012
  • 17:59 andrewbogott: rebooting labvirt1011
  • {{safesubst:SAL entry|1=17:59 urandom: upgrade Cassandra to 3.11.2, restbase2010-{a,b,c} - T178905}}
  • 17:44 andrewbogott: rebooting labvirt1010
  • {{safesubst:SAL entry|1=17:01 urandom: upgrade Cassandra to 3.11.2, restbase2007-{a,b,c} - T178905}}
  • 16:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 fully (duration: 00m 56s)
  • 16:51 andrewbogott: rebooting labvirt1008
  • 16:33 andrewbogott: rebooting labvirt1007
  • 16:25 jynus: stop mysql @ db1051 in preparation for decom
  • 16:25 andrewbogott: rebooting labvirt1006
  • 16:16 andrewbogott: rebooting labvirt1005
  • 16:09 andrewbogott: rebooting labvirt1004
  • 16:05 XioNoX: lvs1002 repooled
  • 16:04 andrewbogott: rebooting labvirt1002
  • 15:57 twentyafterfour: reloading apache on phab1001 to free up some resources
  • 15:56 andrewbogott: rebooting labvirt1014
  • {{safesubst:SAL entry|1=15:54 urandom: upgrade Cassandra to 3.11.2, restbase2001-{a,b,c} - T178905}}
  • 15:51 XioNoX: disable pybal on lvs1002 - T187962
  • 15:39 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@402d729]: Adjust the cirrus concurrencies (duration: 00m 40s)
  • 15:38 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@402d729]: Adjust the cirrus concurrencies
  • 15:32 _joe_: cross enabling videoscalers,jobrunners in their respective pools
  • {{safesubst:SAL entry|1=15:27 urandom: upgrade Cassandra to 3.11.2, restbase2001-{b,c} - T178905}}
  • 15:26 XioNoX: stop pybal on lvs1001
  • 15:26 awight@deploy1001: Finished deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336 (duration: 25m 37s)
  • 15:13 _joe_: adding jobrunners, videoscalers to both pools with equal weight in codfw
  • 15:06 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=videoscaler,dc=codfw,service=nginx
  • {{safesubst:SAL entry|1=15:06 urandom: upgrade Cassandra to 3.11.2, restbase1007-{b,c} - T178905}}
  • 15:01 awight@deploy1001: Started deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336
  • 14:39 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: cluster=videoscaler,dc=codfw,service=nginx,name=mw22(4[1-5]|5[3-8]|6[1-9]).*
  • 14:35 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: cluster=videoscaler,dc=codfw,service=nginx,name=mw21(5[3-9]|6).*
  • 14:22 andrewbogott: rebooting labvirt1009
  • 14:21 jynus: setting m2 on read write
  • 14:18 jynus: setting gerrit on read only
  • 14:14 jynus: starting s2-master switchover from db1051 to db1065
  • 14:06 andrewbogott: rebooting labvirt1003
  • 13:58 jynus: disabling puppet on db1051, db1065
  • 13:35 marostegui: Deploy schema change on db1096:3316 - T191316 T192926 T195193 T89737
  • 13:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 for alter table (duration: 00m 57s)
  • 13:06 moritzm: installing elfutils security updates
  • 13:01 akosiaris: starting slow rolling restart of all VMs on ganeti01.svc.codfw.wmnet
  • 12:59 akosiaris: add +spec_ctrl to ganeti01.svc.codfw.wmnet cluster default cpu_type
  • 12:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low load (duration: 00m 56s)
  • 12:24 mobrovac@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch CirrusSearch jobs to EventBus for all wikis, file 2/2 - T189137 (duration: 00m 56s)
  • 12:23 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@c8d62da]: Enable cirrus for everything T190327 (duration: 00m 47s)
  • 12:23 mobrovac@deploy1001: Synchronized wmf-config/jobqueue.php: Switch CirrusSearch jobs to EventBus for all wikis - T189137 (duration: 00m 57s)
  • 12:22 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@c8d62da]: Enable cirrus for everything T190327
  • 11:38 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase s4 weight for db1097 and db1103 (duration: 00m 56s)
  • 11:35 jynus: stop and reimage db1084
  • 11:27 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 58s)
  • 10:19 awight@deploy1001: Finished deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 03m 44s)
  • 10:16 marostegui: Deploy schema change on dbstore1002:s6 - T191316 T192926 T195193 T89737
  • 10:15 awight@deploy1001: Started deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336
  • 10:15 awight@deploy1001: Finished deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 00m 06s)
  • 10:15 awight@deploy1001: Started deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336
  • 08:29 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1011 - T190704
  • 08:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 56s)
  • 08:24 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: wikidatawiki dispatching: dispatchMaxTime 720 (4 dispatchers at once) (duration: 00m 56s)
  • 08:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 57s)
  • 07:48 marostegui: Stop replication on all sanitarium masters to move labsdb1011 - T190704
  • 07:30 marostegui: Stop MySQL on labsdb1011 to install intel-microcode and reboot
  • 06:32 marostegui: Deploy schema change on s6 codfw master (db2039), this will generate lag on s6 codfw - T191316 T192926 T195193 T89737
  • 06:14 marostegui: Stop slave on db2095:3316 to rebuild archive_insert and archive_update triggers - T192926
  • 06:14 ppchelko@deploy1001: Finished deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402, take 2 (duration: 07m 13s)
  • 06:06 ppchelko@deploy1001: Started deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402, take 2
  • 06:05 ppchelko@deploy1001: Finished deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402 (duration: 11m 45s)
  • 06:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool all sanitariums masters - T190704 (duration: 01m 09s)
  • 05:53 ppchelko@deploy1001: Started deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402
  • 05:46 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 - T190704
  • 05:31 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010 - T190704
  • 05:25 marostegui: Restart MySQL on labsdb1010
  • 05:24 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - T190704
  • 05:17 marostegui: Deploy schema change on db1070 s5 primary master - T191316 T192926 T195193
  • 04:44 kartik@deploy1001: Finished deploy [cxserver/deploy@8ce20ba]: Update cxserver to 391d7b6 (Fixing T196462) (duration: 03m 06s)
  • 04:41 kartik@deploy1001: Started deploy [cxserver/deploy@8ce20ba]: Update cxserver to 391d7b6 (Fixing T196462)
  • 03:10 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jun 6 03:10:14 UTC 2018 (duration 10m 15s)
  • 02:59 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 15m 31s)
  • 02:27 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 08m 23s)
  • 01:07 krinkle@deploy1001: Synchronized php-1.32.0-wmf.7/composer.json: I13dbdba2b9d / T196496 (duration: 00m 57s)
  • 01:06 krinkle@deploy1001: Synchronized php-1.32.0-wmf.7/vendor/: I5a5d7d / T196496 (duration: 01m 35s)
  • 00:15 reedy@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/WikimediaMessages/: respect watchlist preference feature flag (duration: 00m 58s)

2018-06-05

  • 23:54 reedy@deploy1001: Synchronized wmf-config/: Config! (duration: 00m 57s)
  • 23:42 reedy@deploy1001: Synchronized wmf-config/: Config! (duration: 00m 57s)
  • 23:41 reedy@deploy1001: Synchronized rpc/: code updates (duration: 00m 56s)
  • 23:26 reedy@deploy1001: Synchronized multiversion/submodules.json: minus unicodeconverter (duration: 00m 56s)
  • 23:24 reedy@deploy1001: Synchronized wmf-config/: bye to UnicodeConverter (duration: 00m 57s)
  • 23:21 reedy@deploy1001: Synchronized wmf-config/: page creation log on Beta Labs (duration: 00m 56s)
  • 23:07 reedy@deploy1001: Synchronized wmf-config/: Updates (duration: 00m 58s)
  • 23:05 reedy@deploy1001: Synchronized composer.json: minus x! (duration: 00m 56s)
  • 23:04 reedy@deploy1001: Synchronized rpc: 644 (duration: 00m 56s)
  • 23:01 ebernhardson: restore ferm on elastic1018 and logstash1009
  • 22:57 ebernhardson: temporarily disable ferm on elastic1018 and logstash1007 to test theory
  • 22:57 ebernhardson: temporarily disable ferm on elastic1018 to test theory
  • 22:50 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Simplify PHP_SAPI conditionals (duration: 00m 57s)
  • 22:20 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/437638/ (duration: 00m 57s)
  • 21:41 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.7
  • 21:29 thcipriani@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.7 and rebuild l10n cache (duration: 65m 25s)
  • 20:23 thcipriani@deploy1001: Started scap: testwiki to php-1.32.0-wmf.7 and rebuild l10n cache
  • 20:03 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@7ecc3b6]: Update mobileapps to 66727b7 (duration: 05m 21s)
  • 19:57 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@7ecc3b6]: Update mobileapps to 66727b7
  • 19:35 urandom: upgrade Cassandra to 3.11.2, restbase1007-a (canary) - T178905
  • 17:45 urandom: upgrade Cassandra to 3.11.2, restbase2001 (canary) - T178905
  • 17:00 urandom: upgrade Cassandra to 3.11.2, restbase-dev1006 - T178905
  • 16:58 Trey314159: reindexing Slovak wikis on elastic@eqiad (T191545)
  • 16:56 urandom: upgrade Cassandra to 3.11.2, restbase-dev1005 - T178905
  • 16:10 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove if file_exists for /etc/wikimedia-scaler (duration: 00m 51s)
  • 16:07 thcipriani: starting branch cut for 1.32.0-wmf.7
  • 16:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 fully (duration: 00m 49s)
  • 15:58 urandom: upgrade Cassandra to 3.11.2, restbase-dev1004 - T178905
  • 15:42 Trey314159: reindexing Slovak wikis on elastic@codfw (T191545)
  • 15:41 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 fully (duration: 00m 49s)
  • 15:30 bstorm_: reboot labsdb1005 for upgrades T195313
  • 15:24 marostegui: Stop MySQL on labsdb1005
  • 14:55 marostegui: Downtime labsdb1005 and labsdb1004 for maintenance on labsdb1005 - T195313
  • 14:52 urandom: restarting restbase-dev1004 - T178905
  • 14:51 moritzm: installing fixed kernels/microcode for spectre v2 on labvirt*
  • 14:42 jynus: reenabling puppet on all databases
  • 14:09 jynus: disabling puppet on eqiad dbs for 435751 gerrit deploy
  • 13:55 Amir1: ladsgroup@terbium:~$ mwscript deleteAutoPatrolLogs.php --wiki=commonswiki --sleep 2 --check-old
  • 13:26 mobrovac@deploy1001: Finished deploy [restbase/deploy@a39743d]: Fix transform monitoring tests (duration: 03m 15s)
  • 13:24 zeljkof: EU SWAT finished
  • 13:24 zfilipin@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: Remove ruwiki from MFSpecialCaseMainPage (T196223) (duration: 00m 51s)
  • 13:22 mobrovac@deploy1001: Started deploy [restbase/deploy@a39743d]: Fix transform monitoring tests
  • 13:22 mobrovac@deploy1001: Finished deploy [restbase/deploy@a39743d]: Fix transform monitoring tests (duration: 08m 00s)
  • 13:14 mobrovac@deploy1001: Started deploy [restbase/deploy@a39743d]: Fix transform monitoring tests
  • 13:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@a39743d]: Fix transform monitoring tests (duration: 18m 02s)
  • 12:55 mobrovac@deploy1001: Started deploy [restbase/deploy@a39743d]: Fix transform monitoring tests
  • 12:13 kartik@deploy1001: Finished deploy [cxserver/deploy@0a350c3]: Update cxserver to 7fb7671 (duration: 01m 15s)
  • 12:12 kartik@deploy1001: Started deploy [cxserver/deploy@0a350c3]: Update cxserver to 7fb7671
  • 12:06 XioNoX: disable graceful-switchover on dual-RE routers
  • 11:35 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low load (duration: 00m 49s)
  • 11:30 elukey: manually set net.netfilter.nf_conntrack_tcp_timeout_time_wait to 65 (was 120) on mw* hosts
  • 11:21 mobrovac@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch CirrusSearch jobs for all wikis except wp, wd, commons - T189137 (duration: 00m 51s)
  • 11:21 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@aa5e94b]: Enable cirrus jobs in kafka for everything except wikipedia, wikidata and commons T190327 (duration: 00m 41s)
  • 11:21 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@aa5e94b]: Enable cirrus jobs in kafka for everything except wikipedia, wikidata and commons T190327
  • 11:07 elukey: manually set net.netfilter.nf_conntrack_tcp_timeout_time_wait to 65 (was 120)
  • 10:51 jynus: stop db1081 for reimage
  • 10:45 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 (duration: 00m 50s)
  • 10:38 akosiaris: reboot acrab for spec_ctrl CPU flag addition
  • 10:20 akosiaris: reboot acrab for kernel upgrade and qemu upgrade
  • 10:18 akosiaris: upgrade qemu to 1:2.8+dfsg-6+deb9u4 on ganeti01.svc.codfw.wmnet
  • 10:14 moritzm: installing batik security updates
  • 10:08 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 with low load (duration: 00m 50s)
  • 09:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 50s)
  • 09:22 jynus: stop db1080 for reimage
  • 09:09 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 (duration: 00m 49s)
  • 09:01 marostegui: Deploy schema change on db1110 - T191316 T192926 T89737 T195193
  • 09:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 51s)
  • 08:52 mobrovac@deploy1001: Synchronized wmf-config/jobqueue.php: Switch video scaling jobs to EventBus - T190327 (duration: 00m 52s)
  • 08:52 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@63b30a6]: Enable videoscaler jobs in kafka T190327 (duration: 00m 49s)
  • 08:51 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@63b30a6]: Enable videoscaler jobs in kafka T190327
  • 08:16 ema: libvmod-re2 1.3.1-1 uploaded to apt.w.o T196355
  • 08:10 gehel: rebooting elastic10(41|43) for plugin update - T193734
  • 06:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2059 after reimage (duration: 00m 50s)
  • 05:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 00m 50s)
  • 05:44 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2059 for reimage (duration: 00m 51s)
  • 05:14 marostegui: Deploy schema change on db1100 - T191316 T192926 T89737 T195193
  • 05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 00m 51s)
  • 03:25 pnorman@deploy1001: Finished deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a (duration: 06m 02s)
  • 03:19 pnorman@deploy1001: Started deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a
  • 03:12 pnorman@deploy1001: Finished deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a (duration: 04m 50s)
  • 03:07 pnorman@deploy1001: Started deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a
  • 02:51 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): enable v3view (duration: 00m 06s)
  • 02:51 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): enable v3view
  • 02:49 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): enable v3view
  • 02:45 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable more sources (duration: 00m 26s)
  • 02:45 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable more sources
  • 02:43 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable style without labels (duration: 02m 22s)
  • 02:41 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable style without labels
  • 02:30 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jun 5 02:30:57 UTC 2018 (duration 10m 15s)
  • 02:20 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 08m 21s)
  • 00:08 thcipriani: restarting jenkins to finalize updates

2018-06-04

  • 23:57 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable page creation log on Test Wikipedia" T196400 (duration: 00m 49s)
  • 23:50 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable page creation log on Test Wikipedia T196400 (duration: 00m 50s)
  • 23:21 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@825863f]: Potential mitigation for T194325 (duration: 09m 39s)
  • 23:11 smalyshev@deploy1001: Started deploy [wdqs/wdqs@825863f]: Potential mitigation for T194325
  • 22:57 smalyshev@deploy1001: Started deploy [wdqs/wdqs@825863f]: Potential mitigation for T194325
  • 22:35 pnorman@deploy1001: Finished deploy [kartotherian/deploy@a588bf4] (cleartables): Deploy var name parameters (duration: 00m 21s)
  • 22:35 pnorman@deploy1001: Started deploy [kartotherian/deploy@a588bf4] (cleartables): Deploy var name parameters
  • 21:08 pnorman@deploy1001: Finished deploy [kartotherian/deploy@d8dcba3] (cleartables): Redeploy kartotherian to test (duration: 00m 21s)
  • 21:07 pnorman@deploy1001: Started deploy [kartotherian/deploy@d8dcba3] (cleartables): Redeploy kartotherian to test
  • 20:58 awight@deploy1001: Finished deploy [ores/deploy@bf182e2]: roll back ores2001 (duration: 01m 11s)
  • 20:57 awight@deploy1001: Started deploy [ores/deploy@bf182e2]: roll back ores2001
  • 20:50 awight@deploy1001: Finished deploy [ores/deploy@65e979f]: ores2001 canary of drafttopic; T176336 (take 3 after bumping revision) (duration: 01m 47s)
  • 20:48 awight@deploy1001: Started deploy [ores/deploy@65e979f]: ores2001 canary of drafttopic; T176336 (take 3 after bumping revision)
  • 20:40 awight@deploy1001: Finished deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)f (duration: 00m 22s)
  • 20:40 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@276ea43]: Update mobileapps to f579f0d (duration: 05m 54s)
  • 20:40 awight@deploy1001: Started deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)f
  • 20:40 awight@deploy1001: Started deploy [ores/deploy@d77e52c]: l ores2001.codfw.wmnet ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)
  • 20:34 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@276ea43]: Update mobileapps to f579f0d
  • 20:34 awight@deploy1001: Finished deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (duration: 02m 35s)
  • 20:34 arlolra@deploy1001: Finished deploy [parsoid/deploy@828034c]: Updating Parsoid to bd5a840 (duration: 10m 54s)
  • 20:31 awight@deploy1001: Started deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336
  • 20:30 andrew@deploy1001: Finished deploy [horizon/deploy@12aa2d3]: fix for T192179 (duration: 03m 35s)
  • 20:27 andrew@deploy1001: Started deploy [horizon/deploy@12aa2d3]: fix for T192179
  • 20:23 arlolra@deploy1001: Started deploy [parsoid/deploy@828034c]: Updating Parsoid to bd5a840
  • 19:10 gehel: elasticsearch cluster restart on eqiad completed - T193734
  • 19:06 ottomata: bouncing kafka2003 one more time for T196077
  • 19:04 otto@deploy1001: Started restart [eventlogging/eventbus@3a5c395]: bouncing eventbus after upgrading to python-kafka 1.4.3 for T196077
  • 19:00 ottomata: bouncing kafka2003 again to test T196077 with python-kafka 1.4.3
  • 18:52 thcipriani@deploy1001: Synchronized docroot/noc/index.html: SWAT: Fix wrong link to Server Admin Log on noc.wikimedia.org T193848 (duration: 00m 50s)
  • 18:44 ottomata: bouncing kafka on kafka2003 to test T196077
  • 18:36 otto@deploy1001: Finished deploy [eventlogging/eventbus@3a5c395]: T196077 (duration: 04m 07s)
  • 18:35 ottomata: deploying eventlogging-service-eventbus for T196077
  • 18:32 otto@deploy1001: Started deploy [eventlogging/eventbus@3a5c395]: T196077
  • 18:31 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Redeploy to test (duration: 00m 25s)
  • 18:30 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Redeploy to test
  • 18:01 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Enable osm source (duration: 00m 25s)
  • 18:00 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Enable osm source
  • 17:56 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-intl source (duration: 00m 25s)
  • 17:55 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-intl source
  • 17:53 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-pbf source (duration: 00m 25s)
  • 17:52 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-pbf source
  • 17:51 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view (duration: 00m 25s)
  • 17:51 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view
  • 17:48 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view (duration: 00m 24s)
  • 17:47 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view
  • 17:44 tgr: disabled SUL+wikitech 2FA for MarkAHershberger (T196370)
  • 17:37 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view (duration: 00m 26s)
  • 17:37 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view
  • 17:30 gehel@deploy1001: Finished deploy [wdqs/wdqs@fd534fa]: WDQS: new GUI and blazegraph versions (duration: 08m 25s)
  • 17:30 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test (duration: 02m 54s)
  • 17:27 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test
  • 17:22 gehel@deploy1001: Started deploy [wdqs/wdqs@fd534fa]: WDQS: new GUI and blazegraph versions
  • 17:22 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test (duration: 03m 24s)
  • 17:18 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test
  • 15:09 _joe_: performing rolling restart of authdns servers to pick up ip change for the videoscalers
  • 15:05 herron: gzipped large rotated log files in analytics1003:/var/log/hive to clear icinga disk space warning
  • 14:49 _joe_: restarting low-traffic pybals in eqiad,codfw for adding the videoscaler VIP
  • 14:38 ppchelko@deploy1001: Started restart [cpjobqueue/deploy@c6dc83d]: (no justification provided)
  • 14:35 Amir1: ladsgroup@terbium:~$ foreachwikiindblist large deleteAutoPatrolLogs.php --sleep 2 --check-old
  • 14:34 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010 - T190704
  • 14:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 49s)
  • 14:28 moritzm: installing wireshark security updates
  • 14:27 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=videoscaler,name=eqiad
  • 14:21 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: cluster=videoscaler,service=nginx,dc=codfw,name=mw211.*
  • 14:21 oblivian@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: cluster=videoscaler,service=nginx,dc=codfw
  • 14:20 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=.*,service=mathoid,cluster=kubernetes,name=.*
  • 14:19 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=videoscaler,service=nginx,dc=eqiad
  • 14:10 herron: lithium:~# systemctl restart rsyslog.service
  • 14:10 marostegui: Stop replication on all sanitarium masters to move labsdb1010 to another sanitarium host - T190704
  • 14:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool all sanitariums masters - T190704 (duration: 00m 49s)
  • 14:02 moritzm: installing wireshark security updates
  • 14:02 zeljkof: EU SWAT finished
  • 13:49 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Change bewikiquote logo (T196134) (duration: 00m 49s)
  • 13:44 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgMetaNamespace to "Вікіцытатнік" on bewikiquote (T196230) (duration: 00m 49s)
  • 13:39 anomie: Running populateExternallinksIndex60.php on group 2 for T59176. FYI: this will probably take until next Friday to complete.
  • 13:34 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgProofreadPagePageSeparator to empty string on zhwikisource (T194875) (duration: 00m 49s)
  • 13:27 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgProofreadPagePageSeparator to empty string for jawikisource (T195873) (duration: 00m 49s)
  • 13:16 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Temporarily enable MFMobileMainPageCss in ruwiki (T195905) (duration: 00m 50s)
  • 13:11 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Assign movefile to autoreviewrs and patrollers on zhwiki (T195247) (duration: 00m 52s)
  • 12:49 gehel: restart elastic2001 to enable G1 GC - T156137
  • 12:18 krinkle@deploy1001: Finished deploy [performance/navtiming@b229f75]: (no justification provided) (duration: 00m 05s)
  • 12:18 krinkle@deploy1001: Started deploy [performance/navtiming@b229f75]: (no justification provided)
  • 11:38 akosiaris: rebalance row_A, row_C nodegroups in ganeti01.svc.eqiad.wmnet cluster
  • 10:51 akosiaris: reimage ganeti1004, ganeti1008 to stretch
  • 10:39 _joe_: rolling restart of apache on the jobrunners to pick the changed privatetmp setting, rotating logs
  • 10:14 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
  • 10:13 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
  • 09:39 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - https://phabricator.wikimedia.org/T190704
  • 09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 00m 49s)
  • 09:04 addshore: addshore@terbium:~$ for i in {1..2500}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki
  • 08:56 marostegui: Deploy schema change on db1097:3315 - T191316 T192926 T89737 T195193
  • 08:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 00m 49s)
  • 08:53 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2005 (duration: 00m 50s)
  • 08:10 jynus: restarting icinga due to ongoing check/downtime issues
  • 07:57 marostegui: Stop replication on db2094:3315 for testing
  • 07:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 after alter table (duration: 00m 51s)
  • 07:11 gehel: starting elasticsearch cluster restart on eqiad - T193734
  • 06:18 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2059, db2075 - T190704 (duration: 00m 49s)
  • 06:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 - T190704 (duration: 00m 49s)
  • 05:52 marostegui: Stop replication in sync on db1121 and db2051 - T190704
  • 05:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 - T190704 (duration: 00m 49s)
  • 05:29 marostegui: Deploy schema change on db1082 with replication (this will generate lag on labs for s5) - T191316 T192926 T89737 T195193
  • 05:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 00m 53s)
  • 02:53 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jun 4 02:53:16 UTC 2018 (duration 10m 14s)
  • 02:43 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 14m 33s)

2018-06-03

  • 02:18 andrewbogott: rebooting labservices1001; it seems to have crashed

2018-06-02

  • 07:36 legoktm@deploy1001: Synchronized php-1.32.0-wmf.6/skins/MonoBook/: Temporarily revert responsive MonoBook (T195625) (duration: 00m 58s)

2018-06-01

  • 19:21 ebernhar1son: enable query phase slow logging and increase thresholds for fetch phase slow logging for content/general indices on eqiad and codfw elasticsearch clusters
  • 19:14 mutante: zh.planet - fixed issue with corrupt state file and permissions - updated and using new design as well now
  • 17:34 mutante: deployment.eqiad/codfw DNS names switched from tin to deploy1001
  • 17:06 thcipriani@deploy1001: Synchronized README: noop test of new deployment server (duration: 00m 53s)
  • 16:39 mutante: deploy2001 - also fixing file permissions. files owned by 996 -> mwdeploy, files owned by 997 -> trebuchet
  • 16:21 mutante: deployment server has switched away from tin to deploy1001. set global scap lock on deploy1001, re-enabled puppet and ran puppet, disabled tin as deployment server (T175288)
  • 16:13 herron: enabled new logstash tcp input with TLS enabled for syslogs on port 16514 T193766
  • 15:51 gehel: elasticsearch cluster restart on codfw completed - T193734
  • 15:47 mutante: @deploy1001:/srv/deployment# find . -uid 997 -exec chown trebuchet {} \;
  • 15:41 mutante: root@deploy1001:/srv/mediawiki-staging# find . -uid 996 -exec chown mwdeploy {} \;
  • 15:17 mutante: [deploy1001:~] $ scap pull-master tin.eqiad.wmnet
  • 15:12 mutante: tin umask 022 && echo 'switching deploy servers' > /var/lock/scap-global-lock
  • 15:05 mutante: rsyncing /srv/mediawiki-staging to /srv/mediawiki-staging-before-backup/ on tin as a backup
  • 14:52 mutante: deploy1001 - scap pull
  • 14:30 elukey: killed pt-heartbear-wikimedia after https://gerrit.wikimedia.org/r/436748 on db1107
  • 14:24 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 fully (duration: 01m 02s)
  • 14:08 reedy@tin: Synchronized php-1.32.0-wmf.6/extensions/FlaggedRevs: T196139 (duration: 01m 08s)
  • 13:32 marostegui: Deploy schema change on dbstore1002:s5 - T191316 T192926 T89737 T195193
  • 13:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 (duration: 01m 03s)
  • 10:59 _joe_: disabling puppet on all hosts with role::mediawiki::common while installing mcrouter everywhere
  • 10:35 marostegui: Deploy schema change on db1096:3315 - T191316 T192926 T89737 T195193
  • 10:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 (duration: 01m 03s)
  • 10:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 db1096:3315 (duration: 01m 02s)
  • 10:04 Amir1: ladsgroup@terbium:~$ foreachwikiindblist medium deleteAutoPatrolLogs.php --sleep 2 --check-old
  • 09:53 volans@tin: Finished deploy [debmonitor/deploy@fe8df6e]: Release v0.1.1 (duration: 00m 33s)
  • 09:53 volans@tin: Started deploy [debmonitor/deploy@fe8df6e]: Release v0.1.1
  • 09:37 marostegui: Stop replication in sync on db1113:3315 and db1096:3315 for data checks
  • 09:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 db1096:3315 (duration: 01m 03s)
  • 09:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 with low load (duration: 01m 03s)
  • 08:38 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): reenable v3view on 2004 (duration: 04m 53s)
  • 08:33 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): reenable v3view on 2004
  • 08:30 jynus: reimage db1083
  • 08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 after alter table (duration: 01m 03s)
  • 08:24 jynus: temporarily reducing s7-codfw-master consistency to aliviate lag (binlog_sync, flush_log)
  • 08:22 joal@tin: Finished deploy [analytics/refinery@7a72241]: Regular weekly deploy (duration: 12m 31s)
  • 08:20 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 01m 05s)
  • 08:10 joal@tin: Started deploy [analytics/refinery@7a72241]: Regular weekly deploy
  • 06:15 marostegui: Stop MySQL on db2059 to clone db2075 - T190704
  • 06:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2059 (duration: 00m 56s)
  • 05:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2092 and db2062 in s1 (duration: 00m 59s)
  • 05:27 marostegui: Deploy schema change on db1113:3315 - T191316 T192926 T89737 T195193
  • 05:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 for alter table (duration: 00m 57s)
  • 01:34 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 00m 33s)
  • 01:33 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 01:27 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 34s)
  • 01:24 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 01:03 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 24s)
  • 01:00 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 00:59 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 52s)
  • 00:56 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 00:53 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 26s)
  • 00:51 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 00:48 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 03m 11s)
  • 00:45 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 00:03 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 04m 26s)


Archives