Jump to content

Server Admin Log/Archive 35

From Wikitech

2018-08-31

  • 23:35 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/includes/Html.php: I67ceb34eabf2f - T200506 (duration: 00m 50s)
  • 23:00 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/WikidataPageBanner/: I9dc9a4 - T199855 (duration: 00m 51s)
  • 21:14 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/includes/deferred/: Id2852d73d00 (2/2) (duration: 00m 55s)
  • 21:11 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/includes/jobqueue/jobs/: Id2852d73d00 (1/2) (duration: 00m 52s)
  • 19:46 mutante: right when it was fixed on ms-be2043 it also broke on ms-be2040. following the same instructions to fix xfs in a root screen (T199198)
  • 19:12 mutante: ms-be2043 - following instructions at https://wikitech.wikimedia.org/wiki/Graphite#Repair_xfs_misreporting_free_space to repair xfs misreporting free space (T199198), fixing docs, icinga-downtime doesn't want fqdn but short name
  • 18:10 jforrester@deploy1001: Synchronized php-1.32.0-wmf.19/includes/Title.php: Hot-deploy of I05eea553c58 to let users edit Copyright again (duration: 00m 50s)
  • 17:57 mutante: depooled wtp2020 because icinga reported memory errors
  • 17:39 SMalyshev: repooled wdqs1005
  • 16:30 jforrester@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: Hot-deploy of I38eda4aac48 to fix T203213 (duration: 00m 54s)
  • 10:49 moritzm: installing libgd2 security updates on trusty
  • 09:46 moritzm: installing libx11 security updates on trusty
  • 08:57 gehel: elasticsearch data directory migration on all logstash nodes
  • 08:16 godog: repair sde1 on ms-be2041 - T199198
  • 05:54 elukey: resumed the Hadoop workers reboots for kernel upgrades
  • 05:33 elukey: restart pdfrender on scb1003
  • 00:44 mutante: netmon1002 - restarted smokeping, removed radon as target (unblock decome of former dns server), added cobalt instead as a target also in C4

2018-08-30

  • 23:42 mutante: vega - removed rsync config and let puppet regenerate it
  • 23:38 mutante: bromine - remove outdated rsync conf fragment for static-bugzilla, stopping rsync, running puppet
  • 23:36 mutante: releases1001 - killing outdated rsync processes for releases from bromine
  • 22:57 jforrester@deploy1001: Finished scap: Full sync for i18n re-build following Ieaded578ffd (duration: 32m 29s)
  • 22:25 jforrester@deploy1001: Started scap: Full sync for i18n re-build following Ieaded578ffd
  • 22:13 volans: this was a test for the logging to IRC, please ignore ^^^^
  • 22:12 END: (NOTRUN) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (switchdc/volans@sarin)
  • 22:12 START: - Cookbook sre.switchdc.mediawiki.00-disable-puppet (switchdc/volans@sarin)
  • 21:57 jforrester@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/JsonConfig/: Hot-deploy Ieaded578ffd revert of T200968 due to bugs (duration: 00m 51s)
  • 20:23 mutante: cp1080 - powercycled - lots of RECOVERY from Icinga for IPsec connections - leaving depooled so far (T201174)
  • 20:17 mutante: powercycling cp1080
  • 20:06 mutante: dzahn@neodymium conftool action : set/pooled=no; selector: name=cp1080.eqiad.wmnet| reason: Strongswan CRITICALs fom Icinga (T201174)
  • 20:04 dzahn@neodymium: conftool action : set/pooled=no; selector: name=cp1080.eqiad.wmnet
  • 19:47 dduvall@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.19
  • 19:36 marxarelli: Deploying 1.32.0-wmf.19 to all wikis
  • 19:21 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/includes/: I11b390 (duration: 01m 16s)
  • 19:13 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/resources/src/startup: I13a996e01b48 (duration: 01m 06s)
  • 18:52 moritzm: restarting aphlict on phab1001 to pick up nodejs security update
  • 16:59 godog: xfs_repair on ms-be1041 sdf1 - T199198 (retroactive, started at 15:32
  • 16:35 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=wdqs,name=wdqs1005.eqiad.wmnet
  • 16:30 gehel: shutting down wdqs1005 for new SSD and reimaging - T202779
  • 16:29 gehel: shutting down wdqs1005 for new SSD and reimaging - T198351
  • 16:15 gehel: restart of logstash to move data directory - T198351
  • 15:17 bstorm_: increased number of nfsd threads on labstore1004 to 300
  • 14:08 moritzm: upgrading mw1262-1265 to wikidiff 1.7.3
  • 13:46 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119, db1090 (s2 and s7) (duration: 00m 59s)
  • 13:41 moritzm: upgrading mw1261 to wikidiff 1.7.3
  • 13:31 jynus: depooling labsdb1009 due to extra lag
  • 13:10 jynus: sanitizing fixcopyrightwiki on db2094 and children T202820
  • 13:08 jynus: sanitizing fixcopyrightwiki on db1124 and children T202820
  • 13:03 moritzm: upgrading grafana to 4.6.4 (security release)
  • 12:59 elukey: drain + reboot analytics10[28-79]* for kernel updates (will take multiple days)
  • 12:28 godog: roll restart thumbor in eqiad to upgrade to 2.1
  • 12:25 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 36s)
  • 12:23 reedy@deploy1001: Synchronized dblists/: fixcopyrightwiki (duration: 00m 56s)
  • 12:22 moritzm: uploaded grafana 4.6.4 to apt.wikimedia.org
  • 12:13 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: fixcopyrightwiki (duration: 00m 56s)
  • 12:11 reedy@deploy1001: Synchronized static/images/project-logos: fixcopyrightwiki (duration: 00m 55s)
  • 12:10 reedy@deploy1001: rebuilt and synchronized wikiversions files: fixcopyright
  • 12:07 reedy@deploy1001: Synchronized dblists/: fixcopyrightwiki (duration: 00m 56s)
  • 11:41 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2070 (duration: 00m 55s)
  • 11:28 zeljkof: EU SWAT finished
  • 11:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Test Wikidata: Use new item ID formatter for all the items (T203147) (duration: 00m 53s)
  • 11:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Beta Wikidata: Use new item ID formatter for all the items (T203147) (duration: 00m 57s)
  • 11:09 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add editsitejson to everyone who has editinterface (T190015) (duration: 01m 00s)
  • 10:37 moritzm: upgrading deployment-prep to wikidiff 1.7.3
  • 10:04 godog: roll-restart thumbor to upgrade 2.1
  • 08:52 godog: roll-restart swift-proxy to send requests to thumbor in eqiad and codfw - T201858
  • 07:34 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2070 (duration: 00m 57s)
  • 06:46 jynus: upgrade and restart db1090
  • 06:43 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090 (s2 and s7) (duration: 00m 56s)
  • 05:52 jynus: upgrade and restart db1119
  • 05:49 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 (duration: 00m 58s)
  • 03:42 mutante: running manual update of en.planet feeds (T203055)
  • 03:21 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Aug 30 03:21:55 UTC 2018 (duration 10m 14s)
  • 03:11 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.19) (duration: 15m 12s)
  • 02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 15m 20s)

2018-08-29

  • 23:56 catrope@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/Translate/stringmangler/StringMatcher.php: T202058 (duration: 00m 56s)
  • 23:55 catrope@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/Translate/stringmangler/StringMatcher.php: T202058 (duration: 00m 59s)
  • 19:47 bstorm_: failed tools and cloud vps storage services over to labstore1004
  • 19:24 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 to 1.32.0-wmf.19
  • 19:21 marxarelli: Deploying 1.32.0-wmf.19 to group1
  • 18:55 mutante: puppet deploy of new cluster apache site inclusion for fixcopyright.wm, tested on mwdebug100*,mw1269, apache-fast-test (T202819) (gerrit:456194)
  • 18:31 ottomata: debdeploy git-fat update for all nodes - T202100
  • 18:10 mutante: puppet re-enabled on mw* via cumin before last log message (cumin expected to log to SAL automatically?)
  • 18:08 mutante: new apache config in sites-available for new site fixcopyright.wm is being generated by puppet on cluster, but not enabled yet (T202819)
  • 17:51 imarlier@deploy1001: Finished deploy [performance/coal@7457c86]: (no justification provided) (duration: 00m 06s)
  • 17:51 imarlier@deploy1001: Started deploy [performance/coal@7457c86]: (no justification provided)
  • 17:51 mutante: disabling puppet on mw hosts as a precaution before deploying apache change
  • 16:49 marostegui: Force RAID controller to WB policy T202051
  • 16:21 rxy: tgr@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: Wikidata: Use new item ID formatter for Q1-Q10000 (T201834) (duration: 00m 56s) (by logmsgbot)
  • 16:21 rxy: T202323 labstore1005 reboot (by arturo)
  • 16:17 arturo: T202323 failover drbd primary from labstore1005 to labstore1004
  • 15:53 arturo: T202323 labstore1004 now running latest kernel
  • 15:51 marlier: deploy coal
  • 15:51 imarlier@deploy1001: Finished deploy [performance/coal@5d995a3]: (no justification provided) (duration: 00m 06s)
  • 15:51 imarlier@deploy1001: Started deploy [performance/coal@5d995a3]: (no justification provided)
  • 15:48 elukey: roll restart aqs on aqs100[4-9] to pick up the new druid config settings
  • 15:47 arturo: T202323 labstore1004 reboot
  • 15:37 joal@deploy1001: Finished deploy [analytics/aqs/deploy@c33f6e5]: Update wikistats2 endpoints to use user_text and page_title (duration: 11m 38s)
  • 15:26 joal@deploy1001: Started deploy [analytics/aqs/deploy@c33f6e5]: Update wikistats2 endpoints to use user_text and page_title
  • 15:00 arturo: T202323 starting operations
  • 14:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 (duration: 00m 58s)
  • 13:45 gehel: restart relforge for plugin upgrade, data dir migration and kernel upgrade
  • 13:41 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=wdqs,name=wdqs2001.codfw.wmnet
  • 13:39 gehel: shutting down wdqs2001 for new SSD and reimaging - T202777
  • 13:11 kartik@deploy1001: Finished deploy [cxserver/deploy@c3385eb]: Update cxserver to afe0d1f (T202970) (duration: 03m 46s)
  • 13:07 kartik@deploy1001: Started deploy [cxserver/deploy@c3385eb]: Update cxserver to afe0d1f (T202970)
  • 11:53 hashar: European SWAT completed (2)
  • 11:51 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: yphc.ir to the wgCopyUploadsDomains whitelist - T201237 (duration: 00m 56s)
  • 11:46 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use HD logos for various Wikibooks - T177506 (duration: 00m 56s)
  • 11:42 hashar@deploy1001: Synchronized static/images/project-logos: Upload HD logos for various wikibooks - T177506 (duration: 00m 55s)
  • 11:37 hashar: European SWAT completed.
  • 11:37 hashar: deploy1001: "mwscript updateCollation.php --wiki=azwiki --previous-collation=uppercase" completed | T201770
  • 11:35 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new logos for advisorywiki - T202844 (duration: 00m 57s)
  • 11:33 hashar@deploy1001: Synchronized static/images/project-logos: Upload new logos for advisorswiki - T202844 (duration: 00m 55s)
  • 11:30 hashar: last sync did not synchronized due to ssh / keyholder isses
  • 11:30 hashar@deploy1001: Synchronized static/images/project-logos: Upload new logos for advisorswiki - T202844 (duration: 00m 37s)
  • 11:30 moritzm: rearmed keyholder on sarin
  • 11:25 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Allow subpages in main namespace in zhwikiversity - T202007 (duration: 00m 56s)
  • 11:20 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Translation of scnwiktionary sitename was removed, add it back - T202926 (duration: 00m 56s)
  • 11:13 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Allow sysops to remove flood flag - T202599 (duration: 00m 56s)
  • 11:06 hashar: mwscript updateCollation.php --wiki=azwiki --previous-collation=uppercase # T201770
  • 11:06 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set $wgCategoryCollation = uca-az on azwiki - T201770 (duration: 00m 56s)
  • 10:41 elukey@deploy1001: Finished deploy [analytics/refinery@1c6423f]: (no justification provided) (duration: 02m 56s)
  • 10:38 elukey@deploy1001: Started deploy [analytics/refinery@1c6423f]: (no justification provided)
  • 10:30 joal@deploy1001: Finished deploy [analytics/refinery@1c6423f]: Fix over yesterday weekly deploy of analytics Hadoop jobs - try 2 (duration: 08m 28s)
  • 10:22 joal@deploy1001: Started deploy [analytics/refinery@1c6423f]: Fix over yesterday weekly deploy of analytics Hadoop jobs - try 2
  • 09:52 marostegui: Deploy schema change on db1091
  • 09:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 (duration: 00m 58s)
  • 09:48 marostegui: Deploy schema change on db1102:s4
  • 09:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 00m 57s)
  • 09:33 elukey: cr1/2-eqiad: update analytics-in4 filter with the new archiva host, add a new term 'archiva' to analytics-in6 filter - T198623
  • 09:09 volans: upgraded python{,3}-conftool,spicerack on sarin
  • 09:05 joal@deploy1001: Finished deploy [analytics/refinery@1c6423f]: Fix over yesterday weekly deploy of analytics Hadoop jobs (duration: 11m 23s)
  • 09:04 godog: switch statsd and carbon CNAMEs to graphite1004 - T196484
  • 09:03 volans: uploaded spicerack_0.0.2-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T177385
  • 08:56 volans: uploaded python{,3}-conftool_1.0.2-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T177385
  • 08:54 joal@deploy1001: Started deploy [analytics/refinery@1c6423f]: Fix over yesterday weekly deploy of analytics Hadoop jobs
  • 08:49 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp1043.eqiad.wmnet
  • 08:42 volans: uploaded cumin_3.0.2-2+deb9u1 to apt.wikimedia.org stretch-wikimedia - T177385
  • 08:31 godog: repair sdh1 on ms-be2043 - T199198
  • 08:31 godog: repair sdd1 on ms-be2040 - T199198
  • 08:14 marostegui: Force WriteBack policy on db2042 T202051
  • 08:10 addshore@deploy1001: Synchronized wmf-config/InterwikiSortOrders.php: DOCS ONLY: T170745 InterwikiSortOrders.php doc / comment (duration: 00m 57s)
  • 08:07 moritzm: rolling reboot of wtp1* servers for kernel security update
  • 07:47 marostegui: Reboot db2042 - T202051
  • 07:31 moritzm: rebooting cp1008/pinkunicorn
  • 06:03 elukey: migrate archiva.wikimedia.org to archiva1001 (upgrading archiva to its latest upstream version + Debian Stretch + Java 8)
  • 06:02 marostegui: Deploy schema change on db1083 (labswiki) - T89737
  • 05:58 marostegui: Deploy schema change on db1073 (labswiki) - T187089
  • 05:55 marostegui: Deploy schema change on db1073 (labswiki) - T114117 T51191 T67448
  • 05:46 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2088 - T202822 (duration: 00m 55s)
  • 05:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 54s)
  • 05:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 (duration: 01m 07s)
  • 04:28 kart_: Finished old draft purge for ContentTranslation script run on mwmaint1001 (T201895)
  • 03:13 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 29 03:13:13 UTC 2018 (duration 10m 14s)
  • 03:03 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.19) (duration: 16m 23s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 08m 01s)
  • 00:05 tzatziki: reset email for User:Galahad

2018-08-28

  • 22:37 jforrester@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/JsonConfig/includes/JCSingleton.php: Hot-deploy T203006 fix (duration: 00m 56s)
  • 22:36 jforrester@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/JsonConfig/includes/JCSingleton.php: Hot-deploy T203006 fix (duration: 00m 57s)
  • 21:51 chasemp: install nmap on stat1006 && chmod 500 /usr/bin/nmap
  • 20:33 legoktm@deploy1001: Synchronized wmf-config/CommonSettings.php: beta only: Only enable TemplateWizard if TemplateData is also enabled (duration: 00m 56s)
  • 19:47 herron: puppetmaster reboots finished, re-enabling puppet agents
  • 19:26 herron: temporarily disabling puppet agents for rolling kernel updates/reboots on puppetmaster hosts
  • 19:15 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.19
  • 18:37 kaldari@deploy1001: Synchronized wmf-config/CommonSettings.php: for TemplateWizard (duration: 00m 55s)
  • 18:36 kaldari@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: for TemplateWizard (duration: 00m 55s)
  • 18:35 kaldari@deploy1001: Synchronized wmf-config/InitialiseSettings.php: for TemplateWizard (duration: 00m 55s)
  • 18:33 kaldari@deploy1001: Synchronized wmf-config/extension-list: (no justification provided) (duration: 00m 55s)
  • 18:31 kaldari@deploy1001: Synchronized wmf-config/extension-list: (no justification provided) (duration: 00m 57s)
  • 18:11 arlolra: Updated Parsoid to 61086f6 (T199246, T198221)
  • 18:11 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.19 and rebuild l10n cache (duration: 65m 35s)
  • 18:00 arlolra@deploy1001: Finished deploy [parsoid/deploy@d29d829]: Updating Parsoid to 61086f6 (duration: 09m 10s)
  • 17:52 moritzm: rebooting multatuli for microcode tests
  • 17:51 arlolra@deploy1001: Started deploy [parsoid/deploy@d29d829]: Updating Parsoid to 61086f6
  • 17:05 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.19 and rebuild l10n cache
  • 16:25 marxarelli: starting branch cut for 1.32.0-wmf.19
  • 15:38 marostegui: Deploy schema change on db1097:3314
  • 15:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 (duration: 00m 48s)
  • 15:35 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=wdqs,name=wdqs1004.eqiad.wmnet
  • 15:35 gehel: shutting down wdqs1004 to add new disks - T202779
  • 15:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 (duration: 00m 48s)
  • 14:49 moritzm: depooling scb1002 for hardware diagnostics
  • 13:15 godog: repair sdd1 on ms-be2041 - T199198
  • 12:34 tgr@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/TemplateStyles/: SWAT: Hoist selectors for html and body element (T197617) (duration: 00m 48s)
  • 12:31 tgr@deploy1001: Synchronized php-1.32.0-wmf.18/vendor/: SWAT: 455770|Update wikimedia/css-sanitizer to 2.0.0 (T197617) (duration: 01m 16s)
  • 11:34 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Permissions changes in ruwikinews (T201265) (duration: 00m 48s)
  • 11:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create namespace aliases in zhwikiversity (T202821) (duration: 00m 48s)
  • 11:24 zfilipin@deploy1001: Synchronized dblists/commonsuploads.dblist: SWAT: Remove uzwiki from commonsuploads.dblist (T202847) (duration: 00m 48s)
  • 11:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Wikidata: Use new item ID formatter for Q1-Q100 (T201833) (duration: 00m 49s)
  • 11:00 arturo: T202549 delete mysql-server from cloudcontrol100[3,4.wikimedia.org
  • 10:53 joal@deploy1001: Finished deploy [analytics/refinery@8cded22]: Regular weekly deploy of analytics Hadoop jobs (duration: 09m 14s)
  • 10:52 marostegui: Import nova_api_eqiad1 and nova_eqiad1 into m5 master - T202549
  • 10:51 arturo: T202549 downtime cloudcontrol1003.wikimedia.org in icinga for 2h
  • 10:44 joal@deploy1001: Started deploy [analytics/refinery@8cded22]: Regular weekly deploy of analytics Hadoop jobs
  • 10:42 marostegui: Deploy schema change on db1081
  • 10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 (duration: 00m 48s)
  • 10:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 (duration: 00m 50s)
  • 09:12 kartik@deploy1001: Finished deploy [cxserver/deploy@e2e5674]: Update cxserver to 98cbefd and LingoCloud deployment (T186715) (duration: 03m 38s)
  • 09:08 kartik@deploy1001: Started deploy [cxserver/deploy@e2e5674]: Update cxserver to 98cbefd and LingoCloud deployment (T186715)
  • 07:57 moritzm: rebooting multatuli for some tests
  • 07:47 marostegui: Deploy schema change on s3:advisorswiki - T202904 T199368
  • 07:46 marostegui: Deploy schema change on s3:advisorswiki - T202904 T192926
  • 07:45 marostegui: Deploy schema change on s3:advisorswiki - T202904 T196379
  • 07:41 marostegui: Deploy schema change on s3:advisorswiki - T202904 https://phabricator.wikimedia.org/T195193
  • 07:39 marostegui: Deploy schema change on s3:advisorswiki - T202904 T197891
  • 07:37 moritzm: imported python-requests-mock/1.3.0-3~wmf1+stretch to apt.wikimedia.org/stretch-wikimedia
  • 07:36 moritzm: imported linux-meta/1.19 to apt.wikimedia.org/jessie-wikimedia
  • 06:57 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2033 - T201757 (duration: 00m 49s)
  • 06:47 marostegui: Deploy schema change on s8 primary master (db1071)
  • 05:56 volans@deploy1001: Finished deploy [netbox/deploy@5e70423]: Security upgrade of dependency (duration: 00m 34s)
  • 05:55 volans@deploy1001: Started deploy [netbox/deploy@5e70423]: Security upgrade of dependency
  • 05:28 krinkle@deploy1001: Synchronized php-1.32.0-wmf.18/resources/src/startup/startup.js: one line to rule them all - I4942bf (duration: 00m 48s)
  • 05:07 marostegui: Deploy schema change db1103:3314
  • 05:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 (duration: 00m 49s)
  • 04:56 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (beta) 8d773b6cf (duration: 00m 50s)
  • 04:55 krinkle@deploy1001: sync-file aborted: (beta) 8d773b6cf (duration: 00m 02s)
  • 04:21 eileen: process-control config revision is d0de40dec9
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 28 02:45:08 UTC 2018 (duration 10m 12s)
  • 02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 13m 17s)
  • 01:39 eileen: civicrm revision changed from 4e0dcf7cb4 to 863ed3f1a2, config revision is 5a629055fa

2018-08-27

  • 21:36 gehel: repooling wdqs2003, new SSD, reimaged and data loaded - T195285
  • 21:04 krinkle@deploy1001: Synchronized wmf-config/mc.php: doc - Ic7349f4ce (duration: 00m 48s)
  • 20:39 Amir1: the ORES deployment is done
  • 20:37 ladsgroup@deploy1001: Finished deploy [ores/deploy@0e8cc73]: Deploy renaming wp10 models to articlequality (T196240) (duration: 25m 01s)
  • 20:12 ladsgroup@deploy1001: Started deploy [ores/deploy@0e8cc73]: Deploy renaming wp10 models to articlequality (T196240)
  • 20:10 ladsgroup@deploy1001: Finished deploy [ores/deploy@0e8cc73]: Deploy renaming wp10 models to articlequality (T196240) (duration: 05m 38s)
  • 20:09 Amir1: rolling back
  • 20:04 ladsgroup@deploy1001: Started deploy [ores/deploy@0e8cc73]: Deploy renaming wp10 models to articlequality (T196240)
  • 19:34 XioNoX: adding graphite1004 and graphite2003 to analytics-in4 - T202846
  • 19:19 XioNoX: adding IX bgp session to AS10089 on cr1-eqsin
  • 18:58 XioNoX: pushing labs-instance-in4 changes to cr1/2-eqiad - T199437
  • 18:56 catrope@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/PageTriage/includes/ArticleMetadata.php: Allow deferred writes on GET for pages with missing metadata (T202815) (duration: 00m 47s)
  • 18:50 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add mhs.ox.ac.uk to $wgCopyUploadsDomains (T201604) (duration: 00m 48s)
  • 18:47 catrope@deploy1001: Synchronized wmf-config/throttle.php: Throttle exception for 2018-08-29 (T202288) (duration: 00m 48s)
  • 18:34 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable PageTriage log channel (T202815) (duration: 00m 57s)
  • 18:21 XioNoX: merge Per DC alerting on sudden traffic drop (454613) - T201630
  • 18:12 XioNoX: trunking cloud-instances1-b-codfw to labtestneutron2002:eth1 and labtestneutron2001:eth1 - T202636
  • 17:37 XioNoX: pushing the above analytics-in changes to cr1/2-eqiad - T198623
  • 17:27 volans: uploaded spicerack_0.0.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia - T199079
  • 17:17 gehel@deploy1001: Finished deploy [wdqs/wdqs@22869f0]: new version of wdqs GUI and updater (duration: 10m 24s)
  • 17:07 gehel@deploy1001: Started deploy [wdqs/wdqs@22869f0]: new version of wdqs GUI and updater
  • 17:05 gehel@deploy1001: Finished deploy [wdqs/wdqs@22869f0]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 27s)
  • 17:04 gehel@deploy1001: Started deploy [wdqs/wdqs@22869f0]: new version of wdqs GUI and updater (wdqs1009 only)
  • 16:31 anomie: Running populateContentTables.php on advisorswiki for T183488 and T202904
  • 15:50 marostegui: Upgrade kernel and mariadb on db2088
  • 15:49 anomie: Re-running populateContentTables.php on aawikibooks, gotwikibooks, kswikiquote, lvwikibooks, nostalgiawiki, wawikibooks and wikimania2005wiki for T183488
  • 15:48 gehel: reenabling puppet on install[12]002
  • 15:26 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=codfw,cluster=wdqs,name=wdqs2003.eqiad.wmnet
  • 15:26 gehel: taking wdqs2003 offline for SSD installation - T202778
  • 15:12 herron: decreasing logstash elasticsearch index replica count to 1 on indices older than 1 day
  • 15:12 godog: set transient low watermark to 80% for elasticsearch logstash cluster to allow shard replica allocation - T201971
  • 14:52 papaul: up Shutting down db2088 for BIOS upgrade
  • 14:43 anomie: Running deduplicateArchiveRevId.php on gotwikibooks, kswikiquote, lvwikibooks, nostalgiawiki, wawikibooks and wikimania2005wiki for T202032
  • 14:42 anomie: Running deduplicateArchiveRevId.php on aawikibooks for T202032
  • 14:04 mobrovac@deploy1001: Finished deploy [citoid/deploy@fe96789]: Resolve DOIs/URLs all the way to end - T197242 (duration: 05m 37s)
  • 13:58 mobrovac@deploy1001: Started deploy [citoid/deploy@fe96789]: Resolve DOIs/URLs all the way to end - T197242
  • 13:57 mobrovac@deploy1001: Finished deploy [proton/deploy@17fc7bb]: Ignore HTTPS errors for the time being (duration: 00m 33s)
  • 13:57 mobrovac@deploy1001: Started deploy [proton/deploy@17fc7bb]: Ignore HTTPS errors for the time being
  • 13:25 anomie@deploy1001: Synchronized php-1.32.0-wmf.18/includes/Storage/RevisionStore.php: Backport for T202032 (duration: 00m 49s)
  • 12:12 akosiaris: rolling restart of parsoid service on wtp hosts for picking up https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454574/
  • 12:02 addshore: swat done
  • 12:00 marostegui: Create empty databases nova_api_eqiad1 and nova_eqiad1 on m5 master (db1073) - T202549
  • 11:59 addshore@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/Wikibase/lib/includes: gerrit:455542 Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup T202706 (duration: 00m 51s)
  • 11:50 godog: repair on ms-be2042 sdd - T199198
  • 11:49 moritzm: rebooting labweb* for kernel security update
  • 11:45 akosiaris: enable puppet on all hosts including puppet class service::configuration and start a rolling puppet run on them
  • 11:40 tgr@deploy1001: Synchronized php-1.32.0-wmf.18/includes/diff/DifferenceEngine.php: SWAT: Fix DifferenceEngine revision loading logic (T201218, T202454) (duration: 00m 49s)
  • 11:31 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: *.pensoft.net should be in wgCopyUploadsDomains whitelist instead of pensoft.net (T202832) (duration: 00m 48s)
  • 11:28 zfilipin@deploy1001: Synchronized multiversion/MWMultiVersion.php: SWAT: Fix "seperated" typo in MWMultiVersion.php file (T201491) (duration: 00m 48s)
  • 11:26 zfilipin@deploy1001: Synchronized wmf-config/abusefilter.php: SWAT: Enable AbuseFilter "block" on it.wikibooks (T202808) (duration: 00m 48s)
  • 11:19 marostegui: Deploy schema change on dbstore1002:s4
  • 11:16 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Remove sitewide and user CSS/JS editing from old groups Enforce that interface-admin is the only group that can edit non-own CSS/JS (T190015) (duration: 00m 48s)
  • 11:15 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove sitewide and user CSS/JS editing from old groups Enforce that interface-admin is the only group that can edit non-own CSS/JS (T190015) (duration: 00m 48s)
  • 10:58 marostegui: Deploy schema change on db2051 (s4 codfw master) with replication, this will generate lag on s4 codfw
  • 10:38 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 48s)
  • 10:38 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
  • 10:29 akosiaris: enable and run puppet on scb1001, restart services
  • 10:29 akosiaris: enable and run puppet on restbase2001, restart restbase
  • 10:19 akosiaris: disable puppet on hosts including service::configuration for merge of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454574/. See https://phabricator.wikimedia.org/P7486
  • 10:15 akosiaris: disable puppet on maps*, scb*, restbase* hosts for merge of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454574/
  • 10:09 moritzm: reimaging wtp2020 to stretch
  • 09:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 48s)
  • 09:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 48s)
  • 09:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 49s)
  • 09:20 addshore: deploy slot done
  • 09:16 moritzm: rebooting bast4002 for kernel security update
  • 09:16 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:455389 Wikidata: Use new item ID formatter for Q1 T201832 (duration: 00m 49s)
  • 08:59 marostegui: Drop sul, phadmin and phuser from dbstore1002
  • 08:56 marostegui: Drop eventlogcleaner user from dbstore1002
  • 08:53 moritzm: rebooting bast5001 for kernel security update
  • 08:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 48s)
  • 08:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 48s)
  • 08:31 moritzm: installing ca-certificates updates for jessie/stretch
  • 08:27 moritzm: reimaging wtp2017-wtp2019 to stretch
  • 08:22 marostegui: deploy schema change on db1087 with replication (lag on labs:s8 will be generated)
  • 08:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 (duration: 00m 48s)
  • 08:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 (duration: 00m 52s)
  • 08:05 gehel: restarting kartotherian / tilerator on maps* for nodejs upgrade
  • 07:59 moritzm: installing nodejs security updates on maps* hosts
  • 07:44 elukey: force remount of /mnt/hdfs on stat1005 (transport not connected errors)
  • 07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 (duration: 00m 50s)
  • 07:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 48s)
  • 07:10 moritzm: reimaging wtp2014-wtp2016 to stretch
  • 06:53 marostegui: Deploy schema change on labtestwiki - T89737
  • 06:52 moritzm: installing discover updates from stretch 9.5 point release
  • 06:50 marostegui: Deploy schema change on labtestwiki - T187089
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 00m 48s)
  • 06:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 (duration: 00m 49s)
  • 05:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 (duration: 00m 50s)
  • 05:16 moritzm: reimaging wtp2011-wtp2013 to stretch
  • 05:14 marostegui: Deploy schema change on db1066 (s2 primary master)
  • 02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 27 02:46:56 UTC 2018 (duration 10m 12s)
  • 02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 15m 13s)

2018-08-26

  • 05:53 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2088 - T202822 (duration: 00m 54s)

2018-08-24

  • 22:07 krinkle@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/WikimediaEvents/: I17452c (duration: 00m 50s)
  • 17:44 James_F: scap sync-dir php-1.32.0-wmf.18/extensions/WikimediaEvents/ No-op bump this code to avoid dirty deploy repo
  • 17:44 James_F: scap sync-file php-1.32.0-wmf.18/extensions/Echo/includes/gateway/UserNotificationGateway.php Hot-deploy of Ifdfa93059 to fix T202672 log error
  • 17:44 James_F: scap sync-file php-1.32.0-wmf.18/extensions/WikiEditor/modules/jquery.wikiEditor.dialogs.config.js Hot-deploy of I364ac118255 to fix missing icon
  • 15:07 marostegui: Deploy schema change on dbstore1002:s8
  • 13:45 marostegui: Deploy schema change on s8 codfw master with replication (this will generate lag on s8 codfw)
  • 12:33 akosiaris: upload apertium-kaz-tat to apt.wikimedia.org/jessie-wikimedia/main T199962
  • 12:33 akosiaris: upload apertium-eo-fr to apt.wikimedia.org/jessie-wikimedia/main T199962
  • 11:58 elukey: upgrade nodejs packages on aqs* for security upgrade (rolling restart of aqs daemon included)
  • 11:19 ema: powercycle cp3030, stuck rebooting T200445
  • 10:47 moritzm: reimage wtp2008-wtp2010 to stretch
  • 09:28 moritzm: reimage wtp2005-wtp2007 to stretch
  • 09:09 marostegui: Restart MySQL on db2035 (codfw s2 master) for mysql upgrade and pick up new mysql options after merging https://gerrit.wikimedia.org/r/455103
  • 09:04 marostegui: Deploy schema change on db1095:3312
  • 07:56 marostegui: Deploy schema change on db1122
  • 07:53 moritzm: reimage wtp2002-wtp2004 to stretch
  • 06:52 marostegui: Deploy schema change on db1074 with replication (this will generate lag on s2 labs)
  • 06:38 moritzm: reimage wtp2001 to stretch
  • 06:09 moritzm: imported php-igbinary, libzip5, php-apcu-bc to thirdparty/php72 (T202704)
  • 05:26 marostegui: Deploy schema change on db1076
  • 00:21 mutante: ms-be2043 - mount -a ; re-enabled puppet, running puppet

2018-08-23

  • 23:47 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202548 (duration: 01m 01s)
  • 23:43 Reedy: created wikilove tables on zh_yuewiki T202548
  • 23:42 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T201236 (duration: 00m 55s)
  • 23:37 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202599 (duration: 00m 53s)
  • 23:34 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202599 (duration: 00m 49s)
  • 23:28 Reedy: running `mwscript updateCollation.php --wiki=trwiki --previous-collation=uppercase` T184943
  • 23:28 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T184943 (duration: 00m 48s)
  • 23:23 reedy@deploy1001: Synchronized wmf-config/flaggedrevs.php: T202139 (duration: 00m 48s)
  • 23:19 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202680 (duration: 00m 48s)
  • 23:16 reedy@deploy1001: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 23:16 reedy@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 01s)
  • 23:15 reedy@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 00s)
  • 22:53 XioNoX: Add v4 session in eqdfw to AS8075
  • 22:46 XioNoX: Add v6 session in eqsin to 10089
  • 22:39 XioNoX: deactivating down AMSIX peer (no reply to emails)
  • 22:36 XioNoX: clear peer 80.249.208.51 AS 8708 on cr2-esams
  • 22:31 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.18
  • 22:27 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/FlaggedRevs/backend/FlaggedRevs.class.php: Follow-up 413e11e2f: Skip CurrentRevisionCallback if not stabilized T202659 (duration: 00m 56s)
  • 19:34 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: revert all wikis to 1.32.0-wmf.18
  • 19:34 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 5, onthisday is timing out. Just push this through.. (duration: 03m 27s)
  • 19:31 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 5, onthisday is timing out. Just push this through..
  • 19:26 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.18
  • 18:54 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 4, onthisday is timing out. Just push this through.. (duration: 04m 39s)
  • 18:50 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 4, onthisday is timing out. Just push this through..
  • 18:48 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 3, onthisday is timing out. Just push this through.. (duration: 03m 59s)
  • 18:44 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 3, onthisday is timing out. Just push this through..
  • 18:41 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 3, onthisday is failing (duration: 03m 53s)
  • 18:37 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.18 (duration: 00m 54s)
  • 18:37 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 3, onthisday is failing
  • 18:37 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 2, onthisday is failing (duration: 06m 30s)
  • 18:37 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.18
  • 18:33 XioNoX: Equinix updated their MAC filter, IX sessions up - T196941
  • 18:33 thcipriani: starting train a little early to play catchup
  • 18:32 RoanKattouw: Deployed patch for T199993
  • 18:30 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 2, onthisday is failing
  • 18:30 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage (duration: 03m 54s)
  • 18:30 XioNoX: remove MAC filter workaround on cr2-eqdfw - T196941
  • 18:28 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/FlaggedRevs/backend/FlaggedRevs.class.php: Fix incorrect return value in CurrentRevisionCallback T202580 (duration: 00m 54s)
  • 18:28 RoanKattouw: Deployed patch for T151910
  • 18:26 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage
  • 18:15 otto@deploy1001: Finished deploy [analytics/superset/deploy@de75f23]: 0.26.3 to analytics-tool1003 with requirements setup (duration: 00m 03s)
  • 18:15 otto@deploy1001: Started deploy [analytics/superset/deploy@de75f23]: 0.26.3 to analytics-tool1003 with requirements setup
  • 18:09 otto@deploy1001: Finished deploy [analytics/superset/deploy@de75f23]: 0.26.3 to analytics-tool1003 (duration: 00m 36s)
  • 18:08 otto@deploy1001: Started deploy [analytics/superset/deploy@de75f23]: 0.26.3 to analytics-tool1003
  • 17:28 XioNoX: enable v4 neighbors on cr2-eqdfw - T196941
  • 17:16 XioNoX: enable v6 neighbors on cr2-eqdfw - T196941
  • 17:01 XioNoX: powering off cr1-eqdfw - T196941
  • 16:49 filippo@neodymium: conftool action : set/pooled=yes; selector: name=restbase2003.codfw.wmnet
  • 16:22 godog: depool and cassandra-drain restbase2003, then reboot
  • 16:20 filippo@neodymium: conftool action : set/pooled=no; selector: name=restbase2003.codfw.wmnet
  • 16:17 XioNoX: deactivating IX/Transit links on cr1-eqdfw - T196941
  • 16:15 godog: upgrade hp raid firmware on restbase2003 - T201804 T141756
  • 15:15 marostegui: Deploy schema change on db1090:3312
  • 15:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 54s)
  • 15:14 jynus: upgrading db2089
  • 15:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 (duration: 00m 55s)
  • 15:02 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113 (duration: 00m 54s)
  • 14:47 jynus: upgrading db1113
  • 14:37 otto@deploy1001: Finished deploy [analytics/turnilo/deploy@50a8845]: Deploying 1.7.2 to analytics-tool1002 with dep fixes (duration: 00m 08s)
  • 14:37 otto@deploy1001: Started deploy [analytics/turnilo/deploy@50a8845]: Deploying 1.7.2 to analytics-tool1002 with dep fixes
  • 14:36 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113 (duration: 00m 57s)
  • 14:26 otto@deploy1001: Finished deploy [analytics/turnilo/deploy@240b357]: Deploying 1.7.2 to analytics-tool1002 (duration: 05m 07s)
  • 14:21 otto@deploy1001: Started deploy [analytics/turnilo/deploy@240b357]: Deploying 1.7.2 to analytics-tool1002
  • 13:39 jynus: upgrading dbstore1001
  • 13:38 marostegui: Deploy schema change on db1103:3312
  • 13:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 (duration: 00m 54s)
  • 13:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 (duration: 00m 54s)
  • 13:29 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T198396 Revert "Limit page creation and edit rate on Wikidata" gerrit:454785 (duration: 00m 56s)
  • 13:06 jynus: upgrading dbstore200[12], will take a while due to ongoing alter tables
  • 12:34 moritzm: install Java updates on Hadoop nodes
  • 12:06 moritzm: created new archiva-deployers LDAP group (T200454)
  • 12:00 marostegui: Deploy schema change on db1105:3312
  • 12:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 (duration: 00m 55s)
  • 11:56 addshore: SWAT done
  • 11:50 addshore@deploy1001: Synchronized wmf-config: T199800 gerrit:Cleanup wikdiff2 mobile moved paragraph config (duration: 00m 55s)
  • 11:40 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202555 Require autoconfirmed status to edit the 828 namespace at es.wikibooks (duration: 00m 56s)
  • 11:35 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202065 Switch 'wikidata-staff' add/remove 'interface-admin' for own account (duration: 00m 55s)
  • 11:19 moritzm: installing shared-mime-info updates from stretch 9.5 point release
  • 11:18 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: T199800 Enable moved paragrah detection everywhere (duration: 00m 55s)
  • 11:01 marostegui: Deploy schema change on dbstore1002:s2
  • 11:00 gehel: new SSL certs / tlsproxy deployed on elastic nodes - T198351
  • 10:56 marostegui: Deploy schema change on s2 codfw masters (db2035) with replication - this will generate lag on s2 codfw
  • 10:35 moritzm: installing openldap updates from stretch 9.5 point release
  • 10:27 akosiaris: reboot analytics-tool* hosts for the IP renumbering change T202559
  • 10:27 akosiaris: purge rec dns caches for analytics-tool* hosts T202559
  • 10:15 elukey: restart druid-broker on druid100[4-5] due to unresponsiveness (still unclear why)
  • 10:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 (duration: 00m 54s)
  • 09:59 marostegui: Deploy schema change on s6 primary master (db1061)
  • 09:50 moritzm: installing debdeploy update
  • 09:40 moritzm: restarting etherpad-lite for nodejs security update
  • 09:29 moritzm: installing nodejs security updates on wtp* (Parsoid servers)
  • 09:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 (duration: 00m 55s)
  • 09:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 (duration: 00m 56s)
  • 09:17 gehel: starting to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/444610 as part of T198351, including regeneration of SSL certs. Disabling puppet on elastic* during the operation
  • 08:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 (duration: 00m 55s)
  • 08:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 (duration: 00m 55s)
  • 08:42 addshore: done with deploy window
  • 08:41 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: T201832 Stuff for new link formatter on testwikidata gerrit:454510 (duration: 00m 55s)
  • 08:39 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T201832 Stuff for new link formatter on testwikidata gerrit:454510 gerrit:452365 gerrit:454764 (duration: 00m 56s)
  • 07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 (duration: 00m 55s)
  • 07:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 (duration: 00m 56s)
  • 07:30 moritzm: installing openssl updates
  • 07:23 marostegui: Drop blob_orphans and blob_tracking from s3 on codfw (lag will be generated on codfw) T59186
  • 07:12 marostegui: Drop blob_orphans and blob_tracking from s1 T59186
  • 06:47 marostegui: Drop blob_orphans and blob_tracking from s7 T59186
  • 06:31 marostegui: Enable semi-sync on es2 - T202364
  • 06:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3316 (duration: 00m 54s)
  • 06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316 (duration: 00m 55s)
  • 05:57 marostegui: Drop blob_orphans and blob_tracking from s2 T59186
  • 05:50 marostegui: Drop blob_orphans and blob_tracking from s4 T59186
  • 05:47 marostegui: Drop blob_orphans and blob_tracking from s6 T59186
  • 05:19 marostegui: Drop blob_orphans and blob_tracking from s5 T59186
  • 04:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 (duration: 00m 55s)
  • 04:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 (duration: 00m 57s)
  • 03:14 ejegg: updated payments-wiki from 268539c2bd to 763e14b6df
  • 03:06 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Aug 23 03:06:25 UTC 2018 (duration 10m 22s)
  • 02:56 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 13m 27s)
  • 02:34 ejegg: updated CiviCRM from affd1b5e9a to 4e0dcf7cb4
  • 02:29 ejegg: updated SmashPig standalone deploy from 3603e191a3 to cd46beefb7
  • 02:23 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 09m 10s)
  • 00:36 twentyafterfour: phabricator will be down momentarily for apache restart, downtime scheduled in icinga
  • 00:08 twentyafterfour: preparing phabricator update

2018-08-22

  • 23:17 thcipriani@deploy1001: Synchronized wmf-config/WikibaseSearchSettings.php: SWAT: Use proper data types for indexing items T199884 (duration: 00m 55s)
  • 23:15 SMalyshev: repooled wdq3 after recovery
  • 22:45 SMalyshev: temp depooled wdq3 to make it catch up faster
  • 21:57 mutante: releases1001/2001: sudo find /srv/org/wikimedia/releases/wikidiff2 -name wikidiff2\* -exec chown :releasers-wikidiff2 {} \; (T202473)
  • 20:37 ladsgroup@deploy1001: Finished deploy [ores/deploy@8ff4da1]: Update ORES to use git lfs for model T197097 (duration: 31m 42s)
  • 20:32 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@f7fa1df]: Update mobileapps to 141ff20 (T202105 T202237) (duration: 03m 43s)
  • 20:28 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@f7fa1df]: Update mobileapps to 141ff20 (T202105 T202237)
  • 20:26 thcipriani@deploy1001: Synchronized php: Revert "Group1 wikis to 1.32.0-wmf.18" (duration: 00m 54s)
  • 20:24 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Revert "Group1 wikis to 1.32.0-wmf.18"
  • 20:10 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@8dd6d74]: bump to master, msearch fixes (duration: 03m 41s)
  • 20:07 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@8dd6d74]: bump to master, msearch fixes
  • 20:05 ladsgroup@deploy1001: Started deploy [ores/deploy@8ff4da1]: Update ORES to use git lfs for model T197097
  • 19:55 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.18 (duration: 00m 54s)
  • 19:54 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.18
  • 19:38 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/Flow/includes/TemplateHelper.php: Work around exception in DifferenceEngine::showDiffStyle() T202454 (duration: 00m 57s)
  • 19:01 anomie: re-running populateContentTables.php on cawiki for T183488
  • 18:14 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@00671e6]: Add prometheus-client based metrics export (duration: 04m 22s)
  • 18:10 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@00671e6]: Add prometheus-client based metrics export
  • 18:10 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@ebb466b]: Add prometheus-client based metrics export (duration: 01m 36s)
  • 18:08 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@ebb466b]: Add prometheus-client based metrics export
  • 17:50 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@6fdae90]: updates to support msearch daemon (duration: 04m 16s)
  • 17:45 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@6fdae90]: updates to support msearch daemon
  • 17:31 jforrester@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/JsonConfig/includes/JCSingleton.php: T202469 fix UBN trainblocker for wmf.18 (duration: 00m 56s)
  • 17:21 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@32a81be]: Revisit jobs concurrencies T202107 (duration: 00m 49s)
  • 17:20 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@32a81be]: Revisit jobs concurrencies T202107
  • 17:19 Jeff_Green: authdns-update to deploy commit If54277153b6
  • 16:58 godog: start backfilling metrics from graphite1001 into graphite1004 - T196484
  • 16:51 godog: repair on ms-be2042 sdk/sdn - T199198
  • 16:51 godog: correction, ms-be2040 - T199198
  • 16:48 godog: repair on ms-be2020 sdh/sdc - T199198
  • 16:21 ema: prometheus-trafficserver-exporter 0.0.2-1 uploaded to stretch-wikimedia T202381
  • 16:06 marostegui: Deploy schema change on db1096:3316
  • 16:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 (duration: 00m 55s)
  • 16:02 XioNoX: add static NAT for 208.80.155.15 - T202520
  • 15:54 XioNoX: Update NAT for 208.80.152.231 - T202536
  • 14:57 elukey: upload archiva 2.2.3-2 to stretch-wikimedia
  • 14:57 XioNoX: load eqiad-1534946573 on pfw3-eqiad - T202537
  • 14:56 elukey: update pcc's facts
  • 14:55 XioNoX: load codfw-1534946573 on pfw3-codfw - T202537
  • 14:24 marostegui: Deploy schema change on dbstore1002
  • 14:11 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1011 (duration: 00m 55s)
  • 14:03 moritzm: installing openssl updates
  • 13:15 addshore@deploy1001: Synchronized wmf-config: BETA ONLY (1 wikidata patch) (duration: 00m 56s)
  • 13:14 addshore@deploy1001: sync-file aborted: BETA ONLY (1 wikidata patche) (duration: 00m 00s)
  • 13:07 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set default for wgMultiContentRevisionSchemaMigrationStage T197816 (duration: 00m 56s)
  • 12:59 marostegui: Deploy schema change on s6 codfw masters (db2039) this will generate lag on s6 codfw
  • 12:46 addshore@deploy1001: Synchronized wmf-config: BETA ONLY (2 wikidata patches) (duration: 00m 55s)
  • 12:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 (duration: 00m 57s)
  • 12:15 marostegui: Deploy schema change on db1070 (s5 primary master) - T67448 T114117 T51191
  • 11:40 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2069 - T201603 (duration: 00m 56s)
  • 11:37 marostegui: Deploy schema change on db1113:3315
  • 11:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 (duration: 00m 58s)
  • 10:43 elukey: upload archiva 2.2.3-1 to stretch-wikimedia/main - T192639
  • 10:31 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #5 (duration: 06m 15s)
  • 10:30 Amir1: deploying ores gerrit:454283 to beta (T197097)
  • 10:25 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #5
  • 10:25 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #4 (duration: 08m 31s)
  • 10:17 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #4
  • 10:16 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #3 (duration: 08m 28s)
  • 10:08 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #3
  • 10:08 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c]: Expand CSS end points - T202105 (duration: 13m 33s)
  • 09:59 jynus: restarting backups on codfw due to grant issue
  • 09:54 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c]: Expand CSS end points - T202105
  • 09:53 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c]: Expand CSS end points - T202425 (duration: 03m 25s)
  • 09:51 moritzm: uploaded clustershell 1.8-1~deb9u1 to apt.wikimedia.org/stretch-wikimedia
  • 09:49 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c]: Expand CSS end points - T202425
  • 09:49 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c] (dev-cluster): Expand CSS end points (duration: 03m 35s)
  • 09:45 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c] (dev-cluster): Expand CSS end points
  • 09:34 moritzm: installing nodejs security updates on restbase*
  • 09:19 jynus: reimage es1011 to stretch
  • 09:06 addshore@deploy1001: Synchronized php-1.32.0-wmf.18/includes: T202483 The BlobStoreFactory constructor needs an LBFactory (duration: 01m 18s)
  • 08:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1011 (duration: 00m 56s)
  • 08:46 marostegui: Stop replication on x1 codfw master (db2034) for data fixes - T104459 T201603
  • 08:40 moritzm: installing clamav security updates on mendelevium (bundling libmspack security update along)
  • 08:39 ema: repool eqiad for front-edge traffic
  • 07:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Change x1 eqiad weights to give more weight to the most powerful server (duration: 00m 55s)
  • 07:06 moritzm: installing apache update on deploy1001/planet1001
  • 06:57 moritzm: imported libargon-2.1 for thirdparty/php7.2 repo (T202478)
  • 06:30 jynus: finished es2 switchover T202364
  • 06:20 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Switchover es2 master eqiad from es1011 to es1015 (duration: 00m 55s)
  • 06:12 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es2 (duration: 01m 05s)
  • 06:11 jynus: depool es2 to failover master from es1011 to es1015
  • 05:53 marostegui: Start topology changes for es2 failover - T202364
  • 05:53 jynus: restarting heartbeat on es1011
  • 03:06 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 22 03:06:38 UTC 2018 (duration 10m 29s)
  • 02:56 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 15m 36s)
  • 02:22 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 09m 06s)
  • 01:10 mutante: puppetdb2001 - dumping postgres db to local backup path

2018-08-21

  • 23:42 thcipriani@deploy1001: Synchronized wmf-config: SWAT: Switch entity reference type indexing from opt-in to opt-out T199884 (duration: 00m 57s)
  • 22:49 eileen_: civicrm revision changed from 8910c34502 to affd1b5e9a, config revision is 4c055834dd
  • 22:32 mutante: vega - cd /srv/org/wikimedia/TransparencyReport ; git reset --hard origin ; puppet agent -tv (to fix transparency.wikimedia.org content repo and puppet run, just like on bromine)
  • 22:29 mutante: bromine - last log line refers to /srv/org/wikimedia/TransparencyReport (transparency.wikimedia.org)
  • 22:28 mutante: bromine - git reset --hard origin ; git checkout master ; git pull origin ; puppet agent -tv (to fix broken puppet run due to unclean git repo.. unknown why it broke)
  • 21:00 moritzm: rebooting mw2232 for some tests
  • 20:17 herron: re-enabling puppet agents after puppetmaster apache update
  • 20:11 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Group0 to 1.32.0-wmf.18
  • 20:07 herron: temporarily disabling puppet agents during puppetmaster apache updates
  • 19:49 thcipriani@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.18 and rebuild l10n cache (duration: 34m 54s)
  • 19:14 thcipriani@deploy1001: Started scap: testwiki to php-1.32.0-wmf.18 and rebuild l10n cache
  • 19:11 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.13 (duration: 01m 41s)
  • 19:08 mutante: deploy1001 - chmod -R g+w /srv/mediawiki-staging/php-1.32.0-wmf.13/.git
  • 18:58 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.12 (duration: 03m 15s)
  • 18:52 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.10 (duration: 07m 37s)
  • 18:17 volans: restarting ircecho on einsteinium - T202314
  • 17:58 volans: restarting ircecho on einsteinium - T202314
  • 17:13 thcipriani: starting branch cut for 1.32.0-wmf.18
  • 16:32 XioNoX: enabling ae1 on asw-a-eqiad then ae1 on cr1-eqiad - T202075
  • 16:27 godog: upgrade hp raid firmware on ms-be2020 - T141756
  • 16:21 XioNoX: disable ae1 on asw-a-eqiad - T202075
  • 16:18 XioNoX: disable ae1 on cr1-eqiad - T202075
  • 16:17 papaul: replacing PEM2 on cr2-codfw
  • 15:38 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096 (duration: 00m 50s)
  • 15:24 ejegg: restarted CiviCRM deduplication jobs
  • 15:10 moritzm: installing apache updates from stretch 9.5 point release
  • 14:58 moritzm: rolling restart of proton* to pick up nodejs security update
  • 13:44 jynus: stop db1096 (both s5 an s6) for upgrade
  • 13:36 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096 (duration: 00m 49s)
  • 13:27 jynus: stop db2090 for upgrade
  • 12:38 ema: power-cycle ms-be2020: multiple alerts, no ssh access, I/O errors in console
  • 12:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 (duration: 00m 50s)
  • 12:31 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 (duration: 00m 50s)
  • 12:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 49s)
  • 12:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 (duration: 00m 49s)
  • 12:13 moritzm: rolling restart of services in scb/eqiad to pick up the nodejs security update
  • 12:05 jynus: stop es1015 for upgrade
  • 11:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1015 (duration: 00m 50s)
  • 11:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 (duration: 00m 50s)
  • 11:16 zeljkof: EU SWAT finished
  • 11:16 zfilipin@deploy1001: Synchronized wmf-config/Wikibase-production.php: SWAT: Update several Wikidata-related configs (duration: 00m 50s)
  • 11:05 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add IP ranges for ptWiki editathon (T202354) (duration: 00m 52s)
  • 09:58 moritzm: installing libxcursor security updates
  • 09:46 moritzm: installing libxml2 security updates on trusty
  • 09:27 moritzm: installing mutt security updates
  • 09:16 moritzm: installing jetty9 security updates
  • 08:37 marostegui: Rename blob_tracking and blob_orphans on db1089 - T59186
  • 07:55 marostegui: Slowly reload haproxies on dbproxy10XX - T201021
  • 06:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 (duration: 00m 50s)
  • 05:12 marostegui: Stop puppet on dbproxy1006 to do some haproxy logging tests - https://phabricator.wikimedia.org/T201021
  • 05:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 (duration: 00m 51s)
  • 04:54 marostegui: Set max_connections back from 800 to 500 on db1073 - T188589
  • 02:47 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: rm unused Id5f5295d (duration: 00m 50s)
  • 02:43 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: rm unused I741b16452 (duration: 00m 56s)
  • 02:39 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 21 02:39:12 UTC 2018 (duration 10m 18s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 08m 32s)
  • 01:35 aaron@deploy1001: Synchronized wmf-config/mc.php: Enable broadcasted mcrouter operations for all wikis (duration: 00m 51s)
  • 00:07 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@9fbf9fa]: Don't clean up the queue size counters (duration: 00m 46s)
  • 00:07 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@9fbf9fa]: Don't clean up the queue size counters

2018-08-20

  • 23:09 twentyafterfour: Finished evening SWAT
  • 23:08 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/454083/ Phab Task: T202065 (duration: 00m 51s)
  • 23:06 twentyafterfour: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/454083/ deployed to mwdebug1001
  • 22:53 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@c947567]: Use timing for queue sizes as a workaround to grafana not plotting gauges (duration: 00m 45s)
  • 22:52 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@c947567]: Use timing for queue sizes as a workaround to grafana not plotting gauges
  • 22:29 eileen: civicrm revision changed from 3f6dd6c9a6 to 8910c34502, config revision is d787269882
  • 22:27 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@541932a]: Make the queue sized report stats once a second T202107 (duration: 00m 46s)
  • 22:26 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@541932a]: Make the queue sized report stats once a second T202107
  • 22:13 mutante: ganeti1001 - creating 3 new VMs for analytics-tools
  • 21:08 eileen: civicrm revision is 3f6dd6c9a6, config revision is d787269882
  • 21:07 fdans@deploy1001: Finished deploy [analytics/refinery@f59ce0c]: deploying changes to virtualpageview spam site filtering (duration: 10m 17s)
  • 20:56 fdans@deploy1001: Started deploy [analytics/refinery@f59ce0c]: deploying changes to virtualpageview spam site filtering
  • 20:43 aaron@deploy1001: Synchronized wmf-config/mc.php: Set "cluster" parameter for mcrouter broadcast routing prefixes (duration: 00m 50s)
  • 20:36 arlolra: Updated Parsoid to 129d71f (T130224, T199926)
  • 20:35 mbsantos@deploy1001: Finished deploy [mobileapps/deploy@cae24fe]: Update mobileapps to 95e976d (T202105) (duration: 06m 19s)
  • 20:29 mbsantos@deploy1001: Started deploy [mobileapps/deploy@cae24fe]: Update mobileapps to 95e976d (T202105)
  • 20:26 arlolra@deploy1001: Finished deploy [parsoid/deploy@44aa5e8]: Updating Parsoid to 129d71f (duration: 11m 29s)
  • 20:14 arlolra@deploy1001: Started deploy [parsoid/deploy@44aa5e8]: Updating Parsoid to 129d71f
  • 19:35 jforrester@deploy1001: Finished scap: i18n sync for WikimediaMessages clean-up following T202095 (duration: 31m 12s)
  • 19:04 jforrester@deploy1001: Started scap: i18n sync for WikimediaMessages clean-up following T202095
  • 19:00 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/WikimediaMessages/: SWAT T202095 Reinstate "Rename global OTRS-member group to otrs-member" (will need scap sync later) (duration: 00m 49s)
  • 18:56 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/TimedMediaHandler/: SWAT T202208Fix regression: double-reg of file types and incorrect mp4 (duration: 00m 52s)
  • 18:44 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T201783 T202014 Add Contact link to footers of fr,ruwiki (duration: 00m 50s)
  • 18:36 gehel@deploy1001: Finished deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater (duration: 10m 25s)
  • 18:32 jforrester@deploy1001: Synchronized wmf-config/throttle.php: SWAT T202003 Add throttle exemption for 24 August (duration: 00m 51s)
  • 18:31 volans: restarted ircecho on einsteinium
  • 18:26 gehel@deploy1001: Started deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater
  • 18:25 gehel@deploy1001: Finished deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 32s)
  • 18:24 gehel@deploy1001: Started deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater (wdqs1009 only)
  • 18:05 volans: restarting icinga, re-occurrence of T196336
  • 17:44 chasemp: install iotop on stat1004
  • 17:08 XenoRyet: updated payments-wiki from 360bea8331 to 268539c2bd
  • 16:59 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@4989231]: Create metrics to track the actual concurrent job executions T202107 (duration: 00m 55s)
  • 16:58 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@4989231]: Create metrics to track the actual concurrent job executions T202107
  • 14:14 moritzm: rolling upgrade of scb in codfw to latest nodejs security update
  • 13:46 moritzm: installing jetty9 security updates
  • 13:00 marostegui: Increase max_connections from 500 to 800 on db1073 to triage issues - T188589
  • 12:51 jynus: reload dbproxy1005
  • 12:29 moritzm: upgrading scb2006 to latest nodejs along with rolling restarts of node-based services
  • 12:02 zeljkof: EU SWAT finished
  • 12:01 arturo: T202261 extend icinga downtime 1D for cloudcontrol1003.wikimedia.org, cloudcontrol1004.wikimedia.org, clounet1003.eqiad.wmnet, cloudnet1004.eqiad.wmnet neutron not properly syncing with agents
  • 11:56 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Adding Chinese Wikiversitys logos (T202127) (duration: 00m 50s)
  • 11:50 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos for lfnwiki (T202228) (duration: 00m 50s)
  • 11:41 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for lfnwiki (T202228) (duration: 00m 50s)
  • 11:36 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Rollbacker User Group at ru.wikiquote (T200201) (duration: 00m 53s)
  • 11:25 arturo: T202261 icinga downtime 1h for cloudcontrol1003.wikimedia.org, cloudcontrol1004.wikimedia.org, clounet1003.eqiad.wmnet, cloudnet1004.eqiad.wmnet previous to patch merge
  • 11:07 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow all bureaucrats to remove interface-admin (duration: 00m 54s)
  • 11:01 arturo: T202261 disabled puppet in cloudcontrol1003.wikimedia.org, cloudcontrol1004.wikimedia.org, clounet1003.eqiad.wmnet, cloudnet1004.eqiad.wmnet
  • 10:31 moritzm: rebooting mw1261 for kernel update/some tests
  • 10:23 moritzm: rebooting mw2234 for some kernel tests
  • 10:07 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf2 to apt.wikimedia.org
  • 09:31 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf2+jessie to apt.wikimedia.org
  • 08:50 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request (duration: 06m 29s)
  • 08:43 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request
  • 08:43 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request - T201974 (duration: 18m 17s)
  • 08:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 (duration: 00m 50s)
  • 08:31 moritzm: installing systemd updates from stretch 9.5 point release
  • 08:25 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request - T201974
  • 08:24 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request (duration: 06m 02s)
  • 08:18 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request
  • 08:16 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request - T201974 (duration: 04m 16s)
  • 08:15 marostegui: Deploy schema change on db1100 to check for regressions
  • 08:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 50s)
  • 08:12 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request - T201974
  • 08:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db1100 (duration: 01m 02s)
  • 07:29 gehel: resetting management card on elastic1022
  • 06:57 volans: reset-failed debmonitor failed session on ms-be2024 ^^^^
  • 02:47 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 20 02:47:17 UTC 2018 (duration 10m 22s)
  • 02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 14m 23s)

2018-08-17

  • 16:24 gehel: disabling systemd state check for elastic eqiad until T202120 is fixed
  • 14:38 moritzm: rebooting elnath for some kernel tests
  • 13:02 jynus: stopping db2084 for upgrade
  • 12:49 imarlier@deploy1001: Finished deploy [performance/coal@aff3793]: (no justification provided) (duration: 01m 06s)
  • 12:48 imarlier@deploy1001: Started deploy [performance/coal@aff3793]: (no justification provided)
  • 12:46 gehel: rebooting relforge for JVM and kernel upgrade
  • 12:36 moritzm: installing openjdk-8 security updates on elastic* in codfw (eqiad already has it via the recent reimage to stretch)
  • 12:30 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097 (duration: 00m 53s)
  • 12:07 gehel: repooling wdqs2003, catched up on updates
  • 10:58 moritzm: rebooting mw2152-mw2162 for kernel security update (also bundling wikidiff and apache updates)
  • 10:35 moritzm: powercycling mw2286
  • 09:47 moritzm: rebooting mw2243-mw2290 for kernel security update (also bundling wikidiff and apache updates)
  • 09:03 gehel: depool wdqs2003 to catch up on updates
  • 08:29 jynus: stopping db2073 for upgrade
  • 08:13 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097 (duration: 00m 54s)
  • 08:06 gehel: silencing systemd check as mjolnir is flapping
  • 07:59 gehel: restarting mjolnir on all elastic / cirrus nodes
  • 07:42 jynus: stopping db2085 for upgrade
  • 07:41 moritzm: rebooting mw2220-mw2242 for kernel security update (also bundling wikidiff and apache updates)
  • 00:42 krinkle@deploy1001: Synchronized wmf-config/: rm StartProfiler.php / Ia83751 / T201782 (duration: 00m 50s)
  • 00:39 krinkle@deploy1001: Synchronized docroot/noc/: Ia83751 (duration: 00m 50s)

2018-08-16

  • 23:41 ejegg: re-activated CiviCRM scheduled jobs
  • 23:34 catrope@deploy1001: Synchronized php-1.32.0-wmf.16/skins/MinervaNeue/resources/skins.minerva.content.styles.images/magnifying-glass.svg: Correct MinervaNeue search icon (T199000) (duration: 00m 51s)
  • 23:18 ejegg: disabled CiviCRM scheduled jobs for version upgrade
  • 23:15 eileen: civicrm revision changed from b60ceb3d2f to 3f6dd6c9a6, config revision is 74ae910b0e
  • 22:32 XioNoX: re-activating BGP sessions between cr1/2-ulsfo and the office's router2
  • 22:25 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@e70f9d5]: retry git fat init fail (duration: 00m 14s)
  • 22:24 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@e70f9d5]: retry git fat init fail
  • 22:24 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@e70f9d5]: dont override the spark sharelib (duration: 00m 11s)
  • 22:24 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@e70f9d5]: dont override the spark sharelib
  • 21:56 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@651904b]: use spark 2.3.0, oozie still doesnt like 2.3.1 (duration: 00m 16s)
  • 21:56 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@651904b]: use spark 2.3.0, oozie still doesnt like 2.3.1
  • 21:56 ebernhardson@deploy1001: deploy aborted: try again after git-fat init fail (duration: 00m 01s)
  • 21:55 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: try again after git-fat init fail
  • 21:51 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: try again after git-fat init fail (duration: 00m 18s)
  • 21:50 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: try again after git-fat init fail
  • 21:46 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0 (duration: 04m 56s)
  • 21:44 mutante: rebooting cobalt (gerrit.wikimedia.org) for kernel upgrade
  • 21:41 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0
  • 21:39 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0 (duration: 00m 14s)
  • 21:39 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0
  • 21:29 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.1 (duration: 00m 18s)
  • 21:28 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.1
  • 21:23 thcipriani: contint1001 - installing jenkins upgrade
  • 21:08 mutante: contint2001 - installing jenkins upgrade
  • 21:05 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail (duration: 00m 16s)
  • 21:04 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail
  • 21:01 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail (duration: 00m 06s)
  • 21:01 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail
  • 21:00 mutante: releases1001 - installing package upgrade like on releases2001 before, scheduling downtime, reboot
  • 20:53 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: (no justification provided) (duration: 00m 02s)
  • 20:53 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: (no justification provided)
  • 20:50 mutante: releases2001 - rebooting
  • 20:48 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x (duration: 00m 46s)
  • 20:48 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x
  • 20:47 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x (duration: 01m 33s)
  • 20:46 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x
  • 20:38 mutante: releases2001: upgrading openjdk, systemd, jenkins
  • 20:33 mutante: releases1001/2001: upgrading apache2 packages
  • 20:26 mutante: gerrit2001 - scheduled downtime, rebooting
  • 19:54 mutante: phab1002 closing idle root screen that was used for rsyncing repos
  • 19:15 thcipriani: clearing gerrit accounts cache
  • 19:11 twentyafterfour: twentyafterfour and thcipriani performing online maintenance on gerrit All-Users repo
  • 19:10 legoktm: T201314 mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'EricEnfermero' 'Larry Hockett' --ignorestatus
  • 19:06 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@9fb53c4]: (no justification provided) (duration: 00m 17s)
  • 19:05 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@9fb53c4]: (no justification provided)
  • 18:57 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@040690e]: coordinator.properties should reference a coordinator, not bundle (duration: 00m 18s)
  • 18:56 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@040690e]: coordinator.properties should reference a coordinator, not bundle
  • 18:52 arlolra: Updated Parsoid to dbbad6a (T201115)
  • 18:47 arlolra@deploy1001: Finished deploy [parsoid/deploy@59f6585]: Updating Parsoid to dbbad6a (duration: 09m 51s)
  • 18:45 gehel: reimage of elasticsearch eqiad completed - T198391 / T193649
  • 18:37 arlolra@deploy1001: Started deploy [parsoid/deploy@59f6585]: Updating Parsoid to dbbad6a
  • 18:25 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5d87cc0]: now without the shebang (duration: 00m 17s)
  • 18:25 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5d87cc0]: now without the shebang
  • 18:22 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES filters in PageTriage on testwiki (duration: 00m 50s)
  • 18:16 catrope@deploy1001: Synchronized wmf-config/throttle.php: Throttle exemptions for cswiki (T202038) (duration: 00m 53s)
  • 18:14 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@0a704b6]: push new python dependency handling (duration: 03m 49s)
  • 18:10 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@0a704b6]: push new python dependency handling
  • 18:06 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@d994cb9]: push new python dependency handling (duration: 00m 05s)
  • 18:06 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@d994cb9]: push new python dependency handling
  • 17:59 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@122080c]: push new python dependency handling (duration: 00m 20s)
  • 17:59 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@122080c]: push new python dependency handling
  • 17:57 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@fec00bc]: Push updated transfer-to-es oozie job (duration: 00m 08s)
  • 17:57 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@fec00bc]: Push updated transfer-to-es oozie job
  • 17:36 gehel: reimaging elastic1029
  • 16:12 gehel: banning, depooling and shutting down elastic1029 for memory replacement - T201991
  • 15:55 jynus: stopping db2034 for upgrade
  • 15:53 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2019 (duration: 00m 51s)
  • 15:34 jynus: stopping es2019 for upgrade
  • 15:29 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2018, depool es2019 (duration: 00m 50s)
  • 15:16 bblack: restarting pybal on lvs1016 - T201694
  • 15:13 cmjohnson1: lvs1016 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 15:09 bblack: stopping pybal on lvs1016 to fail traffic to lvs1006 for T201694
  • 15:07 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp107[78]\.eqiad\.wmnet
  • 15:05 cmjohnson1: cp107[78] moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 15:05 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp107[78]\.eqiad\.wmnet
  • 15:04 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp107[56]\.eqiad\.wmnet
  • 15:03 cmjohnson1: cp1076 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 15:02 cmjohnson1: cp1075 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 15:01 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp107[56]\.eqiad\.wmnet
  • 15:01 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp107[56]\.eqiad\.wmnet
  • 15:00 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp107[56]\.eqiad\.wmnet
  • 15:00 jynus: stopping es2018 for upgrade
  • 14:58 cmjohnson1: torrelay1001 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:57 cmjohnson1: ms-be1040 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:56 cmjohnson1: dbproxy1013 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:54 cmjohnson1: labstore1009 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:51 papaul: shutting down mw2184 for maintenance
  • 14:50 cmjohnson1: db1066 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:49 cmjohnson1: db1118 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:46 cmjohnson1: db1116 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:44 cmjohnson1: labstore1008 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:43 bblack@neodymium: conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org
  • 14:43 cmjohnson1: dbproxy1012 moving uplink from asw2-a-eqiad to asw-a-eqiad
  • 14:41 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@166eafa]: Update mobileapps to a808c9d (T201979) (duration: 06m 03s)
  • 14:40 cmjohnson1: dns1001 moving uplink from asw2-a eqiad to asw-a-eqiad
  • 14:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=dns1001.wikimedia.org
  • 14:38 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2018 (duration: 00m 55s)
  • 14:34 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@166eafa]: Update mobileapps to a808c9d (T201979)
  • 14:33 cmjohnson1: cloudelastic1001 moving uplink from asw2-a eqiad to asw2-a2
  • 14:27 cmjohnson1: lvs1015 moving cross connect from asw2-a2 to asw2-a5 T201694
  • 14:25 XioNoX: starting moving asw2-a-eqiad servers' uplinks for T201694
  • 14:22 gehel: repooling wdqs[12]003
  • 14:11 moritzm: rebooting labtestweb2001 for kernel security update
  • 13:54 moritzm: rebooting labtestvirt2003 for kernel security update
  • 13:46 moritzm: rebooting labtestservices2002/2003 for kernel security update
  • 13:31 moritzm: rebooting seaborgium for kernel security update
  • 13:15 moritzm: rebooting serpens for kernel security update
  • 13:05 gehel: restarting blazegraph on wdqs[12]003
  • 12:44 moritzm: upgrading wikidiff to 1.7.2 on labweb* (HHVM bytecode cache is pruned during update)
  • 12:08 arturo: T201473 install a new version of `prometheus-pdns-exporter` (0.3) into jessie-wikimedia, due to errors in the postinst script
  • 12:06 gehel: depooling wdqs[12]003 to catchup on updates
  • 12:00 moritzm: installing ruby2.3 security updates
  • 11:37 arturo: T201473 copy `prometheus-pdns-exporter` from trusty-wikimedia to jessie-wikimedia in reprepro
  • 11:34 Amir1: EU mid-day SWAT is done
  • 11:31 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.16/maintenance/populateChangeTagDef.php: SWAT: Add option to populateChangeTagDef not to update the count (duration: 00m 53s)
  • 11:12 jynus: stopping db2042 for maintenance T202051
  • 11:09 moritzm: upgrading wikidiff to 1.7.2 on mw1339-mw1348 (HHVM bytecode cache is pruned during update)
  • 11:03 arturo: manually delete glance rsync image cronjob from the glancesync user in labcontrol1001.wikimedia.org (leftover after glance merge in main/eqiad1)
  • 10:43 moritzm: upgrading wikidiff to 1.7.2 on mw1285-mw1290 and mw1312-mw1317 (HHVM bytecode cache is pruned during update)
  • 10:30 _joe_: restarting changeprop on scb1002, not listening on its tcp port
  • 10:27 _joe_: restarting cpjobqueue on scb1002, not listening on its tcp port
  • 10:00 volans: add jiji to the 'ops' LDAP group - T201849
  • 09:46 gehel: all elasticsearch nodes reimaged (except elastic1029, waiting on memory issue) - T198391 / T193649 / T201991
  • 09:16 gehel: reimaging elastic1022
  • 08:40 moritzm: upgrading wikidiff to 1.7.2 on mw1319-mw1333 (HHVM bytecode cache is pruned during update)
  • 08:37 ema: reboot cp2009 for kernel upgrade
  • 08:35 vgutierrez: Reimaging radon as spare system - T202040
  • 08:32 moritzm: uploaded jenkins 2.121.3 to apt.wikimedia.org (for jessie and stretch)
  • 08:16 moritzm: upgrading wikidiff to 1.7.2 on mw1334-mw1338/mw1307/mw1318/ (HHVM bytecode cache is pruned during update)
  • 07:49 gehel: reimaging elastic10(23|24)
  • 07:45 volans@deploy1001: Finished deploy [debmonitor/deploy@1f01fd1]: Release v0.1.8 (duration: 00m 31s)
  • 07:45 volans@deploy1001: Started deploy [debmonitor/deploy@1f01fd1]: Release v0.1.8
  • 07:27 moritzm: rebooting install1002 for kernel security update
  • 07:24 moritzm: rebooting install2002 for kernel security update
  • 05:12 _joe_: moving away corrupted commitlog file on aqs1007 cassandra-a instance, trying to restart it
  • 02:38 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Aug 16 02:38:46 UTC 2018 (duration 10m 17s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 08m 38s)

2018-08-15

  • 22:43 krinkle@deploy1001: Finished scap: rm StartProfiler.php - T201782 (duration: 31m 46s)
  • 22:33 XioNoX: update old eqiad recdns IPs in cr1/2-eqiad bgp group Anycast4 with dns1001/2
  • 22:11 krinkle@deploy1001: Started scap: rm StartProfiler.php - T201782
  • 21:57 krinkle@deploy1001: Synchronized php-1.32.0-wmf.10: rm StartProfiler.php - T201782 (duration: 03m 01s)
  • 21:48 krinkle@deploy1001: Synchronized wmf-config/: I173a02910c (duration: 00m 50s)
  • 21:46 krinkle@deploy1001: Synchronized scap/plugins/: I173a02910c (duration: 00m 50s)
  • 20:49 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I207421 (duration: 00m 57s)
  • 19:07 gehel: reimaging elastic10(18|19|20)
  • 17:20 XioNoX: Rollback: redirect ns2 to authdns1001 - T201608
  • 17:16 moritzm: reboot eeden for kernel security update
  • 17:07 XioNoX: redirect ns2 to authdns1001 - T201608
  • 17:00 moritzm: reboot radon for kernel security update
  • 16:50 XioNoX: redirect ns0 to authdns1001 - T201608
  • 16:42 XioNoX: rollback: redirect ns1 to radon - T201608
  • 16:35 moritzm: reboot authdns2001 for kernel security update
  • 16:27 XioNoX: redirect ns1 to radon - T201608
  • 14:58 gehel: reimaging elastic1028
  • 14:50 ejegg: disabled CiviCRM dedupe jobs
  • 14:47 ejegg: updated SmashPig free-standing deployment from 2736e41045 to 3603e191a3
  • 14:13 gehel: reimaging elastic102[567]
  • 13:47 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2184.codfw.wmnet
  • 13:35 moritzm: installing samba security updates
  • 12:51 Trey314159: reindexing Polish wikis on elastic@eqiad and elastic@codfw complete (T200037)
  • 12:22 moritzm: rebooting mw2190-mw2219 for kernel security update (also bundling wikidiff and apache updates)
  • 12:19 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2184.codfw.wmnet
  • 12:04 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.16/includes/changetags/ChangeTags.php: Swap SET and WHERE statements in ChangeTags::undefineTag (T201934) (duration: 00m 55s)
  • 10:52 moritzm: rebooting mw2163-mw2189 for kernel security update (also bundling wikidiff and apache updates)
  • 09:39 gehel: reimaging elastic103[012], this will trigger master re-election
  • 09:17 moritzm: rebooting mw2135-mw2147 for kernel security update (also bundling wikidiff and apache updates)
  • 08:59 moritzm: rebooting deployment-mediawiki-09
  • 08:14 gehel: masking cassandra-a instance on aqs1007 since it is flapping - T201986
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 15 02:45:47 UTC 2018 (duration 10m 24s)
  • 02:15 ejegg: updated payments-wiki from 6002d9edc0 to 46ed86f389
  • 00:12 catrope@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/PageTriage/: SWAT: PageTriage fixes (T199357, T201812, T201560, T201373, T201253) (duration: 00m 51s)

2018-08-14

  • 23:55 TimStarling: restarted populateContentTables.php on s2
  • 23:48 tzatziki: deleting three images for legal compliance
  • 23:39 tzatziki: Correction: Change email for User:Textorus
  • 23:38 tzatziki: Change password for User:Textorus
  • 23:20 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Enable CentralNotice EventLogging at a low sample rate (0.01) (duration: 00m 50s)
  • 23:20 fdans@deploy1001: Finished deploy [analytics/refinery@21e07ae]: Deploying revert to prevent partition dropping jobs from failing (duration: 08m 27s)
  • 23:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable wgLegacyJavaScriptGlobals on all group0 wikis (T35837) (duration: 00m 54s)
  • 23:11 fdans@deploy1001: Started deploy [analytics/refinery@21e07ae]: Deploying revert to prevent partition dropping jobs from failing
  • 22:51 ejegg: updated payments-wiki from 7efaea7953 to 6002d9edc0
  • 21:19 jgleeson: payments-wiki changed from 6393e83ce0 to 7efaea7953, config revision is a5bce0fc89
  • 19:35 krinkle@deploy1001: Synchronized php-1.32.0-wmf.16/includes/cache/MessageCache.php: I6093113a / T201893 (duration: 00m 52s)
  • 19:11 mforns@deploy1001: Finished deploy [analytics/refinery@a4d1d99]: adding hashing to EL whitelist (duration: 13m 04s)
  • 19:06 filippo@neodymium: conftool action : set/pooled=no; selector: name=logstash1008,service=gelf
  • 18:58 mforns@deploy1001: Started deploy [analytics/refinery@a4d1d99]: adding hashing to EL whitelist
  • 18:44 XioNoX: delete peer 4589 on cr2-esams (no more direct peering)
  • 18:27 XioNoX: renumber v4 IP of 8560 on cr1-eqord
  • 18:20 mforns@deploy1001: Finished deploy [analytics/refinery@cb57843]: corresponding to refinery-source v0.0.70 (duration: 14m 27s)
  • 18:15 XioNoX: bump PCCW max accepted prefixes on cr2-eqiad
  • 18:13 XioNoX: bump PCCW max accepted prefixes on cr2-esams
  • 18:10 XioNoX: re-activate peer 13285 on cr2-ulsfo
  • 18:07 XioNoX: update NTP servers on pfw
  • 18:05 mforns@deploy1001: Started deploy [analytics/refinery@cb57843]: corresponding to refinery-source v0.0.70
  • 18:02 ejegg: re-enabled CiviCRM Omnimail import jobs
  • 18:00 volans@deploy1001: Finished deploy [netbox/deploy@eae2c9d]: Fix broken dependency (duration: 00m 33s)
  • 17:59 Trey314159: reindexing Polish wikis on elastic@eqiad and elastic@codfw (T200037)
  • 17:59 volans@deploy1001: Started deploy [netbox/deploy@eae2c9d]: Fix broken dependency
  • 17:22 XioNoX: configuring eqiad A switch ports for T201694
  • 17:19 gehel: restarting elasticsearch on elastic1050 (high load)
  • 16:57 gehel: reimage of elastic103[345]
  • 16:18 volans@deploy1001: Finished deploy [netbox/deploy@e2fd41d]: Security upgrade of dependency (duration: 00m 34s)
  • 16:17 volans@deploy1001: Started deploy [netbox/deploy@e2fd41d]: Security upgrade of dependency
  • 16:01 volans@deploy1001: Finished deploy [netbox/deploy@792d4d5]: Security upgrade of dependency (duration: 01m 23s)
  • 15:59 volans@deploy1001: Started deploy [netbox/deploy@792d4d5]: Security upgrade of dependency
  • 15:39 volans@deploy1001: Finished deploy [netbox/deploy@792d4d5]: Security upgrade of dependency (duration: 01m 03s)
  • 15:38 volans@deploy1001: Started deploy [netbox/deploy@792d4d5]: Security upgrade of dependency
  • 15:01 moritzm: rebooting meitnerium/archiva.wikimedia.org for kernel security update
  • 14:59 ema: cache_text eqiad: restart varnish-be without transient storage caps
  • 14:53 moritzm: rebooting cloudelastic* for kernel update
  • 14:50 gehel: reimage of elastic103[678]
  • 14:41 bstorm@deploy1001: Finished deploy [striker/deploy@13da520]: (no justification provided) (duration: 01m 13s)
  • 14:40 bstorm@deploy1001: Started deploy [striker/deploy@13da520]: (no justification provided)
  • 14:35 Krinkle: Copying xenon/logs/daily/2018-*{all,load,index,api,RunSingleJob}.log from mwlog1001 to webperfX002 hosts
  • 14:26 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1102 with full weight (duration: 00m 52s)
  • 14:21 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1089.eqiad.wmnet,service=varnish-be
  • 14:05 ema: restart varnish-be on cp108[79], fetch failures after child crashes
  • 13:47 gehel: restarting elasticsearch on elastic1051 (overloaded)
  • 13:21 gehel: restarting elasticsearch on elastic1043 (overloaded)
  • 13:11 moritzm: upgrading wikidiff to 1.7.2 on mw1308-mw1311/mw1293-mw1296 (HHVM bytecode cache is pruned during update)
  • 12:57 gehel: restarting tilerator on maps eqiad
  • 12:54 Trey314159: reindexing Indonesian wikis on elastic@eqiad and elastic@codfw complete (T200204)
  • 12:49 moritzm: upgrading wikidiff to 1.7.2 on snapshot hosts
  • 11:58 moritzm: upgrading wikidiff to 1.7.2 on mw1276-mw1283 (HHVM bytecode cache is pruned during update)
  • 11:40 TimStarling: restarted populateContentTables.php on s2 (T183488)
  • 11:20 zeljkof: EU SWAT finished
  • 11:19 moritzm: upgrading wikidiff to 1.7.2 on mw1299-mw1306 (HHVM bytecode cache is pruned during update)
  • 11:19 zfilipin@deploy1001: Synchronized vendor/perftools/xhgui-collector/external/header.php: SWAT: Fix "the the" typo in vendor/perftools/xhgui-collector/external/header.php (T201491) (duration: 00m 49s)
  • 11:17 zfilipin@deploy1001: Synchronized wmf-config/CirrusSearch-common.php: SWAT: Fix the the typo in wmf-config/CirrusSearch-common.php (T201491) (duration: 00m 51s)
  • 11:14 moritzm: repooled mw1233 (debugging completed)
  • 10:45 moritzm: repooled mw1227 (was probably overlooked to repool after previous debugging)
  • 10:41 volans: upgraded cumin on labpuppetmaster* to fix cumin with the new openstack region - T201881
  • 10:16 moritzm: upgrading wikidiff to 1.7.2 on mw1221-mw1235 (HHVM bytecode cache is pruned during update)
  • 10:12 volans: uploaded cumin_3.0.2-2_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 09:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1102 with low load (duration: 00m 50s)
  • 09:32 jynus: stop and restart db1122 for maintenance
  • 09:27 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1102 (duration: 00m 52s)
  • 09:23 addshore: for i in {10000..12500}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
  • 09:20 moritzm: upgrading wikidiff to 1.7.2 on mw1238-mw1258 (HHVM bytecode cache is pruned during update)
  • 09:17 addshore: for i in {6000..10000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
  • 09:13 addshore: for i in {3000..6000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
  • 09:08 addshore: for i in {1000..3000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
  • 09:07 addshore: for i in {1..1000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
  • 08:51 addshore: addshore@labweb1001:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki GoranSMilovanovic # T201122
  • 08:19 moritzm: upgrading wikidiff to 1.7.2 on mw1266-mw1275 (HHVM bytecode cache is pruned during update)
  • 07:56 moritzm: installing ghostscript security updates
  • 07:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:s7 and db1101:s8 (duration: 00m 52s)
  • 06:45 _joe_: rolling restart of hhvm in eqiad api, high load
  • 06:37 _joe_: depooling mw1233 from live traffic for debugging
  • 05:13 TimStarling: killed populateContentTables.php for s2 at jcrespo's request
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 14 02:45:46 UTC 2018 (duration 10m 25s)
  • 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 13m 57s)
  • 00:35 aaron@deploy1001: Synchronized wmf-config/mc.php: Enable broadcasted mcrouter cache operations for test wikis and mw.org (duration: 00m 49s)
  • 00:15 mutante: graphite2002 - stopping carbon-local-relay, running puppet to start it again to confirm no issues with gerrit:448779
  • 00:09 aaron@deploy1001: Synchronized wmf-config/mc.php: Only do cache writes to mcrouter for all wikis (duration: 00m 52s)

2018-08-13

  • 23:26 urandom: Restart Cassandra in RESTBase dev env -- T201863
  • 23:24 godog: restart carbon-c-relay on graphite1001 to mirror traffic to graphite1004 - T196484
  • 23:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable wp10 and draftquality ORES models on testwiki (T198997) (duration: 00m 51s)
  • 22:32 bawolff@deploy1001: Synchronized php-1.32.0-wmf.16/resources/src/startup/mediawiki.js: backport I3cd2d7 (CSP fixes) (duration: 00m 50s)
  • 22:31 Trey314159: reindexing Indonesian wikis on elastic@eqiad and elastic@codfw (T200204)
  • 22:30 bawolff@deploy1001: Synchronized php-1.32.0-wmf.16/includes/OutputPage.php: backport I3cd2d7 (CSP fixes) (duration: 00m 50s)
  • 22:29 bawolff@deploy1001: Synchronized php-1.32.0-wmf.16/includes/ContentSecurityPolicy.php: backport I3cd2d7 (CSP fixes) (duration: 00m 51s)
  • 22:29 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw complete (T200204)
  • 21:30 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Deploy adjust CSP settings I74a79dc7defaa (duration: 00m 52s)
  • 21:29 mutante: elastic1049 - started ferm
  • 19:32 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw (T200204)
  • 18:47 otto@deploy1001: Finished deploy [analytics/refinery@c7f68b7]: camus - Add --check-java-opts and --check-emails-to option - T198908 (duration: 11m 10s)
  • 18:35 otto@deploy1001: Started deploy [analytics/refinery@c7f68b7]: camus - Add --check-java-opts and --check-emails-to option - T198908
  • 17:16 gehel@deploy1001: Finished deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI (duration: 10m 54s)
  • 17:15 thcipriani@deploy1001: Synchronized wmf-config/wikitech.php: wikitech.php: update keystone URLs (duration: 00m 53s)
  • 17:05 gehel@deploy1001: Started deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI
  • 17:02 gehel@deploy1001: Finished deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 30s)
  • 17:01 gehel@deploy1001: Started deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI (wdqs1009 only)
  • 15:41 otto@deploy1001: Finished deploy [analytics/refinery@a051125]: fix for T198908 (duration: 08m 06s)
  • 15:33 papaul: shutting down lvs2009 to disable on board NICs
  • 15:33 otto@deploy1001: Started deploy [analytics/refinery@a051125]: fix for T198908
  • 15:10 otto@deploy1001: Finished deploy [analytics/refinery@9006a4e]: refinery changes for T198908 (duration: 10m 30s)
  • 15:00 otto@deploy1001: Started deploy [analytics/refinery@9006a4e]: refinery changes for T198908
  • 14:51 moritzm: installing ghostscript security updates
  • 14:47 cmjohnson1: disabling puppet and shutting down db1052 for final decommissioning
  • 14:25 moritzm: installing dpkg updates from stretch point release
  • 14:10 arturo: T201504 disable keystone in main and eqiad1 deployments, all has been downtimed in icinga
  • 14:09 andrewbogott: stopping nodepool, downtiming horizon for T201504
  • 14:05 moritzm: upgrading mw1262-1265 to wikidiff 1.7.2 (T199801)
  • 12:56 gehel: start reimaging of elasticsearch / cirrus / eqiad cluster (RAID0 / Stretch) - T193649 / T198391
  • 12:56 gehel: reimaging of elasticsearch / cirrus / codfw cluster (RAID0 / Stretch) completed - T193649 / T198391
  • 12:53 moritzm: upgrading mw1261 servers to wikidiff 1.7.2 (T199801)
  • 12:28 moritzm: upgrading mwdebug* servers to wikidiff 1.7.2 (T199801)
  • 12:28 moritzm: upgrading mwdebug* servers to wikidiff 1.7.2
  • 12:24 moritzm: uploaded hhvm-wikidiff2 1.7.2 to apt.wikimedia.org (source package name is still php-wikidiff2 for historical reasons) (T199801)
  • 12:06 zeljkof: EU SWAT finished
  • 12:00 moritzm: rebooting cloudcontrol* for kernel security update
  • 11:58 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set mobile wordmark for pswikivoyage (T200152) (duration: 00m 52s)
  • 11:47 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Translate extension on oldwikisource (T201814) (duration: 00m 50s)
  • 11:42 moritzm: rebooting dbmonitor* hosts for kernel security update
  • 11:29 zfilipin@deploy1001: Synchronized static/images/mobile/copyright/: SWAT: Add missing PNG files in mobile logo folder (T200152) (duration: 00m 49s)
  • 11:05 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 11:04 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
  • 11:01 moritzm: rebooting kraz/irc.wikimedia.org for kernel security update
  • 10:29 jynus: upgrade and restart db2036
  • 08:42 jynus: upgrade and restart db1101
  • 08:39 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:s7 and db1101:s8 (duration: 00m 51s)
  • 07:16 legoktm@deploy1001: Synchronized php-1.32.0-wmf.16/maintenance/dumpTextPass.php: Add the 'full' option explicitly to dumpTextPass.php (T201803) (duration: 00m 58s)
  • 06:53 ema: finish up cache reboots for kernel updates: cp5001.eqsin.wmnet,cp3035.esams.wmnet,cp[4021-4022].ulsfo.wmnet
  • 06:46 oblivian@neodymium: conftool action : set/pooled=inactive; selector: name=restbase2003.codfw.wmnet
  • 06:34 _joe_: powercycling restbase2003, in kernel panic
  • 02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 13 02:46:41 UTC 2018 (duration 10m 28s)
  • 02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 14m 21s)
  • 01:45 TimStarling: on mwmaint1001 running populateContentTables.php on all wikis using ~tstarling/pct-list T183488

2018-08-12

  • 15:43 gehel: full cache invalidation of maps tiles - T201772
  • 06:15 ejegg: stopped CiviCRM omnimail jobs

2018-08-10

  • 21:50 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/UploadWizard: T201708 UBN fix (e498c7d) (duration: 00m 56s)
  • 18:45 bblack: restart varnish-be on cp1087
  • 18:44 bblack: restart varnish-be on cp1079
  • 18:44 bblack: restart varnish-be on cp1075
  • 18:41 bblack: restart varnish-be on cp1081
  • 17:49 jgleeson: Released config update containing Google API Key (37a1a28dfb)
  • 17:34 bblack: cp1085 + cp1089: restart varnish with wiped storage, repool
  • 17:04 ema: restart varnish-be on cp1083
  • 16:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1089.eqiad.wmnet
  • 16:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1085.eqiad.wmnet
  • 16:36 bblack: depool cp1089, cp1085
  • 16:12 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw abandoned (T200204)
  • 16:02 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw (T200204)
  • 14:48 ema: powercycle cp5005, stuck rebooting
  • 12:27 gehel: resetting management card on elastic2017 - T201671
  • 12:18 gehel: restarting icinga on einsteinium - T196336
  • 11:43 moritzm: installing busybox security updates
  • 11:38 moritzm: installing jansson security updates for trusty
  • 11:17 moritzm: installing ant security updates
  • 11:06 moritzm: installing libxcursor security updates
  • 09:35 mobrovac@deploy1001: Finished deploy [citoid/deploy@983d80c]: Remove the bibtex spec.yaml x-ample - T197242 (duration: 03m 04s)
  • 09:32 mobrovac@deploy1001: Started deploy [citoid/deploy@983d80c]: Remove the bibtex spec.yaml x-ample - T197242
  • 09:27 jynus: upgrade and restart db1125
  • 09:19 moritzm: rearmed keyholder on labpuppetmaster*
  • 09:12 moritzm: rebooting labpuppetmaster1002 for kernel security update/microcode update
  • 09:03 jynus: upgrade and restart db1124
  • 08:58 moritzm: rebooting labpuppetmaster1001 for kernel security update/microcode update
  • 08:29 jynus: upgrade and restart db2095
  • 08:28 arturo: rebuild python-pykube for stretch-wikimedia and add it to apt.wikimedia.org T200660
  • 08:03 jynus: upgrade and restart db2094
  • 07:57 ema: powercycle cp3040, kernel crash
  • 07:03 moritzm: installing mutt security updates

2018-08-09

  • 23:40 ejegg: updated payments-wiki from d78c98cf0f to 6393e83ce0
  • 23:33 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/451810/ (duration: 00m 51s)
  • 23:27 maxsem@deploy1001: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/451792/ (duration: 00m 51s)
  • 23:12 maxsem@deploy1001: Synchronized static/images/mobile/copyright/wikivoyage-wordmark-ps.svg: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/451791/ (duration: 00m 52s)
  • 22:31 XioNoX: disable fpc1-fpc3 - T201145
  • 22:09 bblack: re-enabling pybal on lvs1016
  • 21:31 XioNoX: disable fpc3-fpc4 - T201145
  • 21:21 XioNoX: disable fpc5-fpc6 - T201145
  • 21:21 bblack: stop pybal on lvs1016 (should move services to lvs1006)
  • 21:06 XioNoX: disable fpc4-fpc6 - T201145
  • 21:01 XioNoX: enable fpc5-fpc6 - T201145
  • 20:54 XioNoX: enable fpc3-fpc4 - T201145
  • 20:49 XioNoX: enable fpc1-fpc3 - T201145
  • 20:44 XioNoX: disable fpc3-fpc5 and fpc5-fpc7 - T201145
  • 20:31 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@8c6a7f8]: add python snappy library (duration: 07m 25s)
  • 20:23 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@8c6a7f8]: add python snappy library
  • 20:18 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1080.eqiad.wmnet
  • 19:16 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.16 refs T191062
  • 19:09 twentyafterfour: Preparing to deploy 1.32.0-wmf.16 to group2 wikis.
  • 19:02 herron: logstash1008:~# systemctl restart logstash T200960
  • 19:02 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add temporary debug logging to ThumbnailRenderJob T201305 (duration: 00m 57s)
  • 18:58 niharika29@deploy1001: Synchronized php-1.32.0-wmf.16/includes/jobqueue/jobs/ThumbnailRenderJob.php: Add temporary debug logging T201305 (duration: 01m 11s)
  • 18:49 bblack: remaining cache_misc sites will move to cache_text shortly - https://gerrit.wikimedia.org/r/451659
  • 18:29 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@39be980]: Repair daemons failing to load logging config (duration: 03m 33s)
  • 18:26 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@39be980]: Repair daemons failing to load logging config
  • 18:23 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@22a50af]: temp fix while waiting for jenkins (duration: 03m 35s)
  • 18:19 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@22a50af]: temp fix while waiting for jenkins
  • 18:17 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add media.farsnews.com to wgCopyUploadDomains T200872 (duration: 01m 00s)
  • 18:15 niharika29@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 01s)
  • 17:59 XioNoX: connecting asw2-a5-eqiad to asw2-a6-eqiad - T201145
  • 17:54 moritzm: installing gnupg security updates on trusty (Debian already fixed)
  • 17:52 awight@deploy1001: Finished deploy [ores/deploy@1712122]: fawiki wp10; ORES update: T201518 (duration: 31m 58s)
  • 17:35 mforns@deploy1001: Finished deploy [analytics/refinery@9f4267c]: deploy refinery together with refinery-source v0.0.68 (duration: 08m 22s)
  • 17:26 mforns@deploy1001: Started deploy [analytics/refinery@9f4267c]: deploy refinery together with refinery-source v0.0.68
  • 17:21 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@6581b28]: Update mobileapps to 616ffef (T191640) (duration: 06m 35s)
  • 17:20 awight@deploy1001: Started deploy [ores/deploy@1712122]: fawiki wp10; ORES update: T201518
  • 17:14 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@6581b28]: Update mobileapps to 616ffef (T191640)
  • 15:14 mobrovac@deploy1001: Finished deploy [citoid/deploy@92e5071]: Record format stats (duration: 17m 07s)
  • 14:57 mobrovac@deploy1001: Started deploy [citoid/deploy@92e5071]: Record format stats
  • 14:52 moritzm: rebooting dns2002 for kernel security update/enabling microcode updates
  • 14:50 bblack: all cpNNNN: max tcp per-flow rate reducing 1Gbps -> 256Mbps - https://gerrit.wikimedia.org/r/c/operations/puppet/+/451535
  • 14:44 moritzm: rebooting dns2001 for kernel security update/enabling microcode updates
  • 14:34 moritzm: rebooting maerlant for kernel security update/enabling microcode updates
  • 14:28 herron: rolling reboot of mx servers for kernel updates
  • 14:26 moritzm: rebooting nescio for kernel security update/enabling microcode updates
  • 14:18 herron: rebooting fermium (lists) to switch out of -rt kernel variant
  • 14:12 moritzm: rebooting dns4002 for kernel security update/enabling microcode updates
  • 13:55 moritzm: rebooting dns4001 for kernel security update/enabling microcode updates
  • 13:48 moritzm: rebooting dns5002 for kernel security update/enabling microcode updates
  • 13:28 moritzm: rebooting dns5001 for kernel security update/enabling microcode updates
  • 12:22 moritzm: rebooting ununpentium for kernel security update
  • 12:14 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Do not leak local $wgWBShared… variables to the global scope (duration: 00m 56s)
  • 12:13 moritzm: rebooting netmon1003 (servermon.wikimedia.org) for kernel security update
  • 12:10 moritzm: rearmed keyholder on netmon1002
  • 12:09 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable RDF export for lexicographical data (T201153) (duration: 00m 56s)
  • 12:06 moritzm: rebooting netmon1002 for kernel security update
  • 12:05 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Enable RDF export for lexicographical data (T201153) (duration: 00m 58s)
  • 12:05 moritzm: rearmed keyholder on netmon2001
  • 11:57 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos in hewikibooks (T201562) (duration: 00m 55s)
  • 11:50 moritzm: rebooting netmon2001 for kernel security update
  • 11:42 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Update logo for hewikibooks (T201562) (duration: 01m 00s)
  • 11:30 ema: resume rolling reboots of caches for numa_networking T193865
  • 11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Transwiki import in zhwikiversity (T201328) (duration: 00m 56s)
  • 11:19 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove noratelimit from epcoordinator group on cswiki (T201010) (duration: 00m 58s)
  • 11:07 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateStyles everywhere (T199909) (duration: 00m 58s)
  • 10:57 mobrovac: truncating commons and others mobile tables - T201103
  • 10:42 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 and db1123 fully (duration: 00m 59s)
  • 10:38 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2069 due to crash (duration: 01m 00s)
  • 10:38 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #5 - T201103 (duration: 06m 27s)
  • 10:31 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #5 - T201103
  • 10:31 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #4 - T201103 (duration: 12m 11s)
  • 10:31 moritzm: rebooting archiva1001 for kernel security update
  • 10:25 jynus: disabling alerts for db2069
  • 10:21 jynus: handling db2069 crash, no user impact
  • 10:19 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #4 - T201103
  • 10:19 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #3 - T201103 (duration: 04m 20s)
  • 10:14 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #3 - T201103
  • 10:14 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #2 - T201103 (duration: 08m 38s)
  • 10:05 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #2 - T201103
  • 10:03 moritzm: rebooting url downloaders for kernel security update (alsafi, actinium, aluminium, alcyone)
  • 10:02 mobrovac@deploy1001: Finished deploy [restbase/deploy@cb6b4b4]: Drop mobile-sections, feed and media end points from non-WPs - T201103 (duration: 04m 26s)
  • 09:57 mobrovac@deploy1001: Started deploy [restbase/deploy@cb6b4b4]: Drop mobile-sections, feed and media end points from non-WPs - T201103
  • 09:57 moritzm: rebooting dubnium for kernel security update
  • 09:56 mobrovac@deploy1001: deploy aborted: Drop mobile-sections, feed and media end points from non-WPs - T201103 (duration: 06m 14s)
  • 09:50 mobrovac@deploy1001: Started deploy [restbase/deploy@cb6b4b4]: Drop mobile-sections, feed and media end points from non-WPs - T201103
  • 09:42 moritzm: rebooting pollux for kernel security update
  • 09:24 moritzm: rebooting iron for kernel security update
  • 09:23 moritzm: reset racadm on iron, com2 was unresponsive
  • 09:08 moritzm: rebooting bast5001 for kernel security update
  • 09:02 moritzm: rebooting bast4002 for kernel security update
  • 08:57 moritzm: rebooting bast2001 for kernel security update
  • 08:37 moritzm: rebooting bast2002 for kernel security update
  • 07:26 ema: cp3032 mbox lagged: reboot w/ numa_networking
  • 05:52 kartik@deploy1001: Finished deploy [cxserver/deploy@957ff6a]: Update cxserver to 27813b6 (T201085) (duration: 03m 59s)
  • 05:48 kartik@deploy1001: Started deploy [cxserver/deploy@957ff6a]: Update cxserver to 27813b6 (T201085)
  • 05:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool pc2004 and pc2005 after BIOS upgrade - T201387 (duration: 01m 00s)
  • 03:23 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Aug 9 03:23:32 UTC 2018 (duration 10m 34s)
  • 03:12 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 15m 23s)
  • 02:38 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 16m 05s)
  • 02:06 twentyafterfour: restarted apache on phab1001 to hotfix antivandalism bug
  • 01:39 mutante: baham - installing BIOS upgrade (2.4.2 for Dell R320) - server is on role(spare) and the last that did not get the upgrade on T162850
  • 00:14 reedy@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/VisualEditor/: Bug fix (duration: 00m 58s)
  • 00:05 twentyafterfour: finished phabricator upgrade
  • 00:02 twentyafterfour: deploying phabricator release/2018-08-08/1

2018-08-08

  • 23:55 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: satwiki sitename (duration: 01m 05s)
  • 23:48 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: sq* collation (duration: 00m 56s)
  • 23:40 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: extension registration config for GWToolset (duration: 00m 56s)
  • 23:39 reedy@deploy1001: Synchronized wmf-config/extension-list: GWToolset to extension.json (duration: 00m 57s)
  • 23:27 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove old or same image config (duration: 00m 56s)
  • 23:22 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgVariantArticlePath for zhwikiversity (duration: 01m 05s)
  • 21:04 twentyafterfour: 1.32.0-wmf.16 appears to be stable, no noticeable increase in errors logged.
  • 20:59 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.16 refs T191062 (duration: 00m 57s)
  • 20:58 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.16 refs T191062
  • 20:49 James_F: Updated mobileapps to 162ebcf in Production.
  • 20:48 jforrester@deploy1001: Finished deploy [mobileapps/deploy@4ef02e1]: Update mobileapps to 162ebcf (duration: 07m 36s)
  • 20:40 jforrester@deploy1001: Started deploy [mobileapps/deploy@4ef02e1]: Update mobileapps to 162ebcf
  • 19:57 _joe_: powercycling cp1077, network card problems
  • 19:54 bblack: reboot cp1088 for network card
  • 19:53 bblack: reboot cp1090 for network card
  • 19:53 _joe_: powercycling cp1079, network card problems
  • 19:46 bblack: rebooting cp1084 to debug eth hardware
  • 19:35 XioNoX: fix interface description on asw-a-eqiad uplinks
  • 19:19 volans: completed forced puppet run where it was failed
  • 19:18 bblack: depooling front-edge traffic from eqiad in DNS - https://gerrit.wikimedia.org/r/c/operations/dns/+/451401
  • 19:11 volans: force puppet run where it's still failed
  • 19:01 twentyafterfour: Waiting to deploy the train until after I've established that network issues are resolved.
  • 18:23 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f0246f7]: Only rerender mobile-sections for wikipedia T201103 (duration: 01m 29s)
  • 18:22 ppchelko@deploy1001: Started deploy [changeprop/deploy@f0246f7]: Only rerender mobile-sections for wikipedia T201103
  • 17:55 herron: performing rolling restart of logstash instances T200960
  • 17:01 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/includes/logging/LogFormatter.php: SWAT T185049 Unbreak views of invalid title log entries, wmf.16 (duration: 00m 58s)
  • 17:00 jforrester@deploy1001: Synchronized php-1.32.0-wmf.15/includes/logging/LogFormatter.php: SWAT T185049 Unbreak views of invalid title log entries, wmf.15 (duration: 01m 01s)
  • 16:53 bblack: reimagine cp1071-4, cp1099
  • 16:20 bblack: reimaging cp1060, cp1062-8
  • 15:58 bblack: reimaging cp1050.eqiad.wmnet cp1052.eqiad.wmnet cp1053.eqiad.wmnet cp1054.eqiad.wmnet cp1055.eqiad.wmnet cp1059.eqiad.wmnet
  • 15:54 bblack: downtimed all ipsec checks :P
  • 15:45 bblack: reimaging cp1046-9
  • 14:55 jynus: switching over, stopping, upgrading and restarting labsdb1011
  • 13:34 papaul: BIOS update in progress on pc2005
  • 13:10 papaul: BIOS update in progress on pc2004
  • 12:53 jynus: switching over, stopping, upgrading and restarting labsdb1010
  • 12:00 jynus: testing new read only check on labsdb wikireplicas
  • 11:21 jynus: switching over, stopping, upgrading and restarting labsdb1009
  • 11:14 marostegui: Stop MySQL on pc2005 for BIOS upgrade - T201387
  • 11:14 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2005 - T201387 (duration: 00m 59s)
  • 11:02 ariel@deploy1001: Finished deploy [dumps/dumps@d6bd774]: fix up getConfiguration invocation (duration: 00m 04s)
  • 11:02 ariel@deploy1001: Started deploy [dumps/dumps@d6bd774]: fix up getConfiguration invocation
  • 10:27 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 and db1123 with low load after maint (duration: 00m 57s)
  • 10:11 ema: power-cycle cp2022, stuck rebooting
  • 09:08 ema: begin rolling reboots of caches for numa_networking T193865
  • 08:58 marostegui: Drop unused grants from 208.80.153.48
  • 08:44 marostegui: Drop unused grants from 208.80.154.12
  • 08:41 marostegui: Drop unused grants from 208.80.154.136
  • 08:03 jynus: stop db1123 for provisioning and upgrade
  • 07:55 jynus: stop db1100 for provisioning and upgrade
  • 07:47 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 and db1123 (duration: 00m 58s)
  • 07:27 marostegui: Stop MySQL on pc2004 - T201387
  • 06:17 kartik@deploy1001: Finished deploy [cxserver/deploy@6a0cab1]: Update cxserver to 951fdba (T199308, T199512, T199320, T200665, T200453, T106437) (duration: 03m 32s)
  • 06:13 kartik@deploy1001: Started deploy [cxserver/deploy@6a0cab1]: Update cxserver to 951fdba (T199308, T199512, T199320, T200665, T200453, T106437)
  • 05:23 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2004 - T201387 (duration: 00m 56s)
  • 05:12 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 01m 06s)
  • 03:12 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 8 03:12:43 UTC 2018 (duration 10m 26s)
  • 03:02 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 16m 06s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 08m 40s)
  • 00:11 ejegg: updated payments-wiki from 8cb7c86b12 to d78c98cf0f

2018-08-07

  • 23:46 ejegg: updated payments-wiki from 0c9c24d30c to 8cb7c86b12
  • 23:28 TimStarling: restarted populateContentTables.php for commonswiki, it died at rev_id 60164000
  • 22:14 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1048.eqiad.wmnet
  • 21:58 ejegg: updated CiviCRM from d626907f2c to b60ceb3d2f
  • 21:17 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.16 refs T191062
  • 21:07 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1049.eqiad.wmnet
  • 20:45 twentyafterfour@deploy1001: Finished scap: testwikis wikis to 1.32.0-wmf.16 refs T191062 (duration: 68m 20s)
  • 20:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1050.eqiad.wmnet
  • 20:34 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1062.eqiad.wmnet
  • 19:45 herron: upload prometheus-logstash-exporter_0.1.2-1 to apt.wikimedia.org/jessie-wikimedia/main
  • 19:45 herron: upload prometheus-logstash-exporter_0.1.2-1 to apt.wikimedia.org/stretch-wikimedia/main
  • 19:36 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.32.0-wmf.16 refs T191062
  • 19:30 andrewbogott: shutting down labnodepool1002 in advance of a rename. T201439
  • 19:09 otto@deploy1001: Finished deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (all nodes) (duration: 01m 51s)
  • 19:07 otto@deploy1001: Started deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (all nodes)
  • 19:07 otto@deploy1001: Finished deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (scb2001 only) (duration: 00m 20s)
  • 19:06 otto@deploy1001: Started deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (scb2001 only)
  • 19:03 twentyafterfour: Branching 1.32.0-wmf.16
  • 18:24 mutante: netbox - restored database from dump file - backed up and back-up (T190184)
  • 18:13 mutante: netbox - temp dropped databae to test restoring from dump
  • 18:02 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1063.eqiad.wmnet
  • 18:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet
  • 16:51 hoo: Deployed patch for T201418
  • 16:50 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@22a50af]: bump to master, prep for deploy to cirrus servers, take two (duration: 01m 51s)
  • 16:48 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@22a50af]: bump to master, prep for deploy to cirrus servers, take two
  • 16:47 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1052.eqiad.wmnet
  • 16:47 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet
  • 16:44 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@65e6bb9]: bump to master, prep for deploy to cirrus servers (duration: 02m 54s)
  • 16:41 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@65e6bb9]: bump to master, prep for deploy to cirrus servers
  • 15:55 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1019, db1122 and db1081 with full weight (duration: 00m 51s)
  • 15:26 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1053.eqiad.wmnet
  • 15:26 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1077.eqiad.wmnet
  • 14:58 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1054.eqiad.wmnet
  • 14:57 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1079.eqiad.wmnet
  • 14:57 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1064.eqiad.wmnet
  • 14:57 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1078.eqiad.wmnet
  • 14:26 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low load (duration: 00m 47s)
  • 14:18 herron: double again logstash jvm heap size to 1g and rolling restart logstash instances
  • 14:18 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1019 and db1102 with low load after maintenance (duration: 00m 52s)
  • 14:11 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1055.eqiad.wmnet,service=varnish-be
  • 14:11 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=varnish-be
  • 14:11 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1071.eqiad.wmnet,service=varnish-be
  • 14:11 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet,service=varnish-be
  • 13:42 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1055.eqiad.wmnet,service=varnish-fe
  • 13:42 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=varnish-fe
  • 13:41 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1071.eqiad.wmnet,service=varnish-fe
  • 13:41 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet,service=varnish-fe
  • 13:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1072.eqiad.wmnet
  • 13:39 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet
  • 13:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1065.eqiad.wmnet
  • 13:39 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet
  • 13:37 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1055.eqiad.wmnet,service=nginx
  • 13:37 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=nginx
  • 13:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1071.eqiad.wmnet,service=nginx
  • 13:36 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet,service=nginx
  • 13:10 otto@deploy1001: Started restart [analytics/aqs/deploy@6fafc63]: Bouncing AQS for mediawiki_history index update
  • 12:58 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=hydrogen.wikimedia.org
  • 12:58 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=chromium.wikimedia.org
  • 12:52 bblack: rebooting authdns1001
  • 12:44 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns1002.wikimedia.org
  • 12:41 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org
  • 11:55 marostegui: Drop unused grants repl@208.80.155.117
  • 11:19 zeljkof: EU SWAT finished
  • 11:18 mobrovac@deploy1001: Synchronized rpc/RunSingleJob.php: RunSingleJob: remove unneeded base64 decoding (duration: 00m 49s)
  • 11:17 marostegui: Remove unused repl grants for 10.64.0%
  • 11:07 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove hewiki interface-editor group (T200698) (duration: 00m 49s)
  • 10:51 jynus: shutdown and upgrade of db1081 (last message was wrong)
  • 10:51 jynus: shutdown and upgrade of db1082
  • 10:45 jynus: shutdown and upgrade of db1122
  • 10:44 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 and db1081 (duration: 00m 49s)
  • 10:33 godog: bounce logstash on logstash1007 for tests
  • 10:19 jynus: shutting down es1019 for hw maintenance T201132
  • 10:18 ema: reboot lvs-eqiad primaries for kernel upgrade
  • 10:04 godog: reboot einsteinium for kernel upgrade
  • 09:57 marostegui: Deploy schema change on db2075 - T67448 T114117 T5119
  • 09:55 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 00m 48s)
  • 09:43 ema: reboot lvs-codfw primaries for kernel upgrade
  • 09:15 _joe_: rolling restart of HHVM in the api-eqiad cluster
  • 09:13 _joe_: restarting hhvm on mw1231, high load
  • 09:10 _joe_: restarting hhvm on mw1226, then repooling it
  • 09:07 ema: reboot lvs3001 for kernel upgrade
  • 09:02 _joe_: depool mw1226 for investigation
  • 08:50 ema: reboot lvs-eqsin primaries for kernel upgrade
  • 08:29 gehel: start reimaging of elasticsearch / cirrus / codfw cluster (RAID0 / Stretch) - T193649 / T198391
  • 08:28 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 49s)
  • 08:28 ema: reboot lvs-ulsfo primaries for kernel upgrade
  • 08:13 godog: reboot tegmen with a new kernel
  • 07:37 jynus: restarted es1014 prometheus exporter (last message was wrong)
  • 07:37 jynus: restarted es1019 prometheus exporter
  • 07:29 volans: migrated puppetboard and debmonitor to cache_text - T164609
  • 07:05 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool pc2006 T200641 (duration: 00m 49s)
  • 06:58 marostegui: Deploy schema change on db1075 (s3 master) T144010 T51190 T199368
  • 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 (duration: 00m 50s)
  • 06:53 ema: reboot lvs secondaries for kernel upgrade
  • 05:21 marostegui: Deploy schema change on db1123 T144010 T51190 T199368
  • 05:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 (duration: 00m 50s)
  • 03:46 TimStarling: on mwmaint1001 running populateContentTables.php concurrently on wikidatawiki and commonswiki (T183488)
  • 02:34 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 7 02:34:24 UTC 2018 (duration 10m 31s)
  • 02:23 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 08m 46s)
  • 01:06 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1065.eqiad.wmnet,service=nginx
  • 01:06 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet,service=nginx
  • 01:05 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1072.eqiad.wmnet,service=nginx
  • 01:05 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet,service=nginx
  • 00:29 aaron@deploy1001: Synchronized wmf-config/mc.php: Use mcrouter for cache reads on all wikis (duration: 00m 49s)

2018-08-06

  • 23:42 bblack@neodymium: conftool action : set/weight=1; selector: dc=esams,cluster=cache_text,service=nginx
  • 23:42 James_F: SWAT complete.
  • 23:42 bblack@neodymium: conftool action : set/weight=1; selector: dc=esams,cluster=cache_text,service=varnish-fe
  • 23:41 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Revert PageImages: Make it possible to add extra namespaces, part II, fatalmonitor unhappy (duration: 00m 49s)
  • 23:38 jforrester@deploy1001: sync-file aborted: SWAT PageImages: Add NS_CATEGORY for Commons T198716 (duration: 00m 04s)
  • 23:35 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT PageImages: Make it possible to add extra namespaces, part II - I3f225d051 (duration: 00m 48s)
  • 23:35 bblack@neodymium: conftool action : set/weight=1; selector: dc=eqiad,cluster=cache_upload,service=nginx
  • 23:33 bblack@neodymium: conftool action : set/weight=1; selector: dc=eqiad,cluster=cache_upload,service=varnish-fe
  • 23:32 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT PageImages: Make it possible to add extra namespaces, part I - I6c51c1fe1 (duration: 00m 49s)
  • 23:30 jforrester@deploy1001: Synchronized wmf-config/extension-list: SWAT Load TimedMediaHandler i18n via static extension registration (duration: 00m 48s)
  • 23:27 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Load TimedMediaHandler via static extension registration T140852 (duration: 00m 48s)
  • 23:21 jforrester@deploy1001: Synchronized multiversion/: SWAT Delete multiversion/submodules.json (duration: 00m 48s)
  • 23:15 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Add new wikimania wiki to CORS origins I493cadb871 (duration: 00m 49s)
  • 23:11 ejegg: updated CiviCRM from 03012506b9 to d626907f2c
  • 23:07 James_F: jforrester@mwmaint1001 SWAT Updating sewiki collation via updateCollation.php T182431
  • 23:06 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Update sewiki collation config T182431 (duration: 00m 51s)
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1090.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1088.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1086.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1084.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1082.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1078.eqiad.wmnet,service=varnish-fe
  • 22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1076.eqiad.wmnet,service=varnish-fe
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1084.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1082.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1078.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1076.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1086.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1088.eqiad.wmnet,service=nginx
  • 22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1090.eqiad.wmnet,service=nginx
  • 22:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1066.eqiad.wmnet
  • 22:09 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1085.eqiad.wmnet
  • 22:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1073.eqiad.wmnet
  • 22:08 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1086.eqiad.wmnet
  • 20:39 arlolra: Updated Parsoid to e02124b (T199849, T198400, T199577, T199509, T200403, T201054)
  • 20:31 arlolra@deploy1001: Finished deploy [parsoid/deploy@6b16b57]: Updating Parsoid to e02124b (duration: 11m 20s)
  • 20:19 arlolra@deploy1001: Started deploy [parsoid/deploy@6b16b57]: Updating Parsoid to e02124b
  • 20:00 gehel@deploy1001: Finished deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater (duration: 10m 55s)
  • 19:50 gehel@deploy1001: Started deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater
  • 19:46 gehel@deploy1001: Finished deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 22s)
  • 19:46 gehel@deploy1001: Started deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater (wdqs1009 only)
  • 19:14 mutante: bast1002 - rebooting
  • 19:04 mutante: bast1002 - the last log line is about bast1002, not bast1001
  • 19:03 mutante: bast1001 - installing package updates, incl systemd,kernel
  • 18:56 herron: rebooting fermium (lists) for security updates
  • 18:54 herron: rolling reboot of mx servers for security updates
  • 18:16 moritzm: rebooting radium for kernel security update
  • 18:15 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable Special:Block Feedback Request (duration: 00m 52s)
  • 18:09 mutante: rebooting bast3002 for kernel upgrade
  • 18:08 mutante: bast2002 - installing package upgrades (future bastion, to be setup)
  • 18:04 herron: double logstash jvm heap size from 256m to 512m and rolling restart of logstash instances T200960
  • 17:57 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1067.eqiad.wmnet
  • 17:57 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet
  • 17:57 mutante: rebooting bast4001
  • 17:56 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1099.eqiad.wmnet
  • 17:56 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1088.eqiad.wmnet
  • 17:37 mobrovac@deploy1001: Finished deploy [citoid/deploy@4f5ba15]: Use WorldCat for open search as well - T162357 (duration: 03m 29s)
  • 17:36 moritzm: uploaded linux 4.9.110+deb9u1~wmf1 for jessie-wikimedia to apt.wikimedia.org
  • 17:34 mobrovac@deploy1001: Started deploy [citoid/deploy@4f5ba15]: Use WorldCat for open search as well - T162357
  • 17:04 gehel@deploy1001: Finished deploy [wdqs/wdqs@0d3c6a6]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 33s)
  • 17:04 gehel@deploy1001: Started deploy [wdqs/wdqs@0d3c6a6]: new version of wdqs GUI and updater (wdqs1009 only)
  • 16:53 andrewbogott: power down labservices1001 for thermal paste fix, T196252
  • 16:53 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1068.eqiad.wmnet
  • 16:52 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1089.eqiad.wmnet
  • 15:54 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1074.eqiad.wmnet
  • 15:50 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1090.eqiad.wmnet
  • 15:50 bblack: pooling cp1090 (upload@eqiad, first of new hardware)
  • 15:05 papaul: shutting down pc2006 for maintenance
  • 15:02 bblack: rebooting cp1075-90
  • 14:56 marostegui: Stop MySQL for onsite maintenance - T200641
  • 14:30 godog: roll-restart thumbor after adding new private containers - T201187
  • 14:07 marostegui: Reload haproxy to apply new configuration
  • 11:32 zeljkof: EU SWAT finished
  • 11:31 zfilipin@deploy1001: Synchronized wmf-config/LabsServices.php: SWAT: deployment-prep: Change urldownloader host (T201160) (duration: 00m 50s)
  • 11:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add suppressredirect permission to rollbacker and patroller at zhwiki (T201160) (duration: 00m 49s)
  • 11:13 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 48s)
  • 11:13 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
  • 11:08 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Give hewiki interface-admins the rights interface-editors have (T200698) (duration: 00m 50s)
  • 10:54 godog: repair sde on ms-be2040
  • 10:38 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 04m 38s)
  • 10:33 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
  • 10:33 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 04m 44s)
  • 10:29 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
  • 10:28 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 10m 51s)
  • 10:18 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
  • 10:17 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 11m 12s)
  • 10:06 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
  • 09:33 mobrovac@deploy1001: Finished deploy [cpjobqueue/deploy@62716a5]: CirrusSearch jobs: Increase checker concurrency to 10 - T198462 (duration: 00m 44s)
  • 09:32 mobrovac@deploy1001: Started deploy [cpjobqueue/deploy@62716a5]: CirrusSearch jobs: Increase checker concurrency to 10 - T198462
  • 09:31 mobrovac@deploy1001: Finished deploy [cpjobqueue/deploy@62716a5]: (no justification provided) (duration: 00m 27s)
  • 09:31 mobrovac@deploy1001: Started deploy [cpjobqueue/deploy@62716a5]: (no justification provided)
  • 08:24 marostegui: Remove unused mysql users allowed to connect from 208.80.152.226
  • 08:23 _joe_: restarting logstash on logstash1007, losing packets as well
  • 08:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 (duration: 00m 51s)
  • 08:03 _joe_: restarting logstash on logstash1009, losing packets again
  • 07:51 _joe_: restarting logstash on logstash1008, losing packets again
  • 06:49 marostegui: Deploy schema change on db1077 with replication, this will generate lag on labsdb:s3 T144010 T51190 T199368
  • 06:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 (duration: 00m 53s)
  • 04:29 TimStarling: on mwmaint1001 running populateContentTables.php on metawiki T183488
  • 04:09 TimStarling: on mwmaint1001 running populateContentTables.php on mediawikiwiki T183488
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 6 02:45:42 UTC 2018 (duration 10m 28s)
  • 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 13m 40s)

2018-08-05

  • 02:41 andrewbogott: rebooting labservices1001 from mgmt

2018-08-04

  • 16:40 krinkle@deploy1001: Synchronized php-1.32.0-wmf.15/includes/resourceloader/ResourceLoaderWikiModule.php: I5cb93c173 (duration: 00m 52s)
  • 16:37 krinkle@deploy1001: Synchronized php-1.32.0-wmf.15/includes/Storage/DerivedPageDataUpdater.php: I5cb93c173 [1/2] (duration: 01m 03s)
  • 14:50 godog: roll-restart logstash because of increasing packet loss

2018-08-03

  • 23:07 XioNoX: - restart asw2-b5-eqiad into loader - T201145
  • 18:07 marxarelli: restarting jenkins on contint1001
  • 18:05 thcipriani: restarting releases-jenkins
  • 15:45 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1239.eqiad.wmnet
  • 14:13 jynus: stopping haproxy on dbproxy1006
  • 14:11 reedy@deploy1001: Synchronized php-1.32.0-wmf.15/includes/Storage/DerivedPageDataUpdater.php: Fix article counting logic in DerivedPageDataUpdater (duration: 00m 50s)
  • 13:06 ema: reboot cp3030 to SSBD-enabled microcode/kernel
  • 12:51 ema: trafficserver 7.1.3+ds-4wm2 uploaded to stretch-wikimedia T200178
  • 12:05 gehel: reimage relforge to stretch - T193649
  • 11:27 jynus: setting dbstore1001:s1 mariadb as read_only=1
  • 10:54 jynus: setting dbstore2002 instances as read only
  • 10:39 hoo: Running populateSitesTable.php on all Wikidata clients for T201003
  • 10:31 jynus: setting read only=1 on all instances on dbstore2001
  • 09:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 (duration: 00m 49s)
  • 09:50 jynus: testing new monitoring on dbstore2001
  • 09:36 paravoid: asw2-b-eqiad: set disable for xe-2/0/5 (cloudvirt1023), xe-7/0/22 (cloudvirt1024), ge-8/0/23 (rdb1009)
  • 09:36 jynus: disabling puppet on mariadb multiinstance hosts
  • 08:24 filippo@neodymium: conftool action : set/pooled=no; selector: name=logstash1007.eqiad.wmnet
  • 08:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 00m 50s)
  • 02:35 ejegg: updated CiviCRM from a0a7ea846a to 03012506b9
  • 00:57 mutante: restarted postgresql on netmon2001 as part of debugging issue with failed icinga replication check

2018-08-02

  • 23:26 maxsem@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikimediaEvents/+/450080/ (duration: 00m 49s)
  • 23:03 XioNoX: adding package anycast-healthchecker to stretch-wikimedia
  • 22:44 XioNoX: replacing package python3-jsonlogger with python3-json-logger in stretch-wikimedia
  • 22:21 mutante: webperf2002 - systemctl start proc-sys-fs-binfmt_misc.automount to fix "Check systemd degraded" Icinga check
  • 22:11 XioNoX: importing python3-jsonlogger_0.1.9 into stretch-wikimedia
  • 20:30 bd808@deploy1001: Finished deploy [striker/deploy@2329901]: Update Striker to 2329901 (T177407, T198076, T190543) (duration: 01m 26s)
  • 20:28 bd808@deploy1001: Started deploy [striker/deploy@2329901]: Update Striker to 2329901 (T177407, T198076, T190543)
  • 20:23 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/VisualEditor/includes/ApiVisualEditorEdit.php: sync https://gerrit.wikimedia.org/r/450090/ to unbreak prod refs T201083 T191061 (duration: 00m 49s)
  • 19:10 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.15 refs T191061
  • 19:07 twentyafterfour: T191061 is free of blockers, proceeding with the train deployment: group2 wikis to 1.32.0-wmf.15
  • 18:21 Amir1: ladsgroup@mwmaint1001:~$ mwscript extensions/ORES/maintenance/PurgeScoreCache.php --wiki=wikidatawiki and --old (T200680)
  • 18:20 Amir1: Morning SWAT is done
  • 18:20 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/ORES/maintenance/PurgeScoreCache.php: SWAT: Join decomposition on maintenance/PurgeScoreCache.php (T200680) (duration: 00m 57s)
  • 18:15 gehel: un-banning and repooling elastic1030 - T201039
  • 18:10 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix a typo in ORES models config (duration: 00m 57s)
  • 16:00 milimetric@deploy1001: Finished deploy [analytics/refinery@5e5b5a9]: Quick fix for sqoop script (duration: 07m 52s)
  • 15:53 milimetric@deploy1001: Started deploy [analytics/refinery@5e5b5a9]: Quick fix for sqoop script
  • 14:27 dcausse: restarting elastic1030 to trigger a master election
  • 13:45 ema: bounce traffic back from lvs3004 to lvs3002
  • 13:38 ema: reboot lvs3002 to SSBD-enabled microcode/kernel
  • 13:30 reedy@deploy1001: Synchronized dblists/: Update size dblists (duration: 00m 55s)
  • 13:18 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 fully (duration: 00m 56s)
  • 13:12 ema: bounce traffic from lvs3002 to lvs3004 (SSBD-enabled)
  • 13:01 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: fix wikimaniawiki wgserver and wgcanonicalserver (duration: 00m 56s)
  • 12:58 marostegui: Sanitize zhwikiversity satwiki T199599 T198401
  • 12:57 ema: reboot lvs3004 to SSBD-enabled microcode/kernel
  • 12:51 dcausse: banning elastic1030 (master struggling)
  • 12:48 marostegui: Sanitize wikimaniawiki - T201001
  • 12:45 dcausse: depooling elastic1030 (master struggling)
  • 12:35 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: update foundation url (duration: 00m 56s)
  • 12:34 reedy@deploy1001: Synchronized wmf-config/missing.php: Update foundation urls (duration: 00m 55s)
  • 12:21 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 25s)
  • 12:19 Reedy: Wikis created T196748 T198401 T199599
  • 12:18 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: fix id_internal (duration: 00m 55s)
  • 12:15 reedy@deploy1001: rebuilt and synchronized wikiversions files: new wikis!
  • 12:11 reedy@deploy1001: Synchronized wmf-config/: New wikis (duration: 00m 55s)
  • 12:10 reedy@deploy1001: Synchronized dblists/: new wikis (duration: 00m 55s)
  • 12:08 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: add id_internal (duration: 00m 55s)
  • 12:07 reedy@deploy1001: Synchronized langlist: Add sat (duration: 00m 55s)
  • 12:06 reedy@deploy1001: Synchronized static/images/project-logos/: New logos! (duration: 00m 57s)
  • 11:22 godog: temporarily remove multiline filter from logstash100[789] and bump pipeline workers to 4 - T200960
  • 11:21 moritzm: installing jansson security updates on trusty
  • 11:17 godog: temporarily remove multiline filter from logstash to allow using multiple workers - T200960
  • 11:16 moritzm: installing tomcat8 security updates
  • 11:12 zeljkof: EU SWAT finished
  • 11:11 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Enable moved paragraph detection for inline diffs on beta cluster (T200975) (duration: 00m 58s)
  • 11:03 godog: comment "pipeline.workers: 1" from logstash1007 - T200960
  • 10:47 godog: test bumping heap to 512mb on logstash1007 - T200960
  • 10:35 volans@deploy1001: Finished deploy [netbox/deploy@ac54feb]: Security upgrade of dependency (duration: 00m 05s)
  • 10:35 volans@deploy1001: Started deploy [netbox/deploy@ac54feb]: Security upgrade of dependency
  • 10:30 volans@deploy1001: Finished deploy [debmonitor/deploy@73b640d]: Release v0.1.7 (duration: 01m 01s)
  • 10:29 volans@deploy1001: Started deploy [debmonitor/deploy@73b640d]: Release v0.1.7
  • 10:25 volans@deploy1001: Finished deploy [debmonitor/deploy@691d2f8]: Release v0.1.7 (duration: 00m 51s)
  • 10:24 volans@deploy1001: Started deploy [debmonitor/deploy@691d2f8]: Release v0.1.7
  • 10:19 godog: test bumping rmem_default to 4MB on logstash1007 - T200960
  • 10:11 reedy@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/LabeledSectionTransclusion/: Consistency (duration: 00m 55s)
  • 10:10 reedy@deploy1001: Synchronized php-1.32.0-wmf.14/languages/Language.php: Revert out deprecation warning of hook (duration: 00m 57s)
  • 09:18 legoktm@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/LabeledSectionTransclusion/: revert again (duration: 00m 55s)
  • 09:14 legoktm@deploy1001: scap failed: RuntimeError Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org) (duration: 22m 08s)
  • 09:14 legoktm@deploy1001: Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org)
  • 08:52 legoktm@deploy1001: Started scap: try once more
  • 08:40 legoktm@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/LabeledSectionTransclusion/: revert (duration: 00m 56s)
  • 08:36 legoktm@deploy1001: scap failed: RuntimeError Scap failed!: 11/11 canaries failed their endpoint checks(http://en.wikipedia.org) (duration: 24m 33s)
  • 08:36 legoktm@deploy1001: Scap failed!: 11/11 canaries failed their endpoint checks(http://en.wikipedia.org)
  • 08:14 marostegui: Deploy schema change on db2043 (s3 codfw master) this will generate lag on codfw:s3 T144010 T51190 T199368
  • 08:11 legoktm@deploy1001: Started scap: LST: Use modern i18n mechanisms for localization (T198173, T200960)
  • 07:34 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 with low load (duration: 00m 56s)
  • 07:20 TimStarling: restarting logstash on logstash1008 and logstash1009
  • 07:06 TimStarling: on logstash1007 increased net.core.rmem_default from 212992 to 851968 in soft state and restarted logstash
  • 06:59 TimStarling: on logstash1007 restarting logstash
  • 06:58 jynus: stop db1092 for reimage
  • 06:53 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 55s)
  • 06:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 (duration: 00m 55s)
  • 06:18 marostegui: Deploy schema change on db1106 with replication, this will generate lag on labsdb:s1 T144010 T51190 T199368
  • 06:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 (duration: 00m 54s)
  • 06:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 (duration: 00m 55s)
  • 06:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 (duration: 00m 57s)
  • 05:30 TimStarling: on mwmaint1001 running populateContentTables.php as described in T183488
  • 05:14 TimStarling: restarted stashbot
  • 05:07 TimStarling: ran patch-slot-origin.sql on labswiki and labtestwiki to fix editing on labswiki (some SAL entries missing)
  • 02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 42s)

2018-08-01

  • 23:39 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscodeJob.php: SWAT: Work around transcode failures in newer ffmpeg (duration: 00m 56s)
  • 23:37 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscodeJob.php: SWAT: Work around transcode failures in newer ffmpeg (duration: 00m 57s)
  • 23:17 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Re-enable VP8 video transcodes to fix playback regression (duration: 00m 56s)
  • 22:58 krinkle@deploy1001: Synchronized wmf-config: I4db5d03f8af7 (step 2) (duration: 00m 57s)
  • 22:57 krinkle@deploy1001: Synchronized wmf-config: I4db5d03f8af7 (step 1) (duration: 00m 57s)
  • 20:36 twentyafterfour: T191061 Finished deploying 1.32.0-wmf.15 to group1 wikis. Fatalmonitor is quiet and everything appears to be stable. See you tomorrow for group2.
  • 20:24 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.15 refs T191061 (duration: 00m 55s)
  • 20:23 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.15 refs T191061
  • 20:17 ejegg: updated payments-wiki from 626ff1cb72 to 0c9c24d30c
  • 20:10 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@c2448e0]: Update mobileapps to 282f368 (T200464 T200459) (duration: 05m 42s)
  • 20:04 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@c2448e0]: Update mobileapps to 282f368 (T200464 T200459)
  • 19:53 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/Collection/: Sync Change: https://gerrit.wikimedia.org/r/449796 Bug: T189636 unblocks: T191061 (duration: 00m 57s)
  • 19:07 twentyafterfour: preparing to deploy 1.32.0-wmf.15 to group1 wikis. Gonna merge a couple of patches first to fix some logspam.
  • food: updated DjangoBannerStats from 73d74f1b5e to 02be6cbb74
  • 18:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #4 (duration: 03m 21s)
  • 18:10 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #4
  • 18:09 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #3 (duration: 06m 27s)
  • 18:03 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #3
  • 18:03 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #2 (duration: 05m 08s)
  • 17:58 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #2
  • 17:57 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens (duration: 12m 23s)
  • 17:47 ejegg: updated DjangoBannerStats from 43e47f98a2 to 73d74f1b5e
  • 17:45 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens
  • 17:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c] (dev-cluster): Multi-content bucket: delete old revisions when a new render happens (duration: 02m 52s)
  • 17:42 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c] (dev-cluster): Multi-content bucket: delete old revisions when a new render happens
  • 17:22 aaron@deploy1001: Synchronized wmf-config/mc.php: Use mcrouter for cache reads for test wikis (duration: 00m 55s)
  • 17:19 XioNoX: enable puppet on all cp2* servers after merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/449760 - T195365
  • 16:51 mforns: Finished deploying refinery using scap, then refinery-deploy-to-hdfs
  • 16:41 vgutierrez: Turn off AES128-SHA support - T192555 && T147202
  • 16:38 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/PageTriage/modules/ext.pageTriage.views.list/ext.pageTriage.listControlNav.js: SWAT: Fix namespaces shown in NPP filters T200826 T200821 (duration: 00m 55s)
  • 16:36 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/PageTriage/modules/ext.pageTriage.views.list/ext.pageTriage.listControlNav.js: SWAT: Fix namespaces shown in NPP filters T200826 T200821 (duration: 00m 58s)
  • 16:34 mforns@deploy1001: Finished deploy [analytics/refinery@7fba0fc]: fixing EL sanitization whitelist (duration: 07m 54s)
  • 16:26 mforns@deploy1001: Started deploy [analytics/refinery@7fba0fc]: fixing EL sanitization whitelist
  • 16:24 mforns: deploying analytics-refinery using scap
  • 16:08 moritzm: upgrading proton* to chromium 68.0.3440.75-1~deb9u1 (new release tested successfully in deployment-prep)
  • 15:50 moritzm: installing libarchive-zip-perl security updates
  • 15:43 akosiaris: upload mapnik_3.0.20+ds-1 to apt.wikimedia.org/stretch-wikimedia/main T200284
  • 15:16 XioNoX: disable puppet on all codfw cp* servers - T195365
  • 14:52 elukey: created user 'research' on db1108 (eventlogging slave - only select grant for the log database)
  • 13:49 moritzm: upgrade application servers in beta to wikidiff 1.7,2 (T199801)
  • 13:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 (duration: 00m 55s)
  • 13:29 marostegui: Deploy schema change on db1083 T144010 T51190 T199368
  • 13:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 00m 55s)
  • 13:02 volans: restarting ircecho on einsteinium to unblock icinga IRC bot
  • 12:59 moritzm: installing tomcat7 security updates on jessie
  • 12:35 marostegui: Deploy schema change on db1105:3311 T144010 T51190 T199368
  • 12:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 (duration: 00m 55s)
  • 12:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 (duration: 00m 55s)
  • 12:16 moritzm: installing fuse security updates
  • 12:16 marostegui: Deploy schema change on db1099:3311 T144010 T51190 T199368
  • 12:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 (duration: 00m 56s)
  • 11:56 zeljkof: EU SWAT finished
  • 11:54 zfilipin@deploy1001: Synchronized wmf-config/: SWAT: CleanupParent for draftquality model when PageTriage is used (T199357) (duration: 00m 56s)
  • 11:45 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.15/skins/MinervaNeue/: SWAT: Restore page issues (T200867) (duration: 00m 57s)
  • 11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable SandboxLink on the isiZulu Wikipedia (T200522) (duration: 00m 56s)
  • 11:20 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Fix config for $wgLocalisationUpdateRepositories (T148965) (duration: 00m 57s)
  • 11:17 moritzm: installing openjpeg2 security updates on jessie
  • 10:44 ema: reboot cp1045 for kernel update
  • 10:40 marostegui: Reload haproxy on dbproxy1008 and dbproxy1003
  • 09:50 moritzm: installing wireshark security updates
  • 09:50 marostegui: Deploy schema change on db2048 (s1 codfw master) with replication, this will generate lag on codfw s1 T144010 T51190 T199368
  • 09:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 55s)
  • 09:41 ema: cp3044 (upload) repooled after upgrade to stretch T200445
  • 09:39 marostegui: Run community_metrics.sh on phab1001
  • 09:36 moritzm: installing clamav security updates
  • 09:19 marostegui: Deploy schema change on db1092 T144010 T51190 T199368
  • 09:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 55s)
  • 09:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 56s)
  • 08:38 elukey: restart hadoop-yarn-nodemanager on analytics10[31-77] to apply the new memory settings
  • 08:37 marostegui: Deploy schema change on db1109:3318 T144010 T51190 T199368
  • 08:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 55s)
  • 08:22 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s1 populateChangeTagDef.php --sleep 2 (T193873)
  • 08:21 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s8 populateChangeTagDef.php --sleep 3 (T193873)
  • 08:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 56s)
  • 08:16 marostegui: Stop MySQL on db2078
  • 07:59 elukey: restart hadoop-yarn-nodemanager on analytics10[28-30] to test new memory settings
  • 07:12 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db1052 from config as it will be decommissioned - T199861 (duration: 00m 55s)
  • 07:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db1052 from config as it will be decommissioned - T199861 (duration: 00m 56s)
  • 07:09 marostegui: Remove db1052 from tendril - T199861
  • 06:59 elukey: restart eventlogging on eventlog1002 to pick up new logging settings
  • 06:58 elukey@deploy1001: Finished deploy [eventlogging/analytics@762ca2b]: Deploy https://gerrit.wikimedia.org/r/#/c/eventlogging/+/449422/ (duration: 00m 07s)
  • 06:58 elukey@deploy1001: Started deploy [eventlogging/analytics@762ca2b]: Deploy https://gerrit.wikimedia.org/r/#/c/eventlogging/+/449422/
  • 06:47 marostegui: Deploy schema change on db1087 with replication, this will generate lag on labs:s8 T144010 T51190 T199368
  • 06:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 (duration: 00m 55s)
  • 06:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic db1089 (duration: 00m 55s)
  • 06:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 55s)
  • 06:22 brion: restarting transcode batch job (errors believed caused by hhvm config restarts)
  • 06:11 brion: stopping requeueTranscodes.php for now
  • 06:11 brion: seeing spike in errors on video scalers
  • 05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1089 (duration: 00m 57s)
  • 05:02 marostegui: Stop MySQL on db1089 for a binlog format change (and also upgrade kernel)
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 (duration: 00m 57s)
  • 04:47 marostegui: Stop MySQL on db1052 to copy its content to dbstore1001 - https://phabricator.wikimedia.org/T199861
  • 04:46 marostegui: Deploy schema change on db1101:3318 T144010 T51190 T199368
  • 04:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 01m 06s)
  • 03:17 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 1 03:17:12 UTC 2018 (duration 10m 24s)
  • 03:06 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 14m 45s)
  • 02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 14m 25s)
  • 00:15 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php: SWAT: Workaround for job queue reporting 0 length (T200813) (duration: 00m 56s)
  • 00:09 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/ParserFunctions/includes/ExtParserFunctions.php: SWAT: Remove & from $mwDefault variable assignment T200772 (duration: 00m 57s)

2018-07-31

  • 23:55 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php: SWAT: Workaround for job queue reporting 0 length (T200813) (duration: 00m 57s)
  • 23:41 ejegg: updated payments-wiki from 6afaca3de3 to 626ff1cb72
  • 23:32 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable $wgCiteResponsiveReferences for Meta-Wiki T200707 (duration: 00m 56s)
  • 23:21 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateStyles on Meta-Wiki T200613 (duration: 00m 57s)
  • 22:56 eileen: update CiviCRM civicrm revision changed from 24b497072d to a0a7ea846a, config revision is 35a258d015
  • 21:37 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.15
  • 21:05 twentyafterfour@deploy1001: Finished scap: testwikis wikis to 1.32.0-wmf.15 (duration: 44m 49s)
  • 20:20 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.32.0-wmf.15
  • 19:00 urandom: enabling `unchecked_tombstone_compaction` on enwiki_T_mobile__ng_remaining -- T192689
  • 18:20 XioNoX: stopping puppet across the fleet for puppetmaster1001 uplink move
  • 18:04 XioNoX: stopping pybal on lvs1002
  • 18:01 XioNoX: repool lvs1001
  • 17:53 XioNoX: stopping pybal on lvs1001
  • 17:34 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=dns,service=pdns_recursor,name=chromium.wikimedia.org
  • 17:19 twentyafterfour: branching 1.32.0-wmf.15 refs T191061
  • 17:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all hosts in row B - T183585 (duration: 00m 51s)
  • 16:40 elukey: repool druid1005 after network maintenance
  • 16:26 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=dns,service=pdns_recursor,name=chromium.wikimedia.org
  • 16:25 godog: repool thumbor100[12] - T183585
  • 16:25 elukey: pool eventbus on kafka1002 after network maintenance
  • 16:24 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=aqs,service=cassandra,name=aqs1004.eqiad.wmnet
  • 16:24 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=aqs,service=aqs,name=aqs1004.eqiad.wmnet
  • 16:16 elukey: precautionary restart of eventbus on kafka1002 after network downtime (DNS name res errors, Kafka broker conn issues, etc..)
  • 16:07 marostegui: Reload dbproxy1003 and dbproxy1008
  • 16:06 ema: reboot cp3044 for kernel update
  • 16:01 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=prometheus,service=prometheus,name=prometheus1004.eqiad.wmnet
  • 15:51 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=druid-public,service=druid-public-broker,name=druid1005.eqiad.wmnet
  • 15:50 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=aqs,service=cassandra,name=aqs1004.eqiad.wmnet
  • 15:50 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=aqs,service=aqs,name=aqs1004.eqiad.wmnet
  • 15:46 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=eventbus,service=eventbus,name=kafka1002.eqiad.wmnet
  • 15:45 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=prometheus,service=prometheus,name=prometheus1004.eqiad.wmnet
  • 15:36 ema: power-cycle cp3031, stuck rebooting
  • 15:29 _joe_: restarting apache on phab1001
  • 15:20 godog: depool thumbor100[12] ahead of switch move - T183585
  • 15:02 XioNoX: starting the eqiad row B servers move - T183585
  • 13:51 moritzm: installing ffmpeg 3.2.12 security updates on video scalers
  • 13:34 akosiaris: repool mathoid eqiad cluster in discovery
  • 13:18 akosiaris: upgrade kubernetes1004 to 1.9.9
  • 13:14 akosiaris: upgrade kubernetes1002 to 1.9.9
  • 13:14 akosiaris: upgrade kubernetes1003 to 1.9.9
  • 13:03 akosiaris: upgrade kubernetes1001 to 1.9.9
  • 12:58 akosiaris: upgrade chlorine.eqiad.wmnet to kubernetes 1.9.9
  • 12:54 akosiaris: upgrade argon.eqiad.wmnet to kubernetes 1.9.9
  • 12:45 akosiaris: depool eqiad mathoid in discovery
  • 12:29 gehel@puppetmaster1001: conftool action : set/weight=15; selector: dc=eqiad,cluster=wdqs,name=wdqs1005.eqiad.wmnet
  • 12:29 gehel@puppetmaster1001: conftool action : set/weight=15; selector: dc=eqiad,cluster=wdqs,name=wdqs1004.eqiad.wmnet
  • 12:29 gehel: rebalance LVS weights to send less traffic to wdqs1003 - T200563
  • 12:13 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove logging now in mw core (duration: 00m 48s)
  • 12:11 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmg - for Sentry (duration: 00m 48s)
  • 12:09 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove wmg - for Sentry (duration: 00m 48s)
  • 12:06 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Load Sentry with wfLoadExtension (duration: 00m 48s)
  • 12:05 reedy@deploy1001: Synchronized wmf-config/liquidthreads.php: Remove lqt cruft, load with wfLoadExtension (duration: 00m 48s)
  • 11:44 gehel: restarting blazegraph on wdqs1003 - JVM unresponsive
  • 11:38 zeljkof: EU SWAT finished
  • 11:37 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Introduce autopatrolled on bnwikisource (T199475) (duration: 00m 48s)
  • 11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos for hewiktionary (T200470) (duration: 00m 48s)
  • 11:24 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for hewiktionary (T200470) (duration: 00m 48s)
  • 11:16 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Milestone logo for atjwiki (T200713) (duration: 00m 49s)
  • 11:10 zfilipin@deploy1001: Synchronized static/images/project-logos/hewikiquote.png: SWAT: Update hewikiquote logo (T200296) (duration: 00m 50s)
  • 10:55 _joe_: test log
  • 10:40 akosiaris@deploy1001: scap-helm mathoid finished
  • 10:40 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
  • 10:39 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
  • 10:39 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw]
  • 09:46 ema: reboot cp1045 (jessie) for kernel updates
  • 09:29 ema: upload varnishkafka 1.0.13-1 to stretch-wikimedia T200445 T186250
  • 08:44 jynus: stop and reimage db1104 to stretch
  • 08:17 akosiaris: repool mathoid codfw in discovery, the kubernetes upgrade is done
  • 08:08 akosiaris: upgrade kubernetes2004 to 1.9.9
  • 08:05 akosiaris: upgrade kubernetes2003 to 1.9.9
  • 08:00 akosiaris: upgrade kubernetes2002 to 1.9.9
  • 07:55 ema: reboot cp2018 for kernel updates
  • 07:54 moritzm: reboot cp1008 for kernel tests
  • 07:53 akosiaris: upgrade kubernetes2001 to 1.9.9
  • 07:50 ema: reboot cp2025 for kernel updates
  • 07:47 akosiaris: upgrade acrux.codfw.wmnet to kubernetes 1.9.9
  • 07:44 jynus: stopping db1118 mariadb and starting mysql there
  • 07:36 moritzm: reboot multatuli for kernel tests
  • 07:35 akosiaris: upgrade acrab.codfw.wmnet to kubernetes 1.9.9
  • 07:32 moritzm: run decomission_appserver on terbium
  • 07:27 akosiaris: backup kubetcd2001 etcd data before kubernetes upgrade
  • 07:24 akosiaris: depool codfw mathoid from discovery for kubernetes upgrade
  • 07:10 moritzm: installing mutt security updates
  • 07:05 marostegui: Remove unused and old grants from codfw hosts - T146149#4456082
  • 06:49 jynus: dropping unnecesary tendril web users from tendril db backends
  • 06:45 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool row B es hosts, reduce db1089 load (duration: 00m 49s)
  • 06:11 moritzm: installing ant security updates
  • 05:01 marostegui: Deploy schema change on db1099:3318 T144010 T51190 T199368
  • 04:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool all hosts in row B - T183585 (duration: 00m 50s)
  • 04:47 marostegui: Stop change_tag_def populate maintenance script for wikidata and enwiki - T193873
  • 02:37 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 31 02:37:40 UTC 2018 (duration 10m 20s)
  • 02:27 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 08m 12s)
  • 01:20 ejegg: updated payments-wiki from 344233cff8 to 6afaca3de3
  • 00:44 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Switch in WebM VP9/Opus video transcodes to replace WebM VP8/Vorbis T63805 (duration: 00m 48s)
  • 00:39 thcipriani@deploy1001: Finished scap: SWAT: Convert to extension.json T87981 Use maps for wgEnabledTranscodeSet T118080 Adjust VP9 encoding for speed, quality More conservative max bitrate for VP9 video transcodes (duration: 33m 44s)
  • 00:07 ejegg: updated payments-wiki from 8c4d6aaa13 to 344233cff8
  • 00:05 thcipriani@deploy1001: Started scap: SWAT: Convert to extension.json T87981 Use maps for wgEnabledTranscodeSet T118080 Adjust VP9 encoding for speed, quality More conservative max bitrate for VP9 video transcodes

2018-07-30

  • 23:35 XioNoX: re-enabling puppet on cp40* - T195365
  • 23:27 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/RevisionSlider/modules: SWAT: RevisionSlider: Fix missing pin icon T200263 (duration: 00m 49s)
  • 23:15 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:Block Feedback Request (2) T199919 (duration: 00m 49s)
  • 22:35 XioNoX: applying static route + fixed MTU to cp4025 - T195365
  • 22:29 XioNoX: - puppet disabled on cp40* hosts - T195365
  • 22:10 thcipriani: restarting jenkins after plugin updates
  • 21:55 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (cobalt) (duration: 00m 10s)
  • 21:55 thcipriani@deploy1001: Started deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (cobalt)
  • 21:51 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (gerrit2001 only) (duration: 00m 09s)
  • 21:51 thcipriani@deploy1001: Started deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (gerrit2001 only)
  • 21:01 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@debb5b0]: Update spark.yaml with latest training configuration (duration: 04m 12s)
  • 20:56 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@debb5b0]: Update spark.yaml with latest training configuration
  • 20:41 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@69ac0d5]: fix wrong yaml import in cli entry point (duration: 02m 30s)
  • 20:39 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@69ac0d5]: fix wrong yaml import in cli entry point
  • 20:14 ejegg: updated CiviCRM from d76e3997e6 to 24b497072d
  • 20:10 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@61c5f47]: ship missing bc alias for deployed daemon (duration: 01m 35s)
  • 20:09 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@61c5f47]: ship missing bc alias for deployed daemon
  • 19:53 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 3 (duration: 06m 25s)
  • 19:47 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 3
  • 19:46 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 2 (duration: 05m 06s)
  • 19:40 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 2
  • 19:40 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 (duration: 14m 31s)
  • 19:26 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465
  • 19:24 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b] (dev-cluster): Language variants for summaries T198465 (duration: 03m 05s)
  • 19:22 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@b1e18ca]: (no justification provided) (duration: 01m 28s)
  • 19:21 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@b1e18ca]: (no justification provided)
  • 19:20 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b] (dev-cluster): Language variants for summaries T198465
  • 19:19 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 01m 32s)
  • 19:17 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
  • 18:57 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/WikimediaMessages/WikimediaMessages.hooks.php: SWAT: Escape Special:Block Feedback Request Message T194301 (duration: 00m 49s)
  • 18:25 thcipriani@deploy1001: Synchronized wmf-config/mc.php: SWAT: Make all wikis write to both nutcracker and mcrouter (3) T198239 (duration: 00m 48s)
  • 18:10 krinkle@deploy1001: Finished deploy [performance/navtiming@f79b313]: (no justification provided) (duration: 00m 05s)
  • 18:10 krinkle@deploy1001: Started deploy [performance/navtiming@f79b313]: (no justification provided)
  • 17:12 XioNoX: repool ulsfo
  • 17:11 gehel@deploy1001: Finished deploy [wdqs/wdqs@137780f]: new version of wdqs GUI (duration: 09m 17s)
  • 17:02 gehel@deploy1001: Started deploy [wdqs/wdqs@137780f]: new version of wdqs GUI
  • 17:01 gehel@deploy1001: Finished deploy [wdqs/wdqs@137780f]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 30s)
  • 17:00 gehel@deploy1001: Started deploy [wdqs/wdqs@137780f]: new version of wdqs GUI (wdqs1009 only)
  • 15:57 ema: pool cp2012, upgraded to stretch T200445
  • 15:50 ema: reboot cp2012 for kernel update
  • 15:27 jynus: restart and upgrade mariadb at db1118
  • 15:14 bblack: reboot cp1075
  • 15:08 godog: upgrade hp raid firmware on ms-be1028 - T141756
  • 15:05 Amir1: running ladsgroup@mwmaint1001:~$ mwscript extensions/ORES/maintenance/PurgeScoreCache.php on all wikis
  • 13:58 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.14
  • 13:39 ema: reboot cp2006 for kernel update
  • 12:59 zeljkof: EU SWAT finished
  • 12:58 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/AbuseFilter/: SWAT: Fix jQuery selector when editing filters (T200604) (duration: 00m 55s)
  • 12:55 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/FileImporter/: SWAT: Fix flipped array indexes in template removal code (T200406) (duration: 00m 57s)
  • 12:45 godog: fix group write on deploy1001 /srv/mediawiki-staging/php-1.32.0-wmf.14/.git/objects
  • 12:40 godog: reboot ms-be2040 with page_poisoning=1 - T199198
  • 12:33 tgr@deploy1001: Finished scap: T190015 Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default (duration: 60m 10s)
  • 11:39 ema: pool cp2006, upgraded to stretch T200445
  • 11:39 moritzm: upgrading intel-microcode to 20180703 on the servers which have microcode updates enabled (reboots to pick up the new version will be coordinated separately)
  • 11:33 tgr@deploy1001: Started scap: T190015 Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default
  • 11:12 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reading from change_tag_def everywhere (T199334) (duration: 00m 55s)
  • 11:10 volans: upgrading cumin to 3.0.2 on the remaining cumin masters (neodymium, labpuppetmaster*)
  • 11:01 volans: upgrading cumin to 3.0.2 on sarin
  • 10:50 volans: uploaded cumin_3.0.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
  • 10:25 godog: run xfs_repair on sdc1 on ms-be2040 - T199198
  • 10:04 akosiaris@deploy1001: scap-helm help finished
  • 10:04 akosiaris@deploy1001: scap-helm help cluster codfw completed
  • 10:04 akosiaris@deploy1001: scap-helm help cluster eqiad completed
  • 10:04 akosiaris@deploy1001: scap-helm help [namespace: help, clusters: eqiad,codfw]
  • 10:03 akosiaris: upgrade kubernetes staging cluster to 1.9.9
  • 10:01 akosiaris: reboot kubestage1001 for kubernetes upgrade
  • 09:59 akosiaris: reboot kubestage1002 for kubernetes upgrade
  • 09:31 moritzm: uploaded intel-microcode 20180703 for jessie-wikimedia/stretch-wikimedia to apt.wikimedia.org (tested successfully on a number of canary hosts for approx two weeks)
  • 09:30 jynus: migrate dbtree to dbmonitor1001
  • 09:25 akosiaris@deploy1001: scap-helm mathoid finished
  • 09:14 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
  • 09:13 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
  • 09:09 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw]
  • 09:08 akosiaris: upgrade mathoid helm chart from 0.0.5 to 0.0.9
  • 09:07 akosiaris@deploy1001: scap-helm mathoid finished
  • 09:07 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
  • 09:07 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
  • 09:07 akosiaris@deploy1001: scap-helm mathoid upgrade -h [namespace: mathoid, clusters: eqiad,codfw]
  • 08:49 marostegui: Deploy schema change on dbstore1002:s8 T144010 T51190 T199368
  • 08:07 elukey@deploy1001: Finished deploy [eventlogging/analytics@54d43e4]: Band aid for T200630 (duration: 00m 05s)
  • 08:07 elukey@deploy1001: Started deploy [eventlogging/analytics@54d43e4]: Band aid for T200630
  • 07:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Give some traffic to db1120 to on x1 (duration: 00m 53s)
  • 07:43 moritzm: installing opencv security updates
  • 07:39 moritzm: installing mercurial security updates
  • 07:32 marostegui: Deploy schema change on db2045 (s8 codfw master) this will generate lag on s8 codfw T144010 T51190 T199368
  • 07:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Add db1120 to x1 T196376 (duration: 00m 54s)
  • 07:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Add db1120 to x1 T196376 (duration: 00m 59s)
  • 05:28 marostegui: Deploy schema change on db1068 (commonswiki master) - T51190
  • 05:00 marostegui: Deploy schema change on db1062 (s7 primary master) T144010 T51190 T199368
  • 03:13 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 30 03:13:42 UTC 2018 (duration 10m 29s)
  • 03:03 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 33s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 11m 41s)

2018-07-29

  • 09:56 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2006 T200641 (duration: 00m 57s)
  • 02:35 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Neuter EducationProgram T188410 (duration: 00m 58s)

2018-07-28

  • 17:49 Krinkle: Navigation Timing alerts indicate we're losing a significant about of traffic
  • 17:35 elukey: restart eventlogging on eventlog1002 after tons of kafka disconnects (still not clear what happened)
  • 08:16 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 fully (duration: 00m 57s)

2018-07-27

  • 22:05 ejegg: updated payments-wiki from f0138ab066 to 8c4d6aaa13
  • 21:39 ejegg: updated payments-wiki from 5e01f50983 to f0138ab066
  • 20:40 SMalyshev: re-pooled wdqs1003
  • 19:04 mutante: terbium is being removed from ferm rules on elasticsearch/relforge, logstash/collector, mariadb/labtestwikitech and mw-maintenance itself (T192092)
  • 18:57 SMalyshev: temporarily depooled wdq1003 to let it catch up with lag
  • 18:15 mutante: syncing home dirs from mwmaint1001 to mwmaint2001 (once, manually, not currently set to auto-sync, but to keep another copy of former terbium homes in case we failover) (T192092)
  • 18:12 mutante: terbium: deleting rsyncd config fragment, stopping rsyncd, not used anymore, decom prep
  • 17:06 gehel: repooling wdqs1003
  • 16:43 gehel: depooling wdqs1003 to let it catch up on update lag
  • 15:24 joal@deploy1001: Finished deploy [analytics/aqs/deploy@6fafc63]: Update AQS new-pages metric (duration: 04m 12s)
  • 15:20 joal@deploy1001: Started deploy [analytics/aqs/deploy@6fafc63]: Update AQS new-pages metric
  • 15:06 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 with low load (duration: 00m 55s)
  • 14:36 mutante: terbium - removing accountcheck cron job (duplicate mail from the same thing now on mwmaint1001)
  • 14:23 jynus: stop and reimage db1094
  • 14:18 elukey: execute echo 'https://wikimania.wikimedia.org' | mwscript purgeList.php on mwmain1001
  • 13:27 ema: libvmod-re2 1.3.1-2 uploaded to stretch-wikimedia T200445
  • 13:15 ema: libvmod-netmapper 1.7-2 uploaded to stretch-wikimedia T200445
  • 13:09 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 for reimage (duration: 00m 54s)
  • 11:33 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@941e3cf]: Make the main processing loop async - T199813 (duration: 02m 02s)
  • 11:31 mobrovac@deploy1001: Started deploy [eventstreams/deploy@941e3cf]: Make the main processing loop async - T199813
  • 09:37 ema: vhtcpd 0.1.1-2 uploaded to stretch-wikimedia T200445
  • 09:00 godog: adjust aggregation to 'sum' for MediaWiki.edit sum metrics - T199968
  • 08:36 Amir1: ladsgroup@mwmaint1001:~$ foreachwikiindblist s3 populateChangeTagDef.php --sleep 2 (T193873)
  • 05:59 akosiaris: reboot webperf1002, webperf2002 for new disk to appear T199853
  • 05:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 (duration: 00m 55s)
  • 04:59 marostegui: Deploy schema change on db1094 T144010 T51190 T199368
  • 04:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 57s)
  • 01:19 aaron@deploy1001: Synchronized php-1.32.0-wmf.14/includes/libs/objectcache/BagOStuff.php: f208a431f912 (duration: 00m 56s)

2018-07-26

  • 23:04 XioNoX: depool ulsfo, outages on both physical links to the site
  • 21:57 ejegg: updated payments-wiki from 6390e50abd to 5e01f50983
  • 21:33 tgr@deploy1001: Synchronized php-1.32.0-wmf.14/includes/Title.php: SWAT T200456: Handle $title === null in Title::newFromText (duration: 00m 57s)
  • 21:22 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T200346 Update custom domain for thumbnail rendering (duration: 00m 54s)
  • 21:21 tgr@deploy1001: Synchronized wmf-config/LabsServices.php: SWAT: T200346 Update custom domain for thumbnail rendering (duration: 00m 54s)
  • 21:19 tgr@deploy1001: Synchronized wmf-config/ProductionServices.php: SWAT: T200346 Update custom domain for thumbnail rendering (duration: 00m 55s)
  • 21:12 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T190015 gerrit 421122 + 421123 prepare for interface-admin group (duration: 00m 54s)
  • 21:11 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: T190015 gerrit 421122 + 421123 prepare for interface-admin group (duration: 00m 55s)
  • 20:55 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Do not set deprecated value for $wgExternalDiffEngine (duration: 00m 54s)
  • 20:39 addshore@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/Wikibase/repo/includes/Store/Sql/SqlChangeDispatchCoordinator.php: Use getClientLockName value for releaseClientLock when dispatching T200420 (duration: 00m 57s)
  • 20:12 ebernhardson: set merge.policy.reclaim_deletes_weight back to 2.0 for wikidatawiki_content
  • 20:11 thcipriani: gerrit upgrade to 2.15.3 complete
  • 20:06 thcipriani: restart gerrit for notedb config update
  • 20:03 thcipriani: running puppet on cobalt for gerrit notedb changes
  • 19:59 mobrovac@deploy1001: Finished deploy [proton/deploy@883cacd]: Use a more secure MW API template - T198461 (duration: 00m 33s)
  • 19:58 mobrovac@deploy1001: Started deploy [proton/deploy@883cacd]: Use a more secure MW API template - T198461
  • 19:40 thcipriani: restarting gerrit
  • 19:39 thcipriani: running puppet on cobalt for gerrit.config update
  • 19:38 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@b1f577d]: Provide better stream and error handling and stop using compression - T199813 (duration: 02m 13s)
  • 19:36 mobrovac@deploy1001: Started deploy [eventstreams/deploy@b1f577d]: Provide better stream and error handling and stop using compression - T199813
  • 19:31 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@b69b468]: upgrade to gerrit 2.15.3 (duration: 00m 08s)
  • 19:31 thcipriani@deploy1001: Started deploy [gerrit/gerrit@b69b468]: upgrade to gerrit 2.15.3
  • 19:22 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@b69b468]: deploy gerrit 2.15.3 to cold-replica (duration: 00m 10s)
  • 19:22 thcipriani@deploy1001: Started deploy [gerrit/gerrit@b69b468]: deploy gerrit 2.15.3 to cold-replica
  • 18:19 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: Revert "all wikis to 1.32.0-wmf.14"
  • 18:17 ebernhardson: set merge.policy.reclaim_deletes_weight=12.0 for wikidatawiki_content in eqiad to attempt to reduce the deleted documents count
  • 18:13 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.14
  • 18:08 arlolra: Updated Parsoid to cdf8ace (T194806, T110004, T156099, T188478, T194083)
  • 17:59 arlolra@deploy1001: Finished deploy [parsoid/deploy@1dce68c]: Updating Parsoid to cdf8ace (duration: 11m 22s)
  • 17:58 XioNoX: moving row C vrrp master back to cr2 - T187962
  • 17:54 mobrovac@deploy1001: Finished deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test, take #2 - T199491 (duration: 06m 16s)
  • 17:47 arlolra@deploy1001: Started deploy [parsoid/deploy@1dce68c]: Updating Parsoid to cdf8ace
  • 17:47 mobrovac@deploy1001: Started deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test, take #2 - T199491
  • 17:46 mobrovac@deploy1001: Finished deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test - T199491 (duration: 19m 36s)
  • 17:45 mutante: created new archiva user tyler for Tyler Cipriani and gave him role of global repo manager (for gerrit uploads instead of using archiva-deploy user which can cause password sync issues, so personalized users )
  • 17:27 mobrovac@deploy1001: Started deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test - T199491
  • 17:01 XioNoX: moving row C vrrp master to cr1 - T187962
  • 16:08 gehel: rolling restart of elasticsearch / cirrus / eqiad completed - T156137
  • 15:47 mobrovac@deploy1001: Finished deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491 (duration: 01m 04s)
  • 15:46 mobrovac@deploy1001: Started deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491
  • 15:46 mobrovac@deploy1001: Finished deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491 (duration: 14m 56s)
  • 15:31 mobrovac@deploy1001: Started deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491
  • 15:30 mobrovac@deploy1001: Finished deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491 (duration: 37m 37s)
  • 15:27 addshore@deploy1001: Synchronized wmf-config: Remove unused wikibaseDispatchRedisLockManager logging T200420 (duration: 00m 56s)
  • 15:21 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add LockManager logging T200420 (duration: 00m 55s)
  • 14:52 mobrovac@deploy1001: Started deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491
  • 14:38 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add wikibaseDispatchRedisLockManager to wmgMonologChannels T200420 (duration: 00m 54s)
  • 14:35 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Add a logger to wikibaseDispatchRedisLockManager T200420 (duration: 00m 56s)
  • 14:34 addshore@deploy1001: sync-file aborted: Add a logger to wikibaseDispatchRedisLockManager T200420 (duration: 00m 02s)
  • 14:24 akosiaris: sudo gnt-instance modify --disk add:size=150G webperf1002 T199853
  • 14:24 akosiaris: sudo gnt-instance modify --disk add:size=150G webperf2002 T199853
  • 13:44 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new wikibase dispatch lock manager on wikidatawiki T200420 T178652 (duration: 00m 55s)
  • 12:44 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: T200420 - Wikidata, dispatch, select 20 instead of 15 wikis (duration: 00m 55s)
  • 12:38 reedy@deploy1001: rebuilt and synchronized wikiversions files: wikidatawiki back to .13 T200420
  • 11:57 zeljkof: EU SWAT finished
  • 11:56 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos in hewikiquote (T200296) (duration: 00m 52s)
  • 11:45 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Upload HD logos for hewikiquote (T200296) (duration: 00m 54s)
  • 11:37 zfilipin@deploy1001: Synchronized tests/InitialiseSettingsTest.php: SWAT: Test spaces in ExtraNamespaces (T199162) (duration: 00m 55s)
  • 11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create TemplateEditor group on enwikivoyage (T198056) (duration: 00m 55s)
  • 11:20 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix unexpected space in pswikis ExtraNamespaces definition (T199480) (duration: 00m 55s)
  • 11:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Dont assign sysop-level privileges to bureaucrats explicitly (T197095) (duration: 00m 56s)
  • 11:02 Amir1: ladsgroup@mwmaint1001:~$ foreachwikiindblist wikitech populateChangeTagDef.php (T193873)
  • 11:01 Amir1: ladsgroup@mwmaint1001:~$ mwscript sql.php --wiki=labtestwiki /srv/mediawiki-staging/php-1.32.0-wmf.14/maintenance/archives/patch-change_tag_def.sql
  • 10:46 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.14 (duration: 00m 54s)
  • 10:45 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.14
  • 10:33 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES on test2wiki (T200412) (duration: 00m 55s)
  • 10:26 Amir1: ladsgroup@mwmaint1001:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=test2wiki ores (T200412)
  • 10:24 ema: cache-text: upgrade to varnish 5.1.3-1wm9 and apply alternate domains patch T164609
  • 10:08 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.32.0-wmf.14"
  • 10:07 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s7 populateChangeTagDef.php --sleep 2 (T193873)
  • 09:50 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.14 (duration: 00m 53s)
  • 09:49 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.14
  • 09:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 55s)
  • 09:40 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 fully (duration: 00m 55s)
  • 09:27 gehel: rolling restart of elasticsearch / cirrus / eqiad to disable G1 - T156137
  • 09:22 marostegui: Deploy schema change on db1079 with replication, this will generate lag on labsdb:s7 T144010 T51190 T199368
  • 09:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 55s)
  • 09:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 55s)
  • 09:00 jynus: restarting hhvm at mw1226, mw1276, mw1281
  • 08:54 jynus: restarting hhvm at mw1230
  • 08:26 joal@deploy1001: Finished deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts - try 2 (duration: 29m 02s)
  • 08:20 marostegui: Deploy schema change on db1090:3317 T144010 T51190 T199368
  • 08:18 mobrovac@deploy1001: Synchronized wmf-config/CommonSettings.php: Set readOnlyReason to false everywhere for JobQueueEventBus - T199594 (duration: 00m 55s)
  • 08:10 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 with low load after reimage (duration: 00m 55s)
  • 07:57 joal@deploy1001: Started deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts - try 2
  • 07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 54s)
  • 07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 (duration: 00m 54s)
  • 07:24 jynus: stop and reimage es1015
  • 07:19 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1015 (duration: 00m 55s)
  • 06:47 marostegui: Deploy schema change on db1086 T144010 T51190 T199368
  • 06:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 55s)
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 (duration: 00m 55s)
  • 05:02 marostegui: Deploy schema change on db1101:3317 T144010 T51190 T199368
  • 05:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 (duration: 01m 05s)
  • 04:13 kartik@deploy1001: Finished deploy [cxserver/deploy@3c99775]: Update cxserver to e98c81vb (T200283) (duration: 05m 12s)
  • 04:08 kartik@deploy1001: Started deploy [cxserver/deploy@3c99775]: Update cxserver to e98c81vb (T200283)
  • 03:16 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 26 03:16:04 UTC 2018 (duration 10m 24s)
  • 03:06 twentyafterfour: stopped phabricator rebuild-identities migration as it's suspected of causing database connection exhaustion and replag
  • 03:05 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 13s)
  • 02:50 Krinkle: phabricator.wikimedia.org is spewing HTTP 500 ("Can Not Connect to MySQL")
  • 02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 40s)
  • 01:42 mutante: phab1002 - starting rsync of repo data from phab1001 in a screen session after gerrit:447949 T190568
  • 01:32 mutante: phab1001 - rm /usr/local/sbin/sync-srv-repos that has a reference to non-existing server iridium.eqiad.wmnet (formerly phab) (T190568)
  • 00:16 twentyafterfour: phabricator update complete with only momentary downtime
  • 00:04 twentyafterfour: deploying phabricator release/2018-07-25/1, scheduled downtime for phab1001 in icinga
  • 00:01 mutante: temp. disable puppet on netmon1002,netmon2001 before applying puppet ferm change out of abundance of caution. applied gerrit:439554 without issues on netmon1003

2018-07-25

  • 23:35 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWInlineImageNode.js: SWAT: ve.dm.MWInlineImageNode: Fix undefined data-mw in toDomElements output T198941 T200214 (duration: 00m 57s)
  • 23:05 mutante: added dchen to LDAP group "wmf" (was already in admins as shell user, so didn't have to be added in puppet repo) (T200366)
  • 20:23 bblack: pooling cp5006 - T187157
  • 20:05 krinkle@deploy1001: Synchronized wmf-config/: Remove wgUploadThumbnailRenderHttpCustomDomain override - T200346 (duration: 00m 57s)
  • 18:59 bstorm_: Added new disk shelf to labstore1007
  • 17:53 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 54s)
  • 17:45 krinkle@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/Cite/includes/Cite.php: (no justification provided) (duration: 00m 55s)
  • 17:07 ejegg: Updated SmashPig settings: extended Ingenico Connect API timeout to 12 seconds
  • 16:49 Amir1: morning SWAT is done
  • 16:49 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reading from the new backend for Special:Tags in several large wikis (T199334) (duration: 00m 56s)
  • 16:40 akosiaris: remove ORES abuser blocking T200338, let's reevaluate
  • 16:02 joal@deploy1001: Finished deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts (duration: 17m 51s)
  • 15:49 gehel: elasticsearch cluster restart on codfw completed - T156137
  • 15:44 joal@deploy1001: Started deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts
  • 15:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 (duration: 00m 55s)
  • 15:31 anomie: running populateContentTables.php on testwiki for T183488
  • 15:26 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 55s)
  • 15:21 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgMultiContentRevisionSchemaMigrationStage to write-both read-old on testwiki (T197817) (duration: 00m 55s)
  • 14:45 marostegui: Deploy schema change on db1098:3317 T144010 T51190 T199368
  • 14:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 (duration: 00m 54s)
  • 14:39 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: (no justification provided)
  • 14:29 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1014 fully (duration: 00m 55s)
  • 14:00 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.14 (duration: 00m 55s)
  • 13:59 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.14
  • 13:54 jynus: running compare.py on all es3 databases
  • 13:48 ema: text-eqiad: test alternate domains patch with varnish 5.1.3-1wm9 T164609
  • 13:44 Amir1: restarting ores celery workers on codfw
  • 13:34 ema: repool cp1067 w/ alternate domains patch and varnish 5.1.3-1wm9 T164609
  • 13:27 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.14
  • 13:26 ema: depool cp1067 and test alternate domains patch with varnish 5.1.3-1wm9 T164609
  • 13:00 gehel: resetting postgres data on maps1002 after failing replication - T200228
  • 12:21 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s5 populateChangeTagDef.php --sleep 2 (T193873)
  • 12:19 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache (duration: 59m 27s)
  • 11:20 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache
  • 11:16 dcausse: EU SWAT done
  • 11:14 dcausse@deploy1001: Synchronized ./wmf-config/CirrusSearch-common.php: [cirrus] allow term_freq and remove deprecated settings (duration: 00m 48s)
  • 11:01 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.10 [keeping static files] (duration: 05m 13s)
  • 10:21 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1014 with low load after maintenance (duration: 00m 47s)
  • 10:07 marostegui: Deploy schema change on db2040 (s7 codfw master) with replication, this will generate lag on s7 codfw T144010 T51190 T199368
  • 10:05 ema: upgrade varnish to 5.1.3-1wm9 on text-eqiad T164609
  • 09:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 (duration: 00m 47s)
  • 09:53 godog: bounce grafana on krypton
  • 09:31 ema: upload varnish 5.1.3-1wm9 to apt.w.o (fixing POST requests w/ separate VCL) T164609
  • 09:20 gehel: restarting elasticsearch cluster restart on codfw - T156137
  • 08:41 jynus: stop es1014 for reimage
  • 08:14 marostegui: Deploy schema change on db1121 with replication, this will generate lag on labsdb hosts for s4 T144010 T51190 T199368
  • 08:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 (duration: 00m 47s)
  • 08:10 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s4 populateChangeTagDef.php --sleep 2 (T193873)
  • 08:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 db1091 (duration: 00m 47s)
  • 08:02 gehel: pausing elasticsearch cluster restart on codfw - T156137
  • 07:58 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1014 (duration: 00m 48s)
  • 07:35 gehel: rolling restart of elasticsearch / cirrus / codfw to disable G1 - T156137
  • 07:12 marostegui: Deploy schema change on db1091 T144010 T51190 T199368
  • 06:53 gehel: resetting postgres data on maps1004 after failing replication - T200228
  • 06:38 marostegui: Stop replication in sync on db1091 and db1097:3314
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 db1091 (duration: 00m 48s)
  • 06:33 jynus: finished es1014 -> es1017 switch T197073
  • 06:27 jynus: enabling semi-sync master on es1017, disabling it as client
  • 06:21 jynus: deploy es3-master dns change
  • 06:02 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Switchover es3 master eqiad from es1014 to es1017 (duration: 00m 24s)
  • 06:01 jynus: switchover es3 eqiad master from es1014 to es1017
  • 04:35 tstarling@deploy1001: Synchronized php-1.32.0-wmf.13/includes/api/ApiMain.php: record all API requests in statsd (duration: 00m 49s)
  • 02:43 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 25 02:43:05 UTC 2018 (duration 10m 19s)
  • 02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 28s)

2018-07-24

  • 21:59 XioNoX: re-pooling eqsin
  • 21:15 XioNoX: re1 is master routing engine on cr1-eqsin, triggering a re switch
  • 21:10 XioNoX: starting to see recoveries from cr1-eqsin upgrade
  • 21:06 XioNoX: Install done, cr1-eqsin re-rebooting
  • 21:00 XioNoX: restarting cr1-eqsin for software upgrade
  • 20:32 XioNoX: depooling eqsin for cr1-eqsin software upgrade
  • 19:09 gehel: resetting postgres data on maps1003 after failing replication - T200228
  • 18:34 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@690fdad]: Wait for the client to consume the meesage being sent before consuming the next one - T199813 (duration: 02m 18s)
  • 18:32 mobrovac@deploy1001: Started deploy [eventstreams/deploy@690fdad]: Wait for the client to consume the meesage being sent before consuming the next one - T199813
  • 17:40 ema: re-enable puppet on all cache nodes with alternate domains disabled T164609
  • 17:33 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.13
  • 17:18 thcipriani: train window running long, services deploy delayed
  • 17:18 ema: restart varnish-fe on cp1068 to clear "child restarted" alert T164609
  • 17:17 elukey: restart eventstreams on scb2* nodes (hopefully last time before deploying the fix) to avoid mem leaks issues during the EU night
  • 17:06 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.14/includes/page/PageArchive.php: PageArchive: Pass correct overrides to newRevisionFromArchiveRow() (T200072) (duration: 01m 01s)
  • 16:58 jynus: finishing test on es3 hosts T199224
  • 16:42 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.13/includes/page/PageArchive.php: PageArchive: Pass correct overrides to newRevisionFromArchiveRow() (T200072) (duration: 01m 03s)
  • 16:07 jynus: test switchover from es2018 to es2017
  • 16:02 dcausse: T156137: unbanning elastic1031
  • 15:59 dcausse: T156137: restarting elasticsearch on elastic1031 to disable G1GC
  • 15:55 jynus: test switchover from es2017 to es2018
  • 15:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 (duration: 01m 02s)
  • 15:38 jynus: stopping puppet on es2017, es2018; changing mysql configuration for production testing
  • 15:29 gehel: restart postgres on maps1001 - T200228
  • 14:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 01m 02s)
  • 14:47 dcausse: T156137: banning elastic1031 due to high load (same "getEntryAfterMiss" symptoms)
  • 14:09 marostegui: Deploy schema change on db1103:3314 T144010 T51190 T199368
  • 13:45 ema: apply alternate domains patch to text-eqiad T164609
  • 13:43 marostegui: Deploy schema change on db1084 T144010 T51190 T199368
  • 13:42 marostegui: Stop replication in sync db1084 and db1103:3314
  • 13:40 marostegui: Deploy schema change on db1081 T144010 T51190 T199368
  • 13:38 marostegui: Stop replication in sync db1081 and db1103:3314
  • 13:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 01m 59s)
  • 13:07 ema: repool cp1067 with alternate domains support T164609
  • 12:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 (duration: 00m 55s)
  • 12:12 gehel: vacuum full of postgres on maps1001 to try to reclaim space - T200228
  • 12:07 ema: depool cp1067 to test alternate domains patch T164609
  • 11:58 zfilipin@deploy1001: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4179557944" --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 02m 50s)
  • 11:55 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache
  • 11:45 zfilipin@deploy1001: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2212739269" --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 03m 42s)
  • 11:42 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache
  • 11:35 ema: disable puppet on cp-text hosts to merge alternate domains patch T164609
  • 11:21 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s1 populateChangeTagDef.php (T193873)
  • 11:19 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s2 populateChangeTagDef.php --sleep 2 (T193873)
  • 11:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 00m 55s)
  • 11:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 (duration: 01m 06s)
  • 10:44 marostegui: Stop replication in sync on db1084 and db1097:3314
  • 10:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 55s)
  • 09:17 marostegui: Deploy schema change on db1097:3314 T144010 T51190 T199368
  • 09:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 (duration: 00m 54s)
  • 08:38 ema: restart varnish-fe on cache_text instances with cold, labeled VCL T200207
  • 08:21 elukey: rolling restart of kafka jumbo/main-(eqiad|codfw) clusters to pick up the new max open files limit (infinity -> 128k)
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 (duration: 00m 55s)
  • 06:32 marostegui: Deploy schema change on dbstore1002:s4 T144010 T51190 T199368
  • 05:33 kartik@deploy1001: Finished deploy [cxserver/deploy@d378d27]: Update cxserver to d3c9d15 (T198941) (duration: 04m 01s)
  • 05:29 kartik@deploy1001: Started deploy [cxserver/deploy@d378d27]: Update cxserver to d3c9d15 (T198941)
  • 05:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 (duration: 00m 54s)
  • 05:05 marostegui: Deploy schema change on db1081 T144010 T51190 T199368
  • 04:54 marostegui: Stop replication in sync on db1081 and db1121
  • 04:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084, db1121 (duration: 00m 56s)
  • 04:44 marostegui: Deploy schema change on db1066 (s2 primary master) T144010 T51190 T199368
  • 03:03 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 24 03:03:08 UTC 2018 (duration 10m 22s)
  • 02:52 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 09m 19s)
  • 02:23 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 09m 44s)

2018-07-23

  • 22:53 mobrovac@deploy1001: Started restart [eventstreams/deploy@5ed03e0]: A ggodnight restart - T199813
  • 22:27 thcipriani@deploy1001: Finished scap: Revert "Accept BCP 47 codes as aliases for nonstandard variants" T199941 Revert "Ensure LanguageCode::bcp47() returns a valid BCP 47 language code" T199941 (duration: 33m 21s)
  • 21:53 thcipriani@deploy1001: Started scap: Revert "Accept BCP 47 codes as aliases for nonstandard variants" T199941 Revert "Ensure LanguageCode::bcp47() returns a valid BCP 47 language code" T199941
  • lunch: T156137: restarting elastic1035 to disable G1GC
  • lunch: T156137: restarting elastic1027 to disable G1GC
  • 20:27 arlolra: Updated Parsoid to a1e851c (T199808, T194083)
  • 20:22 arlolra@deploy1001: Finished deploy [parsoid/deploy@80384a5]: Updating Parsoid to a1e851c (duration: 09m 45s)
  • 20:13 arlolra@deploy1001: Started deploy [parsoid/deploy@80384a5]: Updating Parsoid to a1e851c
  • 20:09 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@565b41a]: Update mobileapps to 254cef5 (duration: 05m 56s)
  • 20:03 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@565b41a]: Update mobileapps to 254cef5
  • 19:49 gehel@deploy1001: Finished deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (duration: 11m 35s)
  • 19:38 gehel@deploy1001: Started deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI
  • 19:37 gehel@deploy1001: Finished deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 22s)
  • 19:36 gehel@deploy1001: Started deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (wdqs1009 only)
  • 17:56 gehel@deploy1001: Finished deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 25s)
  • 17:56 gehel@deploy1001: Started deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only)
  • 17:53 ejegg: updated payments-wiki from d8d8293b87 to 6390e50abd
  • 17:47 gehel@deploy1001: Finished deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 03s)
  • 17:47 gehel@deploy1001: Started deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only)
  • 17:29 XenoRyet: updated payments-wiki from 63372154d0 to d8d8293b87
  • 17:16 gehel: canceling deployment of new WDQS GUI to all servers, looks like there is a JS issue. Will reschedule once the issue is fixed.
  • 17:07 gehel@deploy1001: Finished deploy [wdqs/wdqs@d7e8292]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 30s)
  • 15:25 gehel: decrease gc_grace_seconds to 4 days on cassandra / maps
  • 15:02 mobrovac@deploy1001: Finished deploy [changeprop/deploy@5cfdf26]: Increase concurrency back to normal levels (from 20 to 50) (duration: 01m 38s)
  • 15:00 mobrovac@deploy1001: Started deploy [changeprop/deploy@5cfdf26]: Increase concurrency back to normal levels (from 20 to 50)
  • 14:57 mobrovac@deploy1001: Started restart [eventstreams/deploy@5ed03e0]: Fresh start of the service after alleviating 404s
  • 14:53 elukey: delete empty/not-used/wrongly-created topics in Kafka main-eqiad - T199510
  • 14:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081, db1097:3314 (duration: 00m 53s)
  • 14:17 marostegui: Stop replication in sync on db1081 and db2051
  • 14:00 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@5ed03e0]: Update node-rdkafka to v2.3.4 - T199813 (duration: 02m 29s)
  • 13:57 mobrovac@deploy1001: Started deploy [eventstreams/deploy@5ed03e0]: Update node-rdkafka to v2.3.4 - T199813
  • 13:53 marostegui: Stop replication in sync on db1081 and db1097:3317
  • 13:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081, db1097:3314 (duration: 00m 54s)
  • 13:44 ema: !log trafficserver 7.1.3+ds-4wm1 uploaded to stretch-wikimedia T200178
  • 13:36 marostegui: Deploy schema change on s4 codfw master (db2051), this will generate lag on s4 codfw T144010 T51190 T199368
  • 13:23 marostegui: Deploy schema change on labswiki and labstestwiki T144010 T51190 T199368
  • 13:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074, db1105:3312 (duration: 00m 55s)
  • 11:34 zeljkof: EU SWAT finished
  • 11:32 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Exempt Template and Module namespace on zhwiktionary (T187783) (duration: 00m 55s)
  • 11:14 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Change zhwikiquote logo (T199863) (duration: 00m 55s)
  • 10:10 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 54s)
  • 10:09 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 55s)
  • 09:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074, db1105:3312 (duration: 00m 53s)
  • 09:42 marostegui: Stop replication in sync on db1105:3312 db1074
  • 09:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122, db1103:3312 (duration: 00m 54s)
  • 09:00 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s8 populateChangeTagDef.php (T193873)
  • 08:47 marostegui: Stop replication in sync on db1103:3312 db2035
  • 08:42 godog: enable deprecation page for status.w.o - T199816
  • 08:23 marostegui: Stop replication in sync on db1122 and db1103:3312
  • 08:21 marostegui: Apply schema change T192926 to labswiki and labstestwiki T200140
  • 08:15 marostegui: Apply schema change T160415 to labswiki and labstestwiki T200140
  • 08:11 marostegui: Apply schema change  T191316 to labswiki and labstestwiki T200140
  • 08:07 marostegui: Apply schema change  T191519 to labswiki and labstestwiki T200140
  • 08:05 marostegui: Apply schema change  T190148 to labswiki and labstestwiki T200140
  • 08:04 marostegui: Apply schema change  T197891 to labswiki and labstestwiki T200140
  • 07:54 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s6 populateChangeTagDef.php (T193873)
  • 07:54 marostegui: Stop replication in sync on db1122 and db2035
  • 07:33 marostegui: Deploy schema change on db1122 T144010 T51190 T199368
  • 07:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122, db1103:3312 (duration: 00m 55s)
  • 06:37 jynus: stop db1102 to clone to db1118
  • 04:42 marostegui: Deploy schema change on db1061 (s6 primary master) T144010 T51190 T199368
  • 03:08 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 23 03:08:23 UTC 2018 (duration 10m 37s)
  • 02:57 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 51s)
  • 02:24 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 09m 42s)

2018-07-22

  • 16:15 elukey: rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix)
  • 12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.76.159
  • 12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.76.150
  • 12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.68.116
  • 12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 196.208.95.30
  • 12:03 addshore@deploy1001: Synchronized wmf-config/throttle.php: T198288 More Wikimania throttle rules (duration: 01m 18s)

2018-07-21

  • 22:29 Reedy: Added ct_tag_id to labswiki and labtestwiki T200139
  • 17:35 elukey: rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix)

2018-07-20

  • 21:50 bawolff: deployed patch T200104
  • 17:16 XioNoX: enable bgp on eqdfw-knams link (GTT)
  • 17:08 XioNoX: enable ospf on eqdfw-knams link (GTT)
  • 15:33 elukey: rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix)
  • 15:19 elukey: powercycle ms-be1016 - RAID errors, no ssh available, I/O errors in com2 console
  • 14:15 reedy@deploy1001: rebuilt and synchronized wikiversions files: wikidatawiki back to .12 T199983
  • 14:11 Reedy: T199983 wikidata wiki abck to .12 on mwdebug1001
  • 13:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085, db1113:3316 (duration: 00m 55s)
  • 13:13 marostegui: Stop replication in sync db1085 and db1113:3316
  • 13:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316, depool db1113:3316 (duration: 00m 54s)
  • 13:06 legoktm@deploy1001: Synchronized wmf-config/throttle.php: Update Wikimania IP address - T198288 (duration: 00m 54s)
  • 12:53 marostegui: Stop replication in sync db1085 and db1096:3318
  • 12:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316, depool db1098:3316 (duration: 00m 54s)
  • 12:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 and db1096:3316 (duration: 00m 56s)
  • 12:32 marostegui: Stop replication in sync db1085 and db1096:3316
  • 12:31 marostegui@deploy1001: sync-file aborted: Depool db1085 and db1096:33156 (duration: 00m 01s)
  • 12:00 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@01fac88]: Lower librdkafka settings related to fetching messages - T199813 (duration: 02m 11s)
  • 11:58 mobrovac@deploy1001: Started deploy [eventstreams/deploy@01fac88]: Lower librdkafka settings related to fetching messages - T199813
  • 10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312, db1105:3312 (duration: 00m 53s)
  • 10:30 marostegui: Stop replication in sync on db1090:3312 and db1105:3312 - T200061
  • 10:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312, depool db1105:3312 (duration: 00m 54s)
  • 10:17 marostegui: Stop replication in sync on db1090:3312 and db2035 - T200061
  • 10:14 marostegui: Stop replication in sync on db1090:3312 and db1103:3312 - T200061
  • 10:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 and db1103:3312 (duration: 00m 55s)
  • 09:46 marostegui: Stop replication on db2063 to fix db2095 - T200061
  • 08:50 marostegui: Stop replication on s7 codfw master to check arwiki.change_tag - T200061
  • 08:47 Amir1: stopping populateChangeTagDef.php (T200061)
  • 08:41 foks: reset email for User:Nomden
  • 08:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 55s)
  • 07:46 marostegui: Stop replicaiton on s8 codfw master - T200061
  • 07:43 jynus: stopping db1095 mariadb and cloning it to db1102
  • 07:14 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099, es1019 fully (duration: 00m 54s)
  • 07:11 marostegui: Stop replication in sync on db1074 and db1125:3312
  • 06:13 marostegui: Deploy schema change on db1074 with replication, this will generate lag on labsdb:s2 T144010 T51190 T199368
  • 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 53s)
  • 06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 after alter table (duration: 00m 54s)
  • 05:49 marostegui: Deploy schema change on db1076 T144010 T51190 T199368
  • 05:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 for alter table (duration: 00m 56s)
  • 05:08 marostegui: Start to remove some unused files on db1067 - T200039
  • 01:38 krinkle@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/CentralAuth/includes/CentralAuthUser.php: T170971 (duration: 00m 55s)
  • 00:31 krinkle@deploy1001: Synchronized php-1.32.0-wmf.13/includes/filerepo/: I58706b5610 and I40f6ad2a3d - T200026 (duration: 00m 56s)
  • 00:30 ejegg: updated CiviCRM from 22a8c47668 to d76e3997e6
  • 00:12 bblack: disabled puppet temporary on cp* and kafka-jumbo* for ipsec unconfiguration

2018-07-19

  • 23:43 ejegg: updated CiviCRM from 08680be68e to 22a8c47668
  • 23:31 ejegg: updated CiviCRM from 2bb0d0f9d0 to 08680be68e
  • 23:11 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable Special:Block Feedback Request" (duration: 00m 55s)
  • 22:51 XioNoX: disable ospf on eqdfw-knams link (GTT)
  • 22:18 XioNoX: enable ospf on eqdfw-knams link (GTT)
  • 21:52 hashar: CI Jenkins has been upgraded successfully!
  • 21:21 hashar: restarting CI on contint1001 and actually upgrading it
  • 21:19 hashar: starting CI jenkins on contint1001
  • 21:16 hashar: stopping the CI jenkins on contint1001
  • 20:20 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099 with low load (duration: 00m 54s)
  • 19:25 ariel@deploy1001: Finished deploy [dumps/dumps@f0c4a70]: handle empty tables properly; script to show runtimes of dump steps (duration: 00m 03s)
  • 19:25 ariel@deploy1001: Started deploy [dumps/dumps@f0c4a70]: handle empty tables properly; script to show runtimes of dump steps
  • 19:21 ejegg: updated CiviCRM from 6f311f48d9 to 2bb0d0f9d0
  • 19:18 XioNoX: setting mtu 9192 to all codfw interface ranges
  • 19:04 elukey: roll restart eventstreams on scb2* hosts to prevent OOM issues over the EU night - T199813
  • 19:00 Amir1: Morning SWAT is done
  • 18:57 ladsgroup@deploy1001: Synchronized wmf-config/abusefilter.php: SWAT: Add abusefilter-modify-restricted right to sysops on itwiktionary (T199783) (duration: 00m 53s)
  • 18:50 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/WikibaseQualityConstraints: SWAT: Report SPARQL errors as TODO, not VIOLATION (T199788) (duration: 00m 56s)
  • 18:27 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/WikibaseQualityConstraints: SWAT: Report SPARQL errors as TODO, not VIOLATION (T199788) (duration: 00m 56s)
  • 18:18 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist all populateChangeTagDef.php (T193873) It will take a couple of days
  • 18:14 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set to write the new change tag backend everywhere, enable reading in frwiki (T194165) (duration: 00m 55s)
  • 16:28 jynus: stop, clone away and upgrade db1099 two mysql instances
  • 16:16 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully depool db1099, including from s8 (duration: 00m 55s)
  • 16:00 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 15:53 hashar: CI: npmjs is slow. Bumping Quibble jobs timeout from 30 to 45 minutes | https://gerrit.wikimedia.org/r/#/c/446863 | T198348
  • 15:50 jynus: droping undocumented grants on labsdb1005 using replication T199186
  • 15:21 volans: reimaging mw2226 to test py3 reimage with new cumin, conftool and apache test from neodymium - T187773
  • 15:12 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2225.codfw.wmnet
  • 15:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 after alter table (duration: 00m 54s)
  • 15:00 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099 (duration: 00m 54s)
  • 14:59 moritzm: rebooting multatuli for kernel test
  • 14:30 marostegui: Deploy schema change on db1105:3312 T144010 T51190 T199368
  • 14:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 for alter table (duration: 00m 54s)
  • 14:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 after alter table (duration: 00m 54s)
  • 14:17 elukey: roll restart kafka on kafka-jumbo* and kafka main-codfw (kafka2*) to pick up new Xmx/Xms settings (1g -> 2g)
  • 14:15 volans: upgrading cumin/debdeploy/reimage and other tools on neodymium - T187773
  • 13:57 elukey: restart kafka on kafka-jumbo1001 to raise Xmx/Xms jvm settings (1g -> 2g)
  • 13:43 marostegui: Deploy schema change on db1103:3312 T144010 T51190 T199368
  • 13:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for alter table (duration: 00m 54s)
  • 13:38 volans: reimaging mw2225 to test py3 reimage with new cumin, conftool and apache test - T187773
  • 13:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1103:3314 (duration: 00m 53s)
  • 13:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
  • 13:22 moritzm: installing python-pysaml2 security updates
  • 13:05 moritzm: installing ffmpeg security updates
  • 12:48 moritzm: installing xerces-c security updates
  • 12:38 moritzm: uploaded debdeploy 0.0.99.5 to apt.wikimedia.org
  • 11:49 zeljkof: EU SWAT finished
  • 11:46 zfilipin@deploy1001: Synchronized dblists/categories-rdf.dblist: SWAT: Remove labs wikis from the categories-rdf list, dont need them (T198356) (duration: 00m 55s)
  • 11:44 moritzm: installing reportbug update from stretch point release to test py3-based debdeploy
  • 11:40 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:Block Feedback Request (T199919) (duration: 00m 55s)
  • 11:33 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Structured Discussions beta feature on orwiki (T199971) (duration: 00m 55s)
  • 11:22 volans: testing reimage of californium (spare host) with the upgraded cumin
  • 11:20 chasemp: maintain-views --all-databases --replace-all --table ores_classification --debug (labsdb)
  • 11:19 chasemp: maintain-views --all-databases --replace-all --table ores_model --debug (labsdb)
  • 11:18 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: enable TemplateStyles on enwiki, frwiki, zhwiki T197603 T191452 T189022 (duration: 00m 55s)
  • 11:00 volans: upgrading cumin on sarin - T187773
  • 10:52 legoktm: made myself a crat on test2wiki
  • 10:06 moritzm: installing file/libmagic security updates
  • 09:55 moritzm: installing check-postgres update from stretch 9.5 point release
  • 09:49 aaron@deploy1001: Synchronized php-1.32.0-wmf.13/includes/libs/objectcache/MultiWriteBagOStuff.php: 2cee0ab (duration: 00m 55s)
  • 09:35 volans: cumin upgrade on labpuppetmaster* hosts completed - T187773
  • 09:33 moritzm: installing openssl security updates
  • 09:33 volans: removed leftover cumin installation from labtestpuppetmaster2001 - T187773
  • 09:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
  • 09:27 godog: run xfs_repair on filesystems reporting negative space available on ms-be1040 - T199198
  • 09:24 moritzm: installing tomcat7 security updates
  • 09:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 54s)
  • 09:21 moritzm: installing ant security updates
  • 09:14 moritzm: uploaded jenkins 2.121.2 to apt.wikimedia.org (T199448)
  • 08:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
  • 08:39 volans: start upgrade of cumin to 3.0.1-1 on labpuppetmaster* - T187773
  • 08:32 marostegui: Deploy schema change on dbstore1002:s2 T144010 T51190 T199368
  • 08:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
  • 08:08 elukey: restart eventstreams on scb2* hosts to pick up new Kafka settings (pointing it to main-codfw) - T199813
  • 08:04 kartik@deploy1001: Finished deploy [cxserver/deploy@fe0d521]: Update cxserver to 92e3d2e (T199380, T199885) (duration: 03m 58s)
  • 08:00 kartik@deploy1001: Started deploy [cxserver/deploy@fe0d521]: Update cxserver to 92e3d2e (T199380, T199885)
  • 07:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
  • 07:55 marostegui: Deploy schema change on s2 codfw masters (db2035) this will generate lag on s2 codfw T144010 T51190 T199368
  • 07:50 jynus: stop es1019 and reimage it
  • 07:48 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 54s)
  • 07:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3312 after MySQL upgrade (duration: 00m 55s)
  • 07:26 moritzm: removing cdrom and sr_mod (related module) kernel modules across the fleet (following the blacklisting of the kernel module via puppet)
  • 06:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3312 after MySQL upgrade (duration: 00m 54s)
  • 05:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3312 after MySQL upgrade (duration: 00m 55s)
  • 05:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1103:3312 after MySQL upgrade (duration: 00m 54s)
  • 05:14 marostegui: Stop MySQL on db1103 to upgrade MySQL
  • 05:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for MySQL upgrade (duration: 00m 56s)
  • 04:12 kart_: Finished ContentTranslation draft purge script on mwmaint1001 (T199658)
  • 04:05 kart_: Starting ContentTranslation draft purge script on mwmaint1001, dry run first and then --really
  • 03:57 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I978f3eda8 - Farewell temporary hack from 2013 (duration: 00m 56s)
  • 03:20 ejegg: updated CiviCRM from b21d783e7f to 6f311f48d9
  • 03:09 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 19 03:09:19 UTC 2018 (duration 10m 35s)
  • 02:58 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 15m 05s)
  • 02:24 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 10m 41s)

2018-07-18

  • 23:35 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.13/includes/logging/LogEventsList.php: SWAT: LogEventsList: Use GET in HTMLForm T199856 (duration: 00m 54s)
  • 23:33 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWMediaDialog.js: SWAT: ve.ui.MWMediaDialog: Fix confusion between #getSetupProcess and #getReadyProcess T185944 T199841 (duration: 00m 56s)
  • 22:16 ejegg: updated SmashPig payments listener from 2b4186715e to 2736e41045
  • 19:24 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Unset wgVipsThumbnailHost T199937 (duration: 00m 55s)
  • 16:28 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable new backend for Special:Tags on fawikisource (T199334) (duration: 00m 54s)
  • 16:14 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES on testwiki (T199913) (duration: 00m 55s)
  • 15:25 volans: installing conftool 1.0.1 on jessie and stretch
  • 14:50 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@09f0efe]: Set the maximum rdkafka receive buffer size to 64MB - T199813 (duration: 02m 28s)
  • 14:48 mobrovac@deploy1001: Started deploy [eventstreams/deploy@09f0efe]: Set the maximum rdkafka receive buffer size to 64MB - T199813
  • 14:47 marostegui: Partition commonswiki.logging table on db1103:3314 - T199790
  • 14:46 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2224.codfw.wmnet
  • 14:43 volans: uploaded conftool (with python-conftool and python3-conftool) version 1.0.1 for jessie and stretch to apt.w.o
  • 14:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table - T199790 (duration: 00m 52s)
  • 14:37 moritzm: rebooting poolcounter2001 for some tests
  • 14:22 moritzm: rebooting vega for some tests
  • 13:59 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 54s)
  • 13:53 moritzm: installing ruby2.1 security updates for jessie
  • 13:17 marostegui: Deploy schema change on db1052 - T187089
  • 13:13 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.13 (duration: 00m 53s)
  • 13:13 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.13
  • 12:58 marostegui: Stop MySQL on db1052 to upgrade socket and MySQL
  • 12:57 elukey: restart eventstreams on scb200[5,6] as precautionary after mem consumption too high
  • 12:57 elukey: restart eventstreams on scb2003 as precautionary after mem consumption too high
  • 12:56 elukey: restart eventstreams on scb2001 as precautionary after mem consumption too high (still investigating a fix)
  • 12:54 dcausse: T156137: unbanning elastic1030
  • 12:52 dcausse: T156137: restarting elastic1030 to disable G1GC
  • 12:51 moritzm: installing PHP5 security updates (remaining hosts which only use -cli)
  • 12:48 moritzm: installing PHP security updates on tungsten
  • 12:36 moritzm: installing PHP security updates on icinga.wikimedia.org
  • 12:24 elukey: manually added queued.max.messages.kbytes: 65535 to eventstreams on scb2002 as test for T199813
  • 12:16 moritzm: installing PHP security updates on yubiauth servers
  • 12:02 moritzm: installing PHP security update on bohrium
  • 11:35 moritzm: installing imagemagick security updates
  • 11:34 zeljkof: EU SWAT finished
  • 11:32 zfilipin@deploy1001: Synchronized wmf-config/abusefilter.php: SWAT: Enable AbuseFilter block option on itwiktionary (T199783) (duration: 00m 54s)
  • 11:25 moritzm: installing policykit security updates on trusty
  • 11:20 Amir1: making ores tables on euwiki
  • 11:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Publisher namespace in Bengali Wikisource (T199028) (duration: 00m 54s)
  • 11:05 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ores extension wp10 storage in Basque Wikipedia (T198358) (duration: 00m 54s)
  • 10:30 moritzm: ran puppet clean/deactivate for lawrencium, long-standing source of errors in package deployments (T191360)
  • 10:24 moritzm: installing cups/libcups security updates for jessie
  • 10:10 dcausse: T156137: banning elastic1030 from the cluster (high load)
  • 10:02 dcausse: clearing internal cache on elastic1030
  • 09:58 hashar: contint1001 dropping unused docker containers
  • 09:36 moritzm: draining restbase1007 for eventual reboot for tests with SSBD microcode
  • 08:51 godog: run xfs_repair on filesystems reporting negative space available on ms-be1043 - T199198
  • 08:50 moritzm: reboot ms-be1040 for tests with SSBD microcode
  • 08:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 53s)
  • 08:38 twentyafterfour: restart apache on phab1001
  • 08:37 chasemp: deploy antivandalism stuff to phab https://gerrit.wikimedia.org/r/c/operations/puppet/+/445329 (needs herald to fully enable)
  • 08:31 elukey: drain + reboot analytics1030 for kernel updates
  • 08:29 marostegui: Deploy schema change on db1085 with replication, this will generate lag on labsdb hosts for s6 T144010 T51190 T199368
  • 08:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 01m 01s)
  • 08:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 after alter table (duration: 00m 53s)
  • 08:06 marostegui: Deploy schema change on db1088 T144010 T51190 T199368
  • 08:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 53s)
  • 07:41 jforrester@deploy1001: Synchronized wmf-config/missing.php: Fix typo in missing.php Ia3aae912d9m (duration: 00m 54s)
  • 07:37 moritzm: upgrading tor on radium to 0.3.3.9
  • 07:25 marostegui: Deploy schema change on db1052 - https://phabricator.wikimedia.org/T191316 https://phabricator.wikimedia.org/T192926 https://phabricator.wikimedia.org/T89737 https://phabricator.wikimedia.org/T195193
  • 07:22 moritzm: updated tor on apt.wikimedia.org to 0.3.3.9
  • 06:45 jynus: enable semi-sync on db1099:3311 and db1105:3311
  • 06:08 marostegui: s1 failover finished T197069
  • 06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: read only OFF after failover T197069 (duration: 00m 53s)
  • 06:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Set db1067 as master T197069 (duration: 00m 53s)
  • 06:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Set s1 on ready only for maintenance T197069 (duration: 01m 08s)
  • 06:00 marostegui: Starting s1 failover from db1052 to db1067 - T197069
  • 05:11 Krinkle: mwscript deleteEqualMessages.php --wiki dewiki (1 page) - T45917
  • 05:11 Krinkle: mwscript deleteEqualMessages.php --wiki ruwiki (4 pages) - T45917
  • 05:11 Krinkle: mwscript deleteEqualMessages.php --wiki frwiki (5 pages) - T45917
  • 05:09 Krinkle: mwscript deleteEqualMessages.php --wiki metawiki (deleted 1 page) - T45917
  • 05:09 Krinkle: mwscript deleteEqualMessages.php --wiki test2wiki (deleted 2 pages) - T45917
  • 04:56 marostegui: Starting s1 failover pre steps - https://phabricator.wikimedia.org/T197069
  • 03:05 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 18 03:05:46 UTC 2018 (duration 10m 27s)
  • 02:55 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 09m 43s)
  • 02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 08m 35s)
  • 00:00 reedy@deploy1001: Finished scap: log event fixes (duration: 33m 18s)

2018-07-17

  • 23:26 reedy@deploy1001: Started scap: log event fixes
  • 22:43 reedy@deploy1001: Synchronized wmf-config/interwiki-labs.php: (no justification provided) (duration: 00m 53s)
  • 22:24 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove override for foundationwiki in wgMobileUrlTemplate (duration: 00m 53s)
  • 22:21 reedy@deploy1001: Synchronized tests/urls.txt: testding! (duration: 00m 53s)
  • 20:57 reedy@deploy1001: Synchronized wmf-config/: make foundation.wikimedia.org writeable T188776 (duration: 00m 53s)
  • 20:50 reedy@deploy1001: Finished scap: updating l10n cache for foundationwiki url changes (duration: 58m 32s)
  • 20:17 ebernhardson: un-ban elastic1030 from eqiad search cluster
  • 19:52 reedy@deploy1001: Started scap: updating l10n cache for foundationwiki url changes
  • 19:35 reedy@deploy1001: Synchronized portals: T199808 (duration: 00m 54s)
  • 19:34 reedy@deploy1001: Synchronized portals/wikipedia.org/assets: T199808 (duration: 00m 53s)
  • 18:46 bblack: re-enabling puppet on mw* for more foundationwiki work - T188776
  • 18:39 ebernhardson: ban elastic1030 from eqiad search cluster for latency issues. Likely related to T156137
  • 18:38 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 53s)
  • 18:36 bblack: disabling puppet on mw* for more foundationwiki work - T188776
  • 18:33 reedy@deploy1001: Synchronized wmf-config/: foundationwiki (duration: 00m 54s)
  • 18:33 elukey: rolling restart eventstreams on scb2* nodes to avoid OOMs during the EU night
  • 18:32 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: foundationwiki (duration: 00m 53s)
  • 18:31 reedy@deploy1001: Synchronized tests/: foundationwiki (duration: 00m 53s)
  • 18:29 reedy@deploy1001: Synchronized docroot/: update foundationwiki urls (duration: 00m 54s)
  • 18:28 reedy@deploy1001: Synchronized errorpages/: update foundationwiki urls (duration: 00m 54s)
  • 18:17 reedy@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/ConfirmEdit: Make a function public again (duration: 00m 56s)
  • 17:55 bblack: re-enabling puppet on mw* for foundationwiki apache config change T188776
  • 17:00 bblack: disabling puppet on most of mw* for foundationwiki apache config deploy - T188776
  • 16:52 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Make foundationwiki readonly (duration: 00m 54s)
  • 16:49 moritzm: installing faad2 security updates
  • 15:51 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/tests/phpunit/includes/libs/objectcache/BagOStuffTest.php: (no justification provided) (duration: 00m 54s)
  • 15:49 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/includes/libs/objectcache/BagOStuff.php: 2d919ad (duration: 00m 56s)
  • 15:42 XioNoX: switching default analytics-in6 term to reject+log - T198623
  • 15:40 moritzm: rebooting ms-fe1006 with SSBD-enabled microcode
  • 15:18 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2224.codfw.wmnet
  • 14:58 volans: repooled mw2224
  • 14:50 moritzm: rebooting ms-fe1005 with SSBD-enabled microcode
  • 14:47 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.13
  • 14:26 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache (duration: 06m 14s)
  • 14:19 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache
  • 14:18 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.999 (duration: 03m 01s)
  • 14:12 moritzm: rebooting wtp1025 with SSBD-enabled microcode
  • 14:07 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.8 (duration: 03m 11s)
  • 14:03 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.7 (duration: 03m 39s)
  • 13:54 moritzm: rebooting mw1261 with SSBD-enabled microcode
  • 13:35 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.6 (duration: 06m 18s)
  • 13:27 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache (duration: 62m 17s)
  • 13:25 moritzm: pruned obsolete hosts from debmonitor
  • 13:06 marostegui: Drop unused grants on db1073, db1117:3323, db1117:3325
  • 12:25 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache
  • 12:03 zeljkof: EU SWAT finished
  • 11:58 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Thesaurus NS at thwikt (T198585) Fix problem with namespaces at patch 446005 (T198585) (duration: 00m 51s)
  • 11:46 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/ContentTranslation: SWAT: Fix colors on CX1 (T199503) (duration: 00m 55s)
  • 11:22 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ULS webfonts by default at Burmese Wikipedia (mywiki) (T196219) (duration: 01m 50s)
  • 11:15 mobrovac@deploy1001: Finished deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #4 (duration: 02m 05s)
  • 11:12 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #4
  • 11:12 mobrovac@deploy1001: Finished deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #3 (duration: 07m 29s)
  • 11:04 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #3
  • 11:03 mobrovac@deploy1001: Finished deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #2 - T199458 T195390 (duration: 19m 43s)
  • 10:43 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #2 - T199458 T195390
  • 10:31 moritzm: restarting archiva to pick up OpenJDK security update
  • 10:31 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20 - T199458 T195390
  • 10:14 marostegui: Deploy schema change on db1093 T144010 T51190 T199368
  • 10:08 volans: reimage mw2224 to test the new stretch netboot and wmf-auto-reimage changes
  • 10:07 moritzm: restarting zookeeper in codfw to pick up Java security updates
  • 09:41 marostegui: Deploy schema change on db1113:3316 T144010 T51190 T199368
  • 09:38 marostegui: Drop unused grants from db2037, db2042, db2078:3323, db2078:3325
  • 09:11 marostegui: Deploy schema change on db1098:3316 T144010 T51190 T199368
  • 08:51 moritzm: installing libipc-run-perl update from stretch 9.5 point release
  • 08:31 marostegui: Deploy schema change on db1096:3316 T144010 T51190 T199368
  • 07:50 volans@deploy1001: Finished deploy [debmonitor/deploy@691d2f8]: Release v0.1.6 (duration: 00m 41s)
  • 07:50 volans@deploy1001: Started deploy [debmonitor/deploy@691d2f8]: Release v0.1.6
  • 07:29 godog: un xfs_repair on filesystems reporting negative space available on ms-be1042 - T199198
  • 07:24 moritzm: installing xapian-core security updates
  • 07:17 marostegui: Drop unused grants on es eqiad hosts
  • 07:03 moritzm: updated stretch netinst image for 9.5
  • 06:42 moritzm: installing patch security updates
  • 06:39 moritzm: installing subversion security updates
  • 06:32 marostegui: Drop unused grants on es codfw hosts
  • 06:30 moritzm: installing postgresql-9.6 updates from stretch point release
  • 05:56 marostegui: Deploy schema change on db2039 (s6 codfw master) with replication, this will generate lag on codfw T144010 T51190 T199368
  • 05:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 00m 49s)
  • 05:34 marostegui: Deploy schema change on db1070 (s5 primary master) T144010 T51190 T199368
  • 05:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 00m 49s)
  • 05:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 after alter table (duration: 00m 49s)
  • 05:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 00m 49s)
  • 05:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 00m 49s)
  • 05:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 00m 50s)
  • 04:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 49s)
  • 04:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 53s)
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 17 02:45:01 UTC 2018 (duration 10m 21s)
  • 02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 13m 51s)

2018-07-16

  • 20:57 ejegg: re-enabled payments listener failmail stream
  • 20:11 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@fcae441]: Update mobileapps to bed7b29 (T174809) (duration: 09m 01s)
  • 20:02 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@fcae441]: Update mobileapps to bed7b29 (T174809)
  • 18:35 mepps: updated payments-wiki config to 4e0c6c3cfd
  • 18:10 niharika29@deploy1001: Synchronized wmf-config/: Rollout Watchlist Structured Filters to all wikis T181193 (duration: 00m 51s)
  • 14:54 moritzm: installing openssh updates from stretch point release
  • 14:52 marostegui: Change expire_log_days on db1067 - https://phabricator.wikimedia.org/T197069
  • 14:49 marostegui: Drop unused ceilometer database from db1073
  • 14:47 marostegui: Remove unused grants from db1073
  • 14:39 marostegui: Remove unused grants from labsdb1004 and labsdb1005
  • 14:37 marostegui: Remove unused grants from es2019
  • 14:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2076 (duration: 00m 49s)
  • 14:24 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 49s)
  • 14:22 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Explicitly set wgMultiContentRevisionSchemaMigrationStage to current default (T174044) (duration: 00m 50s)
  • 13:59 marostegui: Stop replication on db2076 (db2095's master)
  • 13:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2076 for maintenance (duration: 00m 50s)
  • 12:31 moritzm: rebooting mw2136 for microcode tests
  • 12:14 moritzm: rebooting multatuli for microcode tests
  • 11:56 zeljkof: EU SWAT finished
  • 11:55 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Reconstruction NS at frwikt (T199631) (duration: 00m 49s)
  • 11:38 vgutierrez: power cycle cp3033 - T199677
  • 11:36 pmiazga@deploy1001: Synchronized php-1.32.0-wmf.12/includes/WebResponse.php: SWAT: WebReponse: Use values altered in WebResponseSetCookie hook (T198525) (duration: 00m 54s)
  • 11:20 zfilipin@deploy1001: Synchronized robots.txt: SWAT: Remove frwiki outdated entries in robots.txt (T199496) (duration: 00m 50s)
  • 11:18 zfilipin@deploy1001: Synchronized robots.txt: SWAT: Remove frwiki outdated entries in robots.txt (T199496) (duration: 00m 49s)
  • 11:14 mobrovac@deploy1001: Finished deploy [changeprop/deploy@ab8f7e9]: Bug fix: Remove anchors in blacklisting URIs and decode event URIs - T198386 (duration: 01m 33s)
  • 11:12 mobrovac@deploy1001: Started deploy [changeprop/deploy@ab8f7e9]: Bug fix: Remove anchors in blacklisting URIs and decode event URIs - T198386
  • 11:06 Amir1: start of ladsgroup@mwmaint1001:~$ mwscript populateChangeTagDef.php --wiki=frwiki
  • 11:05 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp5001.eqsin.wmnet
  • 11:04 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Write to the new change tag backend in frwiki (T194165) (duration: 00m 50s)
  • 11:02 vgutierrez: Power cycling cp5001 to attempt recovering it
  • 10:13 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 10:12 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 08:29 godog: put back ms-be1036 to full weight - T196873
  • 08:27 marostegui: Drop unused grants on db1108 - this might get dbproxy1009 to complain
  • 08:19 godog: run xfs_repair on filesystems reporting negative space available on ms-be1041 - T199198
  • 08:05 marostegui: Drop unused grants on eqiad hosts
  • 07:33 marostegui: Drop unused grants on codfw hosts
  • 07:19 marostegui: Deploy schema change on dbstore1002:s5 T144010 T51190 T199368
  • 07:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 50s)
  • 07:11 marostegui: Deploy schema change on db1110 T144010 T51190 T199368
  • 07:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 50s)
  • 05:24 marostegui: Deploy schema change on db2059 T144010 T51190 T199368
  • 05:10 marostegui: Deploy schema change on db2075 T144010 T51190 T199368
  • 02:43 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 16 02:43:43 UTC 2018 (duration 10m 19s)
  • 02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 13m 36s)

2018-07-14

  • 03:41 krinkle@deploy1001: Synchronized wmf-config/: I2bff4e - beta only (duration: 00m 51s)
  • 03:39 krinkle@deploy1001: Synchronized wmf-config/ProductionServices.php: Ib079ec90ae515 - clean up (duration: 00m 49s)

2018-07-13

  • 18:48 Trey314159: reindexing Mirandese wikis on elastic@eqiad (T197890)
  • 18:42 Trey314159: reindexing Mirandese wikis on elastic@codfw (T197890)
  • 18:39 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 fully (duration: 00m 50s)
  • 18:33 jynus: droping undocumented grants on m5 T199518
  • 14:34 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 with low load (duration: 00m 50s)
  • 14:03 jynus: stop and reimage db1079
  • 13:54 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 fully, depool db1079 (duration: 00m 50s)
  • 11:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 with low load (duration: 00m 50s)
  • 10:12 moritzm: installing cups/libcups security updates on stretch/trusty
  • 09:45 moritzm: installing libpng security updates on trusty (Debian already updated)
  • 09:36 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/includes/libs/objectcache: 5efa9f6 (duration: 00m 51s)
  • 09:35 aaron@deploy1001: sync-dir aborted: (no justification provided) (duration: 00m 01s)
  • 09:33 moritzm: installing ruby-sprockets security updates on jessie
  • 09:27 jynus: stop and reimage db1078
  • 09:05 jynus: apply updated grants to m5 hosts
  • 08:47 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 00m 50s)
  • 07:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 51s)
  • 06:50 elukey: powercycle ms-be1041 after diagnostic tests
  • 06:42 elukey: unblocked stuck dpkg processes on an107[2,5] that broke puppet
  • 05:59 jynus: stop and reimage db1087
  • 05:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2068 after maintenance (duration: 00m 49s)
  • 05:21 marostegui: Stop MySQL on db2068 for upgrade and check unused grants
  • 05:20 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2068 for grants testing (duration: 00m 50s)
  • 05:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 50s)
  • 05:06 marostegui: Rename wbc_entity_usage.eu_touched column on db1110 - T144010
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 52s)

2018-07-12

  • 23:32 jforrester@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/VisualEditor: SWAT VisualEditor: Add missing i18n keys to manifest T198064 (duration: 00m 52s)
  • 23:22 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Timeless config simplification, part II Ifc02818b6e (duration: 00m 49s)
  • 23:20 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Timeless config simplification, part I Ifc02818b6e (duration: 00m 50s)
  • 21:12 XioNoX: advertising 185.15.56.0/24 to the DFZ - T193496
  • 18:40 niharika29@deploy1001: Synchronized wmf-config/Wikibase.php: Revert - Wikidata dispatch, disable dispatching for testwikidatawiki https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/430925/ (duration: 00m 49s)
  • 18:32 niharika29@deploy1001: Synchronized wmf-config/: Wikidata dispatch, Use a LockManager with short TTL for testwikidata T178652 (duration: 00m 51s)
  • 18:18 niharika29@deploy1001: Synchronized wmf-config/Wikibase.php: Disable dispatching for testwikidatawiki https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/430924/ (duration: 00m 50s)
  • 16:56 chasemp: labcontrol1003:~# neutron net-update 7425e328-560c-4f00-8e99-706f3fb90bb4 --port_security_enabled=true
  • 16:45 moritzm: restarting smokeping on netmon1002 to pick up new config after wasat rename
  • 15:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 (duration: 00m 50s)
  • 15:49 akosiaris: rolling restart apertium-apy on scb nodes
  • 15:49 akosiaris: downgrade apertium-fra-cat to apertium-fra-cat_1.2.0~r78602-1+wmf2
  • 15:46 XioNoX: progressively pushing new (tighter) mgmt firewall policies
  • 15:07 chasemp: cloudvirt1021:~# /sbin/reboot
  • 14:37 addshore@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/Wikibase/repo: T199379 Fix missing label for unit items when displayed on entity page gerrit:445413 (duration: 01m 02s)
  • 14:07 marostegui: Drop unused grants from db2068
  • 13:15 akosiaris: upload apertium-fra-cat_1.3.0~r84327-1+wmf2 to apt.wikimedia.org/jessie-wikimedia/main
  • 13:14 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.12
  • 13:14 moritzm: installing openssh updates from stretch 9.4 point release
  • 13:05 godog: run xfs_repair /dev/sdd1 on ms-be1043 - T199198
  • 12:57 marostegui: Drop unused grants from db1073
  • 11:32 dereckson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix bewikibooks logo (T189218) and bnwikisource namespace (T199161) (duration: 00m 57s)
  • 11:14 dereckson@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/WikimediaEvents/extension.json: Add event logging for WMDE fundraising banners (duration: 00m 58s)
  • 11:08 godog: swift eqiad-prod: more weight to ms-be1036 after hw repair
  • 09:46 godog: shut down ms-be1041 for hardware diagnostics - T199198
  • 09:17 moritzm: ran puppet node clean/deactivate on db2064, hardware is broken for good and caused ongoing connection failures in cumin/debdeploy (T195228)
  • 08:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1083 (duration: 00m 56s)
  • 08:44 ppchelko@deploy1001: Started restart [cpjobqueue/deploy@ba672a3]: It logs disconnected consumers after many rebalances during the outage
  • 08:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1083 (duration: 00m 55s)
  • 08:30 oblivian@puppetmaster1001: conftool action : set/ttl=300; selector: dnsdisc=eventbus,name=.*
  • 08:28 _joe_: forcing puppet run on scb* to pick up the eventstreams change
  • 08:25 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=eventbus,name=.*
  • 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1083 (duration: 00m 56s)
  • 08:20 oblivian@puppetmaster1001: conftool action : set/ttl=10; selector: dnsdisc=eventbus,name=.*
  • 08:11 _joe_: reeabling and running puppet on scb1*
  • 08:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1083 (duration: 00m 56s)
  • 07:37 volans: reimaging spare system californium for testing reimage script changes
  • 07:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 with low weight (duration: 00m 57s)
  • 07:18 marostegui: Stop MySQL on db1083 to pick up new binlog format and MySQL upgrade
  • 07:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 for maintenance (duration: 00m 58s)
  • 07:01 marostegui: Drop unused grants from dbstore2002:3320 dbstore1001:3311 db2037 db2034
  • 06:38 marostegui: Drop unused grants from db1061 db1096:3316 db1113:3316
  • 06:26 elukey: restart rsyslog on wezen - T199406
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 after alter table (duration: 00m 57s)
  • 05:01 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db1075 (s3 primary master) - T187521
  • 05:00 marostegui: Deploy schema change on db1075 (s3 primary master) T146591 T197891 T196379
  • 04:48 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db1123 - T187521
  • 04:48 marostegui: Deploy schema change on db1123 T146591 T197891 T196379
  • 04:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 for alter table (duration: 00m 58s)
  • 03:18 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 12 03:18:58 UTC 2018 (duration 10m 25s)
  • 03:08 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 14m 31s)
  • 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 14m 44s)
  • 00:15 ejegg: updated CiviCRM from d24c2bc08b to b21d783e7f

2018-07-11

  • 23:32 thcipriani@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 39s)
  • 23:26 thcipriani@deploy1001: Synchronized wmf-config/throttle.php: SWAT: throttle: lift limits for Kaqchikel edit-a-thon; clear expired rules T199040 (duration: 00m 56s)
  • 23:17 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable anonymous cookie blocking T192017 (duration: 00m 58s)
  • 22:25 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d (T195325 T189830 T177619 T196523) (duration: 06m 44s)
  • 22:23 Reedy: created wikilove_log on ckbwiki T199336
  • 22:19 ejegg: activated SmashPig recurring donation job
  • 22:18 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d (T195325 T189830 T177619 T196523)
  • 21:53 elukey: restart rsyslog on lithium - in:imtcp stuck in EAGAIN (Resource temporarily unavailable) due to a old socket to tegmen.wikimedia.org
  • 21:47 twentyafterfour: disabled phabricator throttling
  • 21:47 elukey: re-enable kafka mirror maker on kafka100[1-3]
  • 21:41 ppchelko@deploy1001: Finished deploy [changeprop/deploy@45c3807]: Temporary decrease concurrency (duration: 01m 17s)
  • 21:40 ppchelko@deploy1001: Started deploy [changeprop/deploy@45c3807]: Temporary decrease concurrency
  • 21:16 elukey: starting kafka on kafka100[1-3] after zk cleanup
  • 21:14 _joe_: repooling mw1280
  • 20:57 krinkle@deploy1001: Synchronized wmf-config/mc.php: Ifa659de6453 - Revert multi-write mcrouter for most wikis - T198239 (duration: 00m 58s)
  • 20:49 _joe_: depooling mw1280 for debugging
  • 20:48 _joe_: repooled mw1223 after investigation
  • 20:39 _joe_: depooling mw1223 for debugging
  • 20:20 bearND: rolled back mobileapps deploy
  • 20:20 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d (T195325 T189830 T177619 T196523) (duration: 03m 30s)
  • 20:16 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d (T195325 T189830 T177619 T196523)
  • 20:03 volans: rolling restarting mediawiki API in eqiad with the highest load
  • 19:45 gehel: note: updater was not and will not be using kafka
  • 19:45 gehel: restarting wdqs-updater on wdqs1010 (still not using Kafka)
  • 19:31 elukey: cleaned up *change-prop.retry.change-prop.retry* in /srv/kafka/data on kafka100[1-3]
  • 19:22 akosiaris: stop changeprop on all scb hosts
  • 19:16 _joe_: restarting a few hhvm appservers with high load
  • 18:44 akosiaris: ok, change merged, running puppet on scb hosts
  • 18:34 elukey: restarted topic nuke script for kafka main
  • 18:14 elukey: start kafka on kafka1002
  • 18:14 elukey: stop mirror makers on kafka100[1-3]
  • 18:10 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 fully (duration: 00m 57s)
  • 17:36 _joe_: restarted cpjobqueue in codfw
  • 17:31 elukey: restart kafka on kafka1001 (oom registered)
  • 17:12 elukey: restart kafka on kafka1003 with 2G heap settings
  • 17:10 elukey: restart kafka on kafka1002 with 2G heap settings
  • 17:06 elukey: restarted kafka on kafka1001 with Xmx 2G and Xms 2F
  • 17:05 _joe_: restarting cpjobqueue on scb2001
  • 17:00 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=eventbus,name=eqiad
  • 16:50 elukey: stop topics cleaner script
  • 16:40 _joe_: masking and stopping cpjobqueue, changeprop everywhere
  • 16:36 elukey: start topic clean procedure on kafka1001 (tmux root session)
  • 16:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 with low load (duration: 00m 58s)
  • 16:19 elukey: restart kafka on kafka1003
  • 16:19 Pchelolo: restart cpjobqueue
  • 16:02 ejegg: disabled failmail logstream for SmashPig
  • 15:40 _joe_: restarting eventbus on kafka-main in eqiad
  • 15:26 chasemp: install dtach on labnet1003
  • 15:18 _joe_: restarting kafka on kafka1002
  • 15:16 gehel: killing wdqs-updater on wdqs10(09|10) to diminish load on kafka
  • 15:11 elukey: restart again kafka on kafka100[1,2] - failed for OOM
  • 15:08 Pchelolo: stop cpjobqueue in eqiad
  • 15:03 elukey: restart kafka on kafka1003
  • 14:58 papaul: shutting down furud to disconnect disk array shelves
  • 14:57 elukey: rolling restart of eventbus on kafka100[1-3]
  • 14:53 elukey: restart kafka on kafka1002
  • 14:52 elukey: restart kafka on kafka1001
  • 14:18 ppchelko@deploy1001: Finished deploy [changeprop/deploy@2e56855]: Move static blacklisting to change-prop T198386 (duration: 01m 22s)
  • 14:16 ppchelko@deploy1001: Started deploy [changeprop/deploy@2e56855]: Move static blacklisting to change-prop T198386
  • 14:15 volans: reset iLO on hpiLO on conf1005 to see if it fixes the issues with the mgmt interface
  • 13:41 thcipriani@deploy1001: Synchronized README: noop: test scap 3.8.4-1 (duration: 00m 56s)
  • 13:25 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.12 (duration: 00m 57s)
  • 13:24 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.12
  • 13:16 akosiaris: restart apertium-apy for apertium-fra-cat upgrade on scb boxes to 1.3.0~r84327-1+wmf1, T189076
  • 13:14 elukey: roll restart of aqs on aqs* to pick up the new Druid config
  • 13:13 akosiaris: upgrade apertium-fra-cat on scb boxes to 1.3.0~r84327-1+wmf1, T189076
  • 12:59 marostegui: Remove unused grants from db1098:3316
  • 12:43 reedy@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/ContactPage: New config! (duration: 00m 57s)
  • 12:41 reedy@deploy1001: Synchronized wmf-config/MetaContactPages.php: ContactPage new config (duration: 00m 56s)
  • 12:40 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: ContactPage new config (duration: 00m 57s)
  • 12:24 raynor: EU SWAT finished
  • 12:23 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleanup: Remove unsued MFForceSecureLogin (duration: 00m 56s)
  • 12:19 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleanup: Remove unused MinervaDownloadIcon (duration: 00m 57s)
  • 12:11 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleanup: Remove unsued VectorExperimentalPrintStyles (duration: 00m 56s)
  • 12:05 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove ambox images on mobile wikis (T191303) (duration: 00m 57s)
  • 11:53 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable page previews for all new editors (T197719) (duration: 00m 56s)
  • 11:38 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/EventBus/: SWAT: Properly handle null content format (duration: 00m 59s)
  • 11:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable FileExporter for sourceswiki (T198594) (duration: 00m 56s)
  • 11:12 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist wikisource populateChangeTagDef.php
  • 11:10 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgChangeTagsSchemaMigrationStage to write both for Wikisource (T194165) (duration: 00m 58s)
  • 11:09 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere.Take 2, feeds timed out. T169009 (duration: 02m 56s)
  • 11:08 arturo: re-enable puppet in install1002
  • 11:06 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere.Take 2, feeds timed out. T169009
  • 11:06 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere. T169009 (duration: 08m 07s)
  • 11:05 jynus: stop db1086 for reimage
  • 11:03 akosiaris: restart HHVM on mw1282
  • 10:58 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere. T169009
  • 10:45 arturo: disabled puppet in install1002 for a debian installer live hack (T199202)
  • 10:35 moritzm: installing tiff security updates on jessie hosts
  • 10:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully depool db1086 (duration: 00m 56s)
  • 10:11 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/includes/libs/rdbms/ChronologyProtector.php: ee89660 (duration: 00m 57s)
  • 10:09 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/includes/libs/rdbms/ChronologyProtector.php: 552dbdb (duration: 00m 59s)
  • 10:08 vgutierrez: reimage baham as spare system - T199247
  • 09:55 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. Take 2, check timed out. T169009 (duration: 11m 14s)
  • 09:44 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. Take 2, check timed out. T169009
  • 09:41 ppchelko@deploy1001: deploy aborted: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. T169009 (duration: 03m 13s)
  • 09:38 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. T169009
  • 09:34 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3] (dev-cluster): Upgrade cassandra driver to 3.5.0 T169009 (duration: 04m 20s)
  • 09:33 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 57s)
  • 09:30 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3] (dev-cluster): Upgrade cassandra driver to 3.5.0 T169009
  • 09:07 moritzm: installing remaining php7 security updates for stretch
  • 08:37 godog: reboot ms-be1040 to run hardware diagnostics - T199198
  • 08:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 after alter table (duration: 00m 56s)
  • 08:14 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db1077 with replication, this will generate lag on s3 labs hosts - T187521
  • 08:13 marostegui: Deploy schema change on db1077 with replication, this will generate lag on s3 labs hosts T146591 T197891 T196379
  • 08:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 for alter table (duration: 00m 56s)
  • 08:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 after alter table (duration: 00m 57s)
  • 07:57 elukey: roll restart of aqs on aqs* to rollback the druid config
  • 07:51 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on dbstore1002:s3, db1078 - T187521
  • 07:50 marostegui: Deploy schema change on db1078 T146591 T197891 T196379
  • 07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 for alter table (duration: 00m 56s)
  • 07:15 marostegui: Deploy schema change on dbstore1002:s3 T146591 T197891 T196379
  • 06:59 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db2043 (s3 codfw master), this will generate lag on s3 codfw - T187521
  • 06:40 marostegui: Deploy schema change on s3 codfw master (db2043) with replication, this will generate lag on s3 codfw T146591 T197891 T196379
  • 06:30 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1062 (s7 primary master) - T187521
  • 06:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 after alter table (duration: 00m 55s)
  • 06:26 marostegui: Deploy schema change on db1062 (s7 primary master) T146591 T197891 T196379
  • 06:12 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1086 - T187521
  • 06:12 marostegui: Deploy schema change on db1086 T146591 T197891 T196379
  • 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 for alter table (duration: 00m 57s)
  • 06:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 after alter table (duration: 00m 57s)
  • 05:55 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: fix logspam, ParserMigration breakage (duration: 00m 57s)
  • 05:49 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1079 with replication, this will generate lag on s7 labs hosts - T187521
  • 05:31 marostegui: Deploy schema change on db1079 with replication, this will generate lag on s7 labs hosts T146591 T197891 T196379
  • 05:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 for alter table (duration: 00m 56s)
  • 05:23 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1090 - T187521
  • 05:20 marostegui: Deploy schema change on db1090 T146591 T197891 T196379
  • 05:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 after alter table (duration: 00m 56s)
  • 05:06 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1094 - T187521
  • 05:05 marostegui: Deploy schema change on db1094 T146591 T197891 T196379
  • 05:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 for alter table (duration: 01m 07s)
  • 03:15 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 11 03:15:37 UTC 2018 (duration 10m 23s)
  • 03:05 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 15m 35s)
  • 02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 12m 37s)
  • 01:27 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I423f8fac62 - Remove wgGadgetsCacheType setting (duration: 00m 54s)
  • 01:01 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I0c621368d6 - Remove unused tmarray variables (duration: 00m 56s)
  • 00:44 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I3d02da810 - Clean up wgLocaltimezone (duration: 00m 56s)
  • 00:40 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ia017dea257d - Clean up wgLocaltimezone (duration: 00m 56s)
  • 00:35 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I1aad80c3bcb - Remove setting of wgLocalTZOffset (duration: 00m 57s)

2018-07-10

  • 23:50 ejegg: disabled ProcessSmashPigRecurring in Civi scheduled jobs, to be replaced with process-control job
  • 22:27 ejegg: updated payments-wiki from 383f667171 to 63372154d0
  • 21:40 chasemp: reboot labnet100[34]
  • 21:04 XioNoX: re-configure GTT circuit in eqiad/knams
  • 20:50 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/tests/phpunit/includes/libs/objectcache/MultiWriteBagOStuffTest.php: 4fba9f6a032 (duration: 00m 56s)
  • 20:48 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/includes/libs/objectcache/MultiWriteBagOStuff.php: 4fba9f6a032 (duration: 00m 57s)
  • 18:33 aaron@deploy1001: Synchronized wmf-config/mc.php: Make all non-test wikis write to both nutcracker and mcrouter again (duration: 00m 57s)
  • 16:51 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.12
  • 16:03 vgutierrez: replacing baham with authdns2001 - T196664
  • 15:27 ema: merge alternate_domains vcl patch T164609
  • 15:19 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.12 and rebuild l10n cache (duration: 61m 27s)
  • 14:31 marostegui: Set disk #0 offline for replacement - T199056
  • 14:17 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.12 and rebuild l10n cache
  • 14:15 moritzm: installing tiff security updates on jessie
  • 13:53 moritzm: installing ruby-sprockets security updates
  • 13:48 moritzm: installing subversion updates from jessie 9.11 point release
  • 13:40 moritzm: rolling restart of thumbor to pick up new exiv2/openssl
  • 13:29 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.5 (duration: 03m 13s)
  • 13:26 moritzm: installing cups security updates on jessie
  • 13:25 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.4 (duration: 06m 18s)
  • 13:14 thcipriani@deploy1001: Synchronized scap/plugins/clean.py: no op sync for consistancy Scap clean: remove remote cache directory (duration: 00m 51s)
  • 12:53 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/includes/libs/objectcache/ReplicatedBagOStuff.php: 4ad6b70 (duration: 00m 51s)
  • 12:24 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/SpamBlacklist/includes/SpamBlacklist.php: 08a2153f7aa (duration: 00m 51s)
  • 12:20 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/SpamBlacklist/includes/SpamBlacklist.php: 583dc7a92f9b (duration: 00m 51s)
  • 12:06 aaron@deploy1001: Synchronized wmf-config/mc.php: Revert "Make all non-test wikis write to both nutcracker and mcrouter" (duration: 00m 56s)
  • 11:56 ppchelko@deploy1001: Finished deploy [restbase/deploy@2674971]: Roll out cassandra-driver@3.5.0 to restbase2001 (duration: 02m 52s)
  • 11:53 ppchelko@deploy1001: Started deploy [restbase/deploy@2674971]: Roll out cassandra-driver@3.5.0 to restbase2001
  • 11:43 arturo: updated compiler facts `PUPPET_MASTERS=puppetmaster1001.eqiad.wmnet PUPPET_COMPILER=compiler02.puppet3-diffs.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts`
  • 11:32 aaron@deploy1001: Synchronized wmf-config/mc.php: Make all non-test wikis write to both nutcracker and mcrouter (duration: 00m 51s)
  • 11:25 dcausse@deploy1001: Synchronized ./wmf-config/InitialiseSettings.php: [cirrus] cleanup unused config vars 2/2 (duration: 01m 40s)
  • 11:24 moritzm: installing PHP 7 security updates on stretch
  • 11:09 dcausse@deploy1001: Synchronized ./wmf-config/: [cirrus] cleanup unused config vars 1/2 (duration: 00m 53s)
  • 10:27 ppchelko@deploy1001: Finished deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header, take 2, checer timed out (duration: 06m 39s)
  • 10:20 ppchelko@deploy1001: Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header, take 2, checer timed out
  • 10:20 ppchelko@deploy1001: deploy aborted: Fix up the invalid Vary header (duration: 12m 50s)
  • 10:07 ppchelko@deploy1001: Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header
  • 10:07 moritzm: restarting thumbor on thumbor1001 to pick up exiv2 security updates
  • 10:03 elukey: restart analytics100[1,2]'s hadoop resource managers, some I/O socket errors after the ip6 interface change
  • 09:34 elukey: forced umount of /mnt/hdfs on stat1004, several processes hang for it (causing load) and transport not connected
  • 09:25 godog: swift eqiad-prod add ms-be1036 back gradually - T196873
  • 09:05 moritzm: installing libsoup security updates on jessie/stretch
  • 08:38 moritzm: installing libjpeg-turbo security updates on trusty
  • 08:37 godog: drain and restart cassandra-a on restbase2001 to test a restart
  • 08:34 elukey: rolling restart of AQS to apply the new config
  • 08:09 godog: disable puppet on hosts running cassandra before merging 444247 and 443114
  • 08:08 moritzm: installing openslp security updates on trusty
  • 07:58 moritzm: installing ntp security updates on trusty (may trigger some Icinga warnings about clocks, these recover after a while)
  • 07:33 twentyafterfour: deploying fix for phabricator userpage having hard-coded my username
  • 06:53 twentyafterfour: deployed rPHEX03173dd0097451f60faa3a1705abee58a9fe4c5f
  • 06:32 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on s7 codfw master (db2040) with replication, this will generate lag on s7 codfw - T187521
  • 06:23 marostegui: Deploy schema change on codfw s7 master (db2040) with replication, this will generate lag on s7 codfw T146591 T197891 T196379
  • 06:17 marostegui: Deploy schema change on s8 primary master (db1071) T146591 T197891 T196379
  • 06:06 marostegui: Deploy schema change on dbstore1002:s8 T146591 T197891 T196379
  • 06:02 marostegui: Deploy schema change on codfw s8 master (db2045) with replication, this will generate lag on s8 codfw T146591 T197891 T196379
  • 05:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 after alter table (duration: 00m 50s)
  • 05:40 marostegui: Deploy schema change on s4 primary master (db1068) T146591 T197891 T196379
  • 05:36 marostegui: Deploy schema change on db1091 T146591 T197891 T196379
  • 05:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 for alter table (duration: 00m 50s)
  • 05:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 after alter table (duration: 00m 49s)
  • 05:27 marostegui: Deploy schema change on db1081 T146591 T197891 T196379
  • 05:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 for alter table (duration: 00m 50s)
  • 05:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 00m 50s)
  • 05:21 marostegui: Deploy schema change on db1121 with replication, this will generate lag on s4 labs hosts T146591 T197891 T196379
  • 05:17 marostegui: Optimize frwiki.wbc_entity_usage on s6 eqiad hosts T187521
  • 05:16 marostegui: Deploy schema change on db1103:3314 T146591 T197891 T196379
  • 05:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 50s)
  • 05:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 00m 50s)
  • 05:04 marostegui: Deploy schema change on db1097:3314 T146591 T197891 T196379
  • 05:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 for alter table (duration: 00m 51s)
  • 04:59 marostegui: Optimize frwiki.wbc_entity_usage on s6 codfw, this will generate lag on s6 codfw - T187521
  • 04:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 after alter table (duration: 00m 50s)
  • 04:48 marostegui: Deploy schema change on db1084 T146591 T197891 T196379
  • 04:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 for alter table (duration: 00m 52s)
  • 04:36 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1066 (s2 primary master) - T187521
  • 04:28 marostegui: Deploy schema change on s1 primary master (db1052) T146591 T197891 T196379
  • 02:44 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 10 02:44:29 UTC 2018 (duration 10m 18s)
  • 00:21 twentyafterfour: deploying https://phabricator.wikimedia.org/rPHABc6f75c918afa1cc59472c5fe226539e093f6c3ef

2018-07-09

  • 23:58 ejegg: updated payments-wiki from ed40f33c44 to 383f667171
  • 23:58 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop trying to set wgLicenseURL T154069 (duration: 00m 50s)
  • 23:55 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: No need for officewiki-specific upload for MP3s any more I42ffbcd76f6 (duration: 00m 50s)
  • 23:53 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgTmhEnableMp3Uploads Idbf1201813 (duration: 00m 50s)
  • 23:52 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgTmhEnableMp3Uploads I6479dbacd4 (duration: 00m 50s)
  • 23:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgVisualEditorNonAccountEnableProportion I3b8cccc00 (duration: 00m 50s)
  • 23:47 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgVisualEditorNonAccountEnableProportion If2ee24ae (duration: 00m 50s)
  • 23:43 jforrester@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: SWAT Cleanup: Remove Beta Cluster use of wikieditor-preview preference Ib16e6e19b1 (duration: 00m 50s)
  • 23:42 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Remove wgWikiEditorFeatures I7580e94ecf7 (duration: 00m 50s)
  • 23:36 jforrester@deploy1001: Synchronized wmf-config/extension-list: Stop loading the MwEmbedSupport extension, part IV (duration: 00m 50s)
  • 23:34 jforrester@deploy1001: Synchronized multiversion/submodules.json: SWAT MwEmbedSupport extension, part III (duration: 00m 50s)
  • 23:33 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT MwEmbedSupport extension, part II (duration: 00m 51s)
  • 23:26 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT 444650 (duration: 00m 49s)
  • 23:19 jforrester@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/ORES: ORES fix Ia1b09d7711 for RoanKattouw (duration: 00m 50s)
  • 23:18 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Stop loading the MwEmbedSupport extension, part I (duration: 00m 50s)
  • 23:16 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT for bd808 (duration: 00m 51s)
  • 22:54 bstorm_: added disk from new shelf to /srv/dumps on labstore1006
  • 21:48 bawolff: createAndPromote.php --wiki=officewiki BWolff_\(WMF\) --sysop --force
  • 21:45 ejegg: updated payments-wiki from f302c5ffa3 to ed40f33c44 (Outbound cURL logging in SmashPig)
  • 19:27 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: GlobalPrefs everywhere https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/444652/ (duration: 00m 51s)
  • 19:01 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES damaging filter on srwiki (T197012) (duration: 00m 50s)
  • 18:59 ejegg: updated payments-wiki from 198c31a657 to f302c5ffa3 (DI update)
  • 18:52 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES edit quality filters on bswiki (T197010) (duration: 00m 50s)
  • 18:48 XioNoX: apply analytics-in6 firewall filter to cr1/2-eqiad with a default permit+log
  • 18:41 ejegg: updated payments-wiki from 43989ebc96 to 198c31a657 (patches from REL1_27)
  • 18:37 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Avoid losing cached ParserOutput in doEditContent (T198483) (duration: 00m 51s)
  • 18:26 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Resyncing to shut up warnings (duration: 00m 48s)
  • 18:23 catrope@deploy1001: Synchronized wmf-config/: Rollout Watchlist Structured Filters to most wikis (T181193) (duration: 00m 53s)
  • 18:14 XioNoX: updating mr1-eqsin security policies - T199074
  • 18:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable TemplateStyles on huwiktionary (T198725) (duration: 00m 52s)
  • 17:32 XioNoX: refactoring analytics firewall policies
  • 17:24 gehel@deploy1001: Finished deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (duration: 09m 18s)
  • 17:14 gehel@deploy1001: Started deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater
  • 17:12 gehel@deploy1001: Finished deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 32s)
  • 17:12 gehel@deploy1001: Started deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (wdqs1009 only)
  • 16:09 marostegui: Set disk 32:0 offline on db1069 for a replacement - T199056
  • 15:32 elukey: enabled snappy compression for varnishkafka eventlogging
  • 15:09 akosiaris: repool eqiad mathoid discovery RR
  • 14:52 akosiaris: upgrade eqiad kubernetes cluster to 1.8.14
  • 14:25 akosiaris: depool eqiad mathoid discovery RR for kubernetes upgrade
  • 14:16 moritzm: rmmoding floppy kernel module from hosts which had it loaded prior to blacklisting
  • 13:57 akosiaris: repool codfw mathoid discovery
  • 13:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 50s)
  • 13:32 marostegui: Deploy schema change on s4 codfw master (db2051) this will generate lag on s4 codfw T146591 T197891 T196379
  • 13:29 akosiaris: upgrade kubernetes200{1,2,3,4} to kubernetes 1.8.14
  • 13:29 akosiaris: upgrade acrab to kubernetes 1.8.14
  • 13:26 marostegui: Deploy schema change on db1080 T146591 T197891 T196379
  • 13:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 00m 50s)
  • 13:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 00m 50s)
  • 13:11 marostegui: Deploy schema change on db1114 T146591 T197891 T196379
  • 13:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 00m 50s)
  • 13:10 akosiaris: upgrade acrux to kubernetes 1.8.14
  • 13:00 akosiaris: depool codfw for mathoid in preparation for kubernetes cluster update
  • 11:56 zeljkof: EU SWAT finished
  • 11:56 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add importsources to ru.wikinews (T199045) (duration: 00m 50s)
  • 11:51 moritzm: installing chromium 67.0.3396.87-1~deb9u1 security updates on proton* (tested compatability of new release in deployment-prep)
  • 11:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use new wordmark for bnwikivoyage (T196680) (duration: 00m 51s)
  • 11:35 zfilipin@deploy1001: Synchronized static/images/mobile/copyright/wikivoyage-wordmark-bn.svg: SWAT: Upload wordmark for bnwikivoyage (T196680) (duration: 00m 50s)
  • 11:33 moritzm: installing glibc updates from stretch 9.4 point release
  • 11:31 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create group eventparticipant on zhwiki (T198167) (duration: 00m 51s)
  • 10:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 after alter table (duration: 00m 50s)
  • 10:35 marostegui: Deploy schema change on db1083 T146591 T197891 T196379
  • 10:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 for alter table (duration: 00m 50s)
  • 10:31 godog: upgrade hp raid firmware on ms-be1017 - T141756
  • 10:12 godog: powercycle ms-be1017 - can't login on ssh/console
  • 10:07 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 10:06 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
  • 09:22 ppchelko@deploy1001: Finished deploy [restbase/deploy@794d6ee]: Enable language variants for HTML T190689 (duration: 15m 32s)
  • 09:07 ppchelko@deploy1001: Started deploy [restbase/deploy@794d6ee]: Enable language variants for HTML T190689
  • 09:02 vgutierrez: Bump AES128-SHA traffic redirection to 100% - T192555
  • 08:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 00m 49s)
  • 08:35 jynus: updating password for labdb1004/5 for admin users
  • 08:31 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1074 with replication, this will generate lag on s2 on labshosts - T187521
  • 08:26 marostegui: Deploy schema change on db1106 with replication, this will generate lag on labs hosts for s1 T146591 T197891 T196379
  • 08:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 00m 50s)
  • 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 00m 51s)
  • 08:08 marostegui: Deploy schema change on db1105:3311 T146591 T197891 T196379
  • 08:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 00m 49s)
  • 08:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 00m 50s)
  • 07:54 marostegui: Deploy schema change on db1099:3311 T146591 T197891 T196379
  • 07:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 00m 50s)
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 00m 50s)
  • 07:40 marostegui: Deploy schema change on db1067 T146591 T197891 T196379
  • 07:40 ema: reboot lvs canaries for microcode updates: lvs4007 lvs3004 lvs2006 lvs2010 lvs1006 T127825
  • 07:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 00m 50s)
  • 07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 00m 49s)
  • 07:18 marostegui: Deploy schema change on db1119 - T146591 T197891 T196379
  • 07:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 00m 50s)
  • 07:18 elukey: update filter analytics-in4 on cr1/cr2 eqiad
  • 07:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 00m 50s)
  • 06:54 marostegui: Deploy schema change on db1089 - T146591 T197891 T196379
  • 06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 00m 50s)
  • 06:41 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1066 (s2 primary master) T197459
  • 06:30 marostegui: Deploy schema change on dbstore1002:s1 - T146591 T197891 T196379
  • 06:25 elukey: restart hue on thorium to pick up new smtp changes - T196920
  • 06:13 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1076 T197459
  • 06:12 marostegui: Deploy schema change on dbstore1001:s1 - T146591 T197891 T196379
  • 06:03 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on dbstore1002:s2, db1122 and db1105 - T187521
  • 06:01 marostegui: Deploy schema change on s1 codfw master (db2048) with replication, this will generate lag on s1 codfw - T146591 T197891 T196379
  • 05:55 marostegui: Deploy schema change on s2 primary master (db1066) - T146591 T197891 T196379
  • 05:53 marostegui: Deploy schema change on db1076 - T146591 T197891 T196379
  • 05:41 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on s2 codfw master with replication - lag will happen on s2 codfw - T187521
  • 05:31 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1090 T197459
  • 05:31 marostegui: Deploy schema change on db1090 - T146591 T197891 T196379
  • 05:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 51s)
  • 05:12 marostegui: Deploy schema change on db1074 with replication, this will generate lag on s2 on labs hosts - T146591 T197891 T196379
  • 05:04 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1074 with replication, this will generate lag on labs hosts T197459
  • 05:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 52s)
  • 02:44 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 9 02:44:19 UTC 2018 (duration 10m 23s)
  • 02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 33s)

2018-07-08

  • 20:37 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I81d566ba860 (duration: 00m 51s)
  • 20:30 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Id94ca5f55c2c6 (duration: 00m 53s)
  • 19:10 chasemp: disable puppet on labstore1007 to turn down ratelimits for nginx (load is rising and rising)
  • 11:27 elukey: restart rsyslog on lithium - in:imtcp thread stuck at 99% cpu usage

2018-07-07

  • 23:25 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Icbbccf495899 (duration: 00m 50s)
  • 23:23 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I001f68a1b39 (duration: 00m 53s)
  • 23:18 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I1b42c260b5ee3 (duration: 01m 00s)
  • 13:02 _joe_: rebooting lvs2001, panic logs from the ethernet card module

2018-07-06

  • 18:08 foks: reset 2FA for User:Howcheng
  • 16:06 chasemp: labcontrol1003:~# /sbin/reboot T198950
  • 13:36 moritzm: installing openldap security updates on trusty
  • 13:22 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1103:3312 - T197459
  • 13:22 marostegui: Deploy schema change on db1103:3312 - T146591 T197891 T196379
  • 13:05 moritzm: installing bouncycastle security updates
  • 13:00 moritzm: installing mercurial security updates
  • 12:57 hashar: Restarting Zuul
  • 12:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@c42c048]: Update restbase-mod-table-cassandra to v1.1.0 (duration: 15m 32s)
  • 12:38 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1105 and db1122 - T197459
  • 12:33 marostegui: Deploy schema change on db1105:3312 - T146591 T197891 T196379
  • 12:30 marostegui: Deploy schema change on dbstore1002:s2 and db1122 T146591 T197891 T196379
  • 12:29 mobrovac@deploy1001: Started deploy [restbase/deploy@c42c048]: Update restbase-mod-table-cassandra to v1.1.0
  • 12:18 marostegui: Deploy schema change on s2 codfw master (db2035) with replication, this will generate lag on s2 codfw T146591 T197891 T196379
  • 11:51 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on dbstore1002:s2 - T197459
  • 11:15 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db2035 (codfw s2 master), this will generate lag on s2 codfw - T197459
  • 11:09 marostegui: Optimize shwiki.logging on s3 eqiad hosts (one by one) T197459
  • 11:08 marostegui: Optimize shwiki.logging on db2043 (codfw s3 master), this will generate lag on s3 codfw - T197459
  • 11:04 marostegui: Optimize frwiki.logging on db1061 (s6 primary master) - T197459
  • 11:02 marostegui: Deploy schema change on db1061 (s6 primary master) T146591 T197891 T196379
  • 11:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 after alter table (duration: 00m 50s)
  • 10:49 marostegui: Optimize frwiki.logging on db1088 - T197459
  • 10:48 marostegui: Deploy schema change on db1088 T146591 T197891 T196379
  • 10:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 50s)
  • 10:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 after alter table (duration: 00m 51s)
  • 10:02 marostegui: Optimize frwiki.logging on db1093 - T197459
  • 10:01 mobrovac: restbase depool restbase2001 to test the cassandra node driver v3.5.0 - T169009
  • 10:00 marostegui: Deploy schema change on db1093 T146591 T197891 T196379
  • 10:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 for alter table (duration: 00m 51s)
  • 09:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 51s)
  • 09:46 marostegui: Deploy schema change on db1085 with replication, this will generate lag on s6 labsdb T146591 T197891 T196379
  • 09:08 jynus: restart db1115 mariadb instance (this will cause temporary downtime if tendril and dbtree)
  • 08:53 marostegui: Optimize frwiki.logging on db11085 with replication (this will generate lag on s6 on labsdb hosts) - T197459
  • 08:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 00m 51s)
  • 08:18 marostegui: Optimize frwiki.logging on db1113:3316 - T197459
  • 08:15 marostegui: Deploy schema change on db1113:3316 T146591 T197891 T196379
  • 08:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3318 after alter table (duration: 00m 50s)
  • 07:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 for alter table (duration: 00m 50s)
  • 07:47 marostegui: Optimize frwiki.logging on db1098:3316 - T197459
  • 07:46 marostegui@deploy1001: sync-file aborted: Depool db1098:3315 for alter table (duration: 00m 01s)
  • 07:45 marostegui: Deploy schema change on db1098:3316 T146591 T197891 T196379
  • 07:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 50s)
  • 07:22 moritzm: re-enabled puppet on mw2246
  • 07:12 moritzm: upgrading remaining video scalers to vp9-row-mt enabled ffmpeg build (T190333)
  • 07:11 marostegui: Optimize frwiki.logging on db1096:3316 - T197459
  • 07:07 marostegui: Deploy schema change on db1096:3316 T146591 T197891 T196379
  • 07:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 51s)
  • 06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 52s)
  • 06:49 marostegui: Optimize frwiki.logging on dbstore1002 - T197459
  • 06:34 marostegui: Deploy schema change on dbstore1002:s6 T146591 T197891 T196379
  • 06:31 marostegui: Optimize frwiki.logging on db2039 (s6 codfw master), with replication, this will generate lag on s6 codfw - T197459
  • 06:25 marostegui: Deploy schema change on s6 codfw master (db2039) with replication, this will generate lag on s6 codfw T146591 T197891 T196379
  • 06:22 marostegui: Optimize dewiki.logging on db1070 (s5 primary master) - T197459
  • 05:03 kartik@deploy1001: Finished deploy [cxserver/deploy@6f9fcce]: Update cxserver to bfc9c84 (T198830) (duration: 03m 26s)
  • 05:01 marostegui: Optimize dewiki.logging on db1110 - T197459
  • 05:00 kartik@deploy1001: Started deploy [cxserver/deploy@6f9fcce]: Update cxserver to bfc9c84 (T198830)
  • 05:00 marostegui: Deploy schema change on s5 primary master (db1070) T146591 T197891 T196379
  • 04:59 marostegui: Deploy schema change on db1110 T146591 T197891 T196379
  • 04:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 50s)
  • 04:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 after alter table (duration: 00m 51s)
  • 04:42 marostegui: Deploy schema change on db1082 with replication T146591 T197891 T196379
  • 00:08 ejegg: updated CiviCRM from e2c8ddd70e to d24c2bc08b

2018-07-05

  • 23:47 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/Echo/: Handle missing presentation model (T195253) (duration: 00m 52s)
  • 23:39 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Ic87af972e1 - Remove wgRecentEchoInstall (duration: 00m 51s)
  • 23:32 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I8165473c49 - Remove wgVaryOnXFPForAPI (duration: 00m 51s)
  • 23:19 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable static maps on bgwiki (duration: 00m 51s)
  • 23:13 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Use require rather than include_once for $wgInterwikiCache (duration: 00m 52s)
  • 21:55 twentyafterfour: whitelist office IPs in phabricator throttle
  • 21:30 bstorm_: disabled unused P440ar RAID controller on labvirt1020
  • 21:15 bstorm_: disabled unused RAID controller on labvirt1019
  • 19:30 gehel: repool wdqs1003, lag almost completely recovered
  • 19:07 ebernhardson: T194678 un-pause cirrussearch writes to codfw
  • 18:58 gehel: depooling wdqs1003 to help it recover update lag
  • 18:55 ebernhardson: T194678 pause cirrussearch writes to codfw to check how kafka+mirrormaker responds
  • 18:48 gehel: restarting blazegraph on wdqs1003 (updater lag)
  • 18:21 thcipriani@deploy1001: Synchronized wmf-config/Wikibase-production.php: SWAT: Set dispatchLagToMaxLagFactor to 60 for wikidata T194950 (duration: 00m 51s)
  • 18:14 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add "L" as an alias of Lexeme namespace in wikidata T195493 (duration: 00m 59s)
  • 18:10 gehel@deploy1001: Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1010 only) (duration: 00m 28s)
  • 18:09 gehel@deploy1001: Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1010 only)
  • 16:13 jynus: stop and reimage db2045
  • 15:34 mobrovac@deploy1001: Finished scap: Set the Accept-Language header explicitly when making requests to the REST API - T198186 (duration: 30m 50s)
  • 15:26 elukey: upgrade (without jvm restart) prometheus-jmx-exporter on the analytics node listed in debmonitor still not running the last version
  • 15:03 mobrovac@deploy1001: Started scap: Set the Accept-Language header explicitly when making requests to the REST API - T198186
  • 15:02 jynus: stop and reimage db2040
  • 15:00 marostegui: Optimize dewiki.logging on db1082 with replication, this will generate lag on s5 on labsdb hosts/script load irssinotifier - T197459
  • 14:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 00m 50s)
  • 14:50 moritzm: installing php security updates on netmon1002
  • 14:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 00m 50s)
  • 14:15 moritzm: installing dbus updates from stretch point release
  • 14:09 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/EventBus/: SWAT: Dont specify the comment if it is an empty string (duration: 00m 51s)
  • 14:03 jynus: stop and reimage db2052
  • 13:55 moritzm: installing glibc updates from stretch point release
  • 13:55 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Revert "Change logo for eswiki temporarily" (T198846) (duration: 00m 50s)
  • 13:50 moritzm: installing postgresql security updates on maps cluster
  • 13:41 aaron@deploy1001: Synchronized wmf-config/mc.php: Make test wikis just write to both nutcracker and mcrouter (duration: 00m 50s)
  • 13:32 zfilipin@deploy1001: Synchronized wmf-config: SWAT: Enable ORES wp10, draftquality on draft ns (118) for enwiki (T198768) (duration: 00m 51s)
  • 13:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Replace Tidy with RemexHtml everywhere (T175706) (duration: 00m 50s)
  • 13:18 marostegui: Optimize dewiki.logging on db1113:3315 - T197459
  • 13:17 marostegui: Deploy schema change on db1113:3315 T146591 T197891 T196379
  • 13:12 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgChangeTagsSchemaMigrationStage to write both for Wikivoyage, Wikibooks (T194165) (duration: 00m 52s)
  • 11:30 moritzm: installing python-mimeparse updates from jessie point release
  • 11:06 moritzm: rebooting contint1001 for kernel update/enabling microcode
  • 11:05 hashar: Stopping Jenkins CI for kernel upgrade on contint1001
  • 10:31 moritzm: installing ncurses security updates on jessie
  • 09:50 marostegui: Optimize dewiki.logging on db1097:3315 - T197459
  • 09:50 marostegui: Deploy schema change on db1097:3315 T146591 T197891 T196379
  • 09:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 00m 50s)
  • 09:42 moritzm: rebooting ms-be1013/1016/1028/1040 as microcode update canaries
  • 09:40 mobrovac@deploy1001: Finished deploy [zotero/translators@d6702bc]: String.includes() polyfill for Sveriges radio translator - T187023 (duration: 00m 08s)
  • 09:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 52s)
  • 09:40 mobrovac@deploy1001: Started deploy [zotero/translators@d6702bc]: String.includes() polyfill for Sveriges radio translator - T187023
  • 09:25 moritzm: rebooting ms-fe1005 as microcode update canary
  • 09:11 godog: drain restbase200[234] and restart cassandra to pick up updated certificates
  • 08:57 moritzm: draining restbase1007 for reboot to pick up Intel microcode update
  • 08:42 moritzm: draining restbase2010 for reboot to pick up Intel microcode update
  • 08:26 godog: add graphite2003 to carbon-c-relay frontend on graphite1001 - T196483
  • 08:12 moritzm: draining restbase2001 for reboot to pick up Intel microcode update
  • 07:40 marostegui: Optimize dewiki.logging on db1096:3315 - T197459
  • 07:39 marostegui: Deploy schema change on db1096:3315 T146591 T197891 T196379
  • 07:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 50s)
  • 07:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 00m 52s)
  • 07:13 elukey: stop mariadb on analytics1003 to apply https://gerrit.wikimedia.org/r/443893 and enable auth via unix socket
  • 06:39 kartik@deploy1001: Finished deploy [cxserver/deploy@3cb9a21]: Update cxserver to f8c71a1 (T191124, T198779) (duration: 03m 52s)
  • 06:35 kartik@deploy1001: Started deploy [cxserver/deploy@3cb9a21]: Update cxserver to f8c71a1 (T191124, T198779)
  • 06:11 marostegui: Deploy schema change on db1100 T146591 T197891 T196379
  • 06:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 00m 52s)
  • 06:09 marostegui: Optimize dewiki.logging on db1100 and dbstore1002:s5 - T197459
  • 05:50 marostegui: Deploy schema change on dbstore1002:s5 T146591 T197891 T196379
  • 05:29 marostegui: Deploy schema change on s3 primary master (db1075) T191316 T192926 T195193
  • 04:57 marostegui: Deploy schema change on s5 codfw host by host T146591 T197891 T196379
  • 04:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore db1089 original traffic (duration: 00m 51s)
  • 04:42 marostegui: Deploy schema change on s7 primary master (db1062) T191316 T192926 T195193
  • 02:40 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: 370242d13 and 45f4f61c2 (duration: 00m 52s)
  • 02:32 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 5 02:32:29 UTC 2018 (duration 10m 21s)
  • 02:22 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 08m 46s)
  • 00:46 krinkle@deploy1001: Synchronized wmf-config/: Ia3bef874a (duration: 00m 51s)

2018-07-04

  • 22:28 krinkle@deploy1001: Synchronized wmf-config/FeaturedFeedsWMF.php: I004bc9c3e71 (duration: 00m 50s)
  • 22:06 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I5a6e28 (duration: 00m 51s)
  • 21:53 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Iab176f3d205 (duration: 00m 50s)
  • 21:49 krinkle@deploy1001: Synchronized wmf-config/: I6f8cfa8f (duration: 00m 51s)
  • 21:43 krinkle@deploy1001: Synchronized wmf-config/: Ia8deabc5d2625 (duration: 00m 51s)
  • 21:42 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ia8deabc5d2625 (duration: 00m 50s)
  • 21:36 krinkle@deploy1001: Synchronized wmf-config/: If7da2a26bbf (duration: 00m 51s)
  • 21:34 krinkle@deploy1001: Synchronized wmf-config/: I1e78ba4365 (duration: 00m 52s)
  • 21:30 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ie30aeecbe (duration: 00m 52s)
  • 15:55 marostegui: Optimize dewiki.logging on s5 codfw master with replication, this will generate lag on s5 codfw - T197459
  • 15:52 ema: cp300[3-6]: puppet node clean/deactivate T167376
  • 15:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 51s)
  • 14:52 moritzm: installing libipc-run-perl updates from jessie point release
  • 14:36 moritzm: installing perl security updates on trusty (Debian already fixed)
  • 14:25 akosiaris: upgrade kubernetes staging API server to 1.8.14
  • 14:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 50s)
  • 14:10 moritzm: installing file/libmagic security updates on trusty (Debian already fixed)
  • 13:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 50s)
  • 13:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 50s)
  • 13:34 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2038, db2047 (duration: 02m 56s)
  • 12:56 marostegui: Stop MySQL and reboot db1089 to upgrade+change it to statement - T197069
  • 12:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 for maintenance - T197069 (duration: 02m 57s)
  • 12:39 ema: cp3034 repooled after hw maintenance T189305
  • 12:32 volans: shutting down bast3002 for disk replacement
  • 12:04 moritzm: installing ruby 1.9 security updates on trusty
  • 11:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 after alter table (duration: 00m 52s)
  • 11:54 ema: repool cp3043 after hardware maintenance T179953
  • 11:30 ema: shutdown cp3048 and cp3034 (both already depooled) for hardware maintenance T190607 T189305
  • 11:28 moritzm: rolling restart of cassandra on restbase hosts in eqiad completed
  • 11:27 moritzm: resuming rolling restart of cassandra on restbase hosts in eqiad completed
  • 11:17 mark: cp3043: mdadm /dev/md0 --add /dev/sdc1 (sdc is former cp3048:sdb)
  • 11:09 mark: cp3043: mdadm /dev/md0 -- fail /dev/sdb1
  • 11:03 ema: depool cp3043 (cache_upload) for hardware maintenance T179953
  • 10:58 godog: update compiler facts
  • 10:53 jynus: stop db2038 and db2047
  • 10:46 marostegui: Deploy schema change on db1077 with replication, this will generate lag on labs s3 T191316 T192926 T89737 T195193
  • 10:42 marostegui: Stop replication on db1077 to drop triggers on db1124:3313 - T192926
  • 10:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 for alter table (duration: 00m 50s)
  • 10:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 after alter table (duration: 00m 50s)
  • 10:01 moritzm: rolling reboot of sca* for "lazy fpu" kernel updates
  • 09:44 _joe_: stopping all cronjobs via a puppet run on terbium, T192092
  • 09:29 akosiaris: upload kubernetes 1.8.14 to apt.wikimedia.org/stretch-wikimedia/main
  • 09:16 moritzm: uploaded linux-meta 1.18 for jessie-wikimedia to apt.wikimedia.org
  • 09:15 elukey: reimage aqs1009 to Debian Stretch
  • 09:09 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2038, db2047 (duration: 00m 50s)
  • 09:03 marostegui: Optimize recentchanges table on s2 codfw - this will generate lag on codfw s2 - T178290
  • 09:00 akosiaris: manually rebalance the mathoid kubernetes production cluster namespaces pods wise
  • 08:56 moritzm: uploaded linux 4.9.107~wmf1 for jessie-wikimedia to apt.wikimedia.org
  • 08:54 marostegui: Deploy schema change on db1123 T191316 T192926 T89737 T195193
  • 08:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 for alter table (duration: 00m 50s)
  • 08:53 elukey: update analytics-in4 filter rules on cr1/cr2 eqiad - T198623
  • 08:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 after alter table (duration: 00m 50s)
  • 08:19 marostegui: Optimize recentchanges table on s6 codfw - this will generate lag on codfw s6 - T178290
  • 08:09 moritzm: rebooting multatuli for kernel update to 4.9.107~wmf1
  • 08:00 marostegui: Deploy schema change on db1078 T191316 T192926 T89737 T195193
  • 07:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 for alter table (duration: 00m 50s)
  • 07:58 ema: install misc VCL on all text hosts T164609
  • 07:53 volans: reimaging silver (spare host, to-be-decomm'ed) as testing host for the reimage script
  • 07:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 after alter table (duration: 00m 53s)
  • 07:46 ema: install misc VCL on a text host (cp3030) T164609
  • 07:43 moritzm: resuming rolling restart of cassandra on restbase hosts in eqiad to pick up OpenJDK security update
  • 07:38 hashar: rebased /srv/mediawiki-staging on deploy1001 for beta cluster only change https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/443475/
  • 07:22 moritzm: installing imagemagick security updates
  • 07:18 akosiaris: reimage kubernetes100{1,2}.eqiad.wmnet kubernetes200{1,2}.codfw.wmnet without swap
  • 06:43 _joe_: restarting tilerator on maps-test2004, to check if it can recover
  • 06:15 elukey: reimage aqs1008 to Debian Stretch
  • 06:03 marostegui: Optimize recentchanges table on s7 codfw - this will generate lag on codfw s7 - T178290
  • 05:34 marostegui: Optimize recentchanges table on s3 eqiad, host by host T178290
  • 05:28 marostegui: Optimize recentchanges table on s3 codfw - this will generate lag on codfw s3 - T178290
  • 04:59 marostegui: Deploy schema change on db1101:3317 T191316 T192926 T89737 T195193
  • 04:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 for alter table (duration: 00m 52s)
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 4 02:45:51 UTC 2018 (duration 10m 16s)
  • 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 14m 57s)
  • 00:38 Krinkle: Purge https://en.wikipedia.org/static/images/project-logos/eswiki-2x.png
  • 00:38 Krinkle: Purge https://en.wikipedia.org/static/images/project-logos/eswiki-1.5x.png
  • 00:37 Krinkle: Purge https://en.wikipedia.org/static/images/project-logos/eswiki.png
  • 00:36 krinkle@deploy1001: Synchronized static/images/project-logos/: T198761 - Update eswiki logo (duration: 00m 51s)

2018-07-03

  • 23:08 maxsem@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/CodeMirror: SWAT https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CodeMirror/+/443743/ (duration: 00m 52s)
  • 18:21 anomie@deploy1001: Synchronized wmf-config/CommonSettings.php: Fix for T197475 (gerrit:440543) (duration: 00m 52s)
  • 18:09 herron: cobalt:~# systemctl restart gerrit
  • 16:45 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase es1017,db1100 weights (duration: 00m 50s)
  • 16:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 with low weight (duration: 00m 50s)
  • 16:00 addshore: WikibaseLexeme for Wikidata clients, SE01EP05 deploy slot complete
  • 16:00 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY (duration: 00m 51s)
  • 15:46 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: all wikidata clients (duration: 00m 50s)
  • 15:42 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: group1 (duration: 00m 50s)
  • 15:37 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: group0 (duration: 00m 51s)
  • 15:17 godog: reset-failed on ms-be1024 for two user sessions failed when the host was in trouble
  • 15:01 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: testwiki (duration: 00m 51s)
  • 14:59 addshore@deploy1001: sync-file aborted: T197454 EP5 Enable WikibaseLexeme on clients: testwiki (duration: 00m 09s)
  • 14:42 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1017 with low weight (duration: 00m 50s)
  • 14:12 kart_: Finished ContentTranslation draft purge script
  • 14:11 papaul: OS install on graphite2003
  • 14:11 kart_: Running ContentTranslation draft purge script
  • 14:08 moritzm: rolling restart of cassandra on restbase in eqiad to pick up Java security updates
  • 14:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Change es1017 IP and rack T197072 (duration: 00m 50s)
  • 14:06 zeljkof: EU SWAT finished
  • 14:05 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Some wikis bureacurats are able to grant non-grantable groups (T197026) (duration: 00m 51s)
  • 13:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Clean legacy AddGroups/RemoveGroups (T197024) (duration: 00m 50s)
  • 13:38 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow bcts on private&fishbowl wikis advanced privilege manipulation (T197024) (duration: 00m 50s)
  • 13:31 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add namespace alias on pswikivoyage (T197507) (duration: 00m 49s)
  • 13:24 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable SandboxLink on eswikiversity (T198335) (duration: 00m 51s)
  • 13:19 marostegui: Deploy schema change on dbstore1002:s3 T191316 T192926 T89737 T195193
  • 13:14 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Wikimania 2018 (T198288) (duration: 00m 53s)
  • 12:49 jynus: shutting down es1017
  • 11:39 elukey: reimage aqs1007 to Debian Stretch
  • 11:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 after alter table (duration: 00m 52s)
  • 10:46 jynus: stop and reimage to stretch db2051
  • 10:13 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@ba672a3]: Decrease checker job concurrency T198462 (duration: 00m 52s)
  • 10:12 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@ba672a3]: Decrease checker job concurrency T198462
  • 09:59 marostegui: Deploy schema change on s3 codfw primary master (db2043) with replication, this will generate lag on s3 codfw T191316 T192926 T89737 T195193 T197459
  • 09:56 marostegui: Stop replication on db2094:3313 to replace triggers - T192926
  • 09:44 jynus: reimage db1100 for upgrade to stretch
  • 09:13 marostegui: Deploy schema change on db1098:3317 T191316 T192926 T89737 T195193 T197459
  • 09:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 for alter table (duration: 00m 50s)
  • 09:11 elukey: reimage aqs1006 to Debian Stretch
  • 09:09 jynus: deploying sys schema into db1072 (phabricator master)
  • 09:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 after alter table (duration: 00m 50s)
  • 09:04 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 51s)
  • 08:32 jynus: stop es1017 mysql for upgrade and maintenance
  • 08:06 moritzm: resuming rolling restart of cassandra on restbase in codfw to pick up Java security updates
  • 07:38 elukey: reimage aqs1005 to debian stretch
  • 07:33 akosiaris: upload apertium-separable_0.3.1-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main T191404
  • 07:18 moritzm: upgrading ffmpeg on mw1307 (T190333)
  • 07:17 akosiaris: reimage kubestage1001
  • 07:13 marostegui: Deploy schema change on db1090:3317 T191316 T192926 T89737 T195193 T197459
  • 07:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 for alter table (duration: 00m 51s)
  • 07:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 after alter table (duration: 00m 50s)
  • 07:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1017 (duration: 00m 51s)
  • 06:16 marostegui: Stop replication on db1079 to drop triggers on db1125:s7 - T192926
  • 06:14 marostegui: Deploy schema change on db1079 with replication, this will generate lag on s7 on labsdb hosts T191316 T192926 T89737 T195193 T197459
  • 06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 for alter table (duration: 00m 50s)
  • 06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 after alter table (duration: 00m 50s)
  • 05:26 marostegui: Deploy schema change on db1094 T191316 T192926 T89737 T195193 T197459
  • 05:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 for alter table (duration: 00m 49s)
  • 05:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 after alter table (duration: 00m 50s)
  • 05:04 marostegui: Deploy schema change on s8 primary master db1071 T191316 T192926 T195193
  • 04:40 marostegui: Deploy schema change on db1086 T191316 T192926 T89737 T195193 T197459
  • 04:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 for alter table (duration: 00m 52s)
  • 04:27 marostegui: Deploy schema change on s2 primary master db1066 T191316 T192926 T195193
  • 02:43 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 3 02:43:03 UTC 2018 (duration 10m 19s)
  • 02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 07s)

2018-07-02

  • 23:33 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/: Watchlist performance fixes (T198359, T198399) (duration: 00m 51s)
  • 23:09 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Add chapter wikis to CORS domain list (T181165) (duration: 00m 52s)
  • 20:53 urandom: Bringing down Cassandra for hardware testing, restbase2010 - T197477
  • 19:25 urandom: Bringing down Cassandra for hardware testing, restbase2001 - T197477
  • 19:00 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Mark rollbacking revision as patrolled T198449; WikiPage: Do not set undid revision ID for rollbacks T190374 (duration: 00m 50s)
  • 18:47 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/: RCFilters: Hide highlight containers when RCFilters is disabled T198440 (duration: 00m 51s)
  • 18:41 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Fix incorrect arguments to prepareContent() call in WikiPage T198483 (duration: 00m 51s)
  • 18:16 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/PageTriage/: Fix setting of reviewed state based on autopatrol permission T198497 (duration: 00m 52s)
  • 18:11 Niharika: Tried updating interwiki cache but it failed with "<AttributeError> 'Namespace' object has no attribute 'force'"
  • 17:16 gehel@deploy1001: Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (duration: 09m 04s)
  • 17:07 gehel@deploy1001: Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph
  • 17:05 gehel@deploy1001: Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1009 only) (duration: 00m 28s)
  • 17:04 gehel@deploy1001: Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1009 only)
  • 15:55 _joe_: stopping jobrunner, jobchron across all jobrunners T198220
  • 15:38 marostegui: Deploy schema change on dbstore1002:s7 T191316 T192926 T89737 T195193 T197459
  • 15:31 andrewbogott: rebooting labnodepool1002 as part of a paging test
  • 15:06 moritzm: installing file/libmagic security updates
  • 15:06 addshore: Lexeme deploy window done...
  • 15:04 twentyafterfour: restarted apache2 on phab1001: whitelist WMDE
  • 14:29 elukey: copy cassandra-tools-wmf 1.0.2-1 from jessie-wikimedia to stretch-wikimedia
  • 14:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1018 (duration: 00m 52s)
  • 14:16 kart_: Finished: Dry run: ContentTranslation Draft Purge
  • 14:09 kart_: Dry run: ContentTranslation Draft Purge
  • 13:40 moritzm: installing libgcrypt11 updates on trusty
  • 13:34 elukey: reimage aqs1004 to Debian Stretch
  • 13:29 hashar@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/mediawiki.rcfilters/styles/: RCFilters: Fix highlight container selector in Watchlist overrides - T198445 (duration: 00m 51s)
  • 13:28 hashar@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/ContentTranslation: Allow non-Main namespace source titles iff given in URL - T198178 (duration: 00m 52s)
  • 13:21 marostegui: Optimize frwiktionary'.logging on codfw - T197459
  • 13:15 marostegui: Optimize viwiki.logging on codfw - T197459
  • 12:59 ppchelko@deploy1001: Finished deploy [changeprop/deploy@82fd280]: Revert: Move static blacklisting to change-prop T198386 Can't deploy yet due to scap bug T198621 (duration: 01m 26s)
  • 12:57 ppchelko@deploy1001: Started deploy [changeprop/deploy@82fd280]: Revert: Move static blacklisting to change-prop T198386 Can't deploy yet due to scap bug T198621
  • 12:50 marostegui: Stop replication on db2077 to remove triggers from db2095 - T192926
  • 12:49 marostegui: Deploy schema change on s7 codfw master (db2040) with replication, this will generate lag on s7 codfw T191316 T192926 T89737 T195193
  • 12:49 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f20916f]: Move static blacklisting to change-prop T198386 take 2 (duration: 00m 24s)
  • 12:48 ppchelko@deploy1001: Started deploy [changeprop/deploy@f20916f]: Move static blacklisting to change-prop T198386 take 2
  • 12:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 after alter table (duration: 00m 52s)
  • 12:24 ppchelko@deploy1001: Finished deploy [changeprop/deploy@a5a57ff]: Move static blacklisting to change-prop T198386 (duration: 00m 39s)
  • 12:24 ppchelko@deploy1001: Started deploy [changeprop/deploy@a5a57ff]: Move static blacklisting to change-prop T198386
  • 12:10 akosiaris: T197559 upload lttoolbox_3.4.2-2+wmf2 to apt.wikimedia.org/jessie-wikimedia/main
  • 11:28 ema: repool cp3037 T196974
  • 11:02 moritzm: installing libgcrypt20 security updates
  • 10:06 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 10:05 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
  • 09:53 elukey: reboot ms-be1039 (bad disk, spike in I/O and load, not reachable via ssh or mgmt console)
  • 09:37 jynus: SET GLOBAL expire_logs_days = 30 on db1067
  • 09:24 marostegui: Deploy schema change on db1087 with replication, this will generate lag on s8 on labs T191316 T192926 T89737 T195193
  • 09:17 marostegui: Stop replication on db1087 to drop triggers on db1124 - T192926
  • 09:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 for alter table (duration: 00m 51s)
  • 09:00 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@dfdd362]: Disable delayed execution for cirrusSearchCheckerJob T198462 (duration: 00m 50s)
  • 08:59 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@dfdd362]: Disable delayed execution for cirrusSearchCheckerJob T198462
  • 08:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 00m 50s)
  • 08:47 volans@deploy1001: Finished deploy [debmonitor/deploy@15a0776]: GC fix and maintenance command (duration: 00m 24s)
  • 08:46 volans@deploy1001: Started deploy [debmonitor/deploy@15a0776]: GC fix and maintenance command
  • 08:24 moritzm: installing jasper security updates
  • 08:09 jynus: stop and reimage es1018
  • 08:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1018 (duration: 00m 50s)
  • 08:04 gehel: powercycle maps-test2003
  • 04:59 marostegui: Deploy schema change on db1109 T191316 T192926 T89737 T195193
  • 04:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 00m 56s)
  • 04:25 twentyafterfour: Bsadowski1 The throttle is averaged over 5 minutes, you should be unblocked shortly after being blocked
  • 04:24 twentyafterfour: applied security patch on phab1001, source in /srv/patches
  • 02:44 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 2 02:44:40 UTC 2018 (duration 10m 21s)
  • 02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 39s)

2018-07-01

  • 02:44 andrewbogott: forcing reboot of labservices1001 via mgmt

2018-06-30

  • 21:35 ariel@deploy1001: Finished deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063 (duration: 02m 27s)
  • 21:32 ariel@deploy1001: Started deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063
  • 16:28 chasemp: labstore1006:~# service nfs-kernel-server restart

2018-06-29

  • 21:37 chasemp: (late log) of VLAN creations for T184209 cloud-instances2-b-eqiad and cloud-instance-transport1-b-eqiad
  • 19:11 herron: stat1005:~# killall -u ironholds T197895
  • 17:28 XioNoX: Re-activating v6 BGP session to Telia on cr2-eqiad - T198502
  • 17:21 XioNoX: deactivating v6 BGP session to Telia on cr2-eqiad - T198502
  • 14:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 00m 50s)
  • 12:30 moritzm: uploaded ffmpeg 3.2.10-1~deb9u1+wmf1 to apt.wikimedia.org/stretch-wikimedia (component/vp9) (linked against libvpx 1.7 and with backported row-mt support) (T190333)
  • 11:11 moritzm: installing zsh updates from jessie 8.11 point release
  • 11:01 moritzm: installing lame security updates
  • 10:34 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2058 after reimage (duration: 00m 51s)
  • 10:21 mobrovac@deploy1001: Finished deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 - T165105 T197853 (duration: 03m 26s)
  • 10:20 marostegui: Deploy schema change on db1092 T191316 T192926 T89737 T195193
  • 10:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 00m 50s)
  • 10:18 mobrovac@deploy1001: Started deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 - T165105 T197853
  • 10:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 00m 50s)
  • 09:23 marostegui: Stop MySQL on db2058 to reimage it
  • 09:23 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2058 for reimage (duration: 00m 50s)
  • 09:18 moritzm: uploaded libvpx 1.7.0-3+wmf1 to apt.wikimedia.org/stretch-wikimedia (component/vp9) (T190333)
  • 09:08 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2054 after reimage (duration: 00m 51s)
  • 08:39 vgutierrez: Apply new TLS varnishkafka settings in cache::text nodes - T182993
  • 08:27 akosiaris: upgrade php on phab1001 to 5.6.33+dfsg-0+deb8u1
  • 08:03 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1016 (duration: 00m 50s)
  • 07:53 marostegui: Stop MySQL on db2054 for reimage
  • 07:53 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2054 for reimage (duration: 00m 50s)
  • 06:21 jynus: stop and reimage es1016
  • 05:40 elukey: force umount of dumps labstore nfs mountpoints on stat100[56]/notebook100[34] to reduce load (also too many open files)
  • 05:02 marostegui: Deploy schema change on db1104 T191316 T192926 T89737 T195193
  • 05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 for alter table (duration: 00m 50s)
  • 04:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 after alter table (duration: 00m 50s)
  • 04:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 (duration: 00m 52s)
  • 01:53 catrope@deploy1001: Finished scap: Full scap to fix i18n issues on testwiki (duration: 32m 42s)
  • 01:20 catrope@deploy1001: Started scap: Full scap to fix i18n issues on testwiki
  • 01:19 krinkle@deploy1001: Synchronized vendor/: I39e592d837 - remove redundant composer packages for xhgui profiling (duration: 00m 52s)

2018-06-28

  • 23:54 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/mediawiki.rcfilters/: Watchlist perf fixes (T198359, T198399) (duration: 00m 52s)
  • 23:18 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Increase Schema:CitationUsage sampling rate to 100% (T191086) (duration: 00m 51s)
  • 21:55 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/PageTriage/: Update extension directory (duration: 00m 51s)
  • 21:52 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Draft namespace and AfC mode for PageTriage on testwiki T198143 (duration: 00m 53s)
  • 20:53 dduvall@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.10
  • 20:46 marxarelli: Rolling 1.32.0-wmf.10 to group2 following fix and successful re-deploy to group1
  • 20:29 dduvall@deploy1001: rebuilt and synchronized wikiversions files: commonswiki to 1.32.0-wmf.10 otra vez
  • 20:07 dduvall@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Syncing table locking fix (T198350) (duration: 00m 57s)
  • 20:04 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Rollback commonswiki to 1.32.0-wmf.8 (resync following ssh hang)
  • 20:02 dduvall@deploy1001: sync-wikiversions aborted: Rollback commonswiki to 1.32.0-wmf.8 (duration: 15m 24s)
  • 19:34 dduvall@deploy1001: rebuilt and synchronized wikiversions files: commonswiki to 1.32.0-wmf.10
  • 19:17 dduvall@deploy1001: Synchronized php: Group1 (less commons) to 1.32.0-wmf.10 (duration: 00m 57s)
  • 19:16 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Group1 (less commons) to 1.32.0-wmf.10
  • 16:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1015 - crashed (duration: 00m 57s)
  • 16:09 moritzm: restarting Cassandra instances on restbase2005 to pick up Java security update
  • 15:01 marostegui: Deploy schema change on db1101:3318 T191316 T192926 T89737 T195193
  • 15:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 for alter table (duration: 00m 57s)
  • 14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 after alter table (duration: 00m 58s)
  • 14:50 raynor: EU SWAT finished
  • 14:49 pmiazga@deploy1001: Synchronized php-1.32.0-wmf.10/skins/Vector/components/watchstar.less: SWAT: Use exactly calculated value to work around a Chrome bug (T196610) (duration: 01m 00s)
  • 14:46 elukey: upgrade piwik 3.2.1 to matomo (new name/package) 3.5.1 - T192298
  • 14:29 moritzm: installing ghostscript security updates
  • 14:26 hashar: CI jobs running npm might suffer from a 10 minutes delay since June 27th | T198348
  • 14:06 gehel: downgrading cassandra to 2.2.6-wmf3 on maps-test2001 (it should never have been upgraded)
  • 13:49 dcausse@deploy1001: Synchronized ./php-1.32.0-wmf.10/includes/htmlform/: Allow overloading of getLabel() with return ' ' (duration: 00m 59s)
  • 13:49 elukey: downgrade cassadra and cassandra-tools from 2.2.6-wmf5 to 2.2.6-wmf3 in jessie-wikimedia component/cassandra22 - T197062
  • 13:25 dcausse@deploy1001: Synchronized ./wmf-config/Wikibase-production.php: Add cirrussearch settings for wikibase (3/3) (duration: 00m 56s)
  • 13:18 dcausse@deploy1001: Synchronized ./wmf-config/: Add cirrussearch settings for wikibase (2/3) (take 2) (duration: 00m 58s)
  • 13:14 vgutierrez: Apply new TLS varnishkafka settings in cache::upload nodes - T182993
  • 13:13 dcausse@deploy1001: Synchronized ./wmf-config/WikibaseSearchSettings.php: Add cirrussearch settings for wikibase (1.5/3) (duration: 00m 56s)
  • 13:01 elukey: upload matomo (new Piwik) 3.5.1-1 to jessie-wikimedia
  • 13:00 moritzm: installing bwm-ng update from jessie 8.11 point release
  • 12:53 moritzm: installing blktrace update from jessie 8.11 point release
  • 12:49 elukey: stop hadoop daemons on analytics1032 + shutdown to swap BBU -T194234
  • 12:37 akosiaris: repool scb1002 T196901
  • 12:37 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,service=.*,cluster=scb,name=scb1002
  • 12:08 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,service=.*,cluster=scb,name=scb1002
  • 12:04 moritzm: installing patch security updates
  • 10:48 twentyafterfour: deployed fix for PhabricatorDataNotAttachedException - https://phabricator.wikimedia.org/rPHEX03971ea8965d3613df69833a766d1502b6d8dabb
  • 10:19 twentyafterfour: hotfixing phabricator DataNotAttached bug
  • 10:18 joal@deploy1001: Finished deploy [analytics/refinery@4fc20a5]: Regular weekly deploy (duration: 08m 21s)
  • 10:11 twentyafterfour: phabricator database migration complete, service restored and appears stable.
  • 10:10 joal@deploy1001: Started deploy [analytics/refinery@4fc20a5]: Regular weekly deploy
  • 10:07 twentyafterfour: running phabricator database migration
  • 10:00 moritzm: installing reportbug update from jessie 8.11 point release
  • 09:58 twentyafterfour: Resuming deployment of phabricator upgrade tagged release/2018-06-27/1 - details: https://phabricator.wikimedia.org/project/profile/3439/ )
  • 09:48 joal@deploy1001: Finished deploy [analytics/aqs/deploy@8eef2a9]: Deploying AQS pageviews-per-country ceiling-value glue code - Corrected (duration: 02m 48s)
  • 09:45 joal@deploy1001: Started deploy [analytics/aqs/deploy@8eef2a9]: Deploying AQS pageviews-per-country ceiling-value glue code - Corrected
  • 09:29 joal@deploy1001: Finished deploy [analytics/aqs/deploy@194ca96]: Deploying AQS pageviews-per-country ceiling-value glue code (duration: 01m 03s)
  • 09:28 joal@deploy1001: Started deploy [analytics/aqs/deploy@194ca96]: Deploying AQS pageviews-per-country ceiling-value glue code
  • 09:20 arturo: T198377 stop nova-spiceproxy daemon in labcontrol1002.wikimedia.org
  • 09:14 mobrovac@deploy1001: Finished deploy [proton/deploy@8a887b5]: Update to dceaf80 - T186748 (duration: 00m 28s)
  • 09:13 mobrovac@deploy1001: Started deploy [proton/deploy@8a887b5]: Update to dceaf80 - T186748
  • 09:10 vgutierrez: Apply new TLS varnishkafka settings in cache::misc nodes - T182993
  • 08:46 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1016 (duration: 01m 04s)
  • 08:46 arturo: aborrero@labtestnet2001:~ 7s 130 $ sudo service nova-spiceproxy stop # daemon in infinite respawning loop
  • 08:15 elukey: restart-hhvm on mw1227 (some threads stuck in jit-related operations, causing high load)
  • 08:09 vgutierrez: updating librdkafka1 && restart varnishkafka instances in cache::text nodes - T182993
  • 07:12 elukey: upload piwik 3.2.1 to jessie-wikimedia
  • 04:46 marostegui: Deploy schema change on db1099:3318 T191316 T192926 T89737 T195193
  • 04:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 for alter table (duration: 00m 59s)
  • 03:03 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jun 28 03:03:34 UTC 2018 (duration 10m 25s)
  • 02:53 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 47s)
  • 02:21 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 07m 54s)
  • 00:33 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/resources: Watchlist perf patches for SWAT, part 2 (T197168, T198140, T198142) (duration: 00m 57s)
  • 00:32 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/includes: Watchlist perf patches for SWAT, part 1 (T197168, T198140, T198142) (duration: 01m 13s)
  • 00:13 twentyafterfour: rolled back and restored service to previous state
  • 00:12 twentyafterfour: phabricator update failed. unable to apply database migrations: mysql access denied
  • 00:05 twentyafterfour: taking apache offline momentarily on phab1001

2018-06-27

  • 23:57 twentyafterfour: phabricator deployment is coming up in just a couple of minutes. There will be downtime while I run database migrations.
  • 23:10 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turning on page creation log for most wikis T196400 (duration: 00m 58s)
  • 21:50 elukey: piwik maintenance on bohrium completed
  • 21:42 XioNoX: setting BFD of the Zayo eqiad-codfw link to standard of 300
  • 20:02 dduvall@deploy1001: Synchronized php: (no justification provided) (duration: 00m 57s)
  • 20:02 SMalyshev: applied fix for T197447 to eqiad wdqs cluster, which involved restart of the services
  • 19:54 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Group1 rolled back to 1.32.0-wmf.8
  • 19:52 marxarelli: Rolling back group1 due to rise in error rate (T198350)
  • 19:22 marxarelli: errors seem due to "INSERT INTO `revision_comment_temp`" statements and lock wait timeout
  • 19:18 marxarelli: seeing rising "Wikimedia\Rdbms\DBQueryError from line 1443 of /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema update..." errors
  • 19:16 dduvall@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.10 (duration: 00m 58s)
  • 19:15 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.10
  • 17:57 XioNoX: updating NTP servers on network devices
  • 17:53 mobrovac@deploy1001: Finished deploy [proton/deploy@cd6ed94]: Update proton to 491e966 - T186748 T197856 (duration: 00m 35s)
  • 17:52 mobrovac@deploy1001: Started deploy [proton/deploy@cd6ed94]: Update proton to 491e966 - T186748 T197856
  • 16:39 marostegui: Deploy schema change on dbstore1002:s8 T191316 T192926 T89737 T195193
  • 15:38 thcipriani@deploy1001: Synchronized README: Scap 3.8.3-1 noop test sync-file (duration: 00m 56s)
  • 15:36 marostegui: Stop replication on db2094:3318 to update triggers on archive table
  • 15:30 godog: upload scap 3.8.3 - T198277
  • 14:51 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1067 (duration: 00m 56s)
  • 13:57 dcausse: EU swat done
  • 13:42 dcausse@deploy1001: Finished scap: wmf-config Add cirrussearch settings for wikibase (1/3) (duration: 05m 41s)
  • 13:40 moritzm: uploaded debmonitor 0.1.5 to apt.wikimedia.org
  • 13:36 dcausse@deploy1001: Started scap: wmf-config Add cirrussearch settings for wikibase (1/3)
  • 13:24 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Change FileImporter config data location (T198050) (duration: 00m 57s)
  • 13:07 elukey: piwik upgraded to 3.2.1 on bohrium + started the db migration procedure (will last 2/3h probably)
  • 12:49 vgutierrez: Upgrade librdkafka1 and restart varnishkafka-webrequest in cache::upload nodes - T182993
  • 11:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 after alter table (duration: 00m 57s)
  • 11:26 marostegui: Deploy schema change on s8 codfw master (db2045) with replication, this will generate lag on s8 codfw T191316 T192926 T89737 T195193
  • 10:38 gehel: removing maps-test2003 from cluster for reimage - T198290
  • 10:36 volans@deploy1001: Finished deploy [debmonitor/deploy@9536ebf]: CSP header hotfix (duration: 00m 22s)
  • 10:36 volans@deploy1001: Started deploy [debmonitor/deploy@9536ebf]: CSP header hotfix
  • 10:24 marostegui: Deploy schema change on db1076 T191316 T192926 T89737 T195193
  • 10:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 for alter table (duration: 00m 56s)
  • 10:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 57s)
  • 10:11 jynus: stopping db1067 and reimage it
  • 09:38 volans@deploy1001: Finished deploy [debmonitor/deploy@052a9ea]: Release v0.1.5 (duration: 00m 24s)
  • 09:38 volans@deploy1001: Started deploy [debmonitor/deploy@052a9ea]: Release v0.1.5
  • 09:14 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1067 (duration: 00m 56s)
  • 08:35 marostegui: Deploy schema change on db1074 with replication, this will generate lag on s2 on labsdb T191316 T192926 T89737 T195193
  • 08:33 marostegui: Stop replication on db1074 to remove triggers from db1125 - T192926
  • 08:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 57s)
  • 08:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 after alter table (duration: 00m 57s)
  • 07:58 vgutierrez: Reinstall acamar & achernar as spare systems
  • 07:50 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns4001.wikimedia.org
  • 07:38 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=dns4001.wikimedia.org
  • 07:37 vgutierrez: Depool dns4001 for server restart - T198215
  • 05:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 for alter table (duration: 01m 06s)
  • 05:08 marostegui: Deploy schema change on db1090:3312 T191316 T192926 T89737 T195193
  • 04:47 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@2207b66]: Update mobileapps to d7221ba (duration: 05m 42s)
  • 04:42 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@2207b66]: Update mobileapps to d7221ba
  • 03:04 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jun 27 03:04:07 UTC 2018 (duration 10m 32s)
  • 02:53 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 15m 53s)
  • 02:20 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 33s)

2018-06-26

  • 21:26 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Group0 to 1.32.0-wmf.10
  • 21:18 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache (duration: 22m 50s)
  • 20:56 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
  • 20:47 marxarelli: removing orphaned /srv/mediawiki/php-1.31.0-wmf.28 dir on deploy1001
  • 20:38 volans: regenerated debian installer for jessie 8.11 point release (released 3 days ago)
  • 20:34 marxarelli: removed /srv/mediawiki/php-1.32.0-wmf.4 on mwdebug2002 to free disk space
  • 20:22 marxarelli: manually cleaning up on mwdebug2002 due to full disk and scap failures
  • 20:21 dduvall@deploy1001: Pruned MediaWiki: 1.32.0-wmf.4 [keeping static files] (duration: 03m 21s)
  • 19:43 marxarelli: cdb-rebuild failed on mwdebug2002 due to lack of disk space
  • 19:41 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache (duration: 39m 11s)
  • 19:02 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
  • 18:39 marxarelli: setting $wmgUseMwEmbedSupport = false in php-1.32.0-wmf.10/LocalSettings.php to extension registry exception (see T197918 and T191056)
  • 18:14 marxarelli: scap sync for testwiki is currently failing on l10update due to MwEmbedSupport's deprecation but no change yet to mediawiki/extensions to remove the extension. blocking train on T197918
  • 18:01 dduvall@deploy1001: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2905568126" --threads=30 --lang en --quiet' returned non-zero exit status 255 (duration: 02m 23s)
  • 17:59 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
  • 17:16 marxarelli: starting branch cut for 1.32-wmf.10
  • 16:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 after alter table (duration: 00m 57s)
  • 14:43 elukey: rm syslog.1.gz puppet.log.1.gz on tegment to fix cronspam
  • 14:19 ottomata: Re-enable job and change-prop Kafka topic mirroring from main-eqiad -> main-codfw
  • 13:37 hashar: European swat completed
  • 13:36 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable zh-my variant on zhwiki - T193983 (duration: 00m 57s)
  • 13:31 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use uploaded HD logos for ruwikiquote - T197508 (duration: 00m 57s)
  • 13:29 hashar: purgeList.php for the various ruwikiquote logos
  • 13:27 hashar@deploy1001: Synchronized static/images/project-logos: Change logo resources for ruwikiquote - T197508 (duration: 00m 56s)
  • 13:25 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable TemplateStyles on ruwikiquote - T197526 (duration: 00m 57s)
  • 13:22 dcausse: elastic@eqiad: forcemerge (only_expunge_deletes=true) on wikidatawiki_content
  • 13:21 dcausse: elastic@eqiad deleting stale index viwiki_general_1525301649
  • 13:19 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Create a few of namespace aliases for ruwiktionary - T197565 (duration: 00m 57s)
  • 13:06 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add some namespace aliases to ruwikiquote - T197058 (duration: 00m 56s)
  • 12:54 chasemp: labcontrol1003 aptitude install ldap-utils python-ldappool
  • 12:15 marostegui: Deploy schema change on db1105:3312 T191316 T192926 T89737 T195193
  • 12:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 for alter table (duration: 00m 57s)
  • 12:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 after alter table (duration: 00m 57s)
  • 11:09 godog: roll-upgrade thumbor in codfw/eqiad
  • 11:08 godog: upload thumbor 2.0-2 to apt.wikimedia.org
  • 10:40 godog: upgrade thumbor to 2.0-1 on thumbor1001
  • 10:10 godog: test upgrade thumbor 2.0 on thumbor2001
  • 10:10 akosiaris: proceed with apertium-apy upgrade on all scb hosts. T194342
  • 10:01 akosiaris: downgrade erroneously upgraded apertium-fra-cat on scb1001, scb1002 from 1.3.0~r84327-1+wmf1 to 1.2.0~r78602-1+wmf2 T194342
  • 09:40 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw
  • 09:39 akosiaris: re-enable icinga-wm that I 've previously silenced
  • 09:37 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=eqiad
  • 09:36 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=appserver
  • 09:33 mobrovac@deploy1001: Finished scap: Switch the last remaining jobs to EventBus, full scap sync with HHVM restarts, take #3 - T190327 (duration: 05m 07s)
  • 09:31 akosiaris: upgrade apertium-apy, apertium-fra-cat, python3-streamparser, enable puppet, run puppet on scb1002 as a last test before full cluster upgrade. T194342
  • 09:31 vgutierrez: updating librdkafka1 and restarting varnishkafka-webrequest on cache::misc nodes - T182993
  • 09:29 dcausse: elastic@eqiad: forcemerge (only_expunge_deletes=true) on commonswiki_file
  • 09:28 mobrovac@deploy1001: Started scap: Switch the last remaining jobs to EventBus, full scap sync with HHVM restarts, take #3 - T190327
  • 09:25 akosiaris: manually install python3-streamparser on scb1001 for testing. T194342
  • 09:19 mobrovac@deploy1001: Finished deploy [changeprop/deploy@5d8eaf1]: Bug fix: Add restore to WD description checks - T197626 (duration: 01m 14s)
  • 09:18 mobrovac@deploy1001: Started deploy [changeprop/deploy@5d8eaf1]: Bug fix: Add restore to WD description checks - T197626
  • 09:11 mobrovac@deploy1001: Finished scap: Switch the last remaining jobs to EventBus, full scap sync, take #2 - T190327 (duration: 08m 34s)
  • 09:09 vgutierrez: upload librdkafka_0.11.3-1~bpo8+1+wikimedia2 to apt.w.o jessie-wikimedia - T182993
  • 09:08 akosiaris: upgrade scb1001 apertium-apy, apertium-fra-cat, enable puppet and run puppet
  • 09:06 akosiaris: T194342. Disable puppet on scb hosts for apertium-apy upgrade
  • 09:04 dcausse: unbanning elastic1042
  • 09:03 mobrovac@deploy1001: Started scap: Switch the last remaining jobs to EventBus, full scap sync, take #2 - T190327
  • 08:52 mobrovac@deploy1001: scap failed: RuntimeError scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) (duration: 04m 27s)
  • 08:52 mobrovac@deploy1001: scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 08:48 mobrovac@deploy1001: Started scap: Switch the last remaining jobs to EventBus, full scap sync for clean-up - T190327
  • 08:47 mobrovac@deploy1001: Synchronized wmf-config/CommonSettings.php: Switch the last remaining jobs to EventBus, only CommonSettings.php now - T190327 (duration: 00m 58s)
  • 08:47 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327 (duration: 00m 58s)
  • 08:46 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327
  • 07:58 vgutierrez: Replaced acamar and achernar with dns200[12] as main lvs name servers in codfw - T196493
  • 07:13 _joe_: restarting ircecho, was not reporting alerts
  • 07:05 marostegui: Deploy schema change on db1103:3312 T191316 T192926 T89737 T195193
  • 07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for alter table (duration: 00m 56s)
  • 06:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 after alter table (duration: 00m 57s)
  • 05:58 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db1054, it is going to be decommissioned T197063 (duration: 00m 55s)
  • 05:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db1054, it is going to be decommissioned T197063 (duration: 00m 57s)
  • 05:52 marostegui: Stop MySQL on db1054 as it is going to be decommissioned - T197063
  • 05:28 SMalyshev: testing fix for T197447 on wdqs1009
  • 05:15 marostegui: Deploy schema change on db1122 T191316 T192926 T89737 T195193
  • 05:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 for alter table (duration: 00m 59s)
  • 03:05 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 12m 59s)
  • 02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 13m 26s)

2018-06-25

  • 23:40 thcipriani: restarting jenkins for updates
  • 23:30 twentyafterfour: After monitoring logstash, everything appears stable. Evening SWAT is complete.
  • 23:23 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440878/ for SWAT: no-op (duration: 00m 56s)
  • 23:10 twentyafterfour@deploy1001: Synchronized wmf-config/Wikibase-production.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440420/ for SWAT (duration: 00m 57s)
  • 21:04 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@770cdb0]: Update mobileapps to 8c76d52 (duration: 05m 41s)
  • 20:58 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@770cdb0]: Update mobileapps to 8c76d52
  • 20:09 mdholloway: deploying mobileapps to canary (scb2001) failed, rolling back to investigate
  • 20:07 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@fb70d4f]: Update mobileapps to 362f2a0 (duration: 01m 34s)
  • 20:06 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@fb70d4f]: Update mobileapps to 362f2a0
  • 20:02 cscott: Completed deploy of Parsoid to version b068bb51 (T197949, T197702, T196799)
  • 19:53 cscott@deploy1001: Finished deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51 (duration: 06m 57s)
  • 19:46 cscott@deploy1001: Started deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51
  • 19:22 cscott@deploy1001: Started deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51
  • 18:33 thcipriani@deploy1001: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 56s)
  • 18:32 thcipriani@deploy1001: Synchronized portals/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 57s)
  • 18:25 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add import sources to ta.wiktionary T196445 (duration: 00m 58s)
  • 18:13 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Enable logging for Schema:CitationUsage T191086 (noop beta-only sync) (duration: 00m 57s)
  • 18:07 gehel@deploy1001: Finished deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (duration: 08m 19s)
  • 17:58 gehel@deploy1001: Started deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater
  • 17:56 gehel@deploy1001: Finished deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 24s)
  • 17:56 gehel@deploy1001: Started deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (wdqs1009 only)
  • 17:36 ottomata: Re-enable job and change-prop kafka topic mirroring from main-codfw -> main-eqiad
  • 16:45 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Beta only gerrit:441909 (duration: 00m 57s)
  • 16:19 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org
  • 16:13 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=acamar.wikimedia.org
  • 16:04 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns2002.wikimedia.org
  • 16:00 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns2001.wikimedia.org
  • 15:21 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: REVERT: Load WikibaseLexeme on testwiki (again again) T197454 (duration: 00m 57s)
  • 15:16 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Load WikibaseLexeme on testwiki (again again) T197454 (duration: 00m 55s)
  • 15:05 addshore: FileImporter slot done, moving onto Lexeme slot
  • 15:03 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable FileExpoter on ar-, de- and fa-wiki T196969 T197066 (duration: 00m 57s)
  • 14:58 marostegui: Deploy schema change on dbstore1002:s2 T191316 T192926 T89737 T195193
  • 14:57 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable FileImporter on Wikimedia Commons T196969 (duration: 00m 56s)
  • 14:40 addshore@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/FileImporter: Ignore namespace string when checking templates and categories (duration: 00m 56s)
  • 14:37 addshore@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/FileImporter: Ignore namespace string when checking templates and categories (duration: 00m 58s)
  • 14:32 addshore@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/FileImporter: Add a temp config to allow interwiki linking T196976 (duration: 00m 57s)
  • 14:30 addshore@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/FileImporter: Add a temp config to allow interwiki linking T196976 (duration: 00m 58s)
  • 14:23 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: Enable license filters for the FileImporter in production T194502 (duration: 00m 55s)
  • 14:19 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: Add ar, de and fa wikipedia to FileImporter interwiki config T196969 T196976 (duration: 00m 56s)
  • 14:17 hashar@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/Echo: Remove masterPos from the job specification - T192945 (duration: 00m 59s)
  • 14:14 elukey: merging jmxtrans and kafkatee's submodules to operations/puppet - part 2 (moving them back from environments/production)
  • 14:13 gehel: banning and depooling elastic1042 (high response times)
  • 14:01 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix ORES config, part II - T197633 (duration: 00m 57s)
  • 13:59 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Whitelist two Indian government websites - T197944 (duration: 00m 57s)
  • 13:57 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Autoconfirmed should require 10 edits&4 days on zhwikivoyage - T198006 (duration: 00m 56s)
  • 13:54 hashar: mwscript namespaceDupes.php --wiki=zhwikivoyage --fix | T198007
  • 13:53 elukey: merging jmxtrans and kafkatee's submodules to operations/puppet - part 1 (moving them under environments/production)
  • 13:51 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Autoconfirmed should require 10 edits&4 days on zhwikivoyage - T198006 (duration: 00m 56s)
  • 13:50 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add namespace aliases to zhwikivoyage - T198007 (duration: 00m 56s)
  • 13:37 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove BN alias for NS_USER on dewikivoyage - T196905 (duration: 00m 57s)
  • 13:34 hashar@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/Echo: Remove masterPos from the job specification - T192945 (duration: 00m 58s)
  • 13:32 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix ORES config - T197633 (duration: 00m 58s)
  • 13:27 hashar@deploy1001: Synchronized dblists/commonsuploads.dblist: Add wikimania2018wiki to commonsupload.dblist | T197714 (duration: 00m 56s)
  • 13:13 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable logging for Schema:CitationUsage at 1% | T191086 (duration: 00m 57s)
  • 13:05 marostegui: Deploy schema change on db2035 (codfw primary master for s2) with replication, this will generate lag on s2 codfw T191316 T192926 T89737 T195193
  • 13:05 godog: refresh cassandra certificates for 'restbase' cluster
  • 13:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 00m 57s)
  • 12:29 kartik@deploy1001: Finished deploy [cxserver/deploy@cc6dc61]: Update cxserver to ece5e7a (T191874, T196354, T195768, T195768) (duration: 03m 33s)
  • 12:26 kartik@deploy1001: Started deploy [cxserver/deploy@cc6dc61]: Update cxserver to ece5e7a (T191874, T196354, T195768, T195768)
  • 12:04 mobrovac: Restbase: Add the "headers" field to Cassandra schemae in production for T197789
  • 12:00 mobrovac@deploy1001: Finished deploy [restbase/deploy@f521e7e] (dev-cluster): Dev cluster: Include pagelanguage in title_revisions (duration: 04m 05s)
  • 12:00 gehel: repooling wdqs1005 it has catched up on updates - T198042
  • 11:56 mobrovac@deploy1001: Started deploy [restbase/deploy@f521e7e] (dev-cluster): Dev cluster: Include pagelanguage in title_revisions
  • 10:32 akosiaris: increase CPU count for proton machines from 2 to 10. T197862
  • 08:22 marostegui: Deploy schema change on db1106 with replication, this will generate lag on s1 labs T191316 T192926 T89737 T195193
  • 08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 00m 56s)
  • 08:16 marostegui: Stop replication on db1106 to change triggers for db1124 and db1095 - T192926
  • 08:11 gehel: rolling restart of wdqs to enable async logging - T198051
  • 08:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 58s)
  • 07:46 gehel: depooling wdqs1005 to allow it to catch up on updates - T198042
  • 07:33 moritzm: slow rollout of most of the remaining debmonitor client installations
  • 05:43 marostegui: Deploy schema change on db1080 T191316 T192926 T89737 T195193
  • 05:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 01m 17s)
  • 02:39 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 07m 37s)
  • 02:21 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 40s)

2018-06-24

  • 22:26 gehel: restarting wdqs1004 - T198042
  • 21:51 gehel: restarting wdqs1005 after taking a few thread dumps - T198042
  • 19:41 gehel: restart blazegraph on wdqs1004
  • 16:09 godog: restart wdqs-updater on wdqs1005
  • 15:40 godog: restart wdqs-blazegraph on wdqs1005
  • 15:40 jynus: restart graphite2001
  • 15:35 godog: systemctl restart wdqs-blazegraph on wdqs1004
  • 15:33 godog: restart-wdqs on wdqs1004

2018-06-22

  • 19:54 jynus: applying transaction manually on dbstore1002 due to weird bug with savepoint on tokudb+image table

2018-06-21

  • 21:38 ebernhardson: banned elastic1036 from search cluster, waited for all load to shift away, and unbanned
  • 16:38 ejegg: updated CiviCRM from 349d43eb8b to e2c8ddd70e
  • 13:28 vgutierrez: Bump AES128-SHA pageview replacement to 4%
  • 12:49 chasemp: remove labvirt1019 canary to start debug of network
  • 11:08 hashar: Refreshed operations-puppet-tests-docker jenkins job to a new Docker container build that includes isc-dhcp-server | https://gerrit.wikimedia.org/r/c/integration/config/+/441367
  • 02:51 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 07m 40s)
  • 02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 14m 12s)

2018-06-20

  • 21:18 ejegg: updated 'domain' settings in CiviCRM so name is now 'Wikimedia Foundation' and description is 'donate.wikimedia.org'
  • 20:08 bearND: rolled back "Update mobileapps to 9d856ec" (was just on canary)
  • 20:07 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@3420e67]: Update mobileapps to 9d856ec (duration: 03m 51s)
  • 20:03 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@3420e67]: Update mobileapps to 9d856ec
  • 19:54 ottomata: removed Kafka MirrorMaker from kafka10(12|13|14)
  • 16:01 sbisson@deploy1001: Finished deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN (duration: 03m 08s)
  • 15:58 sbisson@deploy1001: Started deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN
  • 15:57 sbisson@deploy1001: Finished deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN (duration: 00m 27s)
  • 15:56 sbisson@deploy1001: Started deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN
  • 15:52 sbisson@deploy1001: Finished deploy [kartotherian/deploy@9af9191]: (no justification provided) (duration: 03m 36s)
  • 15:49 sbisson@deploy1001: Started deploy [kartotherian/deploy@9af9191]: (no justification provided)
  • 14:59 dcausse: disk space issue on elastic1020 is due to shard rebalancing (currently receiving 2 enwiki_general shards but removing one wikidatawiki_content)
  • 14:07 imarlier@deploy1001: Finished deploy [performance/navtiming@742edb0]: (no justification provided) (duration: 00m 04s)
  • 14:07 imarlier@deploy1001: Started deploy [performance/navtiming@742edb0]: (no justification provided)
  • 14:03 imarlier@deploy1001: Finished deploy [performance/navtiming@8914e26]: (no justification provided) (duration: 00m 05s)
  • 14:03 imarlier@deploy1001: Started deploy [performance/navtiming@8914e26]: (no justification provided)
  • 13:43 imarlier@deploy1001: Finished deploy [performance/navtiming@995cb0f]: (no justification provided) (duration: 00m 05s)
  • 13:43 imarlier@deploy1001: Started deploy [performance/navtiming@995cb0f]: (no justification provided)
  • 13:38 anomie: Re-running populateExternallinksIndex60.php on plwiki and ptwiki for phab:T59176 (initial run collided with the s2 master switch).
  • 08:57 _joe_: removing user.log as well on tegmen
  • 08:54 _joe_: running logrotate on tegmen
  • 08:54 _joe_: killall nsca on tegmen
  • 08:51 _joe_: restarting nsca daemon on tegmen (gone wild, hundreds of subprocesses
  • 08:47 _joe_: removing user.log.1 and messages.log.1 on tegmen to save some space
  • 07:40 _joe_: powercycled mw1272, down since yesterday
  • 07:29 hashar: deploy1001: rebased php-1.32.0-wmf.8/extensions/Translate to catch up with a non production merged change ( https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/441127 ).
  • 02:20 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 17s)

2018-06-19

  • 16:10 mobrovac@deploy1001: Finished deploy [proton/deploy@43af7d9]: (no justification provided) (duration: 00m 27s)
  • 16:09 mobrovac@deploy1001: Started deploy [proton/deploy@43af7d9]: (no justification provided)
  • 14:13 herron: re-enabling puppet agents
  • 14:11 herron: increased client_max_body_size on puppetdb nginx frontends from 30m to 60m https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441044/
  • 14:06 herron: temporarily disabling puppet agents for deploy of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441044/
  • 12:41 _joe_: hard reboot of ms-be1019 - unable to ssh, console showing i/o errors only
  • 04:56 ebernhardson: unban elastic1035
  • 04:45 ebernhardson: ban elastic1035 from cluster to allow it to recover
  • 02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 06m 53s)
  • 02:19 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 07m 39s)

2018-06-18

  • 19:49 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@bcb2904]: GUI update (duration: 18m 52s)
  • 19:30 smalyshev@deploy1001: Started deploy [wdqs/wdqs@bcb2904]: GUI update
  • 19:30 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: GUI update (duration: 00m 56s)
  • 19:29 smalyshev@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: GUI update
  • 18:41 Trey314159: reindexing Serbian wikis on elastic@eqiad (T196404)
  • 16:39 urandom: DROP unused Cassandra keyspaces - T197080
  • 10:52 _joe_: removing wtp1043 from all pybal configuration until the disk is replaced T196886
  • 10:04 _joe_: initialize namespace "ci" on the kubernetes staging cluster T196654
  • 09:56 joal@deploy1001: Finished deploy [analytics/refinery@e9dbe79]: Regular weekly deploy (duration: 06m 54s)
  • 09:49 joal@deploy1001: Started deploy [analytics/refinery@e9dbe79]: Regular weekly deploy
  • 03:04 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 14m 30s)
  • 02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 13m 12s)
  • 02:27 Trey314159: reindexing Serbian wikis on elastic@codfw (T196404)

2018-06-17

  • 22:24 no_justification: gerrit: started back up, nvm
  • 22:23 no_justification: gerrit: stopping services for a few minutes
  • 13:05 Trey314159: reindexing Serbo-Croatian wikis on elastic@eqiad (T196658)
  • 03:46 Trey314159: reindexing Serbo-Croatian wikis on elastic@codfw (T196658)

2018-06-16

  • 20:42 mdholloway: disabled puppet on maps-test2004 for testing new map styles setup
  • 19:24 Trey314159: reindexing Croatian wikis on elastic@eqiad (T196658)
  • 13:19 Trey314159: reindexing Croatian wikis on elastic@codfw (T196658)
  • 00:15 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ica4cb6644 (duration: 00m 59s)

2018-06-15

  • 18:48 Trey314159: reindexing Bosnian wikis on elastic@eqiad (T196658)
  • 18:10 Trey314159: reindexing Bosnian wikis on elastic@codfw (T196658)
  • 17:15 krinkle@deploy1001: Synchronized wmf-config/mc.php: I619a2ff5db611 (duration: 00m 58s)
  • 15:49 mutante: install2002 - disabling puppet temp, live hackking DHCP config for debugging backup2001 install issue
  • 15:37 mutante: switching noc.wikimedia.org site from terbium to mwamiant1001 backend, running puppet on all cache::misc cp servers (T192092)
  • 15:19 ottomata: bouncing kafka broker on kafka-jumbo1001 to test https://gerrit.wikimedia.org/r/#/c/440520/
  • 15:19 gehel: rolling restart of elasticsearch eqiad for plugin upgrade completed - T194245
  • 14:49 elukey: restart varnishkafka-eventlogging on cp4028, errors logged
  • 14:43 elukey: restart varnishkafka-eventlogging on cp5012 as attempt to clear out the errors (not needed but logging it anyway)
  • 14:12 papaul: OS install on backup2001
  • 13:41 mepps: updated process control to 3b9f51e0fd
  • 12:56 legoktm@deploy1001: Synchronized wmf-config/mc.php: Make sure that mcrouter BagOStuff goes through ObjectCache::newFromParams() - T197450 (duration: 00m 57s)
  • 12:38 jynus: killing refresh counts on commonswiki
  • 12:30 jynus: reenabling db2040 consistency after slaves caught up
  • 12:13 papaul: OS install on bast2002
  • 10:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 00m 58s)
  • 09:48 godog: delete cpjobqueue metrics older than 10d - T196067
  • 09:45 jynus: reenabling db2048 consistency after slaves caught up
  • 09:43 jynus: reducing temp. db2040 consistency to speed up slave lag catch up
  • 09:20 godog: fully remove ms-be1036 from swift due to hw failure - T196873
  • 09:03 bawolff: deploy patch T197279
  • 08:05 gehel: rolling restart of elasticsearch eqiad for plugin upgrade - T194245
  • 05:50 moritzm: slow rollout of debmonitor
  • 05:34 moritzm: installing gnupg security updates on trusty (Debian already fixed)
  • 05:13 marostegui: Deploy schema change on db1119 T191316 T192926 T89737 T195193
  • 05:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 00m 57s)
  • 05:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 01m 07s)

2018-06-14

  • 23:40 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.3 (duration: 04m 38s)
  • 23:31 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.2 (duration: 04m 50s)
  • 23:24 jforrester@deploy1001: Synchronized php-1.32.0-wmf.8/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.less: T195903 SWAT for Roan (duration: 00m 59s)
  • 23:12 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T196400 for SWAT (duration: 01m 00s)
  • 21:56 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/PageTriage/includes/Hooks.php: Fix event presentation class names T197262 (duration: 00m 57s)
  • 21:47 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.999 T196585
  • 21:44 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/PageTriage/includes/Hooks.php: Fix event presentation class names T197262 (duration: 00m 52s)
  • 21:27 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.1 (duration: 06m 54s)
  • 21:18 thcipriani@deploy1001: Finished scap: testwiki to 1.32.0-wmf.999 (multi-content-revisions T196585) and rebuild l10n cache (duration: 65m 50s)
  • 21:12 bblack: re-enable and run puppet on cache_upload - T192555
  • 20:45 bblack: re-enable and run puppet on rest of cache_text (eqiad, eqsin, esams) - T192555
  • 20:21 bblack: re-enable and run puppet on text@ulsfo - T192555
  • 20:12 thcipriani@deploy1001: Started scap: testwiki to 1.32.0-wmf.999 (multi-content-revisions T196585) and rebuild l10n cache
  • 20:11 bblack: re-enable and run puppet on text@codfw - T192555
  • 19:21 vgutierrez: Reenable puppet in cache:misc nodes - T192555
  • 19:13 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.8
  • 18:17 niharika29@deploy1001: Synchronized wmf-config/CommonSettings.php: Bump ExtensionDistributor default to REL1_31 (duration: 00m 57s)
  • 18:14 niharika29@deploy1001: Synchronized langlist-labs: beta: declare beta sr.wikipedia and beta crh.wikipedia to langlist-labs (duration: 00m 58s)
  • 18:10 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on a few more wikis - T195263 (duration: 01m 00s)
  • 17:34 gehel: rolling restart of elasticsearch codfw completed - T194245
  • 17:10 ottomata: bouncing kafka broker on kafka2003 to make sure ACLs are okk
  • 17:08 ottomata: applying ACLs to Kafka main-codfw and main-eqiad - T196081
  • 16:41 vgutierrez: disable puppet on cache nodes before merging gerrit/440114 - T192555
  • 15:39 ottomata: switching EventStreams service to be backed by main-eqiad - T185225
  • 14:48 ejegg: disabled donations and refund queue consumers
  • 14:45 ejegg: updated CiviCRM from 69091d8 to f54f981
  • 14:40 mutante: moving wikidata query dispatcher from terbium to mwmaint1001 - scheduled downtime - check turned into a WARN - disabling puppet on mwmaint1001, removing crons on terbium, waiting a couple minutes for them to finish, re-enabling puppet on mwmaint1001 (T192092)
  • 14:32 moritzm: (slow) initial rollout of debmonitor-client
  • 13:47 zeljkof: EU SWAT finished
  • 13:46 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/Wikibase: SWAT: Revert "Statement transclusion: when entity of unknown type in statement, display ID as string" (T195615) (duration: 01m 21s)
  • 13:35 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set a few of namespace aliases on ruwikisource (T196719) (duration: 00m 59s)
  • 13:11 volans@deploy1001: Finished deploy [debmonitor/deploy@476fd8b]: Release v0.1.4 (duration: 00m 19s)
  • 13:10 volans@deploy1001: Started deploy [debmonitor/deploy@476fd8b]: Release v0.1.4
  • 12:54 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 12:29 ema: cp3037: run update-ocsp-all
  • 12:23 gehel: rolling restart of elasticsearch codfw for plugin upgrade - T194245
  • 11:38 marostegui: Deploy schema change on db1067 T191316 T192926 T89737 T195193
  • 11:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 00m 57s)
  • 11:16 elukey: upgrade cassandra on aqs* to 2.2.6-wmf5
  • 11:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 01m 01s)
  • 10:04 addshore: @terbium: for i in {2001..3000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki // T197222
  • 09:59 addshore: @terbium: for i in {1001..2000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki // T197222
  • 09:57 addshore: @terbium: for i in {1..1000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki // T197222
  • 09:15 elukey: add debmonitor term to analytics-in4 on cr1/cr2 eqiad
  • 08:59 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@2680ba8]: Decrease the checkerJob delays recheck to 10 minutes (duration: 01m 18s)
  • 08:58 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@2680ba8]: Decrease the checkerJob delays recheck to 10 minutes
  • 08:31 elukey: restart hadoop hdfs master nodes to pick up the new journal node settings
  • 08:29 gehel: restart of elasticsearch / relforge for plugin updates
  • 08:08 mutante: switch backend for dbtree.wikimedia.org away from terbium to mwmaint1001 (T192092)
  • 08:07 elukey: roll restart of hadoop journal nodes to pick up the new configuration (two more journal nodes added)
  • 08:03 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETAONLY: senses for beta wikidatas (duration: 00m 58s)
  • 07:59 moritzm: resuming rolling restart of cassandra on restbase2* to pick up OpenJDK security update
  • 07:41 marostegui: Deploy schema change on db1114 T191316 T192926 T89737 T195193
  • 07:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 00m 58s)
  • 07:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 after alter table (duration: 00m 58s)
  • 05:47 mutante: LDAP - added user mepps to wmf group (T192472)
  • 05:15 marostegui: Deploy schema change on db1083 T191316 T192926 T89737 T195193
  • 05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 for alter table (duration: 00m 58s)
  • 05:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 01m 01s)
  • 05:03 marostegui: Deploy schema change on s4 primary master (db1068) T191316 T192926 T195193
  • 03:06 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jun 14 03:06:19 UTC 2018 (duration 10m 30s)
  • 02:55 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 14m 42s)
  • 02:22 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 09m 41s)

2018-06-13

  • 23:15 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/432093/ (duration: 00m 58s)
  • 23:07 maxsem@deploy1001: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440178/ (duration: 01m 00s)
  • 21:20 awight@deploy1001: Finished deploy [ores/deploy@36037b6]: New badwords for ORES in English: T196468 (duration: 72m 56s)
  • 20:07 awight@deploy1001: Started deploy [ores/deploy@36037b6]: New badwords for ORES in English: T196468
  • 19:26 dduvall@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.8 (duration: 00m 57s)
  • 19:25 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.8
  • 18:28 twentyafterfour: finished swat (28m overtime! :P)
  • 18:22 twentyafterfour: ran namespaceDupes for urwiktionary
  • 18:20 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: sync 5b47244 14ca2ba 63bc100m and 2569a77 refs T196488, T196744, T196727, T196763 (duration: 00m 57s)
  • 17:55 twentyafterfour@deploy1001: Synchronized static/images/project-logos/: sync bnwikivoyage logos refs T196803 (duration: 00m 58s)
  • 17:53 twentyafterfour@deploy1001: Synchronized static/images/project-logos/bnwikivoyage-1.5x.png: static/images/project-logos/bnwikivoyage-2x.png static/images/project-logos/bnwikivoyage.png sync bnwikivoyage logos refs T196803 (duration: 00m 58s)
  • 17:49 twentyafterfour@deploy1001: Synchronized wmf-config: Sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/436430/ for SWAT refs T184244 (duration: 01m 00s)
  • 17:08 mutante: terbium - closing unusued screen sessions for all Amir users (2)
  • 16:53 ppchelko@deploy1001: Finished deploy [restbase/deploy@f521e7e]: Add page_language to title_revision table T197082 (duration: 15m 44s)
  • 16:38 ppchelko@deploy1001: Started deploy [restbase/deploy@f521e7e]: Add page_language to title_revision table T197082
  • 16:31 XioNoX: moving mr1-eqiad interfaces to new router
  • 16:14 mutante: rsyncing /home dirs from terbium to mwmaint1001, they will appear later in a subdir "home-terbium" like it was done for tin->deploy1001 (T192092)
  • 16:13 urandom: ALTERing Cassandra schema - T197082
  • 16:11 moritzm: installing plexus-archiver security updates
  • 16:07 moritzm: installing imagemagick security updates on trusty (Debian already fixed)
  • 15:55 elukey: rolling restart of aqs on aqs100[4-9] to pick up the new config changes
  • 15:25 jynus: stopping db1053 and db1059 in preparation for decomm T194634 T196606
  • 15:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1076 (duration: 00m 58s)
  • 14:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1076 (duration: 00m 57s)
  • 14:15 anomie@deploy1001: Synchronized php-1.32.0-wmf.8/includes/Category.php: Backporting fix for T195397 (gerrit:440053) (duration: 01m 00s)
  • 14:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight (duration: 00m 58s)
  • 14:01 zeljkof: EU SWAT finished
  • 14:00 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Implementing Patroller User Rights for azwiki (T196488) (duration: 00m 57s)
  • 13:54 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: English aliases for extra namespaces on urwiktionary (T196614) (duration: 00m 58s)
  • 13:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix wrong language in ur.wiktionary namespace (T196614) (duration: 00m 58s)
  • 13:28 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@160206f]: (no justification provided) (duration: 04m 11s)
  • 13:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make ProofreadPage operate on correct namespaces in pmswikisource (T197033) (duration: 00m 57s)
  • 13:24 elukey@deploy1001: Started deploy [analytics/aqs/deploy@160206f]: (no justification provided)
  • 13:12 zfilipin@deploy1001: Synchronized wmf-config/: SWAT: Enable FileImporter monolog channel in production (T195370) (duration: 01m 00s)
  • 13:02 moritzm: rolling restart of cassandra in codfw to pick up OpenJDK security update
  • 13:00 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@84fab89]: Update AQS for T190213 (duration: 02m 38s)
  • 12:57 elukey@deploy1001: Started deploy [analytics/aqs/deploy@84fab89]: Update AQS for T190213
  • 12:46 elukey: restart mirror maker on kafka1012->1014 to pick up new openjdk-7 upgrades
  • 12:28 elukey: rolling restart of kafka on kafka1012->23 for openjdk-7 upgrades
  • 12:16 arturo: T196633 extend downtime for labcontrol1003
  • 11:21 moritzm: installing perl security updates
  • 10:59 marostegui: Deploy schema change on db1105:3311 T191316 T192926 T89737 T195193
  • 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 00m 58s)
  • 10:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 00m 59s)
  • 10:10 akosiaris: upload apertium-apy_0.11.3-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 09:00 marostegui: Restart mysql on dbstore1002 for maintenance
  • 08:46 ema: depool cp1053 T165252
  • 08:04 marostegui: Stop MySQL and reboot db1076 - T197063
  • 08:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 for binlog change - T197063 (duration: 00m 57s)
  • 07:59 oblivian@deploy1001: Synchronized wmf-config/ProductionServices.php: Remove unused redis shards from the jobqueue T197003 (duration: 00m 58s)
  • 07:57 Pchelolo: restart cpjobqueue on scb1001 cause, it's lost all it's workers
  • 07:55 marostegui: Deploy schema change on db1089 T191316 T192926 T89737 T195193
  • 07:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 00m 58s)
  • 07:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore s2 default read-only message (duration: 00m 57s)
  • 07:41 marostegui: Stop MySQL on db1054 for socket update and binlog change
  • 07:33 godog: start removing ms-be1036 from swift rings - T196873
  • 06:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 00m 59s)
  • 06:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove read only from s2 - T194870 (duration: 00m 33s)
  • 06:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove read only from s2 - T194870 (duration: 00m 34s)
  • 06:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Set s2 on read-only for primary db master maintnance - T194870 (duration: 01m 08s)
  • 06:00 marostegui: Starting s2 failover from db1054 to db1066 - T194870
  • 05:11 marostegui: Starting topology changes in order to get ready for s2 failover - T194870
  • 05:09 marostegui: Disable gtid on db1066
  • 05:03 marostegui: Deploy schema change on dbstore1001:s1 T191316 T192926 T89737 T195193
  • 03:14 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jun 13 03:14:53 UTC 2018 (duration 10m 19s)
  • 03:04 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 15m 45s)
  • 02:31 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 12m 21s)
  • 01:12 aaron@deploy1001: Synchronized wmf-config/mc.php: Add "memcached-mcrouter" to $wgObjectCaches as default for testwiki (duration: 00m 58s)
  • 01:01 aaron@deploy1001: Synchronized php-1.32.0-wmf.7/tests/phpunit/includes/db/LBFactoryTest.php: (no justification provided) (duration: 00m 57s)
  • 01:00 aaron@deploy1001: Synchronized php-1.32.0-wmf.7/includes/libs/rdbms/lbfactory/LBFactory.php: f83bad6 (duration: 00m 59s)
  • 00:55 aaron@deploy1001: Synchronized php-1.32.0-wmf.8/tests/phpunit/includes/db/LBFactoryTest.php: (no justification provided) (duration: 00m 58s)
  • 00:53 aaron@deploy1001: Synchronized php-1.32.0-wmf.8/includes/libs/rdbms/lbfactory/LBFactory.php: c2df9668d13 (duration: 00m 58s)
  • 00:46 bawolff@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/CentralAuth/AntiSpoof/CentralAuthAntiSpoofHooks.php: Deploy I5f25c5 (prevent users from registering previously renamed users) (duration: 00m 57s)
  • 00:44 bawolff@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/CentralAuth/extension.json: Deploy I5f25c5 (prevent users from registering previously renamed users) (duration: 00m 58s)
  • 00:40 bawolff@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/CentralAuth/AntiSpoof/CentralAuthAntiSpoofHooks.php: Deploy I5f25c5 (prevent users from registering previously renamed users) (duration: 00m 57s)
  • 00:39 bawolff@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/CentralAuth/extension.json: Deploy I5f25c5 (prevent users from registering previously renamed users) (duration: 00m 57s)
  • 00:37 bawolff@deploy1001: Synchronized wmf-config/CommonSettings.php: Deploy I5f25c5 (prevent users from registering previously renamed users) (duration: 00m 59s)

2018-06-12

  • 23:47 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add sites to the wgCopyUploadsDomains whitelist of Wikimedia Commons T195270, T195928 (duration: 00m 59s)
  • 23:05 tzatziki: resetting passwords for compromised accounts (T197046)
  • 23:00 bblack: cp3043 - done, reimaged, in live service for cache_upload
  • 22:59 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp3043.esams.wmnet
  • 22:37 tzatziki: (from yesterday) resetting passwords for compromised accounts (T197046)
  • 22:24 twentyafterfour: phabricator: I scheduled a 24 hour downtime in icinga for the phd service, to give me time to work on this issue. See T196840
  • 22:23 twentyafterfour: phabricator: taking phd offline to relieve the load on the m3 database cluster
  • 21:46 bblack: cp3046 - restart varnish backend for mbox lag
  • 21:40 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp3043.esams.wmnet
  • 21:39 bblack: cp3043 - starting process to move to reimage into cache_upload
  • 19:16 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.8
  • 18:57 herron: restarted icinga service on einsteinium
  • 18:42 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.8 and rebuild l10n cache (duration: 39m 39s)
  • 18:03 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.8 and rebuild l10n cache
  • 17:38 ariel@deploy1001: Finished deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install (duration: 00m 04s)
  • 17:37 ariel@deploy1001: Started deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install
  • 17:37 ariel@deploy1001: Finished deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install (duration: 00m 07s)
  • 17:37 ariel@deploy1001: Started deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install
  • 16:54 marxarelli: starting branch cut for 1.32.0-wmf.8
  • 16:11 volans@deploy1001: Finished deploy [debmonitor/deploy@0eca14a]: Release v0.1.3 (duration: 00m 22s)
  • 16:11 volans@deploy1001: Started deploy [debmonitor/deploy@0eca14a]: Release v0.1.3
  • 15:40 bblack: cp3034 - nevermind, doing different approach later in the day, still pooled in text for now!
  • 15:29 bblack: cp3043 switching from text to upload shortly, downtimed in icinga for 2h - https://gerrit.wikimedia.org/r/c/operations/puppet/+/439936
  • 15:07 ema: cp3039: restart varnish-backend
  • 14:38 addshore: file exporter importer slot done
  • 14:38 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: FileImporter/Exporter Enable FileExporter/Importer on group0 wikis T195370 (duration: 00m 51s)
  • 14:20 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: FileImporter/Exporter Allow setting of export target for FileExporter T195370 (duration: 00m 50s)
  • 14:09 addshore@deploy1001: Finished scap: FileExporter backport - Pre deployment backport (extension not yet deployed) (duration: 30m 37s)
  • 13:38 addshore@deploy1001: Started scap: FileExporter backport - Pre deployment backport (extension not yet deployed)
  • 13:16 moritzm: installing openjdk-8 security updates on restbase-dev along with cassandra restarts
  • 12:38 ema: cp3035: restart varnish-be, mbox lag
  • 12:34 _joe_: repooling mw1230 after reimaging T196881
  • 12:14 marostegui: Deploy schema change on db1099:3311 T191316 T192926 T89737 T195193
  • 12:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 00m 52s)
  • 12:11 marostegui: Deploy schema change on dbstore1002:s1 T191316 T192926 T89737 T195193
  • 12:05 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: dc=.*,service=mathoid,cluster=kubernetes,name=.*
  • 11:47 moritzm: updated component/cassandra311 on apt.wikimedia.org to 3.11.2
  • 10:26 jynus: setting expire_log_days on db1066 as 30
  • 10:21 godog: bounce stuck rsyslog on lithium / wezen - T136312
  • 09:41 vgutierrez: cp3037 has been depooled due to unknown hardware issues T196974
  • 08:48 marostegui: Stop replication on db2094 to change triggers for archive table
  • 08:36 volans: running puppet on failed hosts post small puppet outage and puppetdb reboot
  • 08:35 akosiaris: rebalance ganeti codfw cluster
  • 08:35 ema@neodymium: conftool action : set/pooled=no; selector: name=cp3037.esams.wmnet
  • 08:33 akosiaris: reboot puppetdb1001 for spec-ctrl enable. Bundling it with a minor puppet outage to only have a torrent of harmless puppet failures once
  • 08:15 akosiaris: ganeti2002 reboot for microcode update
  • 08:04 akosiaris: ganeti2006 reboot for microcode update
  • 08:03 marostegui: Deploy schema change on s1 codfw primary master (db2048) with replication, this will generate lag on codfw T191316 T192926 T89737 T195193
  • 07:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 after alter table (duration: 00m 50s)
  • 07:43 akosiaris: ganeti2007 reboot for microcode update
  • 07:41 akosiaris: ganeti2003 reboot for microcode update
  • 07:31 mutante: closing idle screen session on tin (about to be decomed, dont use anymore)
  • 06:37 marostegui: Deploy schema change on db1121 with replication, this will generate lag on labsdb:s4 T191316 T192926 T89737 T195193
  • 06:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 for alter table (duration: 00m 50s)
  • 06:31 marostegui: Stop replication on db1095, db1102, db1125 to change triggers - T192926
  • 06:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 after alter table (duration: 00m 51s)
  • 05:09 marostegui: Deploy schema change on db1091 T191316 T192926 T89737 T195193
  • 05:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 for alter table (duration: 00m 52s)
  • 02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jun 12 02:45:53 UTC 2018 (duration 10m 18s)
  • 02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 14m 10s)
  • 00:35 legoktm: remove non-deployers from wmf-deployment Gerrit group (T196959)
  • 00:16 ejegg: updated CiviCRM from 69091d8b5f to f54f9810ab

2018-06-11

  • 23:45 ebernhardson@deploy1001: Synchronized wmf-config/CirrusSearch-common.php: SWAT: Lower CirrusSearch delayed job drop timeout (duration: 00m 50s)
  • 23:26 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Tune CirrusSearch slow logging (duration: 00m 48s)
  • 23:18 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Promote Cirrus MLR models from AB test to prod (duration: 00m 51s)
  • 23:02 twentyafterfour: phabricator: restarting apache2 on phab1001 to free up apache workers
  • 22:17 awight@deploy1001: Finished deploy [ores/deploy@6ee8775]: ORES: bswiki, euwiki, srwiki models (duration: 33m 58s)
  • 21:43 awight@deploy1001: Started deploy [ores/deploy@6ee8775]: ORES: bswiki, euwiki, srwiki models
  • 20:57 arlolra: Updated Parsoid to 06b74d2 (T191843)
  • 20:49 arlolra@deploy1001: Finished deploy [parsoid/deploy@97cdab8]: Updating Parsoid to 06b74d2 (duration: 17m 09s)
  • 20:32 arlolra@deploy1001: Started deploy [parsoid/deploy@97cdab8]: Updating Parsoid to 06b74d2
  • 20:27 otto@deploy1001: Finished deploy [eventstreams/deploy@6b013f9]: Enable composite stream and timestamp since param - T196009 , T187418 (duration: 09m 52s)
  • 20:17 otto@deploy1001: Started deploy [eventstreams/deploy@6b013f9]: Enable composite stream and timestamp since param - T196009 , T187418
  • 20:02 ottomata: bouncing varnishkafka-webrequest on cp3039,cp3047,cp2007,cp3010
  • 19:56 ottomata: bouncing varnishkafka on cp3032
  • 19:29 aaron@deploy1001: Synchronized php-1.32.0-wmf.7/includes/libs/rdbms/ChronologyProtector.php: 11e596776f940 - add some logging details (duration: 00m 53s)
  • 18:50 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Restore full config (duration: 00m 16s)
  • 18:49 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Restore full config
  • 18:15 urandom: convert timeline indices to time-windowed compaction - T196024
  • 17:59 twentyafterfour: phabricator: taking phd offline while I clear out the queue backlog (downtime is logged in icinga) see T196840
  • 17:51 twentyafterfour: phabricator: rebuilding git parent caches
  • 17:33 gehel@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (duration: 13m 38s)
  • 17:27 twentyafterfour: phabricator: restarting phd for D1067
  • 17:25 twentyafterfour: Phabricator: deploying hotfix (D1067) refs T196840 T196860 T196855
  • 17:20 gehel@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater
  • 17:19 gehel@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (duration: 00m 03s)
  • 17:19 gehel@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater
  • 17:17 gehel@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 26s)
  • 17:16 gehel@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (wdqs1009 only)
  • 17:00 akosiaris: ganeti2008 reboot for microcode update
  • 16:42 marostegui: Set disk 32:1 offline on db1065 to get a new one - T196806
  • 16:02 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 01m 32s)
  • 16:00 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
  • 15:44 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 00m 33s)
  • 15:43 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
  • 15:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 after alter table (duration: 00m 50s)
  • 15:33 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 00m 22s)
  • 15:33 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
  • 15:31 marostegui: Set offline disk 32:3 on db1063 - T196806
  • 15:31 marostegui: Set offline disk 32:1 on db1065 - T196806
  • 15:02 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 35m 33s)
  • 14:55 marostegui: Deploy schema change on db1084 T191316 T192926 T89737 T195193
  • 14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 for alter table (duration: 00m 50s)
  • 14:53 akosiaris: reboot bohrium for kernel upgrades and spec-ctrl enabling. Manually stopped mysql behorehand
  • 14:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 after alter table (duration: 00m 50s)
  • 14:35 akosiaris: reboot mx1001, poolcounter1001 for kernel upgrades and spec-ctrl enabling
  • 14:26 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
  • 14:25 godog: upload scap 3.8.2-1 - T196710
  • 14:21 hoo: Updated operations/dumps/dcat (536bd5b..559dee3) on snapshot1008
  • 13:58 marostegui: Deploy schema change on db1081 T191316 T192926 T89737 T195193
  • 13:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 for alter table (duration: 00m 48s)
  • 13:56 otto@deploy1001: Finished deploy [eventlogging/analytics@08a1dff]: Producing events with kafka timestamp set to event time - T196407 (duration: 00m 04s)
  • 13:56 otto@deploy1001: Started deploy [eventlogging/analytics@08a1dff]: Producing events with kafka timestamp set to event time - T196407
  • 13:54 otto@deploy1001: Finished deploy [eventlogging/eventbus@08a1dff]: Producing events with kafka timestamp set to event time - T196407 (duration: 01m 55s)
  • 13:52 otto@deploy1001: Started deploy [eventlogging/eventbus@08a1dff]: Producing events with kafka timestamp set to event time - T196407
  • 13:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 00m 50s)
  • 13:47 hashar: European SWAT completed
  • 13:35 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix wgMetaNamespace for pswikivoyage - T196837 (duration: 00m 49s)
  • 13:29 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix wgMetaNamespace for pswikivoyage - T196837 (duration: 00m 50s)
  • 13:21 volans@deploy1001: Finished deploy [debmonitor/deploy@81d7333]: Release v0.1.2 (duration: 00m 56s)
  • 13:20 volans@deploy1001: Started deploy [debmonitor/deploy@81d7333]: Release v0.1.2
  • 13:20 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use uploaded HD logo for bewikiquote - T196134 (duration: 00m 50s)
  • 13:17 hashar@deploy1001: Synchronized static/images/project-logos: Change logo files for bewikiquote - T196134 (duration: 00m 50s)
  • 13:13 hashar@deploy1001: Synchronized static/images/project-logos: Revert "Change bewikiquote logo" - T196134 (duration: 00m 51s)
  • 13:01 volans@deploy1001: Finished deploy [debmonitor/deploy@81d7333]: Release v0.1.2 (duration: 07m 16s)
  • 12:54 volans@deploy1001: Started deploy [debmonitor/deploy@81d7333]: Release v0.1.2
  • 11:45 arturo: T196633 downtime labcontrol100[3,4] due to unexpected puppet errors on installation of keystone
  • 11:38 arturo: T196633 deploy keystone to labcontrol100[3,4].wikimedia.org. Dormant daemon, no DB yet
  • 11:30 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@b5396cd]: Tune cirrus jobs concurrencies (duration: 00m 42s)
  • 11:30 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@b5396cd]: Tune cirrus jobs concurrencies
  • 11:23 marostegui: Deploy schema change on db1103:3314 T191316 T192926 T89737 T195193
  • 11:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 50s)
  • 11:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 00m 51s)
  • 10:52 mutante: phab1002 - editing cached scap config /srv/deployment/phabricator/deployment-cache/.config to replace tin.eqiad with deploy1001.eqiad deployment server, run puppet. other options: run scap with --refresh-config, delet cached .config file (T196019) (T175288)
  • 10:29 _joe_: depooling permantently mw1230 for disk replacement, T196881
  • 10:11 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
  • 10:10 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
  • 09:07 marostegui: Deploy schema change on db1097:3314 T191316 T192926 T89737 T195193
  • 09:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 for alter table (duration: 00m 52s)
  • 08:32 moritzm: installing gnupg2 security updates
  • 08:27 gehel: restart elastic1020 to enable G1 GC - T156137
  • 08:25 moritzm: installing gnupg1 security updates
  • 07:52 moritzm: installing gnupg security updates
  • 07:37 marostegui: Deploy schema change on dbstore1002:s4 T191316 T192926 T89737 T195193
  • 07:29 moritzm: installing openjdk-7 security updates
  • 06:39 elukey: restart pdfrender on scb1002
  • 06:14 marostegui: Deploy schema change on s4 codfw master (db2051) this will generate lag on codfw - T191316 T192926 T89737 T195193
  • 06:10 marostegui: Stop replication on db2095 to update triggers - T192926
  • 05:32 marostegui: Restart mysql on codfw sanitariums (db1095, db1102, db1124, db1125) to pick up new replication filters - T196748
  • 05:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool pc2005 - T196339 (duration: 00m 52s)
  • 05:25 marostegui: Restart mysql on codfw sanitariums (db2094, db2095) to pick up new replication filters - T196748
  • 05:17 marostegui: Stop MySQL and reboot pc2005 for intel-microcode update and final HW check - T196339
  • 05:15 marostegui: Deploy schema change on s6 primary master (db1061) - T191316 T192926 T195193
  • 02:42 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jun 11 02:42:42 UTC 2018 (duration 10m 16s)
  • 02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 13m 06s)

2018-06-10

  • 12:30 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: CSP in report mode for group0 (duration: 00m 55s)
  • 12:01 bawolff: disable some botpasswords (T194204)
  • 11:08 _joe_: restarting gerrit on cobalt as the web interface is unresponsive

2018-06-08

  • 22:41 no_justification: gerrit: restarting
  • 19:36 urandom: upgrade Cassandra to 3.11.2, restbase1015 & restbase1018 - T178905
  • 18:49 urandom: upgrade Cassandra to 3.11.2, restbase1009 & restbase1014 - T178905
  • 18:02 urandom: upgrade Cassandra to 3.11.2, restbase1017 & restbase1013 - T178905
  • 17:33 no_justification: gerrit: up mostly, but will see some errors about "wip" label for a bit until reindexing completes.
  • 17:28 cmjohnson: powering flerovium down to move to a different space in the rack
  • 17:14 demon@deploy1001: Finished deploy [gerrit/gerrit@7324140]: 2.15.2 (duration: 00m 11s)
  • 17:14 demon@deploy1001: Started deploy [gerrit/gerrit@7324140]: 2.15.2
  • 17:12 no_justification: gerrit: taking offline for 2.14 -> 2.15 upgrade
  • 15:35 urandom: upgrade Cassandra to 3.11.2, restbase1012 - T178905
  • 15:26 marostegui: Restart MySQL on codfw sanitarium hosts db2094, db2095 - https://phabricator.wikimedia.org/T196748
  • 15:00 urandom: upgrade Cassandra to 3.11.2, restbase1008 - T178905
  • 14:17 urandom: upgrade Cassandra to 3.11.2, restbase1011 & restbase1016 - T178905
  • 12:14 twentyafterfour: running phabricator public_task_dump script manually to confirm that it's working as expected.
  • 09:31 arturo: merging https://gerrit.wikimedia.org/r/#/c/436337/ for the eqia1 openstack deployment (labcontrol1003/labcontrol1004)
  • 09:19 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Unify and update sanitarium comments - T190704 (duration: 00m 50s)
  • 09:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Unify and update sanitarium comments - T190704 (duration: 00m 50s)
  • 09:06 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
  • 08:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1066 after reboot (duration: 00m 50s)
  • 08:18 marostegui: Stop MySQL and reboot db1066 for intel-microcode install - T194870
  • 08:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1066 for reboot (duration: 00m 50s)
  • 07:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 after alter table (duration: 00m 51s)
  • 07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight (duration: 00m 50s)
  • 07:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1091 (duration: 00m 50s)
  • 07:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increse weight for db1091 (duration: 00m 50s)
  • 06:47 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T196630 Remove unneeded description from performance survey definition (duration: 00m 52s)
  • 06:44 elukey: bounce kafka mirror maker main-eqiad-to-main-codfw (kafka200*) due to errors in the logs (also lag metrics not displaying)
  • 06:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight after reimage (duration: 00m 50s)
  • 05:34 marostegui: Deploy sanitarium events on db1124 - T190704
  • 05:23 marostegui: Stop MySQL on db1091 for reimage
  • 05:22 marostegui: Deploy schema change on db1113:3316 - T191316 T192926 T195193 T89737
  • 05:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3316 for alter table (duration: 00m 52s)
  • 02:12 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c (duration: 00m 42s)
  • 02:11 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c

2018-06-07

  • 23:58 demon@deploy1001: Finished deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state (x2) (duration: 00m 14s)
  • 23:58 demon@deploy1001: Started deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state (x2)
  • 23:57 demon@deploy1001: Finished deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state (duration: 00m 06s)
  • 23:57 demon@deploy1001: Started deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state
  • 21:49 mobrovac@deploy1001: Finished deploy [proton/deploy@97ec4bf]: Initial deploy to production - T186748 (duration: 02m 19s)
  • 21:47 mobrovac@deploy1001: Started deploy [proton/deploy@97ec4bf]: Initial deploy to production - T186748
  • 20:37 herron: ircecho restarted
  • 19:47 urandom: rolling Cassandra restart, restbase2005, restbase2006, restbase2012 -- T178905
  • 19:36 XenoRyet: updated payments-wiki config from 3197c729ee to b11d120362
  • 19:29 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.7
  • 19:16 herron: stopped ircecho
  • 18:28 urandom: rolling Cassandra restart, restbase2004, restbase2008, restbase2011 -- T178905
  • 18:20 urandom: restarting Cassandra, restbase2003-c -- T178905
  • 18:13 subbu: Updated Parsoid (T183706, T192726, T194879, T196357, T196360, T43716)
  • 18:10 ssastry@deploy1001: Finished deploy [parsoid/deploy@2f80639]: Updating Parsoid to 7819c9e7 (duration: 10m 59s)
  • 17:59 ssastry@deploy1001: Started deploy [parsoid/deploy@2f80639]: Updating Parsoid to 7819c9e7
  • 17:21 awight@deploy1001: Finished deploy [ores/deploy@65ce165]: New home page for ORES; T196580 (take 2) (duration: 04m 34s)
  • 17:16 awight@deploy1001: Started deploy [ores/deploy@65ce165]: New home page for ORES; T196580 (take 2)
  • 17:16 awight@deploy1001: Finished deploy [ores/deploy@65ce165]: New home page for ORES; T196580 (duration: 03m 19s)
  • 17:16 reedy@deploy1001: Synchronized wmf-config/interwiki-labs.php: labs! (duration: 00m 57s)
  • 17:13 awight@deploy1001: Started deploy [ores/deploy@65ce165]: New home page for ORES; T196580
  • 16:45 urandom: rolling Cassandra restart, restbase2001, restbase2002, restbase2007 - T178905
  • 16:21 urandom: rolling Cassandra restart, restbase1007 - T178905
  • 15:59 hoo: Finished running "foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https" T196360 T183706 T195014 T196357
  • {{safesubst:SAL entry|1=15:38 urandom: upgrade Cassandra to 3.11.2, restbase1010-{a,b,c} - T178905}}
  • 15:30 hoo: Running "foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https" T196360 T183706 T195014 T196357
  • 15:23 hoo: Emptied out the sites and site_identifiers tables on pswikivoyage, pmswikisource, bnwikivoyage and sahwikiquote for T122520,
  • 15:20 hoo@deploy1001: Synchronized dblists/wikidataclient.dblist: Enable WikidataClient on sahwikiquote and pswikivoyage - T183706, T196360 (duration: 00m 57s)
  • 15:08 marostegui: Sanitize wikis on db1124 (current sanitarium for s3) - T196362 T196358 T196359 T195008 T193187
  • 15:04 reedy@deploy1001: Synchronized wmf-config/interwiki.php: (no justification provided) (duration: 00m 56s)
  • 14:56 ottomata: beginning rolling restarts of all cluster kafka brokers to apply log.message.timestamp.type=CreateTime - T196407
  • 14:53 reedy@deploy1001: rebuilt and synchronized wikiversions files: new wikis
  • 14:53 marostegui: Sanitize wikis on db1095 (old sanitarium) - T196362 T196358 T196359 T195008 T193187
  • 14:53 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: new wikis (duration: 00m 57s)
  • 14:50 reedy@deploy1001: Synchronized static/images/: new wikis (duration: 00m 57s)
  • 14:48 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: idwikimedia (duration: 00m 57s)
  • 14:47 reedy@deploy1001: Synchronized dblists/: 5 new wikis (duration: 00m 55s)
  • 14:45 Reedy: created new wiki databases T183706 T192726 T194879 T196357 T196360
  • 14:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 (duration: 00m 57s)
  • 14:11 reedy@deploy1001: Synchronized wmf-config/jobqueue-labs.php: labs (duration: 00m 56s)
  • 14:10 reedy@deploy1001: Synchronized wmf-config/LabsServices.php: labs (duration: 00m 57s)
  • 14:02 reedy@deploy1001: Synchronized docroot/noc/: Add some more symlinked configs (duration: 00m 57s)
  • 13:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1066 (duration: 00m 57s)
  • 13:31 reedy@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/WikimediaMaintenance/addWiki.php: add all the wikis (duration: 00m 56s)
  • 13:29 reedy@deploy1001: Synchronized php-1.32.0-wmf.6/extensions/WikimediaMaintenance/addWiki.php: add all the wikis (duration: 00m 58s)
  • 13:26 reedy@deploy1001: Synchronized dblists/all-labs.dblist: labs (duration: 00m 54s)
  • 13:24 reedy@deploy1001: Synchronized wikiversions-labs.json: labs (duration: 00m 57s)
  • 13:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1066 (duration: 00m 56s)
  • 13:17 zeljkof: EU SWAT finished
  • 13:16 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: When using the FileExporter set it as BeatFeature by default (T195370) (duration: 00m 56s)
  • 13:10 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add FileExporter to BetaFeaturesWhiteList (T195370) (duration: 00m 57s)
  • 12:59 marostegui: Deploy schema change on db1088 - T191316 T192926 T195193 T89737
  • 12:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 57s)
  • 12:54 jynus: stop, clone and reimage db1091
  • 12:52 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 (duration: 00m 57s)
  • 12:46 mutante: planet1001/puppetmaster: revoke old cert, sign new cert request, initial puppet run, reinstalled, will turn service active-active again once done (T168490)
  • 12:26 mutante: planet1001 - schedule downtime, boot to PXE, reinstall with stretch (ganeti) (T168490)
  • 12:15 moritzm: repooled mw1280 after hardware maintenance (T195734)
  • 11:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 57s)
  • 10:29 moritzm: installing openssl security updates
  • 08:50 marostegui: Deploy schema change on db1085 with replication, this will generate lag on labsdb for s6 section - T191316 T192926 T195193 T89737
  • 08:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 00m 56s)
  • 08:49 akosiaris: slow reboot of all ganeti eqiad VMs (except bohrium, puppetdb1001, poolcounter1001, mx1001) for kernel upgrades and picking up spec-ctrl cpu flag
  • 08:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 after alter table (duration: 00m 55s)
  • 08:34 marostegui: Stop replication on db1102:3316 and db1125:3316 to update triggers for archive table - T192926
  • 08:15 jynus: running ANALYZE on db2091 T196526
  • 07:46 marostegui: Deploy schema change on db1093 - T191316 T192926 T195193 T89737
  • 07:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 for alter table (duration: 00m 56s)
  • 07:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316 after alter table (duration: 00m 57s)
  • 07:09 moritzm: installing git security updates on trusty (Debian already fixed)
  • 06:55 jynus: relaxing write consistency on db2048 due to ongoing maintenance (sync_binlog,flush_log)
  • 06:26 marostegui: Deploy sanitarium events on db1125 - T190704
  • 06:24 jynus: phabricator maintenance finished
  • 06:10 jynus: start database maintenance on phabricator- brief interruptions could happen
  • 05:15 marostegui: Deploy event_sanitarium on codfw sanitariums - T190704
  • 05:12 marostegui: Deploy schema change on db1098:3316 - T191316 T192926 T195193 T89737
  • 05:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 for alter table (duration: 00m 56s)
  • 05:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 after alter table (duration: 00m 58s)
  • 04:01 eileen: civicrm revision changed from 1aaa798e9b to 69091d8b5f, config revision is c60375be8d
  • 03:05 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jun 7 03:05:05 UTC 2018 (duration 10m 19s)
  • 02:54 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 07m 02s)
  • 02:30 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 11m 24s)
  • 02:25 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@a07af40]: log
  • 02:00 paravoid: starting exim4 and reenabling puppet on mx1001, due to T196598
  • 01:12 ejegg: disabled fundraising mailing statistics fetch jobs
  • 01:00 ejegg: updated CiviCRM from 7a06a6a387 to 1aaa798e9b
  • 00:51 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@a07af40]: (no justification provided) (duration: 01m 02s)
  • 00:50 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@a07af40]: (no justification provided)
  • 00:33 ejegg: restarted fundraising jobs
  • 00:25 legoktm@deploy1001: Finished scap: Preference for responsive MonoBook, plus set mobile width cutoff to 550px (gerrit:437875, gerrit:437814) (duration: 64m 01s)

2018-06-06

  • {{safesubst:SAL entry|1=23:51 urandom: upgrade Cassandra to 3.11.2, restbase2012-{a,b,c} - T178905}}
  • 23:49 eileen: civicrm revision changed from ce960c0642 to 7a06a6a387, config revision is b7bc5dc31f
  • 23:43 eileen: civicrm revision changed from ac13914d03 to ce960c0642, config revision is b7bc5dc31f
  • {{safesubst:SAL entry|1=23:23 urandom: upgrade Cassandra to 3.11.2, restbase2009-{a,b,c} - T178905}}
  • 23:21 legoktm@deploy1001: Started scap: Preference for responsive MonoBook, plus set mobile width cutoff to 550px (gerrit:437875, gerrit:437814)
  • 23:19 eileen: civicrm revision changed from 0b97f1f5b2 to ac13914d03
  • 23:07 cwd: disabled process-control for civi update
  • 22:22 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c (duration: 05m 33s)
  • 22:16 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c
  • 20:47 ebernhardson: sighup logstash on logstash100[789] to reload config for gerrit.wikimedia.org/r/437657
  • {{safesubst:SAL entry|1=20:42 urandom: upgrade Cassandra to 3.11.2, restbase2005-{a,b,c} - T178905}}
  • 20:19 bearND: rolled back mobileapps deploy
  • 20:18 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@a07af40]: Update mobileapps to 3bf9be5 (T196402 T195948) (duration: 08m 37s)
  • {{safesubst:SAL entry|1=20:17 urandom: upgrade Cassandra to 3.11.2, restbase2011-{a,b,c} - T178905}}
  • 20:10 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@a07af40]: Update mobileapps to 3bf9be5 (T196402 T195948)
  • {{safesubst:SAL entry|1=19:38 urandom: upgrade Cassandra to 3.11.2, restbase2008-{a,b,c} - T178905}}
  • 19:19 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.7 (duration: 00m 56s)
  • 19:18 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.7
  • 18:55 herron: stopped exim on mx1001 in prep for upgrade to stretch
  • {{safesubst:SAL entry|1=18:54 urandom: upgrade Cassandra to 3.11.2, restbase2004-{a,b,c} - T178905}}
  • 18:38 andrewbogott: rebooting labvirt1019, 1020
  • 18:36 andrewbogott: rebooting labvirt1018, 1021, 1022
  • 18:29 andrewbogott: rebooting labvirt1017
  • 18:23 andrewbogott: rebooting labvirt1016
  • 18:21 andrewbogott: rebooting labvirt1015
  • {{safesubst:SAL entry|1=18:19 urandom: upgrade Cassandra to 3.11.2, restbase2003-{a,b,c} - T178905}}
  • 18:14 andrewbogott: rebooting labvirt1013
  • 18:06 andrewbogott: rebooting labvirt1012
  • 17:59 andrewbogott: rebooting labvirt1011
  • {{safesubst:SAL entry|1=17:59 urandom: upgrade Cassandra to 3.11.2, restbase2010-{a,b,c} - T178905}}
  • 17:44 andrewbogott: rebooting labvirt1010
  • {{safesubst:SAL entry|1=17:01 urandom: upgrade Cassandra to 3.11.2, restbase2007-{a,b,c} - T178905}}
  • 16:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 fully (duration: 00m 56s)
  • 16:51 andrewbogott: rebooting labvirt1008
  • 16:33 andrewbogott: rebooting labvirt1007
  • 16:25 jynus: stop mysql @ db1051 in preparation for decom
  • 16:25 andrewbogott: rebooting labvirt1006
  • 16:16 andrewbogott: rebooting labvirt1005
  • 16:09 andrewbogott: rebooting labvirt1004
  • 16:05 XioNoX: lvs1002 repooled
  • 16:04 andrewbogott: rebooting labvirt1002
  • 15:57 twentyafterfour: reloading apache on phab1001 to free up some resources
  • 15:56 andrewbogott: rebooting labvirt1014
  • {{safesubst:SAL entry|1=15:54 urandom: upgrade Cassandra to 3.11.2, restbase2001-{a,b,c} - T178905}}
  • 15:51 XioNoX: disable pybal on lvs1002 - T187962
  • 15:39 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@402d729]: Adjust the cirrus concurrencies (duration: 00m 40s)
  • 15:38 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@402d729]: Adjust the cirrus concurrencies
  • 15:32 _joe_: cross enabling videoscalers,jobrunners in their respective pools
  • {{safesubst:SAL entry|1=15:27 urandom: upgrade Cassandra to 3.11.2, restbase2001-{b,c} - T178905}}
  • 15:26 XioNoX: stop pybal on lvs1001
  • 15:26 awight@deploy1001: Finished deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336 (duration: 25m 37s)
  • 15:13 _joe_: adding jobrunners, videoscalers to both pools with equal weight in codfw
  • 15:06 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=videoscaler,dc=codfw,service=nginx
  • {{safesubst:SAL entry|1=15:06 urandom: upgrade Cassandra to 3.11.2, restbase1007-{b,c} - T178905}}
  • 15:01 awight@deploy1001: Started deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336
  • 14:39 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: cluster=videoscaler,dc=codfw,service=nginx,name=mw22(4[1-5]|5[3-8]|6[1-9]).*
  • 14:35 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: cluster=videoscaler,dc=codfw,service=nginx,name=mw21(5[3-9]|6).*
  • 14:22 andrewbogott: rebooting labvirt1009
  • 14:21 jynus: setting m2 on read write
  • 14:18 jynus: setting gerrit on read only
  • 14:14 jynus: starting s2-master switchover from db1051 to db1065
  • 14:06 andrewbogott: rebooting labvirt1003
  • 13:58 jynus: disabling puppet on db1051, db1065
  • 13:35 marostegui: Deploy schema change on db1096:3316 - T191316 T192926 T195193 T89737
  • 13:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 for alter table (duration: 00m 57s)
  • 13:06 moritzm: installing elfutils security updates
  • 13:01 akosiaris: starting slow rolling restart of all VMs on ganeti01.svc.codfw.wmnet
  • 12:59 akosiaris: add +spec_ctrl to ganeti01.svc.codfw.wmnet cluster default cpu_type
  • 12:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low load (duration: 00m 56s)
  • 12:24 mobrovac@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch CirrusSearch jobs to EventBus for all wikis, file 2/2 - T189137 (duration: 00m 56s)
  • 12:23 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@c8d62da]: Enable cirrus for everything T190327 (duration: 00m 47s)
  • 12:23 mobrovac@deploy1001: Synchronized wmf-config/jobqueue.php: Switch CirrusSearch jobs to EventBus for all wikis - T189137 (duration: 00m 57s)
  • 12:22 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@c8d62da]: Enable cirrus for everything T190327
  • 11:38 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase s4 weight for db1097 and db1103 (duration: 00m 56s)
  • 11:35 jynus: stop and reimage db1084
  • 11:27 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 58s)
  • 10:19 awight@deploy1001: Finished deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 03m 44s)
  • 10:16 marostegui: Deploy schema change on dbstore1002:s6 - T191316 T192926 T195193 T89737
  • 10:15 awight@deploy1001: Started deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336
  • 10:15 awight@deploy1001: Finished deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 00m 06s)
  • 10:15 awight@deploy1001: Started deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336
  • 08:29 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1011 - T190704
  • 08:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 56s)
  • 08:24 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: wikidatawiki dispatching: dispatchMaxTime 720 (4 dispatchers at once) (duration: 00m 56s)
  • 08:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 57s)
  • 07:48 marostegui: Stop replication on all sanitarium masters to move labsdb1011 - T190704
  • 07:30 marostegui: Stop MySQL on labsdb1011 to install intel-microcode and reboot
  • 06:32 marostegui: Deploy schema change on s6 codfw master (db2039), this will generate lag on s6 codfw - T191316 T192926 T195193 T89737
  • 06:14 marostegui: Stop slave on db2095:3316 to rebuild archive_insert and archive_update triggers - T192926
  • 06:14 ppchelko@deploy1001: Finished deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402, take 2 (duration: 07m 13s)
  • 06:06 ppchelko@deploy1001: Started deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402, take 2
  • 06:05 ppchelko@deploy1001: Finished deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402 (duration: 11m 45s)
  • 06:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool all sanitariums masters - T190704 (duration: 01m 09s)
  • 05:53 ppchelko@deploy1001: Started deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402
  • 05:46 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 - T190704
  • 05:31 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010 - T190704
  • 05:25 marostegui: Restart MySQL on labsdb1010
  • 05:24 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - T190704
  • 05:17 marostegui: Deploy schema change on db1070 s5 primary master - T191316 T192926 T195193
  • 04:44 kartik@deploy1001: Finished deploy [cxserver/deploy@8ce20ba]: Update cxserver to 391d7b6 (Fixing T196462) (duration: 03m 06s)
  • 04:41 kartik@deploy1001: Started deploy [cxserver/deploy@8ce20ba]: Update cxserver to 391d7b6 (Fixing T196462)
  • 03:10 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jun 6 03:10:14 UTC 2018 (duration 10m 15s)
  • 02:59 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 15m 31s)
  • 02:27 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 08m 23s)
  • 01:07 krinkle@deploy1001: Synchronized php-1.32.0-wmf.7/composer.json: I13dbdba2b9d / T196496 (duration: 00m 57s)
  • 01:06 krinkle@deploy1001: Synchronized php-1.32.0-wmf.7/vendor/: I5a5d7d / T196496 (duration: 01m 35s)
  • 00:15 reedy@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/WikimediaMessages/: respect watchlist preference feature flag (duration: 00m 58s)

2018-06-05

  • 23:54 reedy@deploy1001: Synchronized wmf-config/: Config! (duration: 00m 57s)
  • 23:42 reedy@deploy1001: Synchronized wmf-config/: Config! (duration: 00m 57s)
  • 23:41 reedy@deploy1001: Synchronized rpc/: code updates (duration: 00m 56s)
  • 23:26 reedy@deploy1001: Synchronized multiversion/submodules.json: minus unicodeconverter (duration: 00m 56s)
  • 23:24 reedy@deploy1001: Synchronized wmf-config/: bye to UnicodeConverter (duration: 00m 57s)
  • 23:21 reedy@deploy1001: Synchronized wmf-config/: page creation log on Beta Labs (duration: 00m 56s)
  • 23:07 reedy@deploy1001: Synchronized wmf-config/: Updates (duration: 00m 58s)
  • 23:05 reedy@deploy1001: Synchronized composer.json: minus x! (duration: 00m 56s)
  • 23:04 reedy@deploy1001: Synchronized rpc: 644 (duration: 00m 56s)
  • 23:01 ebernhardson: restore ferm on elastic1018 and logstash1009
  • 22:57 ebernhardson: temporarily disable ferm on elastic1018 and logstash1007 to test theory
  • 22:57 ebernhardson: temporarily disable ferm on elastic1018 to test theory
  • 22:50 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Simplify PHP_SAPI conditionals (duration: 00m 57s)
  • 22:20 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/437638/ (duration: 00m 57s)
  • 21:41 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.7
  • 21:29 thcipriani@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.7 and rebuild l10n cache (duration: 65m 25s)
  • 20:23 thcipriani@deploy1001: Started scap: testwiki to php-1.32.0-wmf.7 and rebuild l10n cache
  • 20:03 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@7ecc3b6]: Update mobileapps to 66727b7 (duration: 05m 21s)
  • 19:57 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@7ecc3b6]: Update mobileapps to 66727b7
  • 19:35 urandom: upgrade Cassandra to 3.11.2, restbase1007-a (canary) - T178905
  • 17:45 urandom: upgrade Cassandra to 3.11.2, restbase2001 (canary) - T178905
  • 17:00 urandom: upgrade Cassandra to 3.11.2, restbase-dev1006 - T178905
  • 16:58 Trey314159: reindexing Slovak wikis on elastic@eqiad (T191545)
  • 16:56 urandom: upgrade Cassandra to 3.11.2, restbase-dev1005 - T178905
  • 16:10 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove if file_exists for /etc/wikimedia-scaler (duration: 00m 51s)
  • 16:07 thcipriani: starting branch cut for 1.32.0-wmf.7
  • 16:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 fully (duration: 00m 49s)
  • 15:58 urandom: upgrade Cassandra to 3.11.2, restbase-dev1004 - T178905
  • 15:42 Trey314159: reindexing Slovak wikis on elastic@codfw (T191545)
  • 15:41 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 fully (duration: 00m 49s)
  • 15:30 bstorm_: reboot labsdb1005 for upgrades T195313
  • 15:24 marostegui: Stop MySQL on labsdb1005
  • 14:55 marostegui: Downtime labsdb1005 and labsdb1004 for maintenance on labsdb1005 - T195313
  • 14:52 urandom: restarting restbase-dev1004 - T178905
  • 14:51 moritzm: installing fixed kernels/microcode for spectre v2 on labvirt*
  • 14:42 jynus: reenabling puppet on all databases
  • 14:09 jynus: disabling puppet on eqiad dbs for 435751 gerrit deploy
  • 13:55 Amir1: ladsgroup@terbium:~$ mwscript deleteAutoPatrolLogs.php --wiki=commonswiki --sleep 2 --check-old
  • 13:26 mobrovac@deploy1001: Finished deploy [restbase/deploy@a39743d]: Fix transform monitoring tests (duration: 03m 15s)
  • 13:24 zeljkof: EU SWAT finished
  • 13:24 zfilipin@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: Remove ruwiki from MFSpecialCaseMainPage (T196223) (duration: 00m 51s)
  • 13:22 mobrovac@deploy1001: Started deploy [restbase/deploy@a39743d]: Fix transform monitoring tests
  • 13:22 mobrovac@deploy1001: Finished deploy [restbase/deploy@a39743d]: Fix transform monitoring tests (duration: 08m 00s)
  • 13:14 mobrovac@deploy1001: Started deploy [restbase/deploy@a39743d]: Fix transform monitoring tests
  • 13:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@a39743d]: Fix transform monitoring tests (duration: 18m 02s)
  • 12:55 mobrovac@deploy1001: Started deploy [restbase/deploy@a39743d]: Fix transform monitoring tests
  • 12:13 kartik@deploy1001: Finished deploy [cxserver/deploy@0a350c3]: Update cxserver to 7fb7671 (duration: 01m 15s)
  • 12:12 kartik@deploy1001: Started deploy [cxserver/deploy@0a350c3]: Update cxserver to 7fb7671
  • 12:06 XioNoX: disable graceful-switchover on dual-RE routers
  • 11:35 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low load (duration: 00m 49s)
  • 11:30 elukey: manually set net.netfilter.nf_conntrack_tcp_timeout_time_wait to 65 (was 120) on mw* hosts
  • 11:21 mobrovac@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch CirrusSearch jobs for all wikis except wp, wd, commons - T189137 (duration: 00m 51s)
  • 11:21 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@aa5e94b]: Enable cirrus jobs in kafka for everything except wikipedia, wikidata and commons T190327 (duration: 00m 41s)
  • 11:21 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@aa5e94b]: Enable cirrus jobs in kafka for everything except wikipedia, wikidata and commons T190327
  • 11:07 elukey: manually set net.netfilter.nf_conntrack_tcp_timeout_time_wait to 65 (was 120)
  • 10:51 jynus: stop db1081 for reimage
  • 10:45 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 (duration: 00m 50s)
  • 10:38 akosiaris: reboot acrab for spec_ctrl CPU flag addition
  • 10:20 akosiaris: reboot acrab for kernel upgrade and qemu upgrade
  • 10:18 akosiaris: upgrade qemu to 1:2.8+dfsg-6+deb9u4 on ganeti01.svc.codfw.wmnet
  • 10:14 moritzm: installing batik security updates
  • 10:08 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 with low load (duration: 00m 50s)
  • 09:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 50s)
  • 09:22 jynus: stop db1080 for reimage
  • 09:09 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 (duration: 00m 49s)
  • 09:01 marostegui: Deploy schema change on db1110 - T191316 T192926 T89737 T195193
  • 09:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 51s)
  • 08:52 mobrovac@deploy1001: Synchronized wmf-config/jobqueue.php: Switch video scaling jobs to EventBus - T190327 (duration: 00m 52s)
  • 08:52 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@63b30a6]: Enable videoscaler jobs in kafka T190327 (duration: 00m 49s)
  • 08:51 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@63b30a6]: Enable videoscaler jobs in kafka T190327
  • 08:16 ema: libvmod-re2 1.3.1-1 uploaded to apt.w.o T196355
  • 08:10 gehel: rebooting elastic10(41|43) for plugin update - T193734
  • 06:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2059 after reimage (duration: 00m 50s)
  • 05:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 00m 50s)
  • 05:44 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2059 for reimage (duration: 00m 51s)
  • 05:14 marostegui: Deploy schema change on db1100 - T191316 T192926 T89737 T195193
  • 05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 00m 51s)
  • 03:25 pnorman@deploy1001: Finished deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a (duration: 06m 02s)
  • 03:19 pnorman@deploy1001: Started deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a
  • 03:12 pnorman@deploy1001: Finished deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a (duration: 04m 50s)
  • 03:07 pnorman@deploy1001: Started deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a
  • 02:51 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): enable v3view (duration: 00m 06s)
  • 02:51 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): enable v3view
  • 02:49 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): enable v3view
  • 02:45 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable more sources (duration: 00m 26s)
  • 02:45 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable more sources
  • 02:43 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable style without labels (duration: 02m 22s)
  • 02:41 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable style without labels
  • 02:30 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jun 5 02:30:57 UTC 2018 (duration 10m 15s)
  • 02:20 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 08m 21s)
  • 00:08 thcipriani: restarting jenkins to finalize updates

2018-06-04

  • 23:57 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable page creation log on Test Wikipedia" T196400 (duration: 00m 49s)
  • 23:50 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable page creation log on Test Wikipedia T196400 (duration: 00m 50s)
  • 23:21 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@825863f]: Potential mitigation for T194325 (duration: 09m 39s)
  • 23:11 smalyshev@deploy1001: Started deploy [wdqs/wdqs@825863f]: Potential mitigation for T194325
  • 22:57 smalyshev@deploy1001: Started deploy [wdqs/wdqs@825863f]: Potential mitigation for T194325
  • 22:35 pnorman@deploy1001: Finished deploy [kartotherian/deploy@a588bf4] (cleartables): Deploy var name parameters (duration: 00m 21s)
  • 22:35 pnorman@deploy1001: Started deploy [kartotherian/deploy@a588bf4] (cleartables): Deploy var name parameters
  • 21:08 pnorman@deploy1001: Finished deploy [kartotherian/deploy@d8dcba3] (cleartables): Redeploy kartotherian to test (duration: 00m 21s)
  • 21:07 pnorman@deploy1001: Started deploy [kartotherian/deploy@d8dcba3] (cleartables): Redeploy kartotherian to test
  • 20:58 awight@deploy1001: Finished deploy [ores/deploy@bf182e2]: roll back ores2001 (duration: 01m 11s)
  • 20:57 awight@deploy1001: Started deploy [ores/deploy@bf182e2]: roll back ores2001
  • 20:50 awight@deploy1001: Finished deploy [ores/deploy@65e979f]: ores2001 canary of drafttopic; T176336 (take 3 after bumping revision) (duration: 01m 47s)
  • 20:48 awight@deploy1001: Started deploy [ores/deploy@65e979f]: ores2001 canary of drafttopic; T176336 (take 3 after bumping revision)
  • 20:40 awight@deploy1001: Finished deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)f (duration: 00m 22s)
  • 20:40 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@276ea43]: Update mobileapps to f579f0d (duration: 05m 54s)
  • 20:40 awight@deploy1001: Started deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)f
  • 20:40 awight@deploy1001: Started deploy [ores/deploy@d77e52c]: l ores2001.codfw.wmnet ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)
  • 20:34 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@276ea43]: Update mobileapps to f579f0d
  • 20:34 awight@deploy1001: Finished deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (duration: 02m 35s)
  • 20:34 arlolra@deploy1001: Finished deploy [parsoid/deploy@828034c]: Updating Parsoid to bd5a840 (duration: 10m 54s)
  • 20:31 awight@deploy1001: Started deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336
  • 20:30 andrew@deploy1001: Finished deploy [horizon/deploy@12aa2d3]: fix for T192179 (duration: 03m 35s)
  • 20:27 andrew@deploy1001: Started deploy [horizon/deploy@12aa2d3]: fix for T192179
  • 20:23 arlolra@deploy1001: Started deploy [parsoid/deploy@828034c]: Updating Parsoid to bd5a840
  • 19:10 gehel: elasticsearch cluster restart on eqiad completed - T193734
  • 19:06 ottomata: bouncing kafka2003 one more time for T196077
  • 19:04 otto@deploy1001: Started restart [eventlogging/eventbus@3a5c395]: bouncing eventbus after upgrading to python-kafka 1.4.3 for T196077
  • 19:00 ottomata: bouncing kafka2003 again to test T196077 with python-kafka 1.4.3
  • 18:52 thcipriani@deploy1001: Synchronized docroot/noc/index.html: SWAT: Fix wrong link to Server Admin Log on noc.wikimedia.org T193848 (duration: 00m 50s)
  • 18:44 ottomata: bouncing kafka on kafka2003 to test T196077
  • 18:36 otto@deploy1001: Finished deploy [eventlogging/eventbus@3a5c395]: T196077 (duration: 04m 07s)
  • 18:35 ottomata: deploying eventlogging-service-eventbus for T196077
  • 18:32 otto@deploy1001: Started deploy [eventlogging/eventbus@3a5c395]: T196077
  • 18:31 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Redeploy to test (duration: 00m 25s)
  • 18:30 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Redeploy to test
  • 18:01 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Enable osm source (duration: 00m 25s)
  • 18:00 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Enable osm source
  • 17:56 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-intl source (duration: 00m 25s)
  • 17:55 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-intl source
  • 17:53 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-pbf source (duration: 00m 25s)
  • 17:52 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-pbf source
  • 17:51 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view (duration: 00m 25s)
  • 17:51 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view
  • 17:48 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view (duration: 00m 24s)
  • 17:47 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view
  • 17:44 tgr: disabled SUL+wikitech 2FA for MarkAHershberger (T196370)
  • 17:37 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view (duration: 00m 26s)
  • 17:37 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view
  • 17:30 gehel@deploy1001: Finished deploy [wdqs/wdqs@fd534fa]: WDQS: new GUI and blazegraph versions (duration: 08m 25s)
  • 17:30 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test (duration: 02m 54s)
  • 17:27 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test
  • 17:22 gehel@deploy1001: Started deploy [wdqs/wdqs@fd534fa]: WDQS: new GUI and blazegraph versions
  • 17:22 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test (duration: 03m 24s)
  • 17:18 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test
  • 15:09 _joe_: performing rolling restart of authdns servers to pick up ip change for the videoscalers
  • 15:05 herron: gzipped large rotated log files in analytics1003:/var/log/hive to clear icinga disk space warning
  • 14:49 _joe_: restarting low-traffic pybals in eqiad,codfw for adding the videoscaler VIP
  • 14:38 ppchelko@deploy1001: Started restart [cpjobqueue/deploy@c6dc83d]: (no justification provided)
  • 14:35 Amir1: ladsgroup@terbium:~$ foreachwikiindblist large deleteAutoPatrolLogs.php --sleep 2 --check-old
  • 14:34 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010 - T190704
  • 14:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 49s)
  • 14:28 moritzm: installing wireshark security updates
  • 14:27 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=videoscaler,name=eqiad
  • 14:21 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: cluster=videoscaler,service=nginx,dc=codfw,name=mw211.*
  • 14:21 oblivian@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: cluster=videoscaler,service=nginx,dc=codfw
  • 14:20 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=.*,service=mathoid,cluster=kubernetes,name=.*
  • 14:19 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=videoscaler,service=nginx,dc=eqiad
  • 14:10 herron: lithium:~# systemctl restart rsyslog.service
  • 14:10 marostegui: Stop replication on all sanitarium masters to move labsdb1010 to another sanitarium host - T190704
  • 14:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool all sanitariums masters - T190704 (duration: 00m 49s)
  • 14:02 moritzm: installing wireshark security updates
  • 14:02 zeljkof: EU SWAT finished
  • 13:49 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Change bewikiquote logo (T196134) (duration: 00m 49s)
  • 13:44 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgMetaNamespace to "Вікіцытатнік" on bewikiquote (T196230) (duration: 00m 49s)
  • 13:39 anomie: Running populateExternallinksIndex60.php on group 2 for T59176. FYI: this will probably take until next Friday to complete.
  • 13:34 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgProofreadPagePageSeparator to empty string on zhwikisource (T194875) (duration: 00m 49s)
  • 13:27 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgProofreadPagePageSeparator to empty string for jawikisource (T195873) (duration: 00m 49s)
  • 13:16 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Temporarily enable MFMobileMainPageCss in ruwiki (T195905) (duration: 00m 50s)
  • 13:11 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Assign movefile to autoreviewrs and patrollers on zhwiki (T195247) (duration: 00m 52s)
  • 12:49 gehel: restart elastic2001 to enable G1 GC - T156137
  • 12:18 krinkle@deploy1001: Finished deploy [performance/navtiming@b229f75]: (no justification provided) (duration: 00m 05s)
  • 12:18 krinkle@deploy1001: Started deploy [performance/navtiming@b229f75]: (no justification provided)
  • 11:38 akosiaris: rebalance row_A, row_C nodegroups in ganeti01.svc.eqiad.wmnet cluster
  • 10:51 akosiaris: reimage ganeti1004, ganeti1008 to stretch
  • 10:39 _joe_: rolling restart of apache on the jobrunners to pick the changed privatetmp setting, rotating logs
  • 10:14 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
  • 10:13 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
  • 09:39 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - https://phabricator.wikimedia.org/T190704
  • 09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 00m 49s)
  • 09:04 addshore: addshore@terbium:~$ for i in {1..2500}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki
  • 08:56 marostegui: Deploy schema change on db1097:3315 - T191316 T192926 T89737 T195193
  • 08:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 00m 49s)
  • 08:53 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2005 (duration: 00m 50s)
  • 08:10 jynus: restarting icinga due to ongoing check/downtime issues
  • 07:57 marostegui: Stop replication on db2094:3315 for testing
  • 07:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 after alter table (duration: 00m 51s)
  • 07:11 gehel: starting elasticsearch cluster restart on eqiad - T193734
  • 06:18 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2059, db2075 - T190704 (duration: 00m 49s)
  • 06:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 - T190704 (duration: 00m 49s)
  • 05:52 marostegui: Stop replication in sync on db1121 and db2051 - T190704
  • 05:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 - T190704 (duration: 00m 49s)
  • 05:29 marostegui: Deploy schema change on db1082 with replication (this will generate lag on labs for s5) - T191316 T192926 T89737 T195193
  • 05:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 00m 53s)
  • 02:53 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jun 4 02:53:16 UTC 2018 (duration 10m 14s)
  • 02:43 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 14m 33s)

2018-06-03

  • 02:18 andrewbogott: rebooting labservices1001; it seems to have crashed

2018-06-02

  • 07:36 legoktm@deploy1001: Synchronized php-1.32.0-wmf.6/skins/MonoBook/: Temporarily revert responsive MonoBook (T195625) (duration: 00m 58s)

2018-06-01

  • 19:21 ebernhar1son: enable query phase slow logging and increase thresholds for fetch phase slow logging for content/general indices on eqiad and codfw elasticsearch clusters
  • 19:14 mutante: zh.planet - fixed issue with corrupt state file and permissions - updated and using new design as well now
  • 17:34 mutante: deployment.eqiad/codfw DNS names switched from tin to deploy1001
  • 17:06 thcipriani@deploy1001: Synchronized README: noop test of new deployment server (duration: 00m 53s)
  • 16:39 mutante: deploy2001 - also fixing file permissions. files owned by 996 -> mwdeploy, files owned by 997 -> trebuchet
  • 16:21 mutante: deployment server has switched away from tin to deploy1001. set global scap lock on deploy1001, re-enabled puppet and ran puppet, disabled tin as deployment server (T175288)
  • 16:13 herron: enabled new logstash tcp input with TLS enabled for syslogs on port 16514 T193766
  • 15:51 gehel: elasticsearch cluster restart on codfw completed - T193734
  • 15:47 mutante: @deploy1001:/srv/deployment# find . -uid 997 -exec chown trebuchet {} \;
  • 15:41 mutante: root@deploy1001:/srv/mediawiki-staging# find . -uid 996 -exec chown mwdeploy {} \;
  • 15:17 mutante: [deploy1001:~] $ scap pull-master tin.eqiad.wmnet
  • 15:12 mutante: tin umask 022 && echo 'switching deploy servers' > /var/lock/scap-global-lock
  • 15:05 mutante: rsyncing /srv/mediawiki-staging to /srv/mediawiki-staging-before-backup/ on tin as a backup
  • 14:52 mutante: deploy1001 - scap pull
  • 14:30 elukey: killed pt-heartbear-wikimedia after https://gerrit.wikimedia.org/r/436748 on db1107
  • 14:24 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 fully (duration: 01m 02s)
  • 14:08 reedy@tin: Synchronized php-1.32.0-wmf.6/extensions/FlaggedRevs: T196139 (duration: 01m 08s)
  • 13:32 marostegui: Deploy schema change on dbstore1002:s5 - T191316 T192926 T89737 T195193
  • 13:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 (duration: 01m 03s)
  • 10:59 _joe_: disabling puppet on all hosts with role::mediawiki::common while installing mcrouter everywhere
  • 10:35 marostegui: Deploy schema change on db1096:3315 - T191316 T192926 T89737 T195193
  • 10:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 (duration: 01m 03s)
  • 10:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 db1096:3315 (duration: 01m 02s)
  • 10:04 Amir1: ladsgroup@terbium:~$ foreachwikiindblist medium deleteAutoPatrolLogs.php --sleep 2 --check-old
  • 09:53 volans@tin: Finished deploy [debmonitor/deploy@fe8df6e]: Release v0.1.1 (duration: 00m 33s)
  • 09:53 volans@tin: Started deploy [debmonitor/deploy@fe8df6e]: Release v0.1.1
  • 09:37 marostegui: Stop replication in sync on db1113:3315 and db1096:3315 for data checks
  • 09:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 db1096:3315 (duration: 01m 03s)
  • 09:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 with low load (duration: 01m 03s)
  • 08:38 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): reenable v3view on 2004 (duration: 04m 53s)
  • 08:33 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): reenable v3view on 2004
  • 08:30 jynus: reimage db1083
  • 08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 after alter table (duration: 01m 03s)
  • 08:24 jynus: temporarily reducing s7-codfw-master consistency to aliviate lag (binlog_sync, flush_log)
  • 08:22 joal@tin: Finished deploy [analytics/refinery@7a72241]: Regular weekly deploy (duration: 12m 31s)
  • 08:20 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 01m 05s)
  • 08:10 joal@tin: Started deploy [analytics/refinery@7a72241]: Regular weekly deploy
  • 06:15 marostegui: Stop MySQL on db2059 to clone db2075 - T190704
  • 06:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2059 (duration: 00m 56s)
  • 05:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2092 and db2062 in s1 (duration: 00m 59s)
  • 05:27 marostegui: Deploy schema change on db1113:3315 - T191316 T192926 T89737 T195193
  • 05:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 for alter table (duration: 00m 57s)
  • 01:34 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 00m 33s)
  • 01:33 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 01:27 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 34s)
  • 01:24 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 01:03 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 24s)
  • 01:00 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 00:59 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 52s)
  • 00:56 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 00:53 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 26s)
  • 00:51 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 00:48 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 03m 11s)
  • 00:45 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 00:03 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 04m 26s)

2018-05-31

  • 23:59 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 23:38 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 22s)
  • 23:36 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 23:35 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 06m 57s)
  • 23:28 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 23:02 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 22s)
  • 22:59 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
  • 22:51 pnorman@tin: Finished deploy [tilerator/deploy@2a26f1e] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 14s)
  • 22:48 pnorman@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): Redeploy to 2004 to try to reproduce error
  • 22:11 mholloway-shell@tin: Finished deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided) (duration: 12m 46s)
  • 21:58 mholloway-shell@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided)
  • 21:55 jynus: temporarily reducing s4-codfw-master consistency to aliviate lag (binlog_sync, flush_log)
  • 21:00 mutante: dzahn@neodymium:~$ sudo wmf-auto-reimage-host --new phab1002.eqiad.wmnet (T196019)
  • 20:57 mholloway-shell@tin: Finished deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided) (duration: 00m 07s)
  • 20:57 mholloway-shell@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided)
  • 19:47 thcipriani@tin: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.6
  • 19:42 mholloway-shell@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided)
  • 19:40 mholloway-shell@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided)
  • 19:31 mholloway-shell@tin: Started deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided)
  • 19:29 mholloway-shell@tin: Finished deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided) (duration: 00m 06s)
  • 19:29 mholloway-shell@tin: Started deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided)
  • 19:27 mholloway-shell@tin: Started deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided)
  • 19:25 mholloway-shell@tin: Finished deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided) (duration: 00m 05s)
  • 19:25 mholloway-shell@tin: Started deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided)
  • 19:11 mholloway-shell@tin: Finished deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided) (duration: 00m 06s)
  • 19:11 mholloway-shell@tin: Started deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided)
  • 19:04 ariel@tin: Finished deploy [dumps/dumps@038c8b3]: tempdir split into subdirs (duration: 00m 04s)
  • 19:04 ariel@tin: Started deploy [dumps/dumps@038c8b3]: tempdir split into subdirs
  • 16:07 addshore: WikibaseLexeme slot done (7 min overrun)
  • 16:06 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: REVERT Load WikibaseLexeme on testwiki (again) T195615 (duration: 01m 21s)
  • 16:03 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: REVERT Load WikibaseLexeme on group0 T195615 (duration: 01m 21s)
  • 15:59 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Load WikibaseLexeme on group0 T195615 (duration: 01m 18s)
  • 15:52 andrewbogott: rebooting labtestservices2001 to troubleshoot unknown load problems
  • 15:50 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Load WikibaseLexeme on testwiki (again) T195615 (duration: 01m 21s)
  • 15:46 herron: enabling localhost:25 exim smtp listeners in production realm T175361
  • 15:44 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/WikibaseLexeme: Only add repo-specific entity type definition elements in Repo context T195615 (duration: 01m 32s)
  • 15:42 addshore@tin: Synchronized php-1.32.0-wmf.6/extensions/WikibaseLexeme: Only add repo-specific entity type definition elements in Repo context T195615 (duration: 01m 32s)
  • 15:31 jynus: reimage db2079
  • 14:52 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: NOOP patch and revert Load WikibaseLexeme on testwiki (sanity) (duration: 01m 22s)
  • 14:38 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/WikibaseLexeme/src/WikibaseLexemeHooks.php: T195615 Dont run repo only hooks on clients (duration: 01m 20s)
  • 14:36 addshore@tin: Synchronized php-1.32.0-wmf.6/extensions/WikibaseLexeme/src/WikibaseLexemeHooks.php: T195615 Dont run repo only hooks on clients (duration: 01m 24s)
  • 14:27 addshore@tin: Synchronized wmf-config/Wikibase.php: Wikibase.php shift around the loading of WikibaseLexeme (duration: 01m 22s)
  • 14:21 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY gerrit (duration: 01m 21s)
  • 14:20 addshore@tin: Synchronized wmf-config/Wikibase-labs.php: BETA ONLY gerrit (duration: 01m 21s)
  • 14:08 ottomata: beginning restarts of Kafka main-eqiad to enable SSL port - T193778
  • 13:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool sanitarium masters - T190704 (duration: 01m 21s)
  • 13:23 hashar@tin: Synchronized php-1.32.0-wmf.6/vendor: WikibaseLexeme: Encoding problems in labels - T195359 (duration: 03m 12s)
  • 12:54 akosiaris: reimage kubernetes200{3,4}.codfw.wmnet
  • 12:30 marostegui: Stop replication on all sanitarium masters - T190704
  • 11:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool sanitarium masters - T190704 (duration: 01m 22s)
  • 10:45 elukey: removed Pivot from thorium (pivot.wikimedia.org now simply redirects to Turnilo)
  • 09:45 marostegui: Stop MySQL on db2062 to clone db2092
  • 09:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2062 for cloning db2092 (duration: 01m 22s)
  • 09:40 volans: restarting Icinga, issues processing the command file
  • 09:35 jynus: testing icinga alerting
  • 09:28 elukey: reimage druid1006 to Debian Stretch
  • 09:11 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 fully (duration: 01m 21s)
  • 08:42 gehel: power off elastic2018 - T196045
  • 08:39 Amir1: ladsgroup@terbium:~$ mwscript deleteAutoPatrolLogs.php --wiki=commonswiki --sleep 2 --check-old
  • 08:09 gehel: power reset elastic2018
  • 08:08 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=codfw,service=mathoid,cluster=scb,name=scb.*
  • 08:07 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,service=mathoid,cluster=scb,name=scb.*
  • 07:54 akosiaris: reimage kubernetes1004 without swap
  • 07:48 ema: power-cycle elastic2018
  • 07:41 marostegui: Stop Replication on db2066
  • 07:40 akosiaris: re-enable puppet across the fleet
  • 07:30 akosiaris: disable puppet for https://gerrit.wikimedia.org/r/#/c/436468/ merge cross fleet
  • 07:27 marostegui: Deploy schema change on s5 codfw master (db2052) this will generate lag on codfw - T191316 T192926 T89737 T195193
  • 07:24 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2066 after alter table (duration: 01m 22s)
  • 06:38 marostegui: Deploy schema change on db2066 - T191316 T192926 T89737 T195193
  • 06:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2066 for alter table (duration: 01m 21s)
  • 06:30 marostegui: Deploy schema change on db2092:3315 and db2094:3315 - T191316 T192926 T89737 T195193
  • 06:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2084:3315 after alter table (duration: 01m 28s)
  • 06:09 akosiaris: reimage ganeti1003, ganeti1007 to stretch
  • 05:59 elukey: reimage druid1005 to Debian Stretch
  • 05:51 elukey: delete /tmp/scap_l10n_1501525840,scap_l10n_1501525840,l10nstuff,l10nstuff3 from tin to free some space in the root partition (1.9G left)
  • 05:40 elukey: restart pdfrender on scb1001
  • 05:25 marostegui: Deploy schema change on db2084:3315 - T191316 T192926 T89737 T195193
  • 05:24 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2084:3315 for alter table (duration: 01m 27s)
  • 05:13 moritzm: installing python-crypto security updates on trusty
  • 04:19 l10nupdate@tin: ResourceLoader cache refresh completed at Thu May 31 04:19:57 UTC 2018 (duration 14m 48s)
  • 04:18 moritzm: rebooting wtp2001/wtp2015 for microcode updates
  • 04:05 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.6) (duration: 18m 15s)
  • 03:03 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.5) (duration: 16m 18s)
  • 02:09 pnorman@tin: Finished deploy [tilerator/deploy@2a26f1e] (cleartables): Deploy style with fewer fonts (duration: 00m 24s)
  • 02:08 pnorman@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): Deploy style with fewer fonts
  • 02:02 pnorman@tin: Finished deploy [tilerator/deploy@78448de] (cleartables): Deploy style with fewer fonts (duration: 00m 25s)
  • 02:01 pnorman@tin: Started deploy [tilerator/deploy@78448de] (cleartables): Deploy style with fewer fonts
  • 01:39 pnorman@tin: Finished deploy [tilerator/deploy@fad9969] (cleartables): Deploy updated stylesheet (duration: 00m 24s)
  • 01:39 pnorman@tin: Started deploy [tilerator/deploy@fad9969] (cleartables): Deploy updated stylesheet
  • 01:38 pnorman@tin: Finished deploy [tilerator/deploy@78d1b82] (cleartables): Deploy updated stylesheet (duration: 00m 05s)
  • 01:38 pnorman@tin: Started deploy [tilerator/deploy@78d1b82] (cleartables): Deploy updated stylesheet
  • 01:30 pnorman@tin: Finished deploy [tilerator/deploy@78d1b82] (cleartables): Deploy updated stylesheet (duration: 00m 25s)
  • 01:30 pnorman@tin: Started deploy [tilerator/deploy@78d1b82] (cleartables): Deploy updated stylesheet
  • 01:05 mutante: $lang.planet.wikimedia.org is changing software from planet-venus to rawdog

2018-05-30

  • 22:40 thcipriani@tin: Synchronized php-1.32.0-wmf.6/skins/Timeless/includes/TimelessTemplate.php: Fix condition for "emptyPortlet" class T196026 (duration: 01m 21s)
  • 22:22 thcipriani@tin: Synchronized php-1.32.0-wmf.6/extensions/GlobalPreferences/includes/Hooks.php: SWAT: Do not type hint PreferencesFormPreSave hook against PreferencesForm T196023 (duration: 01m 22s)
  • 21:47 thcipriani@tin: rebuilt and synchronized wikiversions files: Group1 to 1.32.0-wmf.6
  • 20:58 thcipriani@tin: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.6
  • 20:45 thcipriani@tin: Finished scap: testwiki to php-1.32.0-wmf.6 and rebuild l10n cache (duration: 114m 02s)
  • 19:38 pnorman@tin: Finished deploy [tilerator/deploy@9e40702]: Restore test2001 test2002 (duration: 00m 27s)
  • 19:37 pnorman@tin: Started deploy [tilerator/deploy@9e40702]: Restore test2001 test2002
  • 19:33 pnorman@tin: Started deploy [tilerator/deploy@bc35971] (cleartables): Use parameterized dbname on test2004
  • 19:31 pnorman@tin: Finished deploy [tilerator/deploy@bc35971] (cleartables): Use parameterized dbname on test2004 (duration: 00m 13s)
  • 19:31 pnorman@tin: Started deploy [tilerator/deploy@bc35971] (cleartables): Use parameterized dbname on test2004
  • 18:51 thcipriani@tin: Started scap: testwiki to php-1.32.0-wmf.6 and rebuild l10n cache
  • 18:05 Amir1: Morning SWAT is done
  • 18:04 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Update Wikidata wgPropertySuggesterDeprecatedIds (duration: 01m 01s)
  • 17:54 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on a bunch of additional wikis (T195263) (duration: 01m 02s)
  • 17:47 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the datetime selector on Sp:Block on all Wikimedia wikis (T193785) (duration: 01m 03s)
  • 17:42 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Enable $wgCookieSetOnIpBlock on test wiki (T195930) (duration: 01m 03s)
  • 16:38 moritzm: upgrading openjdk-7 on conf100[1-3]
  • 16:21 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 with full weight, repool db1089 with low (duration: 01m 02s)
  • 15:49 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2038 after alter table (duration: 01m 01s)
  • 15:43 moritzm: installing spice security updates
  • 15:22 jynus: stop and reimage db1082
  • 15:19 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 with higher load, depool db1082 (duration: 01m 01s)
  • 15:09 marostegui: Deploy schema change on db2038 - T191316 T192926 T89737 T195193
  • 15:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2038 for alter table (duration: 01m 01s)
  • 14:21 herron: changing logstash elasticsearch index prefix for syslogs to 'logstash-syslog' https://gerrit.wikimedia.org/r/431860
  • 14:11 ottomata: enabling SSL port for Kafka main-codfw cluster (take 2 :) ) T193778
  • 13:48 raynor: EU SWAT finished
  • 13:46 pmiazga@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Remove unused PopupsAnonsExperimentalGroupSize config variable (T173952) (duration: 01m 01s)
  • 13:44 pmiazga@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove unused PopupsAnonsExperimentalGroupSize config variable (T173952) (duration: 01m 02s)
  • 13:42 ppchelko@tin: Finished deploy [changeprop/deploy@4503987]: Fix a bug with no content in ores response (duration: 01m 38s)
  • 13:40 ppchelko@tin: Started deploy [changeprop/deploy@4503987]: Fix a bug with no content in ores response
  • 13:39 ppchelko@tin: Finished deploy [eventstreams/deploy@14e0b03]: Recreate config with new puppet and restart service T167180 (duration: 02m 07s)
  • 13:37 ppchelko@tin: Started deploy [eventstreams/deploy@14e0b03]: Recreate config with new puppet and restart service T167180
  • 13:32 elukey: reboot analytics1002 (Hadoop master node standby) to pick up new cpu microcode
  • 13:29 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 with low load (duration: 00m 59s)
  • 13:27 ppchelko@tin: Finished deploy [changeprop/deploy@43310d4]: Emit revision-score event T167180 (duration: 01m 32s)
  • 13:26 ppchelko@tin: Started deploy [changeprop/deploy@43310d4]: Emit revision-score event T167180
  • 13:24 akosiaris: emptying ganeti1003, ganeti1007 for stretch upgrade
  • 13:22 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add WMDS support question feed to mediawikiwiki RSS whitelist (T185087) (duration: 01m 01s)
  • 13:12 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable detection of changes in moved paragraphs on most wikis (T195375) (duration: 01m 02s)
  • 13:10 marostegui: Deploy schema change on dbstore2001:3315 - T191316 T192926 T89737 T195193
  • 13:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2089:3315 after alter table (duration: 01m 02s)
  • 12:40 akosiaris: reimage ganeti1002, ganeti1005 as stretch
  • 12:35 elukey: reboot analytics1029,1042,1070 to pick up the new cpu-microcode
  • 12:34 gehel: starting elasticsearch cluster restart on codfw - T193734
  • 11:04 jynus: stop and reimage db1089
  • 10:44 marostegui: Deploy schema change on db2089:3315 - T191316 T192926 T89737 T195193
  • 10:43 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2089:3315 for alter table (duration: 01m 01s)
  • 10:42 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 (duration: 01m 01s)
  • 10:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2059 after alter table (duration: 01m 02s)
  • 09:47 marostegui: Deploy schema change on db2059 - T191316 T192926 T89737 T195193
  • 09:33 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2059 for alter table (duration: 01m 02s)
  • 09:24 XioNoX: peering setup with RIPE RIS in eqsin
  • 08:47 elukey: reimage druid1004 to Debian Stretch
  • 08:38 marostegui: Stop and reboot db2094 and db2095 for testing - T190704
  • 08:22 akosiaris: upgrade python3-tornado on scb1001 and restart apertium-apy. T194883
  • 07:38 Nikerabbit: running refresh-translatable-pages.php for wikis having Translate
  • 07:14 moritzm: installing git security updates
  • 06:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool sanitariums masters for s1, s3, s5, s8 - T190704 (duration: 01m 01s)
  • 06:43 marostegui: Stop db1087 and db2045 in sync - T190704
  • 06:31 marostegui: Stop db1082 and db2052 in sync - T190704
  • 06:25 marostegui: Stop db1077 and db2043 in sync - T190704
  • 06:17 elukey: reimage druid1001 to Debian stretch
  • 06:11 marostegui: Stop db1106 and db2048 in sync - T190704
  • 06:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool sanitariums masters for s1, s3, s5, s8 - T190704 (duration: 01m 00s)
  • 05:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool sanitariums masters for s2, s4, s6, s7 - T190704 (duration: 01m 00s)
  • 05:50 elukey: restart Kafka mirror maker on kafka10[12-23] - failures to consume after rebalance
  • 05:41 marostegui: Stop db1079 and db2040 in sync - T190704
  • 05:31 marostegui: Stop db1085 and db2039 in sync - T190704
  • 05:23 marostegui: Stop db1074 and db2035 in sync - T190704
  • 05:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool sanitariums masters for s2, s4, s6, s7 - T190704 (duration: 01m 03s)
  • 05:06 marostegui: Deploy schema change on s1 primary master db1052 - T188299
  • 03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Wed May 30 03:15:44 UTC 2018 (duration 14m 3s)
  • 03:01 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.5) (duration: 12m 57s)
  • 01:08 mutante: built added rawdog_2.22-1-wmf1 to apt.wikimedia.org, upgraded rawdog on planet2001. Unpacking rawdog (2.22-1-wmf1) over (2.22-1) (T180498)
  • 00:45 pnorman@tin: Finished deploy [tilerator/deploy@bc35971]: Use parameterized dbname on test2004 (duration: 00m 21s)
  • 00:45 pnorman@tin: Started deploy [tilerator/deploy@bc35971]: Use parameterized dbname on test2004
  • 00:41 pnorman@tin: Finished deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004 (duration: 00m 12s)
  • 00:41 pnorman@tin: Started deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004
  • 00:28 pnorman@tin: Finished deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004 (duration: 00m 05s)
  • 00:28 pnorman@tin: Started deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004
  • 00:25 pnorman@tin: Finished deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004 (duration: 02m 17s)
  • 00:22 pnorman@tin: Started deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004
  • 00:22 pnorman@tin: Finished deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004 (duration: 00m 21s)
  • 00:21 pnorman@tin: Started deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004

2018-05-29

  • 23:10 pnorman@tin: Finished deploy [kartotherian/deploy@2b75c93]: Deploy test of stretch build of Kartotherian to test2004 (duration: 00m 23s)
  • 23:10 pnorman@tin: Started deploy [kartotherian/deploy@2b75c93]: Deploy test of stretch build of Kartotherian to test2004
  • 22:09 mutante: boron - apt-get build-dep rawdog (installed libtidy5 python-feedparser python-tidylib
  • 21:31 thcipriani@tin: rebuilt and synchronized wikiversions files: All wikis to 1.32.0-wmf.5
  • 21:06 thcipriani@tin: Synchronized php-1.32.0-wmf.5/extensions/VisualEditor/lib/ve: SWAT: Update VE core submodule to wmf/1.32.0-wmf.5 HEAD (9032a90ca) T195514 (duration: 01m 23s)
  • 20:38 volans@tin: Finished deploy [debmonitor/deploy@361c94a]: Initial sync (3) (duration: 50m 13s)
  • 19:48 volans@tin: Started deploy [debmonitor/deploy@361c94a]: Initial sync (3)
  • 19:16 thcipriani: cutting branch for wmf.6, will not deploy wmf.6 as wmf.5 is not currently on group2 as the train is blocked on T195514 which is blocked on T195868 which is blocked on T195906
  • 18:56 volans@tin: Finished deploy [debmonitor/deploy@fd06bd3]: Initial sync (2) (duration: 01m 42s)
  • 18:55 volans@tin: Started deploy [debmonitor/deploy@fd06bd3]: Initial sync (2)
  • 18:22 volans@tin: Finished deploy [debmonitor/deploy@e2efb6b]: Initial sync (duration: 00m 35s)
  • 18:21 volans@tin: Started deploy [debmonitor/deploy@e2efb6b]: Initial sync
  • 18:04 arlolra@tin: Finished deploy [parsoid/deploy@e87f54d]: Reverting Parsoid to fd49ab4 (duration: 02m 29s)
  • 18:01 arlolra@tin: Started deploy [parsoid/deploy@e87f54d]: Reverting Parsoid to fd49ab4
  • 17:59 ottomata: beginning rolling restarts of kafka main-codfw to enable SSL listener
  • 17:36 arlolra@tin: Finished deploy [parsoid/deploy@e87f54d]: Updating Parsoid to bf3a2fd2 (duration: 09m 48s)
  • 17:32 moritzm: repooled mw2182 (was down for hardware maintenance)
  • 17:26 arlolra@tin: Started deploy [parsoid/deploy@e87f54d]: Updating Parsoid to bf3a2fd2
  • 17:25 foks: Removing 2FA - T187312
  • 17:12 bsitzmann@tin: Finished deploy [mobileapps/deploy@ac4c6be]: Update mobileapps to b2fb793 (T192664) (duration: 06m 28s)
  • 17:05 bsitzmann@tin: Started deploy [mobileapps/deploy@ac4c6be]: Update mobileapps to b2fb793 (T192664)
  • 16:59 elukey: roll restart of kafka mirror maker on kafka-jumbo100* to pick up the new zookeeper settings
  • 16:56 XioNoX: bounced analytics1031 switchport to fix weird issue of that host not being able to receive traffic from analytics1001
  • 16:44 elukey: roll restart of kafka mirror maker on kafka100[1-3] to pick up new zk settings
  • 16:32 thcipriani: upgrading blubber to 0.4.0 for integration machines
  • 15:57 addshore: really done with wb_terms related syncs now
  • 15:52 addshore@tin: Synchronized wmf-config/Wikibase.php: Revert - Dont load PropertySuggester T195520 (duration: 01m 19s)
  • 15:48 elukey: roll restart kafka on kafka-jumbo* to pick up new zookeeper settings
  • 15:45 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/PropertySuggester: Use CirrusSearch for PropertySuggester (duration: 01m 21s)
  • 15:29 addshore: Wikibase - Re enable wb_terms things window - 2 more patches inbound...
  • 15:27 elukey: restart hadoop yarn/hdfs daemons to pick up the new zookeeper settings
  • 15:11 addshore: Wikibase - Re enable wb_terms things window done
  • 14:57 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: TermSqlIndex::getMatchingTerms actually execute select (duration: 02m 19s)
  • 14:57 marostegui: Move s6 topology back to its normal status
  • 14:49 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: TermSqlIndex::getMatchingTerms actually execute select (duration: 02m 18s)
  • 14:32 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: Re add TermSqlIndex::getMatchingTerms select, but dont call (duration: 02m 18s)
  • 14:29 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: Re add TermSqlIndex::getMatchingTerms select, but dont call (duration: 02m 13s)
  • 14:24 elukey: roll restart kafka on kafka100[1-3] (job queues) to pick up the new zookeeper settings
  • 14:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool all databases in row C - T187962 (duration: 01m 19s)
  • 14:13 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: track all wb_terms table access via statsd (duration: 02m 19s)
  • 14:13 gehel: comleted rolling restart of relforge for plugin upgrade - T193734
  • 14:10 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: track all wb_terms table access via statsd (duration: 02m 21s)
  • 14:07 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/CentralNotice: Convert numerical URL parameters to numbers for AndyRussG (was left on tin) (duration: 01m 25s)
  • 14:03 elukey: swap zookeeper from conf1003 to conf1006
  • 13:56 XioNoX: rolling back ns0 and ping1001 redirects - T187962
  • 13:47 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create 2 extra namespaces for bdwikimedia (T195700) (duration: 01m 39s)
  • 13:42 moritzm: upgrading remaining job runners in eqiad to hhvm-wikidiff 1.7.0
  • 13:39 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Revert "Temp rate limit for arwiki due to mass vandalism""" (T192668) (duration: 01m 51s)
  • 13:32 volans: restarted ircecho
  • 13:28 volans: puppet run on failed hosts completed
  • 13:25 moritzm: powered down mw2182 for hardware diagnosis
  • 13:19 gehel: rolling restart of relforge for plugin upgrade - T193734
  • 13:16 volans: running puppet on failed only hosts
  • 13:12 volans: stopped ircecho temporarily
  • 12:54 moritzm: installing xdg-utils security updates
  • 11:21 marostegui: Restar db1125 mysql - T195595
  • 11:14 moritzm: upgrading snapshot hosts to hhvm-wikidiff 1.7.0 (HHVM is unused, just for completeness)
  • 11:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Disable read only on s6 T194939 T187962 (duration: 01m 37s)
  • 11:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Enable read only on s6 T194939 T187962 (duration: 01m 35s)
  • 10:55 XioNoX: Eqiad row C server move starting - T187962
  • 10:53 XioNoX: Eqiad row C server move starting
  • 10:35 moritzm: upgrading mw1308-mw1311 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 10:09 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch all jobs to EventBus file 2/2 - T190327 T195500 (duration: 01m 47s)
  • 10:06 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus file 1/2 - T190327 T195500 (duration: 01m 39s)
  • 10:05 ppchelko@tin: Finished deploy [cpjobqueue/deploy@c6dc83d]: Enable all jobs apart from exceptions for everything. T190327 (duration: 00m 58s)
  • 10:04 ppchelko@tin: Started deploy [cpjobqueue/deploy@c6dc83d]: Enable all jobs apart from exceptions for everything. T190327
  • 09:20 XioNoX: redirect ns0 to baham - T187962
  • 09:16 XioNoX: disable ping1001 redirect - T187962
  • 09:13 marostegui: Downtime s6 replicas for 4 hours - T195595
  • 09:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool all databases in row C - T187962 (duration: 01m 35s)
  • 09:05 moritzm: upgrading labweb servers to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 08:40 jynus: performing topology changes on s6 ahead of a possible failover
  • 08:24 moritzm: upgrading remaining API servers in eqiad to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 07:56 moritzm: upgrading mw1276-mw1290 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 07:49 elukey: reimage druid1002 to debian stretch
  • 07:47 gilles@tin: Synchronized wmf-config/InitialiseSettings.php: T187299 Launch performance survey on ruwiki (duration: 01m 50s)
  • 07:26 moritzm: upgrading remaining app servers in eqiad to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 06:52 elukey: roll restart hadoop master daemons to pick up the new zookeeper settings
  • 05:20 marostegui: Restart MySQL on db2045 (s8 codfw master) - T195598
  • 05:14 marostegui: Stop MySQL on db2094 and db2095 for testing - T190704
  • 04:12 l10nupdate@tin: ResourceLoader cache refresh completed at Tue May 29 04:12:10 UTC 2018 (duration 14m 32s)
  • 03:57 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.5) (duration: 14m 29s)
  • 02:59 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 13m 18s)

2018-05-28

  • 20:14 twentyafterfour: Test failures on https://gerrit.wikimedia.org/r/#/c/435825/ are preventing deployment of the fix for a critical deployment blocker (see T195514) 1.32.0-wmf.5 still blocked refs T191051
  • 20:10 twentyafterfour: train still held up by test failures: https://gerrit.wikimedia.org/r/#/c/435825/
  • 20:02 elukey: restart kafka on kafka1003 as attempt to solve the under-replicated partitions warning
  • 19:22 twentyafterfour@tin: Synchronized php-1.32.0-wmf.5/extensions/CentralNotice/: sync wmf.5 CentralNotice for AndyRussG (duration: 01m 25s)
  • 19:12 elukey: roll restart of kafka-mirror maker (main eqiad -> jumbo) on kafka-jumbo* for zookeeper conf updates
  • 19:07 twentyafterfour: attempting to get the wmf.5 train back on track. Deploying a fix for T195514 (https://gerrit.wikimedia.org/r/c/435292/) to unblock T191051
  • 18:16 elukey: restart kafka mirror maker on kafka1012->14 - failed after the last round of kafka restarts
  • 17:26 elukey: roll restart of kafka on kafka-jumbo* to pick up the new zookeeper settings
  • 17:20 gehel@tin: Finished deploy [wdqs/wdqs@0e40344]: WDQS updater and GUI (duration: 08m 59s)
  • 17:19 elukey: restart kafka on kafka1012->23 to pick up the new zookeeper settings
  • 17:11 gehel@tin: Started deploy [wdqs/wdqs@0e40344]: WDQS updater and GUI
  • 17:08 moritzm: upgrading mwdebug servers in codfw to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 16:48 moritzm: upgrading codfw video scalers to hhvm-wikidiff 1.7.0
  • 16:31 elukey: roll restart kafka on kafka100[1-3] to pick up new zookeeper settings
  • 16:21 elukey: zookeeper cluster restart completed (main-eqiad / conf1*)
  • 16:18 elukey: stop and mask zookeeper on conf1002
  • 16:16 elukey: restart prometheus-burrow-exporter on kafkamon*
  • 16:12 moritzm: upgrading job runners in codfw to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 16:02 marostegui: Stop MySQL on db2095 for testing - T190704
  • 15:59 elukey: swap zookeeper from conf1002 to conf1005
  • 15:56 marostegui: Reboot db1124 and db1125 for more testing - T190704
  • 15:49 _joe_: uploading cergen 0.2.3
  • 14:44 Amir1: EU SWAT is done
  • 14:39 ladsgroup@tin: Synchronized php-1.32.0-wmf.4/extensions/ArticlePlaceholder: Add config variable to disable SearchHookHandler (T195753) (duration: 01m 21s)
  • 14:34 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Disable search integration with Article Placeholder temporarily (T195753) (duration: 01m 20s)
  • 14:29 ladsgroup@tin: Synchronized php-1.32.0-wmf.5/extensions/ArticlePlaceholder: Add config variable to disable SearchHookHandler (T195753) (duration: 01m 18s)
  • 14:23 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Disable Special:ItemDisambiguation in Wikidata (T195756) (duration: 01m 20s)
  • 14:21 ema: cp1045,cp2001,cp3007,cp5001: reboot with intel-microcode T127825
  • 14:00 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable "File mover" flag on zh.wikipedia (T195247) (duration: 01m 19s)
  • 13:52 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: New protection level on the Hungarian Wikipedia - trusted (T194568) (duration: 01m 20s)
  • 13:40 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable $wgUseRCPatrol on azwiki"" (T194389) (duration: 01m 20s)
  • 13:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use uploaded HD logos for yiwikisource (T193562) (duration: 01m 19s)
  • 13:28 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Upload new logos for yiwikisource (T193562) (duration: 01m 19s)
  • 13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable template editor group on newiki (T195557) (duration: 01m 21s)
  • 13:13 volans: restarted pdfrender on scb1002
  • 13:03 moritzm: upgrading app servers in codfw to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 12:09 arturo: T194665 aborrero@install1002:~$ sudo -i reprepro --noskipold -C 'thirdparty/mono-project-stretch' update stretch-wikimedia
  • 12:08 arturo: T194665 aborrero@install1002:~$ sudo -i reprepro --noskipold -C 'thirdparty/mono-project-jessie' update jessie-wikimedia
  • 12:05 arturo: aborrero@install1002:~$ sudo -i reprepro --noskipold -C 'thirdparty/mono-project-trusty' update trusty-wikimedia
  • 11:31 ema: cp1008: reboot with intel-microcode T127825
  • 11:27 moritzm: upgrading API servers in codfw to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 11:10 marostegui: Stop MySQL on db2092 to copy its content to db2094 - T190704
  • 10:36 moritzm: upgrading mw1266-mw1275 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 10:14 moritzm: upgrading eqiad video scalers to hhvm-wikidiff 1.7.0
  • 10:09 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 20s)
  • 10:08 jdrewniak@tin: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 21s)
  • 09:07 moritzm: upgrading mw1299-mw1306 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 09:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 after alter table (duration: 01m 20s)
  • 09:04 marostegui: Deploy schema change on s1 primary master (db1052) - T191519
  • 09:00 marostegui: Deploy schema change on db1052 (s1 primary master) - T190148
  • 08:27 moritzm: upgrading mw1221-mw1235 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 07:40 XioNoX: enable backup tunnel routing between cr2-ulsfo and cr1-eqdfw - T195584
  • 07:31 marostegui: Deploy schema change on db1083 - T190148 T191519 T188299
  • 07:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 for alter table (duration: 01m 20s)
  • 07:29 moritzm: upgrading mw1238-mw1258 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
  • 07:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 01m 22s)
  • 07:10 moritzm: uploaded hhvm-wikidiff2 1.7.0 (source package name php-wikidiff2) to apt.wikimedia.org
  • 06:44 moritzm: installing ruby-loofah security updates
  • 06:36 elukey: reimage druid1003 to Debian Stretch (Analytics cluster, backend for Pivot/Turnilo)
  • 05:25 marostegui: Stop MySQL on db2075 to copy its content to db2095 - T190704
  • 05:23 marostegui: Deploy schema change on db1106 with replication, this will generate lag on labs - T190148 T191519 T188299
  • 05:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 01m 24s)
  • 03:14 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 14m 30s)

2018-05-26

  • 21:47 reedy@tin: Synchronized composer.json: (no justification provided) (duration: 01m 19s)
  • 21:45 reedy@tin: Synchronized multiversion/: multiversion (duration: 01m 21s)
  • 21:42 reedy@tin: Synchronized vendor/: canhasvendor (duration: 01m 46s)
  • 19:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 after alter table (duration: 01m 21s)
  • 14:25 marostegui: Add tmp1 index back on db1101:3318 - T194273
  • 14:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 for alter table (duration: 01m 07s)
  • 14:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 after alter table (duration: 01m 20s)
  • 09:56 marostegui: Add tmp1 index back on db1087 (sanitarium master), this will generate lag on labsdb hosts - T194273
  • 09:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 for alter table (duration: 01m 20s)
  • 09:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 after alter table (duration: 01m 21s)
  • 05:21 marostegui: Add tmp1 index back on db1099:3318 - T194273
  • 05:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 for alter table (duration: 01m 21s)
  • 05:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 01m 22s)

2018-05-25

  • 22:57 mutante: apt.wikimedia.org - import jenkins-debian-glue_0.18.4-wmf3 for jessie-wikimedia (T193910)
  • 21:27 legoktm@tin: Synchronized php-1.32.0-wmf.5/skins/MonoBook/: Temporarily remove responsive support (T195625) (duration: 01m 21s)
  • 20:33 mutante: LDAP: added user wmde-leszek to group 'nda' (T195358)
  • 17:46 XenoRyet: updated civicrm from 4d797fc592 to 0b97f1f5b2
  • 15:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 01m 20s)
  • 13:47 akosiaris: repool ulsfo, links have been stable for quite a few hours
  • 13:27 marostegui: Deploy schema change on db1119 - https://phabricator.wikimedia.org/T190148 https://phabricator.wikimedia.org/T191519 https://phabricator.wikimedia.org/T188299
  • 13:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 01m 20s)
  • 13:15 marostegui: Add indexes back on s8 codfw primary master (db2045) this will generate lag on codfw - T194273
  • 13:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 01m 20s)
  • 12:28 moritzm: fixed dpkg installation state on mx2001
  • 11:32 akosiaris: switch to SSH RSA 2048 bit keys for eqiad ganeti intracluster communication
  • 11:22 akosiaris: upgrade eqiad ganeti cluster to ganeti 2.15.2-7+deb9u1~bpo8+1
  • 11:21 akosiaris: rebalance row_B codfw ganeti nodegroup. Cluster is now fully upgraded to stretch
  • 11:18 akosiaris: powercycling ms-be1034, box is unresposive, tons of logs "sd 0:1:0:1: rejecting I/O to offline device"
  • 10:37 XioNoX: test force mtu 1400 between cp1074 and cp3039 - T195365
  • 09:20 marostegui: Deploy schema change on db1105:3311 - T190148 T191519 T188299
  • 09:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 01m 20s)
  • 09:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 01m 19s)
  • 09:10 marostegui@tin: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 09:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 01m 20s)
  • 08:44 marostegui: Stop MySQL on db1120 to transfer its content to db1125 - T190704
  • 08:39 marostegui: Add tmp1 back on db1092 - https://phabricator.wikimedia.org/T194273
  • 08:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 20s)
  • 08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 01m 20s)
  • 08:26 gilles@tin: Synchronized wmf-config/InitialiseSettings.php: T187299 Launch performance survey on frwiki (duration: 01m 22s)
  • 07:05 jynus: stop db1117:m2 to clone it to db1065
  • 06:57 marostegui: Deploy schema change on db1114 - T190148 T191519 T188299
  • 06:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 01m 20s)
  • 06:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 01m 20s)
  • 06:37 jynus: reimage db1065 after raid rebuild
  • 05:57 marostegui: Add tmp1 index back on dbstore1002 - T194273
  • 05:25 marostegui: Stop MySQL on db1116 to copy its content to db1124 - T190704
  • 05:23 marostegui: Deploy schema change on db1089 - T190148 T191519 T188299
  • 05:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 01m 20s)
  • 05:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 01m 21s)
  • 05:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 01m 20s)
  • 05:05 marostegui: Add tmp1 index back to db1109 - T194273
  • 04:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 01m 21s)
  • 03:10 papaul: OS install on db209[4-5]

2018-05-24

  • 23:13 ladsgroup@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: Dumps: Allow several --entity-type arguments (T195420) (duration: 02m 25s)
  • 22:21 ladsgroup@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Log the query that would hit wb_terms. (T195520) (duration: 01m 20s)
  • 22:11 ladsgroup@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Log the query that would hit wb_terms. (T195520) (duration: 01m 21s)
  • 20:58 marostegui: Add tmp1 index back on db1104
  • 20:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1104 for alter table (duration: 01m 20s)
  • 20:56 legoktm@tin: Synchronized php-1.32.0-wmf.5/extensions/VisualEditor/: no-op, sync with git state (duration: 01m 21s)
  • 20:54 legoktm@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: no-op, sync with git state (duration: 01m 20s)
  • 20:41 legoktm@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: no-op, sync with git state (duration: 01m 20s)
  • 20:17 legoktm@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: wmf.5 this time (duration: 01m 19s)
  • 20:14 legoktm@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Add debug logging (duration: 01m 19s)
  • 20:11 legoktm@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Disable TermSqlIndex::getMatchingTerms (duration: 01m 20s)
  • 19:49 ejegg: updated payments-wiki from c81e25f8d3 to 43989ebc96
  • 19:44 reedy@tin: Synchronized wmf-config/Wikibase.php: Disable PropSuggester (duration: 01m 21s)
  • 19:30 twentyafterfour@tin: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 19:14 twentyafterfour: Starting the MediaWiki train for Thursday May 24, today I will be deploying wmf/1.32.0-wmf.5 to all wikis
  • 18:31 mutante: netmon1002, netmon2001: systemctl mask uwsgi; systemctl reset-failed - to fix Icinga alert about broken DPKG since last netbox deploy and to match existing status on labtestweb2001
  • 18:21 XioNoX: repool esams
  • 18:01 no_justification: gerrit restarting on cobalt, back soon
  • 17:50 mutante: mwdebug1001 - apt-get remove to clean up packages in "non ii" states
  • 16:59 anomie: Running populateExternallinksIndex60.php on group 1 for T59176
  • 16:50 XioNoX: depooled esams - investigating issues
  • 16:31 anomie: Running deduplicateArchiveRevId.php on group 2 for T193180
  • 16:28 jynus: manually failover the backup host for m2 to db1117:3322
  • 16:22 mutante: tin apt-get clean saved 7% disk space on / - fixing disk space alert
  • 16:19 marostegui: Deploy schema change on db1099:3311 - T191519 T188299 T190148
  • 16:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 01m 13s)
  • 16:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 01m 17s)
  • 16:15 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase/repo/maintenance/dispatchChanges.php: Dont use WikibaseRepo class in dispatchChanges constructor (duration: 10m 28s)
  • 16:13 mutante: deploy2001 - let mwdeploy own .~tmp~ in /srv/mediawiki-staging
  • 15:48 cmjohnson: swapped failed disk 0 db1065
  • 15:06 moritzm: installing glibc updates from stretch point release
  • 14:57 marostegui: Deploy schema change on dbstore1001:s1 - T191519 T188299 T190148
  • 14:56 papaul: shutting down elastic2020 for maintenance
  • 14:51 moritzm: upgrading mwdebug servers in eqiad to wikidiff 1.7.0
  • 14:09 akosiaris: empty ganeti2004 for stretch reimage
  • 14:00 mutante: deploy2001: scap pull to sync. then add as scap master and host (gerrit:433616)
  • 13:37 akosiaris: rebalance row_A codfw ganeti nodegroup. Fully upgrade to stretch now
  • 13:33 marostegui: Deploy schema change on db1067 - T191519 T188299 T190148
  • 13:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 01m 00s)
  • 13:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 56s)
  • 13:27 anomie: Running deduplicateArchiveRevId.php on group 1 for T193180
  • 13:25 moritzm: upgrading mw1262-mw1265 (canary hosts) to wikidiff 1.7.0
  • 13:13 moritzm: upgrading mw1261 (canary host) to wikidiff 1.7.0
  • 11:31 moritzm: rebooting mw1261, mw1276, mw1319, mw1312, mw1258, mw1221 to use Intel microcode updates
  • 11:19 marostegui: Deploy schema change on db1080 - T191519 T188299 T190148
  • 11:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 01m 08s)
  • 11:10 marostegui: Deploy schema change on dbstore1002:s1 - T191519 T188299 T190148
  • 11:05 XioNoX: GTT work on eqiad-esams link starting soon
  • 10:55 krinkle@tin: Synchronized php-1.32.0-wmf.4/includes/resourceloader/ResourceLoaderUserModule.php: T195380 (duration: 01m 08s)
  • 10:53 Krinkle: Unexpected dirty git status at tin:/srv/mediawiki-staging/php-1.32.0-wmf.4/extensions/JADE (1 file is locally deleted, but not committed)
  • 10:51 krinkle@tin: Synchronized php-1.32.0-wmf.5/includes/resourceloader/ResourceLoaderUserModule.php: T195380 (duration: 01m 08s)
  • 09:55 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 2/2 - T190327 (duration: 01m 06s)
  • 09:55 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b537fa1]: Switch all non-special jobs for everything except wikipedia, commons and wikidata T190327 (duration: 00m 42s)
  • 09:54 ppchelko@tin: Started deploy [cpjobqueue/deploy@b537fa1]: Switch all non-special jobs for everything except wikipedia, commons and wikidata T190327
  • 09:53 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 1/2, take #2 - T190327 (duration: 01m 08s)
  • 09:37 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 1/2 - T190327 (duration: 01m 09s)
  • 09:33 pnorman@tin: Finished deploy [kartotherian/deploy@9fc09ef]: Do a fresh deploy of Kartotherian to production (duration: 02m 35s)
  • 09:31 pnorman@tin: Started deploy [kartotherian/deploy@9fc09ef]: Do a fresh deploy of Kartotherian to production
  • 09:27 pnorman@tin: Finished deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004 (duration: 00m 22s)
  • 09:27 pnorman@tin: Started deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004
  • 09:26 pnorman@tin: Finished deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004 (duration: 00m 23s)
  • 09:25 pnorman@tin: Started deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004
  • 09:25 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f66dacb]: Correctly commit offsets for multi-topic rules (duration: 00m 49s)
  • 09:24 ppchelko@tin: Started deploy [cpjobqueue/deploy@f66dacb]: Correctly commit offsets for multi-topic rules
  • 08:55 pnorman@tin: Finished deploy [tilerator/deploy@9e40702] (cleartables): Deploy scap fixes to cleartables map test server in verbose mode (duration: 00m 13s)
  • 08:55 pnorman@tin: Started deploy [tilerator/deploy@9e40702] (cleartables): Deploy scap fixes to cleartables map test server in verbose mode
  • 08:55 pnorman@tin: Finished deploy [tilerator/deploy@9e40702] (cleartables--force): Deploy scap fixes to cleartables map test server in verbose mode (duration: 00m 05s)
  • 08:55 pnorman@tin: Started deploy [tilerator/deploy@9e40702] (cleartables--force): Deploy scap fixes to cleartables map test server in verbose mode
  • 08:51 pnorman@tin: Finished deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode (duration: 00m 13s)
  • 08:51 pnorman@tin: Started deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode
  • 08:47 pnorman@tin: Finished deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode (duration: 00m 05s)
  • 08:47 pnorman@tin: Started deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode
  • 08:45 marostegui: Deploy schema change on s1 codfw primary master (db2048), this will generate lag on codfw - T191519 T188299 T190148
  • 08:45 pnorman@tin: Finished deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode (duration: 00m 01s)
  • 08:45 pnorman@tin: Started deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode
  • 08:40 ayounsi@tin: Finished deploy [netbox/deploy@ac54feb]: Adding service name in scap.cfg (duration: 00m 33s)
  • 08:39 ayounsi@tin: Started deploy [netbox/deploy@ac54feb]: Adding service name in scap.cfg
  • 08:35 pnorman@tin: Finished deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server go 3 (duration: 00m 23s)
  • 08:35 pnorman@tin: Started deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server go 3
  • 08:32 gilles@tin: Synchronized wmf-config/InitialiseSettings.php: T187299 Launch performance survey on cawiki and enwikivoyage (duration: 01m 08s)
  • 08:21 gilles: Deployment of cawiki and enwikivoyage performance survey
  • 08:16 jynus: stop db2037 to clone it to db2078 and upgrade
  • 08:05 marostegui: Deploy schema change on s8 primary master (db1071) - T191519 T188299 T190148 T194270
  • 08:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 01m 09s)
  • 07:48 jynus: stop db2042 to clone it to db2078 and upgrade
  • 07:36 marostegui: Deploy schema change on db1109 - T191519 T188299 T190148 T194270
  • 07:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 01m 08s)
  • 07:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 01m 29s)
  • 07:17 moritzm: installing procps security updates
  • 06:47 marostegui: Deploy schema change on db1104 - T191519 T188299 T190148 T194270
  • 06:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1104 for alter table (duration: 01m 08s)
  • 06:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 after alter table (duration: 01m 08s)
  • 06:24 moritzm: installing remaining curl security updates in eqiad
  • 06:17 marostegui: Deploy schema change on db1087, this will generate lag on labs on s8 - T191519 T188299 T190148 T194270
  • 06:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 for alter table (duration: 01m 09s)
  • 05:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 01m 09s)
  • 04:04 l10nupdate@tin: ResourceLoader cache refresh completed at Thu May 24 04:04:35 UTC 2018 (duration 7m 16s)
  • 03:57 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.5) (duration: 13m 29s)
  • 03:06 mutante: added wmde-fisch to LDAP group nda (T195223)
  • 03:01 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 12m 48s)
  • 00:15 ebernhardson@tin: Synchronized php-1.32.0-wmf.4/extensions/CirrusSearch/includes/Job/CheckerJob.php: Drop cirrus checker job metastore transition check (duration: 01m 08s)
  • 00:14 ebernhardson@tin: Synchronized php-1.32.0-wmf.5/extensions/CirrusSearch/includes/Job/CheckerJob.php: Drop cirrus checker job metastore transition check (duration: 01m 08s)
  • 00:10 ebernhardson: upgrade cirrussearch metastore to 1.0 on eqiad and codfw
  • 00:09 ebernhardson@tin: Synchronized php-1.32.0-wmf.5/extensions/CirrusSearch/: SWAT: Convert cirrus metastore to single type (duration: 01m 24s)
  • 00:07 ebernhardson@tin: Synchronized php-1.32.0-wmf.4/extensions/CirrusSearch/: SWAT: Convert cirrus metastore to single type (duration: 01m 24s)
  • 00:00 ebernhardson@tin: Synchronized php-1.32.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWSaveDialog.js: SWAT: T195323: MWSaveDialog: Fix typo in no-categories branch (duration: 01m 07s)

2018-05-23

  • 23:49 ebernhardson@tin: Synchronized php-1.32.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWPopupTool.js: SWAT: Fix typo in API call for version number help (duration: 01m 08s)
  • 23:45 ebernhardson@tin: Synchronized php-1.32.0-wmf.5/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWSaveDialog.js: SWAT: T195323: MWSaveDialog: Fix typo in no-categories branch (duration: 01m 08s)
  • 23:42 ebernhardson@tin: Synchronized php-1.32.0-wmf.4/extensions/MobileFrontend/resources/mobile.editor.common/editor.less: SWAT: T194832: Fix layout of editor switcher dropdown (duration: 01m 08s)
  • 23:10 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T152296: Remove dangerous unused botadmin group at mlwik{tionary|isource} (duration: 01m 10s)
  • 21:29 mutante: arming keyholder on deploy2001
  • 20:57 mutante: rebooting deploy2001
  • 20:56 arlolra@tin: Finished deploy [parsoid/deploy@de18a58]: Reverting Parsoid deploy (duration: 02m 37s)
  • 20:53 arlolra@tin: Started deploy [parsoid/deploy@de18a58]: Reverting Parsoid deploy
  • 20:52 pnorman@tin: Finished deploy [tilerator/deploy@63617a9]: Deploy scap fixes to cleartables map test server go 2 (duration: 00m 23s)
  • 20:52 pnorman@tin: Started deploy [tilerator/deploy@63617a9]: Deploy scap fixes to cleartables map test server go 2
  • 20:43 pnorman@tin: Finished deploy [tilerator/deploy@18faaa6]: Deploy scap fixes to cleartables map test server (duration: 00m 08s)
  • 20:42 pnorman@tin: Started deploy [tilerator/deploy@18faaa6]: Deploy scap fixes to cleartables map test server
  • 20:30 bsitzmann@tin: Finished deploy [mobileapps/deploy@5896151]: Update mobileapps to 29ebe0f (T192664) (duration: 08m 54s)
  • 20:28 arlolra: Updated Parsoid to dccfeafd (T157418, T194777, T195317, T195174, T194763, T194658)
  • 20:21 bsitzmann@tin: Started deploy [mobileapps/deploy@5896151]: Update mobileapps to 29ebe0f (T192664)
  • 20:18 arlolra@tin: Finished deploy [parsoid/deploy@de18a58]: Updating Parsoid to dccfeafd (duration: 13m 09s)
  • 20:05 arlolra@tin: Started deploy [parsoid/deploy@de18a58]: Updating Parsoid to dccfeafd
  • 19:36 mutante: reinstalling naos as deploy2001, booting to PXE (T193916)
  • 19:23 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.5 refs T191051 (duration: 01m 08s)
  • 19:21 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.5 refs T191051
  • 18:24 twentyafterfour@tin: Synchronized README: testing sync-masters without naos (duration: 01m 09s)
  • 17:54 twentyafterfour: Finished SWATting, thanks everyone!
  • 17:52 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/#/c/433546/ for SWAT refs T194871 (duration: 01m 19s)
  • 17:47 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/extensions/Translate/MessageCollection.php: syncing https://gerrit.wikimedia.org/r/#/c/434700/ refs T195347 (duration: 01m 19s)
  • 17:45 twentyafterfour@tin: Synchronized php-1.32.0-wmf.5/extensions/Translate/MessageCollection.php: syncing https://gerrit.wikimedia.org/r/#/c/434699/ refs T195347 (duration: 01m 20s)
  • 17:18 krinkle@tin: Synchronized w/robots.php: Ib1c6d676 - Bye, wgTitle (duration: 01m 20s)
  • 17:17 krinkle@tin: Synchronized w/extract2.php: Ib1c6d676 - Bye, wgTitle (duration: 01m 19s)
  • 17:14 twentyafterfour@tin: Synchronized wmf-config/Wikibase-production.php: SWAT deploying https://gerrit.wikimedia.org/r/#/c/434658/ refs T194273 (duration: 01m 20s)
  • 16:31 SMalyshev: starting wikidata full reindex for T163642
  • 16:11 addshore@tin: Finished scap: WikibaseLexeme: Do not refer to the spelling variant as language, T193603, Patch 1, Patch 2 (duration: 103m 38s)
  • 16:10 anomie: Running deduplicateArchiveRevId.php on group 0 for T193180
  • 15:53 anomie: Running populateExternallinksIndex60.php on group 0 for T59176
  • 15:30 jynus: stop db2044 for cloning to db2078 + upgrade
  • 14:52 jynus: restart db2078 for upgrade and to convert it to multiinstance
  • 14:30 moritzm: installing curl security updates on mediawiki canaries along with HHVM restart to pick up new library version
  • 14:28 addshore@tin: Started scap: WikibaseLexeme: Do not refer to the spelling variant as language, T193603, Patch 1, Patch 2
  • 14:12 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Enable HD logos for wikimania2018wiki T194340 (duration: 01m 20s)
  • 14:05 zeljkof: EU SWAT finished
  • 14:01 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disable wikidiff2 inline moved paragraphs by default (T194271) (duration: 01m 18s)
  • 13:55 jynus: stopping db1065 database to move it to m2
  • 13:47 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Change logo assets for wikimania2018wiki (T194340) (duration: 01m 20s)
  • 13:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable wikidiff2 inline moved paragraphs on production (T194271) (duration: 01m 21s)
  • 13:22 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Disable wikidiff2 inline moved paragraphs by default" (T194271) (duration: 01m 19s)
  • 13:17 zfilipin@tin: scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
  • 13:05 jynus: starting logical dump of m3-master
  • 13:00 jynus: starting logical dump of m2-master
  • 12:47 addshore: addshore@terbium:~$ echo 'https://www.wikidata.org/wiki/Special:EntityData/L1.rdf' | mwscript purgeList.php
  • 12:43 addshore@tin: Synchronized wmf-config/Wikibase.php: Fix disabledRdfExportEntityTypes for wikidata T168260 (duration: 01m 20s)
  • 12:16 XioNoX: set normal metric on codfw-eqsin link
  • 12:06 elukey: upgrade druid public to druid 0.11 (druid100[4-6])
  • 11:45 akosiaris: reimage ganeti2002, ganeti2006 as debian stretch
  • 11:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@59674ba]: Correctly commit pending messages for multi-topic rules (duration: 00m 47s)
  • 11:44 ppchelko@tin: Started deploy [cpjobqueue/deploy@59674ba]: Correctly commit pending messages for multi-topic rules
  • 11:33 addshore: addshore@mw1317:~$ scap pull # It seemed to be missing changes.....
  • 11:30 moritzm: installing curl security updates
  • 11:09 addshore: Lexeme deploy window probably done (unless something explodes)
  • 10:42 addshore: addshore@terbium mwscript extensions/WikibaseLexeme/maintenance/createBlacklistedLexemes.php --wiki testwikidatawiki
  • 10:40 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikibaseLexeme on wikidata.org T191457 T168260 https://gerrit.wikimedia.org/r/#/c/434453 (duration: 01m 20s)
  • 10:32 jynus: stop db1065 for cloning (proxys will complain)
  • 10:27 ppchelko@tin: Finished deploy [cpjobqueue/deploy@ffdc19a]: Logging improvements (duration: 00m 47s)
  • 10:26 ppchelko@tin: Started deploy [cpjobqueue/deploy@ffdc19a]: Logging improvements
  • 10:21 jynus: restarting db1117 for setup and upgrade
  • 10:17 dcausse: manually updating mapping on wikidatawiki elastic indices to add new lexeme fields
  • 10:15 addshore: updateSearchIndexConfig.php with --justMapping failed :(
  • 10:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 after alter table (duration: 01m 20s)
  • 10:14 addshore: addshore@wasat:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki wikidatawiki --justMapping --cluster codfw
  • 09:35 addshore: Finished, forceSearchIndex.php --wiki testwikidatawiki
  • 09:34 moritzm: upgrading remaining job runners in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 09:30 marostegui: Deploy schema change on db1101:3318 - T191519 T188299 T190148 T194270
  • 09:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 for alter table (duration: 01m 20s)
  • 09:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 after alter table (duration: 01m 19s)
  • 09:02 addshore: addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki testwikidatawiki --queue
  • 09:00 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: [testwikidata] Add Lexeme NS to ContentNamespaces (duration: 01m 20s)
  • 08:31 moritzm: upgrading remaining app/API servers in codfw to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 08:21 marostegui: Deploy schema change on db1099:3318 - T191519 T188299 T190148 T194270
  • 08:20 addshore: all the updateSearchIndexConfig runs from wasat also had --cluster=codfw
  • 08:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 for alter table (duration: 01m 20s)
  • 08:18 addshore: addshore@wasat:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now
  • 08:18 addshore: addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now
  • 08:17 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: testwikidata: Add Lexeme NS to wgNamespacesToBeSearchedDefault (duration: 01m 19s)
  • 08:06 addshore: addshore@wasat:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now --reindexProcesses=10
  • 08:00 moritzm: upgrading remaining app servers in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 07:54 addshore: addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now
  • 07:52 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: testwikidata: Add Property NS to wgNamespacesToBeSearchedDefault (duration: 01m 20s)
  • 07:45 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/WikibaseLexeme: Use ISO code und for missing language code (duration: 01m 30s)
  • 07:44 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Use ISO code und for missing language code (duration: 01m 31s)
  • 07:38 moritzm: upgrading remaining API servers in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 07:29 elukey: upload druid debs 0.11.0-3 to stretch-wikimedia
  • 06:52 moritzm: upgrading mw1299-mw1306 (job runners) to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 06:49 marostegui: Re-add indexes on wb_terms on db1092 - T194273
  • 06:44 elukey: restart zookeeper on druid100[4-6] for openjdk-8 upgrades
  • 06:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 20s)
  • 06:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 01m 20s)
  • 05:43 marostegui: Deploy schema change on db1092 - T191519 T188299 T190148 T194273 T194270
  • 05:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 27s)
  • 05:21 marostegui: Deploy schema change on dbstore1002:s8 - T191519 T188299 T190148 T194273 T194270
  • 05:12 marostegui: Deploy schema change on s8 codfw primary master (db2045), this will generate lag on codfw - T194273
  • 05:06 marostegui: Deploy schema change on s3 primary master (db1075) - T191519 T188299 T190148
  • 04:04 l10nupdate@tin: ResourceLoader cache refresh completed at Wed May 23 04:04:43 UTC 2018 (duration 7m 22s)
  • 03:57 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.5) (duration: 16m 39s)
  • 02:58 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 10m 54s)

2018-05-22

  • 23:29 maxsem@tin: Synchronized dblists/categories-rdf.dblist: https://gerrit.wikimedia.org/r/#/c/433740/ (duration: 01m 17s)
  • 22:59 twentyafterfour: train for group0 1.32.0-wmf.5 completed. Tune in tomorrow for more excitement!
  • 22:58 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.5 refs T191051
  • 22:54 twentyafterfour@tin: Finished scap: testwikis wikis to 1.32.0-wmf.5 refs T191051 (duration: 76m 01s)
  • 21:38 twentyafterfour@tin: Started scap: testwikis wikis to 1.32.0-wmf.5 refs T191051
  • 21:13 twentyafterfour@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2833418486" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 39m 54s)
  • 21:12 ejegg: updated CiviCRM from 5b8c868a00 to 4d797fc592
  • 20:33 twentyafterfour@tin: Started scap: testwikis wikis to 1.32.0-wmf.5 refs T191051
  • 19:45 pnorman@tin: Finished deploy [tilerator/deploy@18faaa6]: Update tilerator, fix variable substitution (duration: 02m 46s)
  • 19:43 pnorman@tin: Started deploy [tilerator/deploy@18faaa6]: Update tilerator, fix variable substitution
  • 19:39 pnorman@tin: Finished deploy [tilerator/deploy@18faaa6]: (no justification provided) (duration: 00m 29s)
  • 19:38 pnorman@tin: Started deploy [tilerator/deploy@18faaa6]: (no justification provided)
  • 19:35 pnorman@tin: Finished deploy [tilerator/deploy@18faaa6]: (no justification provided) (duration: 00m 25s)
  • 19:35 pnorman@tin: Started deploy [tilerator/deploy@18faaa6]: (no justification provided)
  • 19:34 pnorman@tin: Finished deploy [tilerator/deploy@a4a3fc7]: (no justification provided) (duration: 46m 19s)
  • 19:17 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/extensions/ContentTranslation/: sync https://gerrit.wikimedia.org/r/#/c/434529/ to fix T194810 (duration: 01m 02s)
  • 19:14 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group2 wikis to 1.32.0-wmf.4 refs T191050
  • 19:03 twentyafterfour: Today's train: Promoting group2 wikis to 1.32.0-wmf.4 followed by group0 to 1.32.0-wmf.5
  • 18:48 pnorman@tin: Started deploy [tilerator/deploy@a4a3fc7]: (no justification provided)
  • 18:44 pnorman: tilerator deploying a4a3fc7
  • 17:03 marostegui: Deploy schema change on s8 codfw primary master (db2045), this will generate lag on codfw - T194270
  • 16:52 elukey: restart zookeeper on druid100[1,3] to complete the openjdk-8 upgrade
  • 16:51 addshore: addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now
  • 16:43 elukey: upload druid debs 0.11.0-3 to jessie-wikimedia
  • 16:25 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db2064 from config - T195228 (duration: 01m 18s)
  • 16:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db2064 from config - T195228 (duration: 01m 19s)
  • 16:14 moritzm: upgrading remaining video scalers to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 15:20 ppchelko@tin: Finished deploy [cpjobqueue/deploy@4312549]: Increase the concurrency for low traffic topics (duration: 00m 41s)
  • 15:19 ppchelko@tin: Started deploy [cpjobqueue/deploy@4312549]: Increase the concurrency for low traffic topics
  • 14:47 addshore@tin: Synchronized wmf-config/Wikibase.php: Wikidata dispatch, set defaults for dispatchChanges settings (duration: 01m 19s)
  • 14:29 addshore: SWAT done
  • 14:28 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T193680 Use uploaded logo in wgLogo for liwikibooks (duration: 01m 16s)
  • 14:26 addshore@tin: Synchronized static/images/project-logos/liwikibooks.png: SWAT: T193680 Change liwikibooks logo (duration: 01m 18s)
  • 14:24 addshore: that last also included https://gerrit.wikimedia.org/r/#/c/434495/ - Use correct logo-size for wikimania2018wiki
  • 14:24 addshore@tin: Synchronized static/images/project-logos/wikimania2018wiki.png: SWAT: Change logo for wikimania2018wiki T194340 (duration: 01m 19s)
  • 14:07 elukey: upgrading druid on druid100[123] to 0.11
  • 13:51 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Revert Revert Temp rate limit for arwiki due to mass vandalism (duration: 01m 18s)
  • 13:46 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Revert Enable $wgUseRCPatrol on azwiki (duration: 01m 19s)
  • 13:39 elukey: upload druid 0.11 debs to jessie|stretch wikimedia
  • 13:26 moritzm: upgrading API servers in codfw to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 13:25 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable $wgUseRCPatrol on azwiki T194389 (duration: 01m 20s)
  • 13:21 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert: Temp rate limit for arwiki due to mass vandalism T192668 (duration: 01m 18s)
  • 13:18 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Add L171081 to clearBlacklistedLexemes (duration: 01m 28s)
  • 13:13 moritzm: upgrading labweb servers to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 13:06 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Handle invalid lexemeId in data when using wbeditentity new=form && Lexeme term languages: codes beyond MW default (duration: 01m 28s)
  • 13:01 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Lemma validation: language covered in deserializer (duration: 01m 30s)
  • 12:59 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Use the same language validation for representations and lemmas (duration: 01m 25s)
  • 12:57 moritzm: installing xdg-utils security updates on trusty
  • 12:55 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Use ChangeOps consistently throughout API (duration: 01m 30s)
  • 12:42 addshore@tin: Synchronized wmf-config: #1 #2 BETA ONLY profiler stuff (duration: 01m 20s)
  • 12:33 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase/repo: API: when validating change op make sure the edited entity is also validated T190928 (duration: 01m 52s)
  • 12:18 gehel: set `unchecked_tombstone_compaction=true` for maps eqiad - T194966
  • 12:15 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Set wgLexemeLanguageCodePropertyId for wikidatawiki T194248 (duration: 01m 19s)
  • 12:11 moritzm: installing imagemagick security updates
  • 12:07 addshore@tin: Finished scap: WikimediaMessages - wikidata-copyright, include the lexeme namespace T169333 (duration: 56m 45s)
  • 11:31 moritzm: upgrading application servers in deployment-prep to wikidiff 1.7.0 (T190717)
  • 11:10 addshore@tin: Started scap: WikimediaMessages - wikidata-copyright, include the lexeme namespace T169333
  • 11:06 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Add 171081 to wmgWikibaseIdBlacklist for wikibase-lexeme T194248 T187060 (duration: 01m 19s)
  • 11:04 ppchelko@tin: Started restart [cpjobqueue/deploy@b45cd3b]: KafkaConsumer is not connected error
  • 10:32 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch cross-wiki posting jobs to EventBus - T190327 (duration: 01m 18s)
  • 10:31 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b45cd3b]: Switch cross-wiki posting jobs for everything T175210 (duration: 01m 03s)
  • 10:30 ppchelko@tin: Started deploy [cpjobqueue/deploy@b45cd3b]: Switch cross-wiki posting jobs for everything T175210
  • 10:24 moritzm: upgrading video scalers in codfw to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 10:23 moritzm: upgrading snapshot hosts to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 09:56 jynus: stop and reimage db2043
  • 09:47 moritzm: upgrading mw123[8-9], mw1266-mw1275 to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 09:22 moritzm: upgrading mw1280-mw1290 to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 07:50 jynus: stop and reimage db2050
  • 07:42 jynus: stop and reimage db2057
  • 07:21 marostegui: Deploy schema change on s8 codfw master (db2045) with replication, this will generate lags on codfw - T191519 T188299 T190148
  • 07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 after alter table (duration: 01m 19s)
  • 05:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 - T193835 (duration: 01m 19s)
  • 05:10 marostegui: Deploy schema change on db1078 - T191519 T188299 T1901482
  • 05:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 for alter table (duration: 01m 44s)
  • 03:39 l10nupdate@tin: ResourceLoader cache refresh completed at Tue May 22 03:39:38 UTC 2018 (duration 7m 36s)
  • 03:32 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 12m 10s)
  • 02:48 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 12m 02s)

2018-05-21

  • 22:50 Krinkle: webperf1001: restart navtiming service, to test with new ipv6 capabilities
  • 22:13 mutante: enabled IPv6 on webperf machines
  • 21:59 foks: adding email to User:Quuxplusone - T194929
  • 21:24 ottomata: granted User:CN=kafka_fundraising_client read permissions for group fundraising* on kafka-jumbo (for kafkatee webrequest consumption: kafka acls --add --allow-principal User:CN=kafka_fundraising_client --consumer --topic '*' --group 'fundraising*'
  • 20:32 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.4 refs T191050 (duration: 01m 18s)
  • 20:31 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.4 refs T191050
  • 19:19 gehel: clearing cassandra snaphosts on maps* nodes to regain some space - T194966
  • 17:13 gehel@tin: Finished deploy [wdqs/wdqs@e01dd03]: new WDQS GUI version (duration: 08m 44s)
  • 17:04 gehel@tin: Started deploy [wdqs/wdqs@e01dd03]: new WDQS GUI version
  • 16:53 fdans@tin: Finished deploy [analytics/refinery@16cb3be]: deploying new jar for upgraded ua parsing (duration: 05m 40s)
  • 16:47 fdans@tin: Started deploy [analytics/refinery@16cb3be]: deploying new jar for upgraded ua parsing
  • 15:32 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 with low load (duration: 01m 18s)
  • 15:23 demon@tin: Pruned MediaWiki: 1.32.0-wmf.2 [keeping static files] (duration: 01m 56s)
  • 15:05 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool fully db1109 (duration: 01m 18s)
  • 14:57 demon@tin: Pruned MediaWiki: 1.31.0-wmf.30 (duration: 03m 19s)
  • 14:50 jynus: stop and reimage db1074
  • 14:46 demon@tin: Pruned MediaWiki: 1.31.0-wmf.29 (duration: 03m 29s)
  • 14:42 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 and pool db1077 with 100% weight (duration: 01m 20s)
  • 14:41 demon@tin: Pruned MediaWiki: 1.31.0-wmf.28 (duration: 03m 54s)
  • 14:08 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 with low weight (duration: 01m 20s)
  • 13:37 jynus: stop and reimage db2035
  • 13:22 bawolff@tin: Synchronized wmf-config/InitialiseSettings.php: Increase edit rate limits on commons (T194864) (duration: 01m 24s)
  • 12:51 jynus: stop and reimage db1077
  • 10:20 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 20s)
  • 10:18 jdrewniak@tin: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 22s)
  • 09:57 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105 with full weight (duration: 01m 21s)
  • 09:14 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 with full weight, repool db1105 with low weight (duration: 01m 21s)
  • 08:02 marostegui: Deploy schema change on db1077 with replication, this will generate lags on labs - T191519 T188299 T1901482
  • 08:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 for alter table (duration: 01m 22s)
  • 07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1123 after alter table (duration: 01m 21s)
  • 05:54 marostegui: Restart MySQL on db2075 and db2092 for testing
  • 05:50 marostegui: Restart MySQL on db1116 and db1120 for testing
  • 05:40 marostegui: Deploy schema change on db1123 - T191519 T188299 T1901482
  • 05:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1123 for alter table (duration: 01m 20s)
  • 05:27 marostegui: Stop MySQL and reboot db1067 - T194852
  • 05:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 - T195228 (duration: 01m 44s)
  • 05:09 marostegui: Deploy schema change on s7 primary master (db1062) - T191519 T188299 T1901482
  • 04:14 l10nupdate@tin: ResourceLoader cache refresh completed at Mon May 21 04:14:03 UTC 2018 (duration 7m 40s)
  • 04:06 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 15m 28s)
  • 03:08 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 14m 23s)
  • 01:20 ottomata: bouncing main -> jumbo MirrorMaker with increased max.request.size - T189464

2018-05-20

  • 12:21 arturo: reboot labtestneutron2002.codfw.wmnet
  • 10:50 tstarling@tin: Synchronized wmf-config/CommonSettings.php: Scribunto maxLangCacheSize (duration: 01m 23s)
  • 08:24 ariel@tin: Finished deploy [dumps/dumps@5438d41]: sync after reimage of snapshot1007 (duration: 00m 03s)
  • 08:24 ariel@tin: Started deploy [dumps/dumps@5438d41]: sync after reimage of snapshot1007

2018-05-19

  • 13:36 mholloway-shell@tin: Finished deploy [kartotherian/deploy@58d2b0a]: Update maps/kartotherian/package to 7520fa5 (duration: 02m 28s)
  • 13:33 mholloway-shell@tin: Started deploy [kartotherian/deploy@58d2b0a]: Update maps/kartotherian/package to 7520fa5
  • 08:31 volans: restarted pdfrender on scb1001

2018-05-18

  • 21:45 tgr: updated some OAuth consumer for a hackathon project: update oauth_registered_consumer set oarc_callback_url = 'http://localhost' where oarc_consumer_key = '2828bd9ca9bdcd81a960721819f25e90';
  • 21:17 demon@tin: Finished deploy [gerrit/gerrit@a07d943]: quota plugin (duration: 00m 11s)
  • 21:16 demon@tin: Started deploy [gerrit/gerrit@a07d943]: quota plugin
  • 19:36 Reedy: tstarling manually loaded Tor Exit Nodes on wikitech
  • 19:01 chasemp: unlock bawolff gerrit account
  • 18:11 mutante: neon, netmon1002 - start ferm service
  • 18:09 mutante: einsteinium: started ferm service
  • 18:03 jynus: stop replication and start schema change on db1105
  • 17:48 faidon@tin: Finished deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches (duration: 00m 05s)
  • 17:48 faidon@tin: Started deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches
  • 17:47 faidon@tin: Finished deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches (duration: 00m 05s)
  • 17:47 faidon@tin: Started deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches
  • 17:45 faidon@tin: Finished deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches (duration: 00m 32s)
  • 17:45 faidon@tin: Started deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches
  • 16:00 gehel: reduce replication of maps v4 keyspace to 3
  • 15:59 reedy@tin: Synchronized php-1.32.0-wmf.4/extensions/VisualEditor/: Fix dialog (duration: 01m 19s)
  • 15:45 reedy@tin: Synchronized wmf-config/throttle.php: throttling! (duration: 01m 22s)
  • 15:44 marostegui: Stop MySQL and reboot db1067 - T194852
  • 15:38 reedy@tin: Synchronized php-1.32.0-wmf.3/extensions/VisualEditor/: Fix dialog (duration: 01m 25s)
  • 15:31 gehel: rolling restart of cassandra on maps1* (repair was started on each node, instead of sequentially)
  • 15:23 gehel: clear cassandra snapshots on maps1002
  • 13:59 marostegui: Manually fail disk #6 on db1066 - T194870
  • 13:19 bawolff_: reset 2FA for Trizek_(WMF)
  • 12:26 jynus: stop and reimage db2041
  • 10:15 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 with low load (duration: 01m 20s)
  • 09:47 marostegui: Stop MySQL on db1116 for testing
  • 09:44 marostegui: Stop MySQL on db2092 for testing
  • 09:43 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
  • 09:33 gehel: cleared v3 snapshot on maps servers
  • 09:25 jynus: stop and reimage db1085
  • 09:24 gehel: drop v3 keyspace on cassandra maps (unused since migration to i18n)
  • 09:01 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1085 (duration: 01m 20s)
  • 08:32 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105:s6 and other vslow hosts (duration: 01m 21s)
  • 06:14 XioNoX: bumping eqsin-codfw link OSPF metric to 5000 (due to packet loss on link)
  • 05:37 marostegui: Stop MySQL on db1120 to copy its content to db2075 - T190704
  • 05:33 marostegui: Deploy schema change on dbstore1002:s3 - T191519 T188299 T190148
  • 05:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1066 - T193847 (duration: 01m 22s)
  • 05:24 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1011
  • 05:18 marostegui: Stop MySQL and reboot db1067 - T194852
  • 01:34 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4: sync wmf.4 to deploy https://gerrit.wikimedia.org/r/#/c/433673/ (duration: 09m 54s)
  • 01:27 twentyafterfour: syncing wmf.4 again to deploy https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 T191050
  • 00:19 mutante: rdb2004 - down in Icinga since >1d, nothing on console, dont see a SAL entry. powercycling

2018-05-17

  • 23:35 twentyafterfour: MediaWiki Train for 1.32.0-wmf.4 remains blocked by critical bugs, see T191050 for a list of blockers.
  • 23:34 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.3 refs T191050 (duration: 01m 20s)
  • 23:32 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.3 refs T191050
  • 23:29 twentyafterfour: rolling back
  • 23:29 twentyafterfour: still seeing Notice: Undefined variable: nonce in /srv/mediawiki/php-1.32.0-wmf.4/includes/resourceloader/ResourceLoaderClientHtml.php on line 272
  • 23:28 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.4 refs T191050 (duration: 01m 17s)
  • 23:26 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.4 refs T191050
  • 23:22 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/: sync https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 (duration: 09m 54s)
  • 22:53 twentyafterfour: deploying https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 T191050
  • 19:46 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.3 (duration: 01m 20s)
  • 19:44 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.3
  • 19:41 twentyafterfour: rolling back due to spike of undefined variable notices in resourceloader and ApiCSPReport.php
  • 19:39 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.4 (duration: 01m 21s)
  • 19:38 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.4
  • 19:33 twentyafterfour: getting the train back on track. Starting with group1 to 1.32.0-wmf.4 right now, will do all wikis to wmf.4 after verifying that group1 looks stable.
  • 19:28 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/extensions/Echo/: unbreak T194848 (duration: 01m 24s)
  • 19:11 twentyafterfour: train is still blocked by T194848
  • 17:23 arlolra: Updated Parsoid to fd49ab4 (T194821, T194687)
  • 17:15 arlolra@tin: Finished deploy [parsoid/deploy@091b891]: Updating Parsoid to fd49ab4 (duration: 09m 35s)
  • 17:06 arlolra@tin: Started deploy [parsoid/deploy@091b891]: Updating Parsoid to fd49ab4
  • 16:11 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 T174047 T194341
  • 16:04 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 (duration: 01m 21s)
  • 15:29 marostegui: Manually fail disk #6 on db1064 to get it replaced
  • 15:28 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 with full weight (duration: 01m 21s)
  • 15:00 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010
  • 14:39 papaul: shutting down furud for shelves swap
  • 14:35 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 https://phabricator.wikimedia.org/T174047 https://phabricator.wikimedia.org/T194341
  • 14:17 marostegui: Manually fail disk #2 on db1064 to get it replaced
  • 14:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db1066 IP - T193847 (duration: 01m 21s)
  • 13:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db1066 IP - T193847 (duration: 01m 17s)
  • 13:50 marostegui: Power off db1066 for a rack change - T193847
  • 13:46 marostegui: Stop MySQL on db1066 for a rack change - T193847
  • 13:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 for a rack change - T193847 (duration: 01m 21s)
  • 13:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1105 (duration: 01m 21s)
  • 13:36 jynus: restarted db1105 by mistake, turning it back on
  • 13:15 jynus: stop and reimage db1106
  • 12:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 (duration: 01m 20s)
  • 12:50 marostegui: Deploy schema change on s3 codfw primary master (db2043) this will generate lag on codfw - T191519 T188299 T190148
  • 11:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 after alter table (duration: 01m 21s)
  • 11:08 marostegui: Stop MySQL and poweroff db1067 - T194852
  • 10:12 mobrovac@tin: Finished deploy [citoid/deploy@8a26508]: Update citoid to 2f35126 - T179123 T185217 (duration: 02m 52s)
  • 10:09 mobrovac@tin: Started deploy [citoid/deploy@8a26508]: Update citoid to 2f35126 - T179123 T185217
  • 09:30 reedy@tin: Synchronized wmf-config/throttle.php: Throttle for Barcelona Hackathon (duration: 01m 22s)
  • 08:41 jynus: stop and reimage db2049
  • 08:04 jynus: stop and reimage db2056
  • 07:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 with low load (duration: 01m 20s)
  • 07:25 elukey: bounced all the prometheus burrow exporters on kafkamon* hosts to refresh their metrics and drop old/expired cgroups
  • 07:22 marostegui: Deploy schema change on db1090:3317 - T191519 T188299 T190148
  • 07:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 for alter table (duration: 01m 21s)
  • 07:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Specify the sanitarium masters in codfw (duration: 01m 21s)
  • 07:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 after alter table (duration: 01m 20s)
  • 07:05 jynus: stop and reimage db1093
  • 07:02 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 (duration: 01m 22s)
  • 05:52 marostegui: Disable BBU auto-learn on new hosts - T192979
  • 05:31 marostegui: Deploy schema change on db1079 with replication (this will generate lag on labs s7) - T191519 T188299 T190148
  • 05:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 for alter table (duration: 01m 22s)
  • 05:20 marostegui: Force BBU learn cycle on db1054 - T194867
  • 05:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 after alter table (duration: 01m 48s)
  • 05:09 marostegui: Deploy schema change on s4 primary master (db1068) - T191519 T188299 T190148
  • 04:10 l10nupdate@tin: ResourceLoader cache refresh completed at Thu May 17 04:10:13 UTC 2018 (duration 7m 42s)
  • 04:02 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 14m 08s)
  • 03:04 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 14m 27s)
  • 00:57 mutante: installing OS on webperf1002, webperf2002

2018-05-16

  • 21:58 cmjohnson1: swapping PEM 1 asw-c8-eqiad
  • 20:13 marostegui: Force WriteBack policy on db1067 - T194852
  • 19:38 twentyafterfour: The train for 1.32.0-wmf.4 is blocked by fatals in Echo extension. See T194848
  • 19:28 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/extensions/CongressLookup: sync CongressLookup extension on wmf.4 refs T191050 (duration: 01m 22s)
  • 17:35 milimetric@tin: Finished deploy [analytics/refinery@a205447]: Fix drop partitions script (duration: 15m 12s)
  • 17:33 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RemexHtml on wikis with < 100 ns0 errors in high priority cats (T193685) (duration: 01m 22s)
  • 17:20 milimetric@tin: Started deploy [analytics/refinery@a205447]: Fix drop partitions script
  • 17:16 milimetric@tin: Started deploy [analytics/refinery@15be6ae]: Fix drop partitions script
  • 16:38 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 with low weight (duration: 01m 21s)
  • 16:34 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db1067 IP - T193835 (duration: 01m 21s)
  • 16:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db1067 IP - T193835 (duration: 01m 17s)
  • 16:22 elukey: upgrade burrow on kafkamon1001 from 1.0 to 1.1
  • 16:18 marostegui: Power off db1067 for rack move - T193835
  • 16:18 godog: move ms-be1036 for asw2-c-eqiad - T187962
  • 16:03 godog: move ms-be1035 for asw2-c-eqiad - T187962
  • 16:03 _joe_: installing cergen 0.2.2 on the puppetmaster frontends
  • 15:50 andrewbogott: upgrading kernel, microcode and rebooting labnet1002
  • 15:50 jynus: stop and reimage db1088
  • 15:49 godog: move ms-be1034 for asw2-c-eqiad - T187962
  • 15:36 godog: move ms-be1025 for asw2-c-eqiad - T187962
  • 15:35 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 (duration: 03m 26s)
  • 15:27 _joe_: uploading cergen 0.2.2 to stretch, jessie
  • 15:18 godog: move ms-be1024 for asw2-c-eqiad - T187962
  • 15:17 elukey: upgrade burrow from 1.0.0 to 1.1.0 on kafkamon* hosts
  • 15:17 elukey: upload burrow 1.1.0 to stretch|jessie-wikimedia
  • 15:17 bstorm_: labsdb1004 reboot and upgrade kernel to 4.9.0-0.bpo.6-amd64
  • 15:10 godog: pool ms-fe1008 for asw2 move - T187962
  • 15:02 marostegui: Stop MySQL on labsdb1004
  • 15:01 marostegui: Stop MySQL on db1067 - T193835
  • 14:54 godog: depool ms-fe1008 for asw2 move - T187962
  • 14:50 moritzm: upgrading codfw job runners to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 14:49 godog: pool ms-fe1007 for asw2 move - T187962
  • 14:19 filippo@neodymium: conftool action : set/pooled=no; selector: name=ms-fe1007.eqiad.wmnet
  • 14:17 godog: depool ms-fe1007 for asw2 move - T187962
  • 14:04 jynus: stop and reimage db2048 (s1 master)
  • 13:56 marostegui: Restart mysql on db1116 for testing
  • 13:54 marostegui: Deploy schema change on db1098:3317 - T191519 T188299 T190148
  • 13:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 for alter table (duration: 01m 20s)
  • 13:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 after alter table (duration: 01m 21s)
  • 13:43 addshore: SWAT all done
  • 13:42 addshore@tin: Synchronized php-1.32.0-wmf.3/autoload.php: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 21s)
  • 13:41 addshore@tin: Synchronized php-1.32.0-wmf.3/includes/installer/DatabaseUpdater.php: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 21s)
  • 13:40 ottomata: rolling restart of Kafka jumbo brokers to apply jdk.tls.namedGroups=secp256r1 https://phabricator.wikimedia.org/T182993
  • 13:39 addshore@tin: Synchronized php-1.32.0-wmf.3/maintenance/: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 30s)
  • 13:38 addshore@tin: Synchronized php-1.32.0-wmf.4/autoload.php: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 21s)
  • 13:37 jynus: restart dbstore2002 for upgrade
  • 13:36 addshore@tin: Synchronized php-1.32.0-wmf.4/includes/installer/DatabaseUpdater.php: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 21s)
  • 13:34 addshore@tin: Synchronized php-1.32.0-wmf.4/maintenance/: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 31s)
  • 13:26 addshore@tin: Synchronized wmf-config/Wikibase.php: SWAT: Configure WikibaseLexeme after Repo & Client T191458 T194250 (duration: 01m 21s)
  • 13:20 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/ContentTranslation/modules/ve-cx/init/ve.init.mw.CXTarget.js: SWAT: Fix mistake in 84caceee that causes exceptions with MT card T194811 (duration: 01m 21s)
  • 13:17 jynus: restarting mariadb processes on dbstore2001 T194516
  • 13:07 moritzm: rebooting deployment-ms-be03 for tests related to IBPB passthrough
  • 12:56 jynus: stop and reimage db2039 (s6 master)
  • 11:28 addshore: WikibaseLexeme slot done
  • 11:10 moritzm: rebooting multatuli for some tests
  • 11:01 addshore: addshore@terbium mwscript extensions/WikibaseLexeme/maintenance/createBlacklistedLexemes.php --wiki testwikidatawiki
  • 10:57 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikibaseLexeme on test.wikidata.org T191458 (duration: 01m 20s)
  • 10:47 addshore@tin: Synchronized wmf-config/Wikibase.php: Prepare Lexeme config for test.wikidata.org T194250 PT 2/2 (duration: 01m 21s)
  • 10:45 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Prepare Lexeme config for test.wikidata.org T194250 PT 1/2 (duration: 01m 22s)
  • 10:41 moritzm: uploaded linux 4.9.88-1+deb9u1~bpo8+1 to apt.wikimedia.org/jessie-wikimedia
  • 10:34 jynus: stop and reimage db2046
  • 10:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 as it will be moved to a different rack - T193835 (duration: 01m 21s)
  • 10:14 marostegui: Drop unused tables msg_resource msg_resource_links from s2 - T194663
  • 09:57 jynus: stop and reimage db2053
  • 09:25 moritzm: upgrading video scalers to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 09:22 marostegui: Deploy schema change on db1101:3317 - T191519 T188299 T190148
  • 09:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 for alter table (duration: 01m 21s)
  • 09:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1094 after alter table (duration: 01m 21s)
  • 08:54 jynus: restart dbproxy1002 for upgrade
  • 08:48 moritzm: upgrading mw1221-mw1235 to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 08:45 jynus: restart dbproxy1003 for upgrade
  • 08:20 marostegui: Stop MySQL on db1120 on s2, s4, s6 and s7 to copy its content to db2075 - T190704
  • 07:57 _joe_: depooled ULSFO from live traffic in dns
  • 07:41 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 to convert it to temporary sanitarium (duration: 01m 20s)
  • 07:36 moritzm: installing systemd updates from stretch SUA
  • 07:33 marostegui: Deploy schema change on db1094 - T191519 T188299 T190148
  • 07:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1094 for alter table (duration: 01m 20s)
  • 07:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 after alter table (duration: 01m 34s)
  • 07:25 marostegui: Drop unused tables msg_resource msg_resource_links from s1 - T194663
  • 07:18 moritzm: upgrading mw1240-mw1258 to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 06:56 moritzm: upgrading mwdebug servers to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 06:45 marostegui: Drop unused tables msg_resource msg_resource_links from s3 codfw - T194663
  • 06:25 marostegui: Drop unused tables msg_resource msg_resource_links from s7 - T194663
  • 06:19 marostegui: Drop unused tables msg_resource msg_resource_links from s4 - T194663
  • 06:19 elukey: update analytics-in4 on cr1/cr2 eqiad to allow conf100[4-6] (new zookeeper hosts)
  • 06:17 marostegui: Drop unused tables msg_resource msg_resource_links from s8 - T194663
  • 06:15 marostegui: Drop unused tables msg_resource msg_resource_links from s6 - T194663
  • 06:11 marostegui: Drop unused tables msg_resource msg_resource_links from s5 - T194663
  • 05:54 elukey: removed acpi_power_meter manually from conf1004 (blacklisted module in puppet), Acpi errors in dmesg
  • 05:22 marostegui: Deploy schema change on db1086 - T191519 T188299 T190148
  • 05:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 for alter table (duration: 01m 23s)
  • 05:14 marostegui: Deploy schema change on s2 primary master db1054 - T191519 T188299 T190148
  • 04:43 kartik@tin: Finished deploy [cxserver/deploy@7e898c7]: Update cxserver to 112a1a1 (T191285) (duration: 03m 52s)
  • 04:39 kartik@tin: Started deploy [cxserver/deploy@7e898c7]: Update cxserver to 112a1a1 (T191285)
  • 03:46 l10nupdate@tin: ResourceLoader cache refresh completed at Wed May 16 03:46:12 UTC 2018 (duration 7m 41s)
  • 03:38 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 16m 26s)
  • 02:40 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 08m 25s)

2018-05-15

  • 23:33 mutante: creating ganeti VM webperf2002.eqiad.wmnet on ganeti2004 (link: private, row: A, cpus: 4, ram: 8, disk: 50) (T194390)
  • 23:11 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2139.codfw.wmnet
  • 23:10 mutante: mw2139 - reimaged, scap pull, apache-fast-test baseurls from naos, repooled with confctl (T194426)
  • 23:06 mutante: creating ganeti VM webperf1002.eqiad.wmnet on ganeti1004 (link: private, row: A, cpus: 4, ram: 8, disk: 50) (T194390)
  • 22:39 samwilson@tin: Synchronized wmf-config/InitialiseSettings.php: Deploying GlobalPreferences T190425 (duration: 01m 21s)
  • 22:21 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.4 refs T191050
  • 22:18 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/includes/api/ApiLogin.php: fix syntax error (duration: 01m 39s)
  • 21:34 twentyafterfour@tin: Finished scap: testwikis to 1.32.0-wmf.4 refs T191050 (duration: 66m 19s)
  • 20:27 twentyafterfour@tin: Started scap: testwikis to 1.32.0-wmf.4 refs T191050
  • 20:22 mutante: [radium:~] $ sudo apt-get autoremove
  • 19:53 cwd: re-enabled process-control
  • 19:42 twentyafterfour@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_770814178" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 02m 47s)
  • 19:40 cwd: disabled process-control for sql trigger refresh
  • 19:39 twentyafterfour@tin: Started scap: testwikis wikis to 1.32.0-wmf.4
  • 19:05 mutante: mw2139 - wmf-auto-reimage --conftoool --new (because it got "Failed to icinga_downtime" and has a new mainboard (T194426)
  • 19:03 mutante: mw2139 - wmf-auto-reimage --conftoool --no-verify (T194426)
  • 17:27 twentyafterfour: branching 1.32.0-wmf.4 refs T191050
  • 17:15 ottomata: rolling restart kafka-jumbo100[456]
  • 16:36 elukey: rolling restart of aqs on aqs* nodes to pick up the new druid config
  • 16:27 milimetric@tin: Finished deploy [analytics/refinery@679cf09]: Update partition drop script after schema change (duration: 07m 13s)
  • 16:25 mobrovac@tin: Started restart [cpjobqueue/deploy@58935d5]: (no justification provided)
  • 16:21 mobrovac@tin: Started restart [changeprop/deploy@e468d8e]: (no justification provided)
  • 16:20 milimetric@tin: Started deploy [analytics/refinery@679cf09]: Update partition drop script after schema change
  • 16:12 elukey: roll restart kafka on kafka-jumbo to pick up new zookeeper settings
  • 16:11 joal@tin: Finished deploy [analytics/aqs/deploy@a736558]: Deploying druid-configuration patch (duration: 05m 47s)
  • 16:05 joal@tin: Started deploy [analytics/aqs/deploy@a736558]: Deploying druid-configuration patch
  • 15:52 elukey: rolling restart of hadoop master daemons to pick up new zookeeper settings
  • 15:20 elukey: roll restart of Kafka Analytics to pick up new zookeeper settings
  • 14:59 elukey: roll restart of kafka daemons on kafka100[1-3] to pick up new zookeeper settings and group.initial.rebalance.delay.ms = 10s
  • 14:28 mobrovac@tin: Started restart [changeprop/deploy@e468d8e]: Restart after Kafka settings change
  • 14:28 mobrovac@tin: Started restart [cpjobqueue/deploy@58935d5]: Restart after Kafka settings change
  • 14:19 ottomata: temporarily disabling puppet on analytics1003 to run refine-eventbus after jumbo based camus eventbus import finishes
  • 14:14 elukey: swap conf1001 with conf1004 in the zookeeper main eqiad's config + roll restart of the service
  • 14:10 mobrovac@tin: Started restart [cpjobqueue/deploy@58935d5]: Restart after Kafka settings change
  • 14:09 mobrovac@tin: Started restart [changeprop/deploy@e468d8e]: Restart after Kafka settings change
  • 14:00 andrewbogott: rebooting labnet1001
  • 13:50 elukey: roll restart of kafka main codfw (kafka200[1-3]) to pick up group.initial.rebalance.delay.ms = 10s
  • 13:31 jynus: stop db2055 for reimage
  • 13:09 chasemp: disable puppet for all openstack things in eqiad
  • 13:07 andrewbogott: stopping nodepool and puppet on labnodepool1001 for T193579
  • 12:59 andrewbogott: stopping puppet on labnet1001 and 1002, silencing icinga for T193579
  • 12:42 jynus: stop db2060 for reimage
  • 12:14 moritzm: uploaded intel-microcode 20180425 for jessie-wikimedia/stretch-wikimedia
  • 10:57 jynus: stop db2067 for reimage
  • 10:49 joal@tin: Finished deploy [analytics/refinery@25abeec]: Fix for regular weekly deploy (duration: 06m 45s)
  • 10:46 jynus: stop db2066 for reimage
  • 10:43 joal@tin: Started deploy [analytics/refinery@25abeec]: Fix for regular weekly deploy
  • 10:16 jynus: stop db2065 for reimage
  • 10:15 moritzm: installing uwsgi security update on graphite servers in eqiad
  • 10:07 moritzm: installing php5 security updates on trusty
  • 09:51 moritzm: upgrading API server canaries to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 09:47 jynus: stop and restart db2091 for upgrade
  • 09:36 joal@tin: Finished deploy [analytics/refinery@b2f4c3c]: Regular weekly deploy (duration: 05m 38s)
  • 09:30 joal@tin: Started deploy [analytics/refinery@b2f4c3c]: Regular weekly deploy
  • 09:26 jynus: stop and restart db2088 for upgrade
  • 09:21 moritzm: upgrading app server canaries to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
  • 09:03 jynus: stop db2061 for reimage
  • 08:42 jynus: stop db2068 for reimage
  • 03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Tue May 15 03:10:59 UTC 2018 (duration 7m 11s)
  • 03:03 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 06m 32s)
  • 02:33 bawolff@tin: Finished scap: Backport https://gerrit.wikimedia.org/r/#/c/433096/ - log js loads of unregistered user js subpages (duration: 56m 27s)
  • 01:37 bawolff@tin: Started scap: Backport https://gerrit.wikimedia.org/r/#/c/433096/ - log js loads of unregistered user js subpages
  • 01:17 bawolff@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/433095/ log security channel (duration: 01m 02s)

2018-05-14

  • 23:39 bawolff@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/433089/ - re-enable sending csp logs to logstash (duration: 01m 01s)
  • 23:36 bawolff@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/432334/ Use english messages in badpass log (duration: 01m 16s)
  • 22:24 eileen: update process-control to 97fd42b3ab disabling dedupe catch up
  • 21:54 mutante: mwdebug1001 - temp modifying apache 08-wikimedia.conf to test gerrit:429863
  • 21:03 arlolra: Updated Parsoid to 945ed23 (T194082, T194083, T194084)
  • 20:54 arlolra@tin: Finished deploy [parsoid/deploy@28fcc4e]: Updating Parsoid to 945ed23 (duration: 11m 29s)
  • 20:42 arlolra@tin: Started deploy [parsoid/deploy@28fcc4e]: Updating Parsoid to 945ed23
  • 20:09 bsitzmann@tin: Finished deploy [mobileapps/deploy@ccffa6b]: Update mobileapps to 39c16e4 (T193440 T193439 T194065) (duration: 07m 49s)
  • 20:02 bsitzmann@tin: Started deploy [mobileapps/deploy@ccffa6b]: Update mobileapps to 39c16e4 (T193440 T193439 T194065)
  • 19:11 urandom: rolling cassandra restart, restbase dev environment
  • 18:20 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Create the eventcoordinator user group on enwiki - T193075 (duration: 01m 02s)
  • 17:12 gehel@tin: Finished deploy [wdqs/wdqs@ef37bf7]: new WDQS GUI version (duration: 08m 43s)
  • 17:03 gehel@tin: Started deploy [wdqs/wdqs@ef37bf7]: new WDQS GUI version
  • 16:19 elukey: umount/remount /mnt/hdfs on stat1005 to pick up new openjdk upgrades
  • 15:11 jynus: stop db2064 for reimage
  • 14:56 marostegui: Drop unused table long_run_profiling from enwiki - T194661
  • 14:45 moritzm: rebooting labtestnet2002 for some microcode/kernel tests
  • 14:44 mobrovac@tin: Finished deploy [restbase/deploy@75dc661]: API: Add /transform/list/tool/{tool}{/from}{/to}, take #2 (duration: 03m 58s)
  • 14:40 mobrovac@tin: Started deploy [restbase/deploy@75dc661]: API: Add /transform/list/tool/{tool}{/from}{/to}, take #2
  • 14:39 mobrovac@tin: Finished deploy [restbase/deploy@75dc661]: API: Add /transform/list/tool/{tool}{/from}{/to} - T163203 (duration: 23m 35s)
  • 14:30 milimetric@tin: Finished deploy [analytics/refinery@541823e]: deploying refinery to update python logic for cron jobs (duration: 19m 46s)
  • 14:30 otto@tin: Finished deploy [analytics/turnilo/deploy@9b2c8f0]: initial deploy (duration: 00m 28s)
  • 14:29 otto@tin: Started deploy [analytics/turnilo/deploy@9b2c8f0]: initial deploy
  • 14:19 moritzm: rebooting silver for some microcode/kernel tests
  • 14:15 mobrovac@tin: Started deploy [restbase/deploy@75dc661]: API: Add /transform/list/tool/{tool}{/from}{/to} - T163203
  • 14:10 milimetric@tin: Started deploy [analytics/refinery@541823e]: deploying refinery to update python logic for cron jobs
  • 13:58 herron: added temporary iptables drop rules on fermium for IPs with many hits logged against the list subscribe rate limit
  • 13:53 reedy@tin: Synchronized php-1.32.0-wmf.3/extensions/CirrusSearch: Partially revert deprecation of global namespace handling in prefix (duration: 01m 20s)
  • 13:50 reedy@tin: Synchronized wmf-config/throttle.php: T194630 (duration: 01m 02s)
  • 13:41 jynus: stop db2063 for reimage
  • 13:18 marostegui: Deploy schema change on dbstore1002:s7 - T191519 T188299 T190148
  • 13:08 akosiaris: remove ganeti2003, ganeti2007 from the ganeti cluster. stretch reimaging in progress
  • 13:06 akosiaris: reboot ganeti2008 for kernel ugprade
  • 12:38 kartik@tin: Finished deploy [cxserver/deploy@a7ef01b]: Update cxserver to 176b507 (duration: 03m 26s)
  • 12:35 kartik@tin: Started deploy [cxserver/deploy@a7ef01b]: Update cxserver to 176b507
  • 11:23 akosiaris: upload apertium-streamparser to apt.wikimedia.org/jessie-wikimedia/main T192978
  • 10:17 moritzm: installing systemd updates from stretch SUA update
  • 09:41 moritzm: installing gunicorn security updates
  • 09:28 moritzm: installing ghostscript security updates on trusty
  • 09:23 moritzm: installing libmad security updates
  • 09:11 moritzm: installing wavpack security updates for stretch (jessie/trusty not affected)
  • 08:41 moritzm: uploaded HHVM 3.18.5+dfsg-1+wmf8+deb9u1 to apt.wikimedia.org
  • 07:57 marostegui: Deploy schema change on s7 codfw master (db2040) with replication, this will generate lag on codfw - T191519 T188299 T190148
  • 07:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 after alter table (duration: 01m 02s)
  • 06:38 elukey: rolling restart of cassandra on aqs* for openjdk-8 upgrades
  • 06:35 moritzm: installing wget security updates on trusty (Debian already fixed)
  • 06:16 marostegui: Drop unused flaggedrevs from s3 testwiki - T174801
  • 05:22 marostegui: Deploy schema change on db1091 - T191519 T188299 T190148
  • 05:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 for alter table (duration: 01m 03s)
  • 03:18 l10nupdate@tin: ResourceLoader cache refresh completed at Mon May 14 03:18:25 UTC 2018 (duration 7m 25s)
  • 03:11 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 16m 20s)

2018-05-13

  • 21:10 bawolff: Reset some botpasswords associated with T194204
  • 20:09 bawolff: Deployed patch for T194605

2018-05-12

  • 06:50 foks: rm 2FA from User:Fatemi (at 6:38 PM UTC, Friday May 11)

2018-05-11

  • 21:28 herron: updated lists rate limit to 1 subscribe per 60 minutes as a temporary measure until problem requests slow down T194032
  • 20:59 herron: deployed rate limiting for POST requests to mailman list subscription URIs https://gerrit.wikimedia.org/r/432168 T194032
  • 20:27 chasemp: temp changes on fermium for T194032
  • 19:26 chasemp: on fermium
  • 19:26 chasemp: disable puppet and temp block a few IPs I believe are bad actors hammering mailman
  • 18:21 chasemp: change 'Advertise this list when people ask what lists are on this machine' to no for cloud-admin-l and cloud-admin-feed
  • 17:21 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with full weight (duration: 01m 02s)
  • 16:26 jynus: restarting mariadb@s8 at dbstore2001
  • 16:20 jynus: removing x1 from dbstore2001
  • 14:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1076 with low load, increase db1066 load (duration: 01m 02s)
  • 14:30 jynus: reset labsdb proxies to its config defaults after rolling restart for upgrade
  • 14:28 elukey: restart kafka brokers on kafka10[20,22,23] to pick up openjdk-7 security upgrades
  • 14:14 andrewbogott: rebooting labvirt1001 for T194258
  • 14:13 elukey: restart Hadoop daemons on analytics100[12] for openjdk security upgrades
  • 13:39 jynus: stopping and restarting labsdb1009 for upgrade
  • 13:32 jynus: reloading haproxy configuration for dbproxy1011 to point to labsdb1011
  • 12:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1066 with low load (duration: 01m 02s)
  • 11:48 Amir1: making change_tag_def table on all wikis (T194302)
  • 10:07 elukey: reimage analytics1052 to Debian Stretch (Hadoop Journal node)
  • 09:30 jynus: stopping db1076 for maintenance
  • 09:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 (duration: 01m 02s)
  • 07:59 elukey: reimage analytics1035 to Debian Stretch
  • 07:00 jynus: depool and upgrade/restart of dbproxy1011
  • 02:09 ejegg: updated payments-wiki from 55d0808853 to c81e25f8d3

2018-05-10

  • 23:18 ejegg: updated CiviCRM from 4c6a9c9c1c to 5b8c868a00
  • 23:09 thcipriani@tin: Synchronized dblists/categories-rdf.dblist: SWAT: Add wikis with more that 1000 categories to categories dump T194139 (duration: 01m 02s)
  • 20:27 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2140.codfw.wmnet
  • 20:27 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2140.co3dfw.wmnet
  • 20:26 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2144.codfw.wmnet
  • 20:24 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2202.codfw.wmnet
  • 20:17 sbisson@tin: Finished deploy [kartotherian/deploy@06572bb]: Kartotherian: add fonts for Syriac and Inuktituk (duration: 03m 40s)
  • 20:13 sbisson@tin: Started deploy [kartotherian/deploy@06572bb]: Kartotherian: add fonts for Syriac and Inuktituk
  • 20:05 XenoRyet: updated payments-wiki from 4a8aada491 to 55d0808853
  • 20:02 twentyafterfour: New branch 1.32.0-wmf.3 appears to be stable on all wikis. This completes the train for the week. Tune in again next week, same bat time, same bat channel. refs T191049
  • 19:57 twentyafterfour@tin: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.3
  • 19:53 twentyafterfour: all deployment blockers resolved, proceeding to deploy mediawiki 1.32.0-wmf.3 to all wikis
  • 19:14 twentyafterfour@tin: Synchronized php-1.32.0-wmf.3/includes/libs/rdbms/: deploy https://gerrit.wikimedia.org/r/#/c/432415/ refs T194308 (duration: 01m 23s)
  • 18:56 maxsem@tin: Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/432416/ labs only (duration: 01m 20s)
  • 18:30 sbisson@tin: Finished deploy [tilerator/deploy@a5ec109]: Tilerator: load source and style from yaml (duration: 08m 45s)
  • 18:21 sbisson@tin: Started deploy [tilerator/deploy@a5ec109]: Tilerator: load source and style from yaml
  • 18:16 sbisson@tin: Finished deploy [kartotherian/deploy@3aa87ff]: Kartotherian: load style from yaml (duration: 06m 16s)
  • 18:09 sbisson@tin: Started deploy [kartotherian/deploy@3aa87ff]: Kartotherian: load style from yaml
  • 18:02 ppchelko@tin: Finished deploy [restbase/deploy@fb306e3]: Logging improvements (duration: 15m 33s)
  • 17:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1123, Remove db1072 (duration: 01m 15s)
  • 17:57 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove db1072 (duration: 01m 20s)
  • 17:46 ppchelko@tin: Started deploy [restbase/deploy@fb306e3]: Logging improvements
  • 15:54 elukey: drain and reimage analytics1028 to Debian Jessie (Hadoop Journal node)
  • 15:00 elukey: rolling restart of Hadoop HDFS datanodes on analytics workers to pick up the new openjdk-8 security upgrades
  • 14:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1123 with low weight (duration: 01m 20s)
  • 14:38 elukey: rolling restart of Hadoop Yarn nodemanagers on analytics worker nodes for openjdk-8 security upgrades
  • 14:32 jynus: restarting db1123
  • 14:28 otto@tin: Started restart [eventlogging/eventbus@aa9eb2c]: apply log level changes
  • 14:27 ottomata: rolling restart of eventbus service to apply new log level settings
  • 14:07 elukey: restart hive/oozie Hadoop daemons on analytics1003 for openjdk-8 upgrades
  • 13:56 mutante: mw2139,mw2140,mw2144,mw2202 - reinstall with --no-verify - last few special cases that didnt have puppet certs or failed before - after that all appservers done
  • 13:46 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2138.codfw.wmnet
  • 13:43 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2137.codfw.wmnet
  • 13:42 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2136.codfw.wmnet
  • 13:41 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2135.codfw.wmnet
  • 13:31 elukey: rolling restart of kafka on kafka-jumbo1* for openjdk-8 security upgrades
  • 13:25 zeljkof: EU SWAT finished
  • 13:19 elukey: reimage analytics1029 to Debian Stretch
  • 13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: cawiki: remove gendered namespace aliases, already on MW core (T113616) (duration: 01m 20s)
  • 13:06 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateStyles for nowiki (T193786) (duration: 01m 20s)
  • 12:21 jynus@tin: Synchronized wmf-config/db-eqiad.php: Add db1123 (duration: 01m 19s)
  • 12:07 jynus@tin: Synchronized wmf-config/db-codfw.php: Add db1123 (duration: 01m 19s)
  • 12:04 Reedy: that was for "Add throttle rule for Netherlands Hackathon 2018 - Women Tech Storm"
  • 12:04 reedy@tin: Synchronized wmf-config/throttle.php: (no justification provided) (duration: 01m 48s)
  • 11:46 elukey: reimage analytics1030/31 to Debian Stretch
  • 11:07 volans: updated puppet compiler facts
  • 09:07 jynus: stop db1072 for maintenance
  • 08:10 jynus: shutdown and restart labsdb1010 for upgrade
  • 07:12 jynus: depool labsdb1010 from wikireplicas for maintenace
  • 03:04 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.2) (duration: 15m 35s)
  • 02:51 mutante: mw2137,mw2138,mw22139 - reinstall with strech (last message was wrong, already running)
  • 02:48 mutante: mw2136,mw2137,mw22138 - reinstall with stret h
  • 02:44 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2141.codfw.wmnet
  • 02:41 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2142.codfw.wmnet
  • 02:39 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2143.codfw.wmnet
  • 01:16 twentyafterfour@tin: Synchronized php-1.32.0-wmf.3/includes/libs/rdbms: sync https://gerrit.wikimedia.org/r/#/c/432297/ refs T194308 T191049 (duration: 01m 24s)
  • 00:33 mutante: mw2135, mw2136, mw2137 - reinstall with stretch, depooled, downtimed
  • 00:28 hoo@tin: Synchronized php-1.32.0-wmf.3/extensions/Wikibase/lib/WikibaseLib.php: Remove wgHooks entry for GalleryGetModes (T194316) (duration: 01m 16s)
  • 00:26 hoo@tin: Synchronized php-1.32.0-wmf.2/extensions/Wikibase/lib/WikibaseLib.php: Remove wgHooks entry for GalleryGetModes (T194316) (duration: 01m 20s)
  • 00:16 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Fix notices about missing index "max" (duration: 01m 20s)
  • 00:06 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable ORES on arwiki (T192498) (duration: 01m 20s)
  • 00:01 twentyafterfour: no phabricator deployment this evening.

2018-05-09

  • 23:53 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable ORES on cawiki, lvwiki, huwiki (T192501, T192499, T192496) (duration: 01m 29s)
  • 23:24 awight@tin: Finished deploy [ores/deploy@bf182e2]: Rollback ores1001 (duration: 00m 03s)
  • 23:24 awight@tin: Started deploy [ores/deploy@bf182e2]: Rollback ores1001
  • 23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on some wikibooks wikis (T192821) (duration: 01m 21s)
  • 23:06 awight@tin: Finished deploy [ores/deploy@bf1e2b1]: ORES: drafttopic (duration: 25m 51s)
  • 23:02 mutante: mw2141,mw2143,mw2142 - reinstalling with stretch - mw2144: puppet cert not found
  • 22:59 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2147.codfw.wmnet
  • 22:58 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2146.codfw.wmnet
  • 22:56 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2145.codfw.wmnet
  • 22:40 awight@tin: Started deploy [ores/deploy@bf1e2b1]: ORES: drafttopic
  • 22:35 awight@tin: Finished deploy [ores/deploy@1b13ef1]: ORES: drafttopic (duration: 03m 15s)
  • 22:33 addshore@tin: Synchronized wmf-config/extension-list: extension-list Add WikibaseLexeme to extension-list (duration: 01m 19s)
  • 22:32 awight@tin: Started deploy [ores/deploy@1b13ef1]: ORES: drafttopic
  • 22:28 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Add default edit rate limit of 90 edits/minute for all users (except wikidata) (duration: 01m 19s)
  • 22:20 awight@tin: Finished deploy [ores/deploy@2a09939]: ORES: force git-lfs install (take 3) (duration: 03m 05s)
  • 22:18 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T184745 T191459 BETA ONLY Enable WikibaseLexeme on BETA wikidatawiki (duration: 01m 19s)
  • 22:17 awight@tin: Started deploy [ores/deploy@2a09939]: ORES: force git-lfs install (take 3)
  • 22:09 awight@tin: Finished deploy [ores/deploy@c0db102]: ORES: force git-lfs install (take 2) (duration: 03m 15s)
  • 22:08 addshore@tin: Synchronized wmf-config/Wikibase-labs.php: T184745 BETA ONLY WikibaseLexeme config (duration: 01m 20s)
  • 22:06 awight@tin: Started deploy [ores/deploy@c0db102]: ORES: force git-lfs install (take 2)
  • 22:06 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T184745 BETA ONLY WikibaseLexeme config (duration: 01m 19s)
  • 22:05 awight@tin: Finished deploy [ores/deploy@c0db102]: ORES: force git-lfs install (duration: 02m 50s)
  • 22:03 awight@tin: Started deploy [ores/deploy@c0db102]: ORES: force git-lfs install
  • 21:27 ejegg: re-started refund queue consumer
  • 21:25 ejegg: updated CiviCRM from ca8acdccf6 to 4c6a9c9c1c
  • 21:20 reedy@tin: Synchronized php-1.32.0-wmf.3/includes/DefaultSettings.php: Add default edit rate limit of 90 edits/minute for all users (duration: 01m 20s)
  • 21:18 reedy@tin: Synchronized php-1.32.0-wmf.2/includes/DefaultSettings.php: Add default edit rate limit of 90 edits/minute for all users (duration: 01m 20s)
  • 20:59 ejegg: disabled refund queue consumer
  • 20:30 maxsem@tin: Synchronized php-1.32.0-wmf.2/extensions/CongressLookup/: https://gerrit.wikimedia.org/r/#/c/432146/ (duration: 01m 20s)
  • 20:28 maxsem@tin: Synchronized php-1.32.0-wmf.3/extensions/CongressLookup/: https://gerrit.wikimedia.org/r/#/c/432146/ (duration: 01m 19s)
  • 20:21 arlolra: Updated Parsoid to 5ce2608 (T194081, T188118)
  • 20:17 arlolra@tin: Finished deploy [parsoid/deploy@181e3b1]: Updating Parsoid to 5ce2608 (duration: 08m 52s)
  • 20:16 ejegg: restarted refund queue consumer
  • 20:08 arlolra@tin: Started deploy [parsoid/deploy@181e3b1]: Updating Parsoid to 5ce2608
  • 20:01 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.3 (duration: 01m 20s)
  • 20:00 twentyafterfour: group1 to 1.32.0-wmf.3 refs T191049
  • 20:00 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.3
  • 20:00 ejegg: updated CiviCRM from 4ff35ad4 to ca8acdccf6
  • 19:53 ottomata: rolling restart eventbus service to deploy logstash config
  • 19:51 ejegg: updated SmashPig standalone from 99585c8084 to 2b4186715e
  • 19:51 otto@tin: Finished deploy [eventlogging/eventbus@aa9eb2c]: logstash T193230 (duration: 01m 17s)
  • 19:49 otto@tin: Started deploy [eventlogging/eventbus@aa9eb2c]: logstash T193230
  • 19:49 mutante: graphite1001/2001 - rm check_uwsgi-coal NRPE check config, reloading nagios-nrpe-server (T194283)
  • 19:48 otto@tin: Finished deploy [eventlogging/eventbus@aa9eb2c]: logstash T193230 (duration: 00m 15s)
  • 19:48 otto@tin: Started deploy [eventlogging/eventbus@aa9eb2c]: logstash T193230
  • 19:38 otto@tin: Finished deploy [eventlogging/eventbus@c70e8c5]: logstash - T193230 (duration: 03m 33s)
  • 19:36 mutante: graphite1001, graphite2001 - deleting uwsgi-coal and coal sytemd unit files; systemctl daemon-reload (T194283)
  • 19:35 otto@tin: Started deploy [eventlogging/eventbus@c70e8c5]: logstash - T193230
  • 19:26 mutante: mw2145, mw2146, mw2147 - reinstall with stretch, depooled, downtimed
  • 19:23 Krinkle: Stop and disable coal-web (uwsgi-coal) service on graphite1001/graphite2001 (T194283)
  • 19:23 Krinkle: Disable coal service on graphite1001/graphite2001 (T194283)
  • 18:41 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2212.codfw.wmnet
  • 18:39 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2213.codfw.wmnet
  • 18:37 sbisson@tin: Finished deploy [tilerator/deploy@a86f8f8]: Make tilerator store up to zoom 15 (duration: 06m 36s)
  • 18:37 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2214.codfw.wmnet
  • 18:31 sbisson@tin: Started deploy [tilerator/deploy@a86f8f8]: Make tilerator store up to zoom 15
  • 18:29 sbisson@tin: Finished deploy [kartotherian/deploy@ef61ad7]: Make kartotherian serve up to z15 (duration: 02m 20s)
  • 18:27 sbisson@tin: Started deploy [kartotherian/deploy@ef61ad7]: Make kartotherian serve up to z15
  • 18:22 imarlier@tin: Finished deploy [performance/coal@8e57e4a]: Deploy only to webperf (duration: 00m 06s)
  • 18:22 imarlier@tin: Started deploy [performance/coal@8e57e4a]: Deploy only to webperf
  • 18:19 moritzm: installing jenkins security updates on releases*
  • 18:04 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/432030/ (duration: 01m 21s)
  • 18:00 maxsem@tin: Finished scap: Deploy CongressLookup on testwiki T194230 (duration: 105m 19s)
  • 16:56 ottomata: disabled 0.9 MirrorMaker on kafka102[023], enabled 1.x MirrorMaker on kafka-jumbo*
  • 16:39 gehel: restarting blazegraph and updater on wdqs1003
  • 16:15 maxsem@tin: Started scap: Deploy CongressLookup on testwiki T194230
  • 15:56 urandom: starting revision cleanup job, wikipedia_T_mobile__ng_lead keyspace - T192689
  • 15:31 thcipriani: upgrading jenkins on contint2001/contint1001
  • 15:26 vgutierrez: Replacing lvs1003 with lvs1016 - T184293
  • 15:22 mutante: mw2212,mw2213,mw2214 - reinstall with stretch
  • 14:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@58935d5]: Allow protocol version negotiation. T167039 (duration: 00m 34s)
  • 14:45 ppchelko@tin: Started deploy [cpjobqueue/deploy@58935d5]: Allow protocol version negotiation. T167039
  • 14:41 ppchelko@tin: Finished deploy [changeprop/deploy@e468d8e]: Allow protocol version negotiation. T167039 (duration: 00m 53s)
  • 14:40 ppchelko@tin: Started deploy [changeprop/deploy@e468d8e]: Allow protocol version negotiation. T167039
  • 14:27 ejegg: disabled refund queue consumer
  • 13:59 ottomata: beginning upgrade of Kafka main-eqiad cluster from 0.9.0.1 to 1.1.0 - T167039
  • 13:55 milimetric@tin: Finished deploy [analytics/refinery@a5a8cbc]: Renaming geoeditors druid datasource (duration: 05m 27s)
  • 13:49 milimetric@tin: Started deploy [analytics/refinery@a5a8cbc]: Renaming geoeditors druid datasource
  • 13:28 zeljkof: EU SWAT finished
  • 13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable mapframe on all but a few wikis (T191585) (duration: 01m 20s)
  • 13:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove unused wgKartographerDfltStyle setting (T191655) (duration: 01m 20s)
  • 13:10 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T192064 [cirrus] Increase the number of shards for wikidatawiki_content, enwiki_general (duration: 01m 20s)
  • 13:08 elukey: reimage analytics103[2,3] to Debian Stretch
  • 13:05 milimetric@tin: Finished deploy [analytics/refinery@640bc35]: Renaming geoeditors druid datasource (duration: 06m 11s)
  • 12:59 milimetric@tin: Started deploy [analytics/refinery@640bc35]: Renaming geoeditors druid datasource
  • 11:26 moritzm: installing wget security updates on Debian systems
  • 11:14 moritzm: reimage mw2206 (earlier reimage failed since the host lacked a puppet cert)
  • 11:03 moritzm: updated jenkins packages
  • 10:55 moritzm: reimaging mw2246 to stretch (video scaler with a deprecated one-off partman recipe)
  • 10:50 demon@tin: Synchronized wmf-config/missing.php: remove vendor dep (duration: 01m 20s)
  • 10:45 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 (duration: 01m 19s)
  • 10:40 demon@tin: Synchronized multiversion/getMWVersion: remove vendor dep (duration: 03m 27s)
  • 10:30 godog: cycle-load edac kernel modules for scb1002 to reset counters
  • 10:28 godog: cycle-load edac kernel modules for cp1068 to reset counters
  • 10:02 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: improve 5xx/4xx error handling (duration: 01m 27s)
  • 09:29 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1064 with full weight (duration: 01m 27s)
  • 08:52 moritzm: reimaging mw1336, mw1337 (job runners) to stretch
  • 08:19 no_justification: gerrit: restarting for version bump 2.14.7 -> 2.14.8
  • 08:18 jynus: stop and upgrade db1053
  • 08:17 demon@tin: Finished deploy [gerrit/gerrit@c421c91]: 2.14.7 -> 2.14.8 (duration: 00m 11s)
  • 08:17 demon@tin: Started deploy [gerrit/gerrit@c421c91]: 2.14.7 -> 2.14.8
  • 07:12 moritzm: reimaging mw2152 to stretch (video scaler with a deprecated one-off partman recipe)
  • 07:10 moritzm: reimaging mw2162 to stretch (last jessie job runner in codfw)
  • 07:05 moritzm: reimaging mw1334, mw1335 (job runners) to stretch
  • 06:04 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2211.codfw.wmnet
  • 06:01 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2210.codfw.wmnet
  • 06:00 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2209.codfw.wmnet
  • 05:26 marostegui: Stop MySQL on db2092 to do some clean ups
  • 05:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 01m 36s)
  • 05:19 marostegui: Stop slave on db1116:s3 to do some gtid cleanups and tests
  • 04:02 l10nupdate@tin: ResourceLoader cache refresh completed at Wed May 9 04:02:56 UTC 2018 (duration 7m 26s)
  • 03:55 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 16m 34s)
  • 02:57 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.2) (duration: 09m 06s)
  • 02:09 maxsem@tin: Synchronized wmf-config: Preparation for T194230 (duration: 01m 15s)
  • 02:06 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Preparation for T194230 (duration: 01m 16s)
  • 02:02 maxsem@tin: Synchronized php-1.32.0-wmf.2/extensions/CongressLookup/: Preparation for T194230 (duration: 01m 17s)
  • 02:00 maxsem@tin: Synchronized php-1.32.0-wmf.3/extensions/CongressLookup/: Preparation for T194230 (duration: 01m 22s)
  • 01:36 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2211.codfw.wmnet
  • 01:36 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2210.codfw.wmnet
  • 01:36 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2209.codfw.wmnet
  • 01:36 XioNoX: progressively push updated BGP_sanitize_in prefix-length-range to routers - T190317
  • 01:33 mutante: mw2209,mw2210,mw2211 - reinstall wtih stretch
  • 01:27 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2204.codfw.wmnet
  • 01:26 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2203.codfw.wmnet
  • 01:22 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2208.codfw.wmnet
  • 01:20 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2207.codfw.wmnet
  • 01:19 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2205.codfw.wmnet
  • 00:54 ejegg: running refund queue consumer overtime
  • 00:35 maxsem@tin: Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/432022/ - noop in prod (duration: 01m 20s)

2018-05-08

  • 23:58 thcipriani@tin: Synchronized php-1.32.0-wmf.2/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Helper/LoggingHelper.php: SWAT: Do not try to access null message message key T194140 (duration: 01m 32s)
  • 23:42 XioNoX: progressively push BGP_sanitize_in as-path too-many-hops to routers - T190317
  • 23:25 ejegg: updated CiviCRM from d9d412e496 to 4ff35ad4bb
  • 23:22 thcipriani@tin: Synchronized wmf-config: SWAT: Add string and external-id types to Wikibase indexing T163642 T99899 (duration: 01m 26s)
  • 23:12 XioNoX: lowering ospf metric of ulsfo-codfw to 390
  • 22:53 awight@tin: Finished deploy [ores/deploy@bf182e2]: Rollback ores1002 to master (duration: 00m 19s)
  • 22:52 awight@tin: Started deploy [ores/deploy@bf182e2]: Rollback ores1002 to master
  • 22:49 awight@tin: Finished deploy [ores/deploy@5b27205]: Deploy LFS files to ores1002 (duration: 01m 59s)
  • 22:47 awight@tin: Started deploy [ores/deploy@5b27205]: Deploy LFS files to ores1002
  • 22:35 XioNoX: remove PREFERRED-TRANSIT Tele2-DTAG from esams/knams routers
  • 22:05 XioNoX: progressively push updated BGP_sanitize_in bogon ASN filters to routers - T190317
  • 21:59 twentyafterfour: MediaWiki train for 1.32.0-wmf.3 group0 is complete. Will resume with group1 tomorrow, same bat time, same bat channel (refs T191049)
  • 21:45 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.3
  • 21:36 eileen: civicrm revision changed from 81e54c850d to d9d412e496, config revision is 96fbded693
  • 21:09 twentyafterfour@tin: Finished scap: testwikis wikis to 1.32.0-wmf.3 (duration: 114m 21s)
  • 20:59 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2204.codfw.wmnet
  • 20:58 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2203.codfw.wmnet
  • 20:24 milimetric@tin: Finished deploy [analytics/refinery@2a4633c]: Deploying renamed geowiki jobs as geoeditors (duration: 07m 07s)
  • 20:17 milimetric@tin: Started deploy [analytics/refinery@2a4633c]: Deploying renamed geowiki jobs as geoeditors
  • 20:02 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2201.codfw.wmnet
  • 20:01 mutante: mw2205,mw2206,mw2207 - reinstalling with stretch - mw2202 - wmf-auto-reimage failed: Timeout of 60 minutes reached waiting for reboot
  • 19:15 twentyafterfour@tin: Started scap: testwikis wikis to 1.32.0-wmf.3
  • 19:14 twentyafterfour: testwikis to 1.32.0-wmf.3 - https://gerrit.wikimedia.org/r/#/c/431821/ refs T191049
  • 19:11 twentyafterfour: updated mediawiki changelog https://www.mediawiki.org/wiki/MediaWiki_1.32/wmf.3/Changelog refs T191049
  • 18:55 mutante: mw2202, mw2203, mw2204 - reinstall with stretch
  • 18:52 marostegui: Manually fail disk #7 on db1073 to get it replaced
  • 18:22 mutante: mwmaint1001 - rebooting
  • 18:18 twentyafterfour: Branching MediaWiki master to wmf/1.32.0-wmf.3 refs T191049
  • 18:08 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2252.codfw.wmnet
  • 18:07 andrew@tin: Finished deploy [horizon/deploy@9245ca9]: rolling out member dashboard (duration: 03m 18s)
  • 18:06 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2251.codfw.wmnet
  • 18:03 andrew@tin: Started deploy [horizon/deploy@9245ca9]: rolling out member dashboard
  • 17:00 bawolff: Clear botpassword throttle for User:TaxonBot (T194160)
  • 16:45 thcipriani@tin: Synchronized README: Testing Scap 3.8.1-1 (duration: 01m 02s)
  • 16:43 godog: upload scap 3.8.1-1 - T127762
  • 16:40 XioNoX: re-pooling lvs2001 - T193677
  • 16:36 mutante: mwmaint1001 - reinstalling one more time after proxysql issues are resolved, PXE booting (T192092)
  • 16:27 mutante: mw2251,mw2252,mw2201 - reinstall with stretch
  • 16:22 herron: cleared low count edac counters on hosts mw2205 dbstore1002 db1051 elastic1029 T183177
  • 16:19 urandom: force (split) compaction of wikipedia_T_mobile__ng_lead.data, restbase1016 - T192689
  • 16:15 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2223.codfw.wmnet
  • 16:14 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2222.codfw.wmnet
  • 16:10 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2215.codfw.wmnet
  • 16:09 XioNoX: failing traffic over lvs2004 - T193677
  • 16:04 ppchelko@tin: Finished deploy [changeprop/deploy@e468d8e]: Allow protocol version negotiation. Codfw only. T167039 (duration: 01m 03s)
  • 16:03 ppchelko@tin: Started deploy [changeprop/deploy@e468d8e]: Allow protocol version negotiation. Codfw only. T167039
  • 16:01 ppchelko@tin: Finished deploy [cpjobqueue/deploy@58935d5]: Allow protocol version negotiation. Codfw only. T167039 (duration: 00m 42s)
  • 16:01 ppchelko@tin: Started deploy [cpjobqueue/deploy@58935d5]: Allow protocol version negotiation. Codfw only. T167039
  • 15:53 mutante: switching performance.wikimedia.org from graphite to webperf backends - running puppet on cache::misc servers (T158837)
  • 15:46 demon@tin: Pruned MediaWiki: 1.32.0-wmf.1 [keeping static files] (duration: 01m 47s)
  • 15:30 XioNoX: starting pybal on lvs2001 - T193677
  • 15:26 godog: (un)load edac kernel modules on thumbor1004 to test resetting counters - T183177
  • 15:09 XioNoX: stopping pybal on lvs2001 - T193677
  • 15:06 ottomata: beginnng Kafka upgrade of main-codfw: T167039
  • 14:53 XioNoX: re-enable pybal on lvs2004 - T193677
  • 14:48 XioNoX: disabling pybal on lvs2004 - T193677
  • 14:37 mutante: LDAP: added 'sbailey' to group 'wmf' (T194091)
  • 14:19 ppchelko@tin: Started restart [changeprop/deploy@7e86531]: Restart changeprop to try forcing it rebalancing topics
  • 14:15 mutante: mw2215,mw2222,mw2223 - reinstalling with stretch
  • 13:43 zeljkof: EU SWAT finished
  • 13:42 zfilipin@tin: Synchronized php-1.32.0-wmf.2/extensions/Translate: SWAT: Refactor TranslationUpdateJob to use only primitive types for parameters (T192111) (duration: 01m 11s)
  • 13:25 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable maps i18n everywhere (T191655) (duration: 01m 00s)
  • 13:14 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable AdvancedSearch BetaFeature on all wikis (T193182) (duration: 01m 00s)
  • 13:02 marostegui: Manually fail disk #9 on db1073 to get it replaced
  • 12:20 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove db1055 (duration: 00m 59s)
  • 12:19 moritzm: reimaging mw2159, mw2160, mw2161 (job runners) to stretch
  • 12:18 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove db1055 (duration: 00m 59s)
  • 12:17 moritzm: upgrading app servers in beta to wikidiff 1.6.0 (T190717)
  • 12:16 moritzm: upgrading app servers in beta to
  • 12:02 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1064 with low load (duration: 00m 59s)
  • 11:36 marostegui: Deploy schema change on db1103:3314 - T191519 T188299 T190148
  • 11:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 59s)
  • 11:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Really depool db2092 (duration: 00m 53s)
  • 10:29 moritzm: reimaging mw1347, mw1348 (API servers) to stretch (last two remaining API servers in eqiad)
  • 10:22 jynus: stop mariadb on db1055 to clone it to db1064
  • 10:15 moritzm: reimaging mw1310, mw1311 (job runners) to stretch
  • 09:58 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 (duration: 00m 54s)
  • 09:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1121 after alter table (duration: 01m 00s)
  • 09:20 elukey: forced a BBU re-learn cycle on analytics1032
  • 09:17 gehel: reducing replication factor on cassandra v3 (unused) keyspace for maps
  • 08:56 moritzm: reimaging mw1345, mw1346 (API servers) to stretch
  • 08:30 moritzm: reimaging mw2156, mw2157, mw2158 (job runners) to stretch
  • 08:27 moritzm: reimaging mw1308, mw1309 (job runners) to stretch
  • 08:03 marostegui: Stop MySQL on db1116 to transfer its content to db2092 - T190704
  • 07:59 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2092 T190704 (duration: 00m 57s)
  • 07:53 elukey: second attempt to remove the cassandra-metrics-collector (+ cleanup) from aqs*
  • 07:30 jynus: cleaning up maintenance hosts (terbium, etc.) from tendril maintenance files
  • 06:51 marostegui: Stop MySQL on db1060 as it will be decommissioned - T193732
  • 06:50 moritzm: reimaging mw1313, mw1343, mw1344 to stretch
  • 06:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1060 from config - T193732 (duration: 01m 01s)
  • 06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1060 from config - T193732 (duration: 00m 59s)
  • 06:05 marostegui: Read_only=off on db1069 to finish with the x1 failover
  • 06:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Promote db1069 as new x1 master (duration: 01m 00s)
  • 06:00 marostegui: Set db1055 ready only
  • 06:00 marostegui: Start x1 failover
  • 05:41 marostegui: Move db2034 under db1069 for x1 failover - T186320
  • 05:36 marostegui: Move dbstore1002:x1 under db1069 for x1 failover - T186320
  • 05:29 marostegui: Disable puppet on db1055 and db1069 before x1 failover - T186320
  • 05:28 marostegui: Disable gtid on db1069 an db2034 before x1 failover - T186320
  • 05:26 marostegui: Deploy schema change on db1121 with replication (this will generate lag on labs on s4) - T191519 T188299 T190148
  • 05:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1121 for alter table (duration: 01m 00s)
  • 05:19 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1011 - https://phabricator.wikimedia.org/T174047
  • 05:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 01m 00s)
  • 04:27 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2221.codfw.wmnet
  • 04:26 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2220.codfw.wmnet
  • 04:24 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2219.codfw.wmnet
  • 02:33 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.2) (duration: 05m 45s)
  • 00:12 ejegg: updated CiviCRM from 9752607052 to 81e54c850d

2018-05-07

  • 23:41 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2218.codfw.wmnet
  • 23:39 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2217.codfw.wmnet
  • 23:39 mutante: mw2219,mw2220,mw2221 - reinstall with stetch
  • 23:37 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2216.codfw.wmnet
  • 22:31 bstorm_: labsdb1009,labsdb1010,labsdb1011 are now on up-to-date views per T174047
  • 22:23 ppchelko@tin: Started restart [changeprop/deploy@7e86531]: Restart changeprop to try forcing it rebalancing topics
  • 21:28 mutante: mw2216,mw2217,mw2218 - wmf-auto-reimage --conftool , reinstall with stretch
  • 21:25 XioNoX: re-pool eqsin - T193897
  • 20:48 imarlier@tin: Started restart [performance/coal@50fe0dd]: Restart coal-web service everywhere, hopefully
  • 20:48 arlolra: Updated Parsoid to 6e38948 (T192909)
  • 20:41 arlolra@tin: Finished deploy [parsoid/deploy@cd5e875]: Updating Parsoid to 6e38948 (duration: 12m 25s)
  • 20:30 otto@tin: Finished deploy [statsv/statsv@c186340]: Configure api.version via CLI opt -- prep for Kafka main upgrade T167039 (duration: 00m 05s)
  • 20:30 otto@tin: Started deploy [statsv/statsv@c186340]: Configure api.version via CLI opt -- prep for Kafka main upgrade T167039
  • 20:29 arlolra@tin: Started deploy [parsoid/deploy@cd5e875]: Updating Parsoid to 6e38948
  • 20:25 bsitzmann@tin: Finished deploy [mobileapps/deploy@e20f23d]: Update mobileapps to c1f4de6 (T191538) (duration: 06m 09s)
  • 20:19 bsitzmann@tin: Started deploy [mobileapps/deploy@e20f23d]: Update mobileapps to c1f4de6 (T191538)
  • 19:58 XioNoX: removing onboard ports license from cr1-eqsin config - T193897
  • 17:34 bawolff@tin: Synchronized php-1.32.0-wmf.2/extensions/LoginNotify/includes/Hooks.php: https://gerrit.wikimedia.org/r/#/c/431611/ Do not send loginnotify emails for throttled logins (duration: 01m 08s)
  • 17:29 cmjohnson1: updating f/w lvs1016
  • 17:08 gehel@tin: Finished deploy [wdqs/wdqs@bd4b3ed]: new wdqs GUI, updater and blazegraph (duration: 05m 18s)
  • 17:03 gehel@tin: Started deploy [wdqs/wdqs@bd4b3ed]: new wdqs GUI, updater and blazegraph
  • 16:57 elukey: executed sudo megacli -AdpBbuCmd -BbuLearn -aALL -NoLog on analytics1032 - BBU alerts flapping
  • 16:37 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 - https://phabricator.wikimedia.org/T174047
  • 16:26 sbisson@tin: Finished deploy [kartotherian/deploy@9935fdb]: Kartotherian: remove temporary and unused osm-intl-i18n source (duration: 03m 28s)
  • 16:23 sbisson@tin: Started deploy [kartotherian/deploy@9935fdb]: Kartotherian: remove temporary and unused osm-intl-i18n source
  • 15:34 marostegui: Deploy schema change on db1097:3314 - T191519 T188299 T190148
  • 15:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 for alter table (duration: 01m 00s)
  • 15:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 after alter table (duration: 00m 57s)
  • 14:59 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010 - https://phabricator.wikimedia.org/T174047
  • 14:30 imarlier@tin: Finished deploy [performance/coal@50fe0dd]: Coal version that uses the graphite API to fetch data, instead of reading directly from whisper files (duration: 00m 13s)
  • 14:30 imarlier@tin: Started deploy [performance/coal@50fe0dd]: Coal version that uses the graphite API to fetch data, instead of reading directly from whisper files
  • 14:13 herron: upgraded prometheus-jmx-exporter to 0.3.0-1 on puppetdb servers
  • 14:12 herron: puppetdb updates complete — re-enabling puppet agents
  • 14:01 herron: temporarily disabling puppet agents for puppetdb security update
  • 13:36 zeljkof: EU SWAT finished
  • 13:35 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgKartographerEnableMapFrame to true for mrwiki (T193371) (duration: 01m 00s)
  • 13:16 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make test and test2 using maps i18n correctly (T191655) (duration: 01m 00s)
  • 12:55 sbisson@tin: Finished deploy [kartotherian/deploy@425f279]: Kartotherian: stop using tilerator_storage_id; Add babel upstream of osm-intl (duration: 05m 41s)
  • 12:50 sbisson@tin: Started deploy [kartotherian/deploy@425f279]: Kartotherian: stop using tilerator_storage_id; Add babel upstream of osm-intl
  • 12:27 moritzm: reimaging mw1333 to stretch (last app server in eqiad)
  • 11:28 moritzm: installing Java security updates on elastic* hosts
  • 10:59 moritzm: reimaging mw1330, mw1331, mw1332 (app servers) to stretch
  • 10:50 moritzm: depooled service nginx for mw1221-mw1231 (API servers)
  • 10:10 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 59s)
  • 10:09 jdrewniak@tin: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 00s)
  • 09:52 moritzm: reimaging mw1305, mw1306 (job runners) to stretch
  • 09:52 elukey: stop graphite cassandra-metrics-collector on aqs* (touch /etc/cassandra-metrics-collector/disable)
  • 09:46 moritzm: rolling restart of Kibana logstash nodes to pick up Java security updates
  • 09:41 marostegui: Manually enable innodb_strict_mode on db1084 - T150949
  • 09:30 ema: cp-text/upload: start varnish upgrades to 5.1.3-1wm8 T192368
  • 09:30 marostegui: Deploy schema change on db1084 - T191519 T188299 T190148
  • 09:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 for alter table (duration: 01m 03s)
  • 09:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 after alter table (duration: 01m 02s)
  • 09:02 moritzm: reimaging mw1275 (T192902)
  • 08:58 mobrovac@tin: Started restart [recommendation-api/deploy@ac66089]: Use the internal WDQS cluster LVS - T190266
  • 08:56 elukey: drain + reimage analytics103[7,8] to Debian Stretch
  • 08:30 godog: eqiad-prod: more weight to ms-be104[0-3] - T190081
  • 08:19 moritzm: rolling restart of logstash to pick up Java security updates
  • 08:19 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - https://phabricator.wikimedia.org/T174047
  • 08:04 gehel: restart cassandra on maps* for JVM upgrade
  • 07:59 gehel: restart cassandra on maps-test* for JVM upgrade
  • 07:53 moritzm: installing libdatetime-timezone-perl stable update for jessie/stretch
  • 07:38 gehel: restart elasticsearch on relforge for JVM upgrade
  • 07:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: db1074 is now db1102's master - T193732 (duration: 00m 59s)
  • 07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T193732 (duration: 00m 59s)
  • 07:28 marostegui: Change master from db1102:s2 from db1060 to db1074
  • 07:19 marostegui: Stop replication in sync on db1060 and db1074 - T193732
  • 07:19 moritzm: reimaging mw1303, mw1304 (job runners) to stretch
  • 07:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T193732 (duration: 00m 59s)
  • 07:06 moritzm: reimaging mw1233, mw1234, mw1235 (API servers) to stretch
  • 07:02 moritzm: reimaging mw1327, mw1328, mw1329 (app servers) to stretch
  • 05:37 marostegui: Unused databases devwikiinternal and rel13testwiki from s3 - T118764
  • 05:27 marostegui: Deploy schema change on db1081 - T191519 T188299 T190148
  • 05:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 for alter table (duration: 01m 01s)
  • 05:15 marostegui: Deploy schema change on s6 primary master db1061 - T191519 T188299 T190148
  • 03:06 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.2) (duration: 11m 22s)

2018-05-05

  • 03:36 XioNoX: commenting rigel out from smokeping
  • 02:16 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2191.codfw.wmnet
  • 02:03 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2193.codfw.wmnet
  • 01:37 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2187.codfw.wmnet
  • 00:35 Krinkle: Delete navtiming 'mediaWikiLoadComplete' metrics (T160315)
  • 00:26 Krinkle: Purging values for navtiming 'mediaWikiLoadEnd' sub-properties from 2018-04-21 to 2018-05-04T04:40:00 (T160315)

2018-05-04

  • 22:50 mutante: mwmaint1001 - now using mw-maintenance role, upcoming terbium replacement
  • 22:49 mutante: mw2187 - scap proxy - reinstalling with stretch
  • 22:46 mutante: mw2191, mw2193 - wmf-auto-reimage with --no-verify because puppet certs didnt exist
  • 22:02 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2199.codfw.wmnet
  • 22:00 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2198.codfw.wmnet
  • 21:55 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2197.codfw.wmnet
  • 19:50 bblack: rebooting lvs1016 (downtimed, also new and not in service!)
  • 19:02 sbisson@tin: Finished deploy [kartotherian/deploy@8e6b35b]: Use new keyspace (v4) for both i18n and non-i18n sources (duration: 03m 57s)
  • 18:58 sbisson@tin: Started deploy [kartotherian/deploy@8e6b35b]: Use new keyspace (v4) for both i18n and non-i18n sources
  • 18:38 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2196.codfw.wmnet
  • 18:36 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2194.codfw.wmnet
  • 18:35 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2195.codfw.wmnet
  • 18:16 mutante: mw2197,mw2198,mw2199 - reinstall with stretch
  • 18:12 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2192.codfw.wmnet
  • 18:05 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2190.codfw.wmnet
  • 18:03 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2189.codfw.wmnet
  • 17:16 XioNoX: depolled eqsin
  • 16:57 bawolff: adjusted login throttling code (T193762)
  • 16:50 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2196.codfw.wmnet
  • 16:50 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2194.codfw.wmnet
  • 16:50 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2195.codfw.wmnet
  • 16:50 dzahn@neodymium: conftool action : set/pooled=no; selector: name=mw2194.codfw.wmnet
  • 16:49 dzahn@neodymium: conftool action : set/pooled=no; selector: name=mw2196.codfw.wmnet
  • 16:25 XioNoX: adding BGP graceful shutdown to routers - T190323
  • 16:19 bawolff: Logging adjustment in mediawiki for T193762
  • 15:57 arturo: enabled puppet in labcontrol1001, labnodepol100[1-2] and labtestvirt10[01-22] after patches deployed for T193657
  • 15:16 mutante: mw2194, mw2195, mw2196 - reinstall with stretch - mw2193 - puppet cert not found
  • 15:06 arturo: disabling puppet in labnodepool100[1-2] for T193657
  • 15:05 arturo: disabling puppet in labcontrol1001 for T193657
  • 14:52 arturo: disabling puppet in labvirt10[01-22] to deploy https://gerrit.wikimedia.org/r/#/c/430581/ and https://gerrit.wikimedia.org/r/#/c/430614/ T193657
  • 14:05 mutante: mw2189, mw2190, mw2192 - reinstall with stretch, mw2191 - puppet cert not found
  • 13:33 marostegui: Manually enable innodb_strict_mode on labsdb1009 - T150949
  • 12:09 hashar: restarted Jenkins on contint1001 (java update)
  • 12:04 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2081 (duration: 00m 59s)
  • 11:22 jynus: stopping db1056 and moving it to spare
  • 10:55 marostegui: Manually enable innodb_strict_mode just on dbstore2001:3315 - T150949
  • 10:51 moritzm: reimaging mw2153, mw2154, mw2155 (job runners) to stretch
  • 09:52 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove db1056 (duration: 01m 00s)
  • 09:39 moritzm: uploaded openjdk-8 8u171-b11 for jessie-wikimedia to apt.wikimedia.org
  • 09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1119 in s1 API (duration: 00m 59s)
  • 09:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1119 (duration: 00m 59s)
  • 08:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1119 (duration: 01m 07s)
  • 08:45 moritzm: reimaging mw2249, mw2250, mw2253 (job runners) to stretch
  • 08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1119 in s1 API (duration: 00m 59s)
  • 08:30 mobrovac@tin: Finished deploy [cpjobqueue/deploy@193cf6f]: Config: Exclude refreshLinks from the RegEx rule (duration: 00m 47s)
  • 08:29 mobrovac@tin: Started deploy [cpjobqueue/deploy@193cf6f]: Config: Exclude refreshLinks from the RegEx rule
  • 08:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool recently new cloned db1119 (duration: 01m 00s)
  • 07:08 moritzm: reimaging mw2243, mw2247, mw2248 (job runners) to stretch
  • 07:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1119 to the config (duration: 01m 20s)
  • 07:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1119 to the config (duration: 01m 06s)
  • 06:42 marostegui: Stop MySQL on db1066 to clone db1119 - T192979
  • 06:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 - T192979 (duration: 01m 11s)
  • 05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Clarify that db1060 will be decommissioned (duration: 00m 53s)
  • 05:21 marostegui: Deploy schema change on dbstore1002:s4 - T191519 T188299 T190148
  • 04:24 demon@tin: rebuilt and synchronized wikiversions files: group2 to wmf.2
  • 04:17 demon@tin: Synchronized multiversion/getMWVersion: clean up getRealmSpecificFilename() (duration: 01m 07s)
  • 04:07 demon@tin: Synchronized wmf-config/InitialiseSettings.php: disabling LQT on a few closed/unloved testwikis (duration: 01m 11s)
  • 03:55 demon@tin: Synchronized scap/plugins: No-op plugin style fixes (duration: 01m 11s)
  • 00:55 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Temporary low-level activation of Eventlogging impression data for testing T183978 (duration: 01m 16s)
  • 00:50 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Temporarily add a higher threshold to trigger login attempt notices T193762 (duration: 01m 17s)
  • 00:01 demon@tin: rebuilt and synchronized wikiversions files: group1 to wmf.2

2018-05-03

  • 23:56 demon@tin: Synchronized php: symlink bump (duration: 01m 16s)
  • 23:43 demon@tin: Synchronized scap/plugins/: cleanup, no-op (duration: 01m 17s)
  • 23:16 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2186.codfw.wmnet
  • 23:13 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2185.codfw.wmnet
  • 23:12 krinkle@tin: Synchronized php-1.32.0-wmf.2/extensions/NavigationTiming/: If293a156ca / T193570 (duration: 01m 16s)
  • 23:10 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2188.codfw.wmnet
  • 23:09 krinkle@tin: Synchronized php-1.32.0-wmf.1/extensions/NavigationTiming/: If293a156cac / T193570 (duration: 01m 17s)
  • 22:52 aaron@tin: Finished scap: Deploy db9acea (bug T193668) (duration: 103m 50s)
  • 22:21 ejegg: updated CiviCRM from 0fdef242a3 to 9752607052
  • 21:21 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2183.codfw.wmnet
  • 21:18 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2184.codfw.wmnet
  • 21:08 aaron@tin: Started scap: Deploy db9acea (bug T193668)
  • 20:34 XioNoX: started peering/transit with Deutsche Telekom on cr2-esams
  • 20:30 mutante: mw1297 - reinstalling as mwmaint1001
  • 20:09 krinkle@tin: Synchronized php-1.32.0-wmf.1/extensions/NavigationTiming: I1e7f091cba1 (duration: 01m 18s)
  • 20:00 mutante: mw2185,mw2186,mw2188 - reinstall with stretch
  • 19:53 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2254.codfw.wmnet
  • 19:45 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2240.codfw.wmnet
  • 19:34 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2231.codfw.wmnet
  • 19:32 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2229.codfw.wmnet
  • 19:18 mutante: mw1297 - puppet node clean, puppet node deactivate - renaming to mwmaint1001 (T192185)
  • 19:02 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: CentralNotice EventLogging banner impression data test T183978 (duration: 01m 04s)
  • 18:59 thcipriani@tin: Synchronized php-1.32.0-wmf.2/extensions/NavigationTiming/modules/ext.navigationTiming.js: SWAT: Emit SaveTiming without relying on getNavTiming() T193693 (duration: 01m 16s)
  • 18:49 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2240.codfw.wmnet
  • 18:48 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2229.codfw.wmnet
  • 18:48 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2231.codfw.wmnet
  • 18:44 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ULS webfonts by default at Bengali Wikisource T193367 (duration: 01m 18s)
  • 18:09 mutante: mw2229,mw2231,mw2240 - wmf-auto-reimage with --new switch because their puppet cert wasn't found on puppetmaster, treated as new hosts that didnt exist before
  • 17:59 mutante: mw2254,mw2183,mw2184 - wmf-auto-reimage with stretch and raid/lvm
  • 17:36 ejegg: updated CiviCRM from 401344fb30 to 0fdef242a3
  • 16:34 Jeff_Green: authdns-update to add frbast.wikimedia.org service alias
  • 15:04 imarlier@tin: Finished deploy [performance/coal@762d160]: verify coal is deploying properly after shutdown on graphite hosts (duration: 00m 14s)
  • 15:04 imarlier@tin: Started deploy [performance/coal@762d160]: verify coal is deploying properly after shutdown on graphite hosts
  • 14:47 marostegui: Manually set offline disk #1 on db1063 so it can be replaced
  • 14:44 imarlier@tin: Started deploy [performance/coal@bd7568a]: verify coal is deploying properly after shutdown on graphite hosts
  • 14:13 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 (duration: 01m 17s)
  • 14:00 ema: cp3030 (text): upgrade varnish to 5.1.3-1wm8 T192368
  • 13:21 addshore: swat done
  • 13:19 addshore@tin: Synchronized wmf-config/: Switch to extension.json for Wikidata.org (duration: 01m 19s)
  • 13:13 addshore@tin: Synchronized wmf-config/: Switch to extension.json for PropertySuggester (duration: 01m 35s)
  • 13:05 godog: stop and mask coal service on graphite hosts - T186774
  • 13:04 gehel: rolling restart of wdqs for jvm upgrade
  • 13:03 marostegui: Deploy schema change on s4 codfw master db2051 with replication (this will generate lag on codfw) - T191519 T188299 T190148
  • 12:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Revert: Clarify that db1060 is running an alter table (duration: 01m 17s)
  • 12:26 moritzm: installing openjdk-8 security updates on stretch-based Hadoop workers
  • 11:57 moritzm: reimaging mw1301, mw1302 (job runners) to stretch
  • 10:51 moritzm: reimaging mw1319, mw1325, mw1326 (app servers) to stretch
  • 09:29 moritzm: reimaging mw1227, mw1231, mw1232 (API servers) to stretch
  • 09:09 gehel: rolling restart of elasticsearch completed - T191543 / T191236
  • 09:06 moritzm: reimaging mw1300 (job runner) to stretch
  • 08:33 moritzm: installing Java security updates on wdqs*
  • 08:31 addshore@tin: Synchronized php-1.32.0-wmf.2/extensions/WikimediaEvents/WikimediaEventsHooks.php: T191500 Update campaign prefix for onBeforeInitializeWMDECampaign hook (duration: 01m 16s)
  • 08:29 addshore@tin: Synchronized php-1.32.0-wmf.1/extensions/WikimediaEvents/WikimediaEventsHooks.php: T191500 Update campaign prefix for onBeforeInitializeWMDECampaign hook (duration: 01m 17s)
  • 08:27 mobrovac@tin: Finished deploy [changeprop/deploy@7e86531]: Bug fix: Resubscribe to the proper list of topics on metadata change (duration: 01m 12s)
  • 08:26 mobrovac@tin: Started deploy [changeprop/deploy@7e86531]: Bug fix: Resubscribe to the proper list of topics on metadata change
  • 08:18 mobrovac@tin: Finished deploy [cpjobqueue/deploy@5c1dcb9]: Bug fix: Resubscribe to the proper list of topics on metadata change (duration: 00m 54s)
  • 08:17 mobrovac@tin: Started deploy [cpjobqueue/deploy@5c1dcb9]: Bug fix: Resubscribe to the proper list of topics on metadata change
  • 08:13 ema: cp-misc: upgrade varnish to 5.1.3-1wm8 T192368
  • 08:08 godog: eqiad-prod: more weight to ms-be104[0-3] - T190081
  • 07:54 moritzm: reimaging mw1284, mw1289, mw1290 (API servers) to stretch
  • 07:39 moritzm: reimaging mw1256, mw1257, mw1258 (app servers) to stretch
  • 07:26 marostegui: Drop table flaggedrevs from eswikibooks - T193676
  • 07:11 moritzm: reimaging mwdebug1002 to stretch
  • 05:59 marostegui: Drop mostly empty flagged* tables from metawiki (s7) - T193390
  • 05:57 elukey: reimage analytics10[39,40] to Debian Stretch
  • 05:31 marostegui: Drop empty flagged* tables from eswiki (s7) - T193678
  • 05:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Clarify that db1060 is running an alter table (duration: 01m 15s)
  • 05:22 marostegui: Deploy schema change on db1060 with replication (this will generate lag on labs - s2) - T191519 T188299 T190148
  • 03:32 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2182.codfw.wmnet
  • 03:28 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2181.codfw.wmnet
  • 03:26 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2180.codfw.wmnet
  • 02:43 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.1) (duration: 07m 20s)
  • 00:33 krinkle@tin: Synchronized php-1.32.0-wmf.1/extensions/NavigationTiming/modules/: Ie77e77de3b8 (duration: 01m 18s)
  • 00:20 mutante: mw2180,mw2181,mw2182 - reinstalling with stretch (in case there are alerts that's why)
  • 00:02 twentyafterfour: no phabricator upgrade tonight.

2018-05-02

  • 23:39 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2179.codfw.wmnet
  • 23:35 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2178.codfw.wmnet
  • 23:29 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2177.codfw.wmnet
  • 23:17 catrope@tin: Synchronized php-1.32.0-wmf.2/extensions/Kartographer/: Add maintenance script to purge pages with map tags (T193525) (duration: 01m 18s)
  • 22:24 XioNoX: Re-enabled the link between fasw-eqiad and pfw3b-eqiad (backup) - T192104
  • 22:18 ebernhardson: start reindex of viwiki on eqiad elasticsearch, failed on last run due to unrelated issues
  • 22:17 XioNoX: Disabling the link between fasw-eqiad and pfw3b-eqiad (backup) - T192104
  • 22:17 XioNoX: Re-enabled the link between fasw-codfw and pfw3b-codfw (backup) - T192104
  • 21:42 XioNoX: Disabling the link between fasw-codfw and pfw3b-codfw (backup) - T192104
  • 21:33 XioNoX: failing-over RG1 to node0 on pfw3-codfw - T192104
  • 19:54 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2173.codfw.wmnet
  • 19:38 mutante: mw2173 - scap pull (wasn't pooled but should have, bring up to date)
  • 19:25 thcipriani@tin: rebuilt and synchronized wikiversions files: revert group1 to 1.32.0-wmf.2
  • 19:23 thcipriani@tin: rebuilt and synchronized wikiversions files: group1 to 1.32.0-wmf.2
  • 19:18 mutante: mw2177, mw2178, mw2179 - reinstalling with stretch
  • 19:14 ppchelko@tin: Finished deploy [restbase/deploy@1093d1d]: Sample log action api 4xx with 1% probability (duration: 16m 18s)
  • 19:10 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2176.codfw.wmnet
  • 19:08 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2175.codfw.wmnet
  • 18:58 ppchelko@tin: Started deploy [restbase/deploy@1093d1d]: Sample log action api 4xx with 1% probability
  • 17:59 mutante: mw2174 - repooled
  • 17:58 imarlier@tin: Started deploy [performance/coal@bd7568a]: deploy coal to webperf1001
  • 17:55 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on wikis with <100 high-prio issues (T192299) (duration: 01m 17s)
  • 17:48 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on metawiki (T192386) (duration: 01m 17s)
  • 17:40 imarlier@tin: Finished deploy [performance/coal@bd7568a]: deploy coal to webperf1001 (duration: 00m 06s)
  • 17:40 imarlier@tin: Started deploy [performance/coal@bd7568a]: deploy coal to webperf1001
  • 17:31 catrope@tin: Synchronized php-1.32.0-wmf.2/extensions/MobileFrontend/: T193564 (duration: 01m 20s)
  • 17:29 catrope@tin: Synchronized php-1.32.0-wmf.2/extensions/Kartographer/: Add missing util dependency (duration: 01m 14s)
  • 17:12 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable $wgCiteResponsiveReferences on kowiki (T193491) (duration: 01m 17s)
  • 16:35 bawolff: run recountCategories.php on huwiki T169964
  • 16:29 gehel: restarting blazegraph to increase TasksMax
  • 15:18 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 (duration: 01m 17s)
  • 15:07 joal@tin: Finished deploy [analytics/refinery@318d449]: Regular weekly deploy (duration: 08m 46s)
  • 14:58 joal@tin: Started deploy [analytics/refinery@318d449]: Regular weekly deploy
  • 14:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: Add and pool db1121 (duration: 01m 17s)
  • 14:33 jynus@tin: Synchronized wmf-config/db-codfw.php: Add db1121 (duration: 01m 16s)
  • 14:23 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Update comment next to WMDE wmgMonologChannels entry T191500 (duration: 01m 17s)
  • 13:58 gilles: End of mid-day EU SWAT
  • 13:49 ottomata: beginning upgrade of kafka-jumbo brokers from 1.0.0 -> 1.1.0 : T193495
  • 13:47 vgutierrez: Update puppet compiler facts
  • 13:40 vgutierrez: Repool lvs2001 - T191897
  • 13:37 gilles@tin: Synchronized wmf-config/InitialiseSettings.php: T187299 Add performance perception QuickSurvey definition (duration: 01m 17s)
  • 13:29 elukey: upgrade zookeeper to 3.4.9 on druid100[4-6] (wikistats 2 backend) - T164008
  • 13:20 elukey: restart druid broker on druid100[1-3] to enable the 'druid.sql.enable' feature
  • 13:15 gilles: T193225 mwscript namespaceDupes.php --wiki=euwikisource --fix
  • 13:13 gilles@tin: Synchronized wmf-config/InitialiseSettings.php: T193225 Add Author namespace on eu.wikisource (duration: 01m 20s)
  • 13:05 gilles: Starting mid-day EU SWAT
  • 13:01 moritzm: re-attempting to reimage mw1250, mw1254, mw1255 (app servers) to stretch, those ran into a timeout earlier which is now fixed in the reimage script
  • 12:48 moritzm: reimaging mw1340, mw1341, mw1342 (API servers) to stretch
  • 12:17 vgutierrez: Depool and reimage lvs2001 as stretch - T191897
  • 11:41 vgutierrez: Repool lvs2002 - T191897
  • 11:12 jynus: stopping db1064 for cloning to db1121 (will create temporary lag on commons wikireplicas)
  • 11:02 kartik@tin: Finished deploy [cxserver/deploy@0aa3532]: Update cxserver to a20bf75 (duration: 06m 01s)
  • 10:56 kartik@tin: Started deploy [cxserver/deploy@0aa3532]: Update cxserver to a20bf75
  • 10:48 moritzm: reimaging mw2200 to stretch
  • 10:30 moritzm: installing openjdk-8 security updates on stat hosts
  • 10:25 vgutierrez: Depool and reimage lvs2002 as stretch - T191897
  • 10:21 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 (duration: 01m 17s)
  • 10:17 vgutierrez: Repool lvs2003 - T191897
  • 09:10 vgutierrez: Depool lvs2003 and reimage as stretch - T191897
  • 08:54 moritzm: reimaging mw1250, mw1254, mw1255 (app servers) to stretch
  • 08:36 moritzm: reimaging mw1228, mw1229, mw1230 to stretch (those were logged to SAL before, but failed with IPMI issues before)
  • 08:22 ema: varnish 5.1.3-1wm8 uploaded to apt.w.o T192368
  • 08:11 elukey: upgrading Druid to 0.10 on druid100[4-6] (wikistats 2 backend) - T164008
  • 07:42 elukey: remove openjdk-7 related packages from druid100[1-3] after zookeeper upgrade
  • 07:36 gehel: elasticsearch eqiad rolling restart for plugin update and NUMA config - T191543 / T191236
  • 07:31 elukey: upgrade zookeeper on druid100[1-3] to 3.4.9 - T164008
  • 07:27 jynus: restart db1098 for upgrade and validation
  • 02:44 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.1) (duration: 07m 11s)
  • 00:17 ebernhardson: start reindex for commonswiki, eqiad elasticsearch, commonswiki_general appears to have failed previous reindex

2018-05-01

  • 22:29 ejegg: updated CiviCRM from 46883844a3 to 401344fb30
  • 22:19 andrewbogott: rebooting labnet1002 for T193579
  • 22:04 krinkle@tin: Synchronized php-1.32.0-wmf.1/extensions/EducationProgram/resources/: I7ca59823ffbf2 (duration: 01m 16s)
  • 22:01 awight@tin: Finished deploy [ores/deploy@bf182e2]: Rollback ores1001 to master (duration: 01m 13s)
  • 22:00 awight@tin: Started deploy [ores/deploy@bf182e2]: Rollback ores1001 to master
  • 21:57 krinkle@tin: Synchronized php-1.32.0-wmf.1/includes/libs/rdbms/: Iba663c (duration: 01m 19s)
  • 21:54 awight@tin: Finished deploy [ores/deploy@52347e0]: Test LFS deployment for ORES; T180627 (duration: 03m 21s)
  • 21:50 awight@tin: Started deploy [ores/deploy@52347e0]: Test LFS deployment for ORES; T180627
  • 21:48 awight@tin: Finished deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627 (duration: 00m 26s)
  • 21:48 awight@tin: Started deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627
  • 21:48 awight@tin: Started deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627
  • 21:44 demon@tin: Finished scap: group0 to wmf.2 (duration: 68m 18s)
  • 21:42 XioNoX: re-enabling eqsin-codfw link
  • 21:23 urandom: restbase: begin culling leaked revisions, enwiki_T_mobile__ng_{lead,remaining} - T192689
  • 20:43 urandom: restbase: begin culling leaked revisions, commons_T_mobile__ng_remaining - T192689
  • 20:35 demon@tin: Started scap: group0 to wmf.2
  • lunch: update stale apifeatureusage-search-svc-eqiad-wmnet template in eqiad elasticsearch and delete unused apifeatureusage template
  • 20:28 urandom: restbase: begin culling leaked revisions, commons_T_mobile__ng_lead - T192689
  • 20:06 demon@tin: Pruned MediaWiki: 1.31.0-wmf.27 (duration: 03m 08s)
  • 20:00 ottomata: rolling restart of eventbus to apply new logstash formatter version T193230
  • 19:59 demon@tin: Pruned MediaWiki: 1.31.0-wmf.30 [keeping static files] (duration: 01m 43s)
  • 19:49 urandom: restbase: begin culling leaked revisions, others_T_mobile__ng_remaining - T192689
  • 19:27 ottomata: rolling restart of eventbus to apply logstash tag https://phabricator.wikimedia.org/T193230
  • 19:17 mutante: mw2174,mw2175,mw2176 ff - reinstalling with wmf-auto-reimage to stretch
  • 19:00 ottomata: rolling restart of eventlogging-service-eventbus to apply logstash logging configs - T193230
  • 18:53 ariel@tin: Finished deploy [dumps/dumps@5438d41]: keep running even if file we want to report on is moved/gone (duration: 00m 04s)
  • 18:53 ariel@tin: Started deploy [dumps/dumps@5438d41]: keep running even if file we want to report on is moved/gone
  • 17:40 XioNoX: disabling eqsin<->codfw link for high packet loss on link
  • 17:39 otto@tin: Finished deploy [eventlogging/eventbus@c70e8c5]: remove occasional logging of request.body in prep for T193230 (duration: 02m 29s)
  • 17:36 otto@tin: Started deploy [eventlogging/eventbus@c70e8c5]: remove occasional logging of request.body in prep for T193230
  • 16:15 ebernhardson: T192972 change eqiad elasticsearch disk watermarks from 85/85 to 80/80 to match disk space alerts
  • 16:15 herron: manually kicked off mirror@sodium:~$ /usr/local/sbin/update-ubuntu-mirror to clear ubuntu mirror out of sync alert
  • 15:50 urandom: restbase: begin culling leaked revisions, others_T_mobile__ng_lead -- T192689
  • 15:37 ejegg: updated SmashPig standalone from a4de12d415 to 99585c8084
  • 13:35 herron: restarted hhvm on mw1233
  • 02:47 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.1) (duration: 07m 04s)
  • 01:20 ejegg: re-enabled fundraising jobs
  • 01:01 ejegg: updated CiviCRM from 47197006d5 to 46883844a3
  • 00:51 eileen: email sent advising of minor interuption
  • 00:50 ejegg: disabled fundraising jobs for db update
  • 00:21 ebernhardson: increase cluster.routing.allocation.balance.threshold from 1.0 to 1.5 for eqiad elasticsearch cluster to reduce rebalancing agressiveness

2000s

2010s

2020s