Server Admin Log

From Wikitech

2024-04-13

  • 15:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60475 and previous config saved to /var/cache/conftool/dbconfig/20240413-154240-marostegui.json
  • 15:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 15:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 15:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60474 and previous config saved to /var/cache/conftool/dbconfig/20240413-154217-marostegui.json
  • 15:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60473 and previous config saved to /var/cache/conftool/dbconfig/20240413-152709-marostegui.json
  • 15:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60472 and previous config saved to /var/cache/conftool/dbconfig/20240413-151201-marostegui.json
  • 14:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60471 and previous config saved to /var/cache/conftool/dbconfig/20240413-145653-marostegui.json
  • 06:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60470 and previous config saved to /var/cache/conftool/dbconfig/20240413-060646-marostegui.json
  • 06:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 06:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 00:52 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp1115.eqiad.wmnet

2024-04-12

  • 21:03 cdanis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:03 cdanis@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:43 cdanis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:43 cdanis@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:36 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_codfw
  • 19:36 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_codfw
  • 18:56 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudbackup2002.codfw.wmnet
  • 18:56 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:56 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 18:55 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 18:52 andrew@cumin1002: START - Cookbook sre.dns.netbox
  • 18:47 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudbackup2002.codfw.wmnet
  • 18:46 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudbackup2001.codfw.wmnet
  • 18:46 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:46 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 18:44 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 18:40 andrew@cumin1002: START - Cookbook sre.dns.netbox
  • 18:35 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudbackup2001.codfw.wmnet
  • 17:00 mutante: crm2001 - on initial puppet run adding envoy build-envoy-config failed building config and service failed due to dependency issue. manual run of "sudo /usr/local/sbin/build-envoy-config -c /etc/envoy/" and restarted envoyproxy.service
  • 16:19 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host matomo1003.eqiad.wmnet with OS bookworm
  • 16:16 elukey: move cassandra instances on cassandra-dev to the new truststore (allowing PKI certs) - T352647
  • 15:59 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 15:56 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues
  • 15:56 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues
  • 15:55 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'readability' for release 'main' .
  • 15:53 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 15:52 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on elastic2090.codfw.wmnet with reason: T353878
  • 15:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on elastic2090.codfw.wmnet with reason: T353878
  • 15:51 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 15:50 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 15:50 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic2090 for reboot to get rid of broken systemd units - bking@cumin2002 - T353878
  • 15:50 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Banning hosts: elastic2090 for reboot to get rid of broken systemd units - bking@cumin2002 - T353878
  • 15:50 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:49 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 15:49 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 15:48 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp1115.eqiad.wmnet
  • 15:47 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 15:46 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:46 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 15:32 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 15:31 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bookworm
  • 15:23 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "magru - ayounsi@cumin1002"
  • 15:22 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:21 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "magru - ayounsi@cumin1002"
  • 15:07 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
  • 15:03 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 15:03 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "magru - ayounsi@cumin1002"
  • 15:03 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 15:02 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 15:01 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "magru - ayounsi@cumin1002"
  • 14:59 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
  • 14:22 hashar@deploy1002: Finished scap: Backport for Parser::statelessFetchTemplate: don't add interwiki redirects to dependencies (T362221) (duration: 16m 29s)
  • 14:19 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 14:18 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 14:18 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 14:17 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 14:09 hashar@deploy1002: hashar and jforrester: Continuing with sync
  • 14:08 hashar@deploy1002: hashar and jforrester: Backport for Parser::statelessFetchTemplate: don't add interwiki redirects to dependencies (T362221) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:08 sukhe: depool cp1115 for PXE boot issue testing: T350179
  • 14:07 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
  • 14:05 hashar@deploy1002: Started scap: Backport for Parser::statelessFetchTemplate: don't add interwiki redirects to dependencies (T362221)
  • 12:53 jayme: updated rsyslog to 8.2404.0-1~bpo11+1 on staging-codfw and staging-eqiad k8s clusters - T357616
  • 12:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P60466 and previous config saved to /var/cache/conftool/dbconfig/20240412-122045-marostegui.json
  • 12:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P60464 and previous config saved to /var/cache/conftool/dbconfig/20240412-120537-marostegui.json
  • 12:02 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
  • 11:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T356166)', diff saved to https://phabricator.wikimedia.org/P60463 and previous config saved to /var/cache/conftool/dbconfig/20240412-115029-marostegui.json
  • 11:33 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
  • 10:55 urbanecm: mwmaint1002: mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=frwiki --search-index (T362367)
  • 09:58 urbanecm: mwmaint1002: mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=eswiki --search-index (T362367)
  • 09:36 moritzm: installing postgresql-common bugfix updates from Bullseye point release
  • 09:26 moritzm: installing debootstrap bugfix updates from Bullseye point release
  • 09:25 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on matomo1003.eqiad.wmnet with reason: Still in setup
  • 09:25 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on matomo1003.eqiad.wmnet with reason: Still in setup
  • 08:56 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
  • 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60461 and previous config saved to /var/cache/conftool/dbconfig/20240412-072435-root.json
  • 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60460 and previous config saved to /var/cache/conftool/dbconfig/20240412-070930-root.json
  • 06:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60459 and previous config saved to /var/cache/conftool/dbconfig/20240412-065424-root.json
  • 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60458 and previous config saved to /var/cache/conftool/dbconfig/20240412-063918-root.json
  • 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60457 and previous config saved to /var/cache/conftool/dbconfig/20240412-062412-root.json
  • 06:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60456 and previous config saved to /var/cache/conftool/dbconfig/20240412-060907-root.json
  • 05:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2109.codfw.wmnet with OS bookworm
  • 05:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60455 and previous config saved to /var/cache/conftool/dbconfig/20240412-055401-root.json
  • 05:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2109.codfw.wmnet with reason: host reimage
  • 05:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2109.codfw.wmnet with reason: host reimage
  • 05:23 moritzm: prune obsolete nginx debs on apt-staging after switch to new nginx provider scheme T329529
  • 05:17 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2109.codfw.wmnet with OS bookworm
  • 05:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2109', diff saved to https://phabricator.wikimedia.org/P60454 and previous config saved to /var/cache/conftool/dbconfig/20240412-051606-root.json
  • 03:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1249 (T356166)', diff saved to https://phabricator.wikimedia.org/P60453 and previous config saved to /var/cache/conftool/dbconfig/20240412-033317-marostegui.json
  • 03:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
  • 03:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
  • 03:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T356166)', diff saved to https://phabricator.wikimedia.org/P60452 and previous config saved to /var/cache/conftool/dbconfig/20240412-033254-marostegui.json
  • 03:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P60451 and previous config saved to /var/cache/conftool/dbconfig/20240412-031744-marostegui.json
  • 03:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P60450 and previous config saved to /var/cache/conftool/dbconfig/20240412-030237-marostegui.json
  • 02:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T356166)', diff saved to https://phabricator.wikimedia.org/P60449 and previous config saved to /var/cache/conftool/dbconfig/20240412-024729-marostegui.json
  • 01:05 denisse: Manually deleting /srv/syslog/.linux.dhcp.DictModel/syslog.log from November 30 on centrallog1002 and centrallog2002 after the prune_old_srv_syslog_directories.service failed to delete the non-empty directory - T362376

2024-04-11

  • 23:04 cstone: civicrm upgraded from c2569254 to 4d5a4fc3
  • 20:20 urbanecm@deploy1002: Finished scap: Backport for ext-EventLogging: Add mediawiki.product_metrics.wikifunctions_ui to $wgEventLoggingStreamNames (duration: 17m 38s)
  • 20:08 urbanecm@deploy1002: urbanecm and phuedx: Continuing with sync
  • 20:05 urbanecm@deploy1002: urbanecm and phuedx: Backport for ext-EventLogging: Add mediawiki.product_metrics.wikifunctions_ui to $wgEventLoggingStreamNames synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:03 urbanecm@deploy1002: Started scap: Backport for ext-EventLogging: Add mediawiki.product_metrics.wikifunctions_ui to $wgEventLoggingStreamNames
  • 19:41 eevans@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 19:40 eevans@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 19:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1248 (T356166)', diff saved to https://phabricator.wikimedia.org/P60448 and previous config saved to /var/cache/conftool/dbconfig/20240411-193537-marostegui.json
  • 19:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
  • 19:35 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
  • 19:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T356166)', diff saved to https://phabricator.wikimedia.org/P60447 and previous config saved to /var/cache/conftool/dbconfig/20240411-193514-marostegui.json
  • 19:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P60446 and previous config saved to /var/cache/conftool/dbconfig/20240411-192006-marostegui.json
  • 19:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P60445 and previous config saved to /var/cache/conftool/dbconfig/20240411-190459-marostegui.json
  • 18:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T356166)', diff saved to https://phabricator.wikimedia.org/P60443 and previous config saved to /var/cache/conftool/dbconfig/20240411-184951-marostegui.json
  • 17:50 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.26 refs T360158
  • 17:27 swfrench@deploy1002: Finished scap: (no justification provided) (duration: 07m 57s)
  • 17:20 swfrench@deploy1002: Started scap: (no justification provided)
  • 17:12 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
  • 17:12 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
  • 17:11 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
  • 17:11 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
  • 17:11 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
  • 17:10 bd808@deploy1002: helmfile [staging] START helmfile.d/services/developer-portal: apply
  • 16:45 hashar@deploy1002: Finished scap: Backport for Revert "Update mobile search for dark mode, remove unused functions in MobilePage.php" (T362297) (duration: 16m 47s)
  • 16:33 hashar@deploy1002: hashar: Continuing with sync
  • 16:31 hashar@deploy1002: hashar: Backport for Revert "Update mobile search for dark mode, remove unused functions in MobilePage.php" (T362297) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 16:29 hashar@deploy1002: Started scap: Backport for Revert "Update mobile search for dark mode, remove unused functions in MobilePage.php" (T362297)
  • 16:27 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host matomo1003.eqiad.wmnet with OS bookworm
  • 16:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 100%: repool', diff saved to https://phabricator.wikimedia.org/P60442 and previous config saved to /var/cache/conftool/dbconfig/20240411-161536-arnaudb.json
  • 16:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 100%: repool', diff saved to https://phabricator.wikimedia.org/P60441 and previous config saved to /var/cache/conftool/dbconfig/20240411-161522-arnaudb.json
  • 16:03 herron: beginning rolling hardware upgrades for titan100[12] T361251
  • 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 75%: repool', diff saved to https://phabricator.wikimedia.org/P60440 and previous config saved to /var/cache/conftool/dbconfig/20240411-160030-arnaudb.json
  • 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 75%: repool', diff saved to https://phabricator.wikimedia.org/P60439 and previous config saved to /var/cache/conftool/dbconfig/20240411-160016-arnaudb.json
  • 15:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60438 and previous config saved to /var/cache/conftool/dbconfig/20240411-155836-root.json
  • 15:56 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe2002.codfw.wmnet with OS bookworm
  • 15:51 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe1002.eqiad.wmnet with OS bookworm
  • 15:47 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp
  • 15:45 btullis@cumin1002: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 50%: repool', diff saved to https://phabricator.wikimedia.org/P60437 and previous config saved to /var/cache/conftool/dbconfig/20240411-154524-arnaudb.json
  • 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 50%: repool', diff saved to https://phabricator.wikimedia.org/P60436 and previous config saved to /var/cache/conftool/dbconfig/20240411-154510-arnaudb.json
  • 15:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60435 and previous config saved to /var/cache/conftool/dbconfig/20240411-154330-root.json
  • 15:39 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe2002.codfw.wmnet with reason: host reimage
  • 15:36 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe2002.codfw.wmnet with reason: host reimage
  • 15:35 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: host reimage
  • 15:33 btullis@cumin1002: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 15:31 mvernon@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: host reimage
  • 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 25%: repool', diff saved to https://phabricator.wikimedia.org/P60434 and previous config saved to /var/cache/conftool/dbconfig/20240411-153019-arnaudb.json
  • 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 25%: repool', diff saved to https://phabricator.wikimedia.org/P60433 and previous config saved to /var/cache/conftool/dbconfig/20240411-153003-arnaudb.json
  • 15:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60432 and previous config saved to /var/cache/conftool/dbconfig/20240411-152825-root.json
  • 15:24 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 15:24 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 15:24 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 15:23 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 15:20 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host moss-fe2002.codfw.wmnet with OS bookworm
  • 15:18 mvernon@cumin1002: START - Cookbook sre.hosts.reimage for host moss-fe1002.eqiad.wmnet with OS bookworm
  • 15:14 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60431 and previous config saved to /var/cache/conftool/dbconfig/20240411-151319-root.json
  • 15:12 dancy@deploy1002: Finished scap: Backport for static.php: Handle mediawiki.org/ontology/ontology.owl (T171807 T359643) (duration: 17m 41s)
  • 15:11 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:00 dancy@deploy1002: dancy: Continuing with sync
  • 14:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 100%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60430 and previous config saved to /var/cache/conftool/dbconfig/20240411-145841-arnaudb.json
  • 14:58 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 14:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60429 and previous config saved to /var/cache/conftool/dbconfig/20240411-145813-root.json
  • 14:57 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp
  • 14:57 dancy@deploy1002: dancy: Backport for static.php: Handle mediawiki.org/ontology/ontology.owl (T171807 T359643) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 100%: Repool', diff saved to https://phabricator.wikimedia.org/P60428 and previous config saved to /var/cache/conftool/dbconfig/20240411-145658-arnaudb.json
  • 14:54 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and not P{cp2042.codfw.wmnet} and A:cp
  • 14:54 dancy@deploy1002: Started scap: Backport for static.php: Handle mediawiki.org/ontology/ontology.owl (T171807 T359643)
  • 14:52 sukhe: sudo cumin "A:cp and A:esams" "run-puppet-agent --enable 'merging CR 1014571'"
  • 14:52 dreamyjazz@deploy1002: Finished scap: Backport for Set wgMFFallbackEditor to visual for most VE wikis (T361134) (duration: 24m 11s)
  • 14:47 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:47 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:47 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:45 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:44 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot-master (exit_code=0) rolling restart_daemons on A:maps-master
  • 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 100%: reool', diff saved to https://phabricator.wikimedia.org/P60427 and previous config saved to /var/cache/conftool/dbconfig/20240411-144416-arnaudb.json
  • 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60426 and previous config saved to /var/cache/conftool/dbconfig/20240411-144336-arnaudb.json
  • 14:43 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot-master rolling restart_daemons on A:maps-master
  • 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 100%: repool', diff saved to https://phabricator.wikimedia.org/P60425 and previous config saved to /var/cache/conftool/dbconfig/20240411-144311-arnaudb.json
  • 14:43 sukhe: sudo cumin "A:cp and A:esams" "disable-puppet 'merging CR 1014571'"
  • 14:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60424 and previous config saved to /var/cache/conftool/dbconfig/20240411-144307-root.json
  • 14:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 75%: Repool', diff saved to https://phabricator.wikimedia.org/P60423 and previous config saved to /var/cache/conftool/dbconfig/20240411-144152-arnaudb.json
  • 14:39 dreamyjazz@deploy1002: dreamyjazz and esanders: Continuing with sync
  • 14:36 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 14:34 moritzm: installing distro-info-data updates from Bullseye point release
  • 14:31 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-eqiad
  • 14:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2149.codfw.wmnet with OS bookworm
  • 14:30 dreamyjazz@deploy1002: dreamyjazz and esanders: Backport for Set wgMFFallbackEditor to visual for most VE wikis (T361134) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 75%: reool', diff saved to https://phabricator.wikimedia.org/P60422 and previous config saved to /var/cache/conftool/dbconfig/20240411-142910-arnaudb.json
  • 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 50%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60421 and previous config saved to /var/cache/conftool/dbconfig/20240411-142830-arnaudb.json
  • 14:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp2042.codfw.wmnet,service=(cdn|ats-be)
  • 14:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3073.esams.wmnet,service=(cdn|ats-be)
  • 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 75%: repool', diff saved to https://phabricator.wikimedia.org/P60420 and previous config saved to /var/cache/conftool/dbconfig/20240411-142806-arnaudb.json
  • 14:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60419 and previous config saved to /var/cache/conftool/dbconfig/20240411-142801-root.json
  • 14:27 dreamyjazz@deploy1002: Started scap: Backport for Set wgMFFallbackEditor to visual for most VE wikis (T361134)
  • 14:27 Dreamy_Jazz: Extending UTC Afternoon backport window
  • 14:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 50%: Repool', diff saved to https://phabricator.wikimedia.org/P60418 and previous config saved to /var/cache/conftool/dbconfig/20240411-142645-arnaudb.json
  • 14:26 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2042.codfw.wmnet with OS bullseye
  • 14:24 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3073.esams.wmnet with OS bullseye
  • 14:19 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 14:18 elukey: drain and restart cassandra-b on aqs2007 - didn't pick up the new truststore during the past roll restart - T352647
  • 14:15 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-codfw
  • 14:14 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 50%: reool', diff saved to https://phabricator.wikimedia.org/P60417 and previous config saved to /var/cache/conftool/dbconfig/20240411-141404-arnaudb.json
  • 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 25%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60416 and previous config saved to /var/cache/conftool/dbconfig/20240411-141324-arnaudb.json
  • 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 50%: repool', diff saved to https://phabricator.wikimedia.org/P60415 and previous config saved to /var/cache/conftool/dbconfig/20240411-141300-arnaudb.json
  • 14:12 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 25%: Repool', diff saved to https://phabricator.wikimedia.org/P60414 and previous config saved to /var/cache/conftool/dbconfig/20240411-141139-arnaudb.json
  • 14:11 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:10 elukey: move cassandra instances on aqs1010 to PKI TLS certs - T352647
  • 14:10 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2149.codfw.wmnet with reason: host reimage
  • 14:09 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:09 Dreamy_Jazz: Afternoon UTC backport window finished
  • 14:09 moritzm: installing NSS security updates
  • 14:08 dreamyjazz@deploy1002: Finished scap: Backport for Ignore missing title/page in CheckUserLookupUtils::getManualLogEntryFromRow (T362284) (duration: 17m 42s)
  • 14:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2149.codfw.wmnet with reason: host reimage
  • 14:06 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2042.codfw.wmnet with reason: host reimage
  • 14:06 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
  • 14:03 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2042.codfw.wmnet with reason: host reimage
  • 14:01 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3073.esams.wmnet with reason: host reimage
  • 13:59 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on aqs1010.eqiad.wmnet with reason: Upgrade to PKI
  • 13:59 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on aqs1010.eqiad.wmnet with reason: Upgrade to PKI
  • 13:59 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 25%: reool', diff saved to https://phabricator.wikimedia.org/P60413 and previous config saved to /var/cache/conftool/dbconfig/20240411-135858-arnaudb.json
  • 13:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60412 and previous config saved to /var/cache/conftool/dbconfig/20240411-135846-root.json
  • 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 20%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60411 and previous config saved to /var/cache/conftool/dbconfig/20240411-135819-arnaudb.json
  • 13:58 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3073.esams.wmnet with reason: host reimage
  • 13:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 25%: repool', diff saved to https://phabricator.wikimedia.org/P60410 and previous config saved to /var/cache/conftool/dbconfig/20240411-135754-arnaudb.json
  • 13:57 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and not P{cp2042.codfw.wmnet} and A:cp
  • 13:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 10%: Repool', diff saved to https://phabricator.wikimedia.org/P60409 and previous config saved to /var/cache/conftool/dbconfig/20240411-135634-arnaudb.json
  • 13:55 dreamyjazz@deploy1002: dreamyjazz: Continuing with sync
  • 13:55 dreamyjazz@deploy1002: dreamyjazz: Backport for Ignore missing title/page in CheckUserLookupUtils::getManualLogEntryFromRow (T362284) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:54 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2008.wikimedia.org
  • 13:54 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:54 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 13:53 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 13:51 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2149.codfw.wmnet with OS bookworm
  • 13:50 dreamyjazz@deploy1002: Started scap: Backport for Ignore missing title/page in CheckUserLookupUtils::getManualLogEntryFromRow (T362284)
  • 13:49 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2149', diff saved to https://phabricator.wikimedia.org/P60408 and previous config saved to /var/cache/conftool/dbconfig/20240411-134932-root.json
  • 13:49 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:46 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:46 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp2042.codfw.wmnet with OS bullseye
  • 13:45 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp2042.codfw.wmnet,service=(cdn|ats-be)
  • 13:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60407 and previous config saved to /var/cache/conftool/dbconfig/20240411-134341-root.json
  • 13:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 10%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60406 and previous config saved to /var/cache/conftool/dbconfig/20240411-134312-arnaudb.json
  • 13:41 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 13:36 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-eqiad
  • 13:34 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp3073.esams.wmnet with OS bullseye
  • 13:32 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3073.esams.wmnet,service=(cdn|ats-be)
  • 13:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db2160.codfw.wmnet with reason: reboot multiinstance replica
  • 13:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db2160.codfw.wmnet with reason: reboot multiinstance replica
  • 13:32 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:31 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2133.codfw.wmnet
  • 13:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60405 and previous config saved to /var/cache/conftool/dbconfig/20240411-132834-root.json
  • 13:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 5%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60404 and previous config saved to /var/cache/conftool/dbconfig/20240411-132807-arnaudb.json
  • 13:27 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and not P{cp[5030,5032].eqsin.wmnet} and A:cp
  • 13:26 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2133.codfw.wmnet
  • 13:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[2133,2160].codfw.wmnet with reason: reboot
  • 13:25 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db[2133,2160].codfw.wmnet with reason: reboot
  • 13:23 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2135.codfw.wmnet
  • 13:18 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2135.codfw.wmnet
  • 13:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[2135,2160].codfw.wmnet with reason: reboot
  • 13:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db[2135,2160].codfw.wmnet with reason: reboot
  • 13:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2134.codfw.wmnet
  • 13:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60403 and previous config saved to /var/cache/conftool/dbconfig/20240411-131327-root.json
  • 13:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 4%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60402 and previous config saved to /var/cache/conftool/dbconfig/20240411-131301-arnaudb.json
  • 13:12 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:12 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2134.codfw.wmnet
  • 13:12 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[2134,2160].codfw.wmnet with reason: reboot
  • 13:11 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db[2134,2160].codfw.wmnet with reason: reboot
  • 13:00 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
  • 12:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2132.codfw.wmnet
  • 12:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60401 and previous config saved to /var/cache/conftool/dbconfig/20240411-125821-root.json
  • 12:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 2%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60400 and previous config saved to /var/cache/conftool/dbconfig/20240411-125755-arnaudb.json
  • 12:54 akosiaris: lower weight of mw1437 back to 10 from the 30 I had upped it to yesterday. The backlog of videoscaling is apparently now served and CPU usage has reached "normal" levels
  • 12:54 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2132.codfw.wmnet
  • 12:54 jayme@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:53 akosiaris@cumin1002: conftool action : set/weight=10; selector: name=mw1437.*.wmnet,dc=eqiad
  • 12:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2132,2160].codfw.wmnet with reason: reboot
  • 12:53 jayme@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db[2132,2160].codfw.wmnet with reason: reboot
  • 12:52 jayme@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:52 jayme@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:51 jayme@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 12:50 jayme@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 12:49 jayme@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:49 jayme@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 12:45 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 12:24 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
  • 12:24 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
  • 12:23 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
  • 12:22 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
  • 12:21 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2008.wikimedia.org with OS bookworm
  • 12:21 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:20 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
  • 12:20 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:20 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2008.wikimedia.org on all recursors
  • 12:20 btullis@deploy1002: helmfile [staging] START helmfile.d/services/editor-analytics: apply
  • 12:19 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2008.wikimedia.org on all recursors
  • 12:19 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:19 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:18 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:16 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 12:16 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 12:16 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org
  • 12:16 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 12:16 ayounsi@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2008.wikimedia.org
  • 12:16 ayounsi@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=97)
  • 12:16 ayounsi@cumin1002: END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:16 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:15 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 12:15 moritzm: installing gnutls28 security updates
  • 12:14 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 12:13 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 12:13 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org
  • 12:13 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 12:13 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2008.wikimedia.org
  • 12:13 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:13 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 12:13 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bullseye
  • 12:12 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 12:10 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 12:06 ayounsi@cumin1002: START - Cookbook sre.hosts.decommission for hosts testvm2008.wikimedia.org
  • 12:06 ayounsi@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host testvm2008.wikimedia.org
  • 12:06 ayounsi@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host testvm2008.wikimedia.org with OS bookworm
  • 12:02 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 12:01 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 12:01 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 11:59 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 11:59 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:58 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:58 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2008.wikimedia.org with OS bookworm
  • 11:57 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 11:57 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 11:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2177.codfw.wmnet with OS bookworm
  • 11:50 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 11:50 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 11:49 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2008.wikimedia.org on all recursors
  • 11:49 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2008.wikimedia.org on all recursors
  • 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 11:47 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 11:47 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 11:45 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 11:45 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org
  • 11:33 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bullseye
  • 11:31 moritzm: installing postgresql-15 security updates
  • 11:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2177.codfw.wmnet with reason: host reimage
  • 11:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2177.codfw.wmnet with reason: host reimage
  • 11:24 effie: upload prometheus-memcached-exporter 0.14.2-1~wmf1 to bookworm-wikimedia main - T350807
  • 11:22 effie: upload memkeys 20181031-2-s1 to bookworm-wikimedia main - T362160
  • 11:22 effie: upload memkeys 20181031-2-s1 to bookworm-wikimedia main
  • 11:10 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2177.codfw.wmnet with OS bookworm
  • 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2177', diff saved to https://phabricator.wikimedia.org/P60394 and previous config saved to /var/cache/conftool/dbconfig/20240411-110938-root.json
  • 10:53 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw2412.codfw.wmnet|mw2413.codfw.wmnet|mw2414.codfw.wmnet|mw2415.codfw.wmnet|mw2416.codfw.wmnet|mw2417.codfw.wmnet|mw2418.codfw.wmnet),cluster=kubernetes,service=kubesvc
  • 10:52 claime: Pooling and uncordoning mw2412.codfw.wmnet,mw2413.codfw.wmnet,mw2414.codfw.wmnet,mw2415.codfw.wmnet,mw2416.codfw.wmnet,mw2417.codfw.wmnet,mw2418.codfw.wmnet - T351074
  • 10:43 moritzm: installing modsecurity-apache security updates
  • 10:37 claime: Running homer 'cr*codfw*' commit 'T351074'
  • 10:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2413.codfw.wmnet with OS bullseye
  • 10:31 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2418.codfw.wmnet with OS bullseye
  • 10:30 moritzm: installing xerces-c security updates
  • 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60393 and previous config saved to /var/cache/conftool/dbconfig/20240411-103005-root.json
  • 10:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2412.codfw.wmnet with OS bullseye
  • 10:25 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2415.codfw.wmnet with OS bullseye
  • 10:22 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2416.codfw.wmnet with OS bullseye
  • 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1247 (T356166)', diff saved to https://phabricator.wikimedia.org/P60392 and previous config saved to /var/cache/conftool/dbconfig/20240411-102153-marostegui.json
  • 10:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
  • 10:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
  • 10:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 100%: post schema update', diff saved to https://phabricator.wikimedia.org/P60391 and previous config saved to /var/cache/conftool/dbconfig/20240411-102031-arnaudb.json
  • 10:19 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2417.codfw.wmnet with OS bullseye
  • 10:17 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2413.codfw.wmnet with reason: host reimage
  • 10:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2414.codfw.wmnet with OS bullseye
  • 10:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60390 and previous config saved to /var/cache/conftool/dbconfig/20240411-101500-root.json
  • 10:13 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2418.codfw.wmnet with reason: host reimage
  • 10:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2412.codfw.wmnet with reason: host reimage
  • 10:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2415.codfw.wmnet with reason: host reimage
  • 10:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: post schema update', diff saved to https://phabricator.wikimedia.org/P60389 and previous config saved to /var/cache/conftool/dbconfig/20240411-100525-arnaudb.json
  • 10:03 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2416.codfw.wmnet with reason: host reimage
  • 10:00 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2417.codfw.wmnet with reason: host reimage
  • 09:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60388 and previous config saved to /var/cache/conftool/dbconfig/20240411-095954-root.json
  • 09:57 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2418.codfw.wmnet with reason: host reimage
  • 09:57 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2414.codfw.wmnet with reason: host reimage
  • 09:57 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "testvm2007 - ayounsi@cumin1002"
  • 09:57 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2417.codfw.wmnet with reason: host reimage
  • 09:56 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2416.codfw.wmnet with reason: host reimage
  • 09:56 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "testvm2007 - ayounsi@cumin1002"
  • 09:56 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2415.codfw.wmnet with reason: host reimage
  • 09:55 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2413.codfw.wmnet with reason: host reimage
  • 09:55 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2412.codfw.wmnet with reason: host reimage
  • 09:55 ayounsi@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2007.codfw.wmnet
  • 09:55 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2007.codfw.wmnet with OS bookworm
  • 09:54 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2414.codfw.wmnet with reason: host reimage
  • 09:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 50%: post schema update', diff saved to https://phabricator.wikimedia.org/P60387 and previous config saved to /var/cache/conftool/dbconfig/20240411-095019-arnaudb.json
  • 09:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60386 and previous config saved to /var/cache/conftool/dbconfig/20240411-094448-root.json
  • 09:40 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2418.codfw.wmnet with OS bullseye
  • 09:40 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2007.codfw.wmnet with reason: host reimage
  • 09:40 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2417.codfw.wmnet with OS bullseye
  • 09:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2416.codfw.wmnet with OS bullseye
  • 09:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2415.codfw.wmnet with OS bullseye
  • 09:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2414.codfw.wmnet with OS bullseye
  • 09:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2413.codfw.wmnet with OS bullseye
  • 09:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2412.codfw.wmnet with OS bullseye
  • 09:38 ayounsi@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2007.codfw.wmnet with reason: host reimage
  • 09:37 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3072.esams.wmnet
  • 09:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 25%: post schema update', diff saved to https://phabricator.wikimedia.org/P60384 and previous config saved to /var/cache/conftool/dbconfig/20240411-093513-arnaudb.json
  • 09:32 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3072.esams.wmnet with OS bullseye
  • 09:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 09:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 09:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60383 and previous config saved to /var/cache/conftool/dbconfig/20240411-092942-root.json
  • 09:27 arnaudb@cumin1002: dbctl restore of MediaWiki config (dc=all) from /var/cache/conftool/dbconfig/20240411-092622-arnaudb.json
  • 09:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P60382 and previous config saved to /var/cache/conftool/dbconfig/20240411-092622-arnaudb.json
  • 09:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 09:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 09:25 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2007.codfw.wmnet with OS bookworm
  • 09:25 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
  • 09:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 depool', diff saved to https://phabricator.wikimedia.org/P60381 and previous config saved to /var/cache/conftool/dbconfig/20240411-092501-arnaudb.json
  • 09:24 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
  • 09:24 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2007.codfw.wmnet on all recursors
  • 09:24 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2007.codfw.wmnet on all recursors
  • 09:24 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:24 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
  • 09:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 weight bump T362302', diff saved to https://phabricator.wikimedia.org/P60380 and previous config saved to /var/cache/conftool/dbconfig/20240411-092318-arnaudb.json
  • 09:20 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
  • 09:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2114 to s6 primary T362302', diff saved to https://phabricator.wikimedia.org/P60379 and previous config saved to /var/cache/conftool/dbconfig/20240411-092012-arnaudb.json
  • 09:19 arnaudb: Starting s6 codfw failover from db2129 to db2114 - T362302
  • 09:16 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 09:16 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2007.codfw.wmnet
  • 09:13 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 09:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60378 and previous config saved to /var/cache/conftool/dbconfig/20240411-091255-root.json
  • 09:12 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 09:06 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
  • 09:03 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
  • 08:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2114 with weight 0 T362302', diff saved to https://phabricator.wikimedia.org/P60377 and previous config saved to /var/cache/conftool/dbconfig/20240411-085926-arnaudb.json
  • 08:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s6 T362302
  • 08:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s6 T362302
  • 08:58 ayounsi@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2006.codfw.wmnet
  • 08:58 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2006.codfw.wmnet with OS bookworm
  • 08:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60376 and previous config saved to /var/cache/conftool/dbconfig/20240411-085749-root.json
  • 08:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1198.eqiad.wmnet with OS bookworm
  • 08:50 jelto@cumin1002: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 08:45 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2006.codfw.wmnet with reason: host reimage
  • 08:45 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 08:45 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on matomo1003.eqiad.wmnet with reason: Adding disk
  • 08:45 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on matomo1003.eqiad.wmnet with reason: Adding disk
  • 08:42 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 08:42 ayounsi@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2006.codfw.wmnet with reason: host reimage
  • 08:40 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3072.esams.wmnet with OS bullseye
  • 08:40 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
  • 08:37 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
  • 08:36 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 08:36 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1198.eqiad.wmnet with OS bookworm
  • 08:31 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2006.codfw.wmnet with OS bookworm
  • 08:29 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
  • 08:29 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
  • 08:28 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2006.codfw.wmnet on all recursors
  • 08:28 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2006.codfw.wmnet on all recursors
  • 08:28 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:28 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
  • 08:27 marostegui@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1198.eqiad.wmnet with OS bookworm
  • 08:27 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
  • 08:26 fabfur@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp3072.esams.wmnet with OS bullseye
  • 08:25 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 08:25 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2006.codfw.wmnet
  • 08:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
  • 08:20 hashar: MediaWiki train is blocked
  • 08:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
  • 08:13 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
  • 08:10 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
  • 08:06 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp2002.wikimedia.org
  • 08:06 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:06 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp2002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 08:06 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1198.eqiad.wmnet with OS bookworm
  • 08:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1198', diff saved to https://phabricator.wikimedia.org/P60374 and previous config saved to /var/cache/conftool/dbconfig/20240411-080502-root.json
  • 08:03 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp2002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 08:01 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
  • 07:56 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp2002.wikimedia.org
  • 07:47 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3072.esams.wmnet with OS bullseye
  • 07:44 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3072.esams.wmnet
  • 07:39 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 07:39 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 07:25 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60373 and previous config saved to /var/cache/conftool/dbconfig/20240411-072503-root.json
  • 07:10 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp1002.wikimedia.org
  • 07:10 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 07:10 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60372 and previous config saved to /var/cache/conftool/dbconfig/20240411-070958-root.json
  • 07:08 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 07:05 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
  • 07:00 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp1002.wikimedia.org
  • 06:57 slyngshede@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts idp1002.wikimedia.org
  • 06:56 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp1002.wikimedia.org
  • 06:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60371 and previous config saved to /var/cache/conftool/dbconfig/20240411-065452-root.json
  • 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60370 and previous config saved to /var/cache/conftool/dbconfig/20240411-063946-root.json
  • 06:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T360332)', diff saved to https://phabricator.wikimedia.org/P60369 and previous config saved to /var/cache/conftool/dbconfig/20240411-062728-arnaudb.json
  • 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60368 and previous config saved to /var/cache/conftool/dbconfig/20240411-062440-root.json
  • 06:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P60367 and previous config saved to /var/cache/conftool/dbconfig/20240411-061220-arnaudb.json
  • 06:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60366 and previous config saved to /var/cache/conftool/dbconfig/20240411-060934-root.json
  • 05:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P60365 and previous config saved to /var/cache/conftool/dbconfig/20240411-055712-arnaudb.json
  • 05:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60364 and previous config saved to /var/cache/conftool/dbconfig/20240411-055428-root.json
  • 05:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1189.eqiad.wmnet with OS bookworm
  • 05:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T360332)', diff saved to https://phabricator.wikimedia.org/P60363 and previous config saved to /var/cache/conftool/dbconfig/20240411-054205-arnaudb.json
  • 05:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2216 (T360332)', diff saved to https://phabricator.wikimedia.org/P60362 and previous config saved to /var/cache/conftool/dbconfig/20240411-053903-arnaudb.json
  • 05:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2216.codfw.wmnet with reason: Maintenance
  • 05:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2216.codfw.wmnet with reason: Maintenance
  • 05:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60361 and previous config saved to /var/cache/conftool/dbconfig/20240411-053840-arnaudb.json
  • 05:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1189.eqiad.wmnet with reason: host reimage
  • 05:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1189.eqiad.wmnet with reason: host reimage
  • 05:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P60360 and previous config saved to /var/cache/conftool/dbconfig/20240411-052333-arnaudb.json
  • 05:15 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1189.eqiad.wmnet with OS bookworm
  • 05:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1189', diff saved to https://phabricator.wikimedia.org/P60359 and previous config saved to /var/cache/conftool/dbconfig/20240411-051341-root.json
  • 05:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P60358 and previous config saved to /var/cache/conftool/dbconfig/20240411-050825-arnaudb.json
  • 04:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60357 and previous config saved to /var/cache/conftool/dbconfig/20240411-045317-arnaudb.json
  • 04:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60356 and previous config saved to /var/cache/conftool/dbconfig/20240411-045024-arnaudb.json
  • 04:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2212.codfw.wmnet with reason: Maintenance
  • 04:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2212.codfw.wmnet with reason: Maintenance
  • 04:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60355 and previous config saved to /var/cache/conftool/dbconfig/20240411-045011-arnaudb.json
  • 04:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P60354 and previous config saved to /var/cache/conftool/dbconfig/20240411-043502-arnaudb.json
  • 04:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P60353 and previous config saved to /var/cache/conftool/dbconfig/20240411-041954-arnaudb.json
  • 04:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60352 and previous config saved to /var/cache/conftool/dbconfig/20240411-040447-arnaudb.json
  • 04:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60351 and previous config saved to /var/cache/conftool/dbconfig/20240411-040147-arnaudb.json
  • 04:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2188.codfw.wmnet with reason: Maintenance
  • 04:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2188.codfw.wmnet with reason: Maintenance
  • 04:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T360332)', diff saved to https://phabricator.wikimedia.org/P60350 and previous config saved to /var/cache/conftool/dbconfig/20240411-040124-arnaudb.json
  • 03:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P60349 and previous config saved to /var/cache/conftool/dbconfig/20240411-034617-arnaudb.json
  • 03:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P60348 and previous config saved to /var/cache/conftool/dbconfig/20240411-033109-arnaudb.json
  • 03:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T360332)', diff saved to https://phabricator.wikimedia.org/P60347 and previous config saved to /var/cache/conftool/dbconfig/20240411-031602-arnaudb.json
  • 03:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2176 (T360332)', diff saved to https://phabricator.wikimedia.org/P60346 and previous config saved to /var/cache/conftool/dbconfig/20240411-031310-arnaudb.json
  • 03:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 03:12 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 03:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T360332)', diff saved to https://phabricator.wikimedia.org/P60345 and previous config saved to /var/cache/conftool/dbconfig/20240411-031247-arnaudb.json
  • 02:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P60344 and previous config saved to /var/cache/conftool/dbconfig/20240411-025740-arnaudb.json
  • 02:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P60343 and previous config saved to /var/cache/conftool/dbconfig/20240411-024232-arnaudb.json
  • 02:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 02:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 02:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60342 and previous config saved to /var/cache/conftool/dbconfig/20240411-023125-marostegui.json
  • 02:30 cstone: civicrm upgraded from a382a7b0 to c2569254
  • 02:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T360332)', diff saved to https://phabricator.wikimedia.org/P60341 and previous config saved to /var/cache/conftool/dbconfig/20240411-022725-arnaudb.json
  • 02:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2174 (T360332)', diff saved to https://phabricator.wikimedia.org/P60340 and previous config saved to /var/cache/conftool/dbconfig/20240411-022433-arnaudb.json
  • 02:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 02:24 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 02:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T360332)', diff saved to https://phabricator.wikimedia.org/P60339 and previous config saved to /var/cache/conftool/dbconfig/20240411-022410-arnaudb.json
  • 02:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60338 and previous config saved to /var/cache/conftool/dbconfig/20240411-021617-marostegui.json
  • 02:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P60337 and previous config saved to /var/cache/conftool/dbconfig/20240411-020903-arnaudb.json
  • 02:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60336 and previous config saved to /var/cache/conftool/dbconfig/20240411-020110-marostegui.json
  • 01:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P60335 and previous config saved to /var/cache/conftool/dbconfig/20240411-015355-arnaudb.json
  • 01:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60334 and previous config saved to /var/cache/conftool/dbconfig/20240411-014602-marostegui.json
  • 01:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T360332)', diff saved to https://phabricator.wikimedia.org/P60333 and previous config saved to /var/cache/conftool/dbconfig/20240411-013848-arnaudb.json
  • 01:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2173 (T360332)', diff saved to https://phabricator.wikimedia.org/P60332 and previous config saved to /var/cache/conftool/dbconfig/20240411-013657-arnaudb.json
  • 01:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 01:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 01:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 01:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 01:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T360332)', diff saved to https://phabricator.wikimedia.org/P60331 and previous config saved to /var/cache/conftool/dbconfig/20240411-013618-arnaudb.json
  • 01:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P60330 and previous config saved to /var/cache/conftool/dbconfig/20240411-012110-arnaudb.json
  • 01:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P60329 and previous config saved to /var/cache/conftool/dbconfig/20240411-010601-arnaudb.json
  • 00:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T360332)', diff saved to https://phabricator.wikimedia.org/P60328 and previous config saved to /var/cache/conftool/dbconfig/20240411-005054-arnaudb.json
  • 00:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2170 (T360332)', diff saved to https://phabricator.wikimedia.org/P60327 and previous config saved to /var/cache/conftool/dbconfig/20240411-004758-arnaudb.json
  • 00:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 00:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 00:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T360332)', diff saved to https://phabricator.wikimedia.org/P60326 and previous config saved to /var/cache/conftool/dbconfig/20240411-004735-arnaudb.json
  • 00:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60325 and previous config saved to /var/cache/conftool/dbconfig/20240411-004536-marostegui.json
  • 00:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 00:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 00:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T356166)', diff saved to https://phabricator.wikimedia.org/P60324 and previous config saved to /var/cache/conftool/dbconfig/20240411-004514-marostegui.json
  • 00:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P60323 and previous config saved to /var/cache/conftool/dbconfig/20240411-003226-arnaudb.json
  • 00:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P60322 and previous config saved to /var/cache/conftool/dbconfig/20240411-003005-marostegui.json
  • 00:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P60321 and previous config saved to /var/cache/conftool/dbconfig/20240411-001718-arnaudb.json
  • 00:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P60320 and previous config saved to /var/cache/conftool/dbconfig/20240411-001458-marostegui.json
  • 00:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T360332)', diff saved to https://phabricator.wikimedia.org/P60319 and previous config saved to /var/cache/conftool/dbconfig/20240411-000211-arnaudb.json

2024-04-10

  • 23:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T356166)', diff saved to https://phabricator.wikimedia.org/P60318 and previous config saved to /var/cache/conftool/dbconfig/20240410-235950-marostegui.json
  • 23:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2153 (T360332)', diff saved to https://phabricator.wikimedia.org/P60317 and previous config saved to /var/cache/conftool/dbconfig/20240410-235920-arnaudb.json
  • 23:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 23:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 23:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T360332)', diff saved to https://phabricator.wikimedia.org/P60316 and previous config saved to /var/cache/conftool/dbconfig/20240410-235857-arnaudb.json
  • 23:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P60315 and previous config saved to /var/cache/conftool/dbconfig/20240410-234350-arnaudb.json
  • 23:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P60314 and previous config saved to /var/cache/conftool/dbconfig/20240410-232842-arnaudb.json
  • 23:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T360332)', diff saved to https://phabricator.wikimedia.org/P60313 and previous config saved to /var/cache/conftool/dbconfig/20240410-231335-arnaudb.json
  • 23:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2146 (T360332)', diff saved to https://phabricator.wikimedia.org/P60312 and previous config saved to /var/cache/conftool/dbconfig/20240410-231032-arnaudb.json
  • 23:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 23:10 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 23:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T360332)', diff saved to https://phabricator.wikimedia.org/P60311 and previous config saved to /var/cache/conftool/dbconfig/20240410-231008-arnaudb.json
  • 22:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P60310 and previous config saved to /var/cache/conftool/dbconfig/20240410-225500-arnaudb.json
  • 22:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P60309 and previous config saved to /var/cache/conftool/dbconfig/20240410-223953-arnaudb.json
  • 22:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T360332)', diff saved to https://phabricator.wikimedia.org/P60308 and previous config saved to /var/cache/conftool/dbconfig/20240410-222445-arnaudb.json
  • 22:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2145 (T360332)', diff saved to https://phabricator.wikimedia.org/P60307 and previous config saved to /var/cache/conftool/dbconfig/20240410-222150-arnaudb.json
  • 22:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 22:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 22:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 22:20 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 22:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T360332)', diff saved to https://phabricator.wikimedia.org/P60306 and previous config saved to /var/cache/conftool/dbconfig/20240410-222028-arnaudb.json
  • 22:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P60305 and previous config saved to /var/cache/conftool/dbconfig/20240410-220521-arnaudb.json
  • 21:56 mutante: prometheus - recreating deleted TLS certs/keys in private repo
  • 21:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P60304 and previous config saved to /var/cache/conftool/dbconfig/20240410-215014-arnaudb.json
  • 21:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T360332)', diff saved to https://phabricator.wikimedia.org/P60303 and previous config saved to /var/cache/conftool/dbconfig/20240410-213506-arnaudb.json
  • 21:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2130 (T360332)', diff saved to https://phabricator.wikimedia.org/P60302 and previous config saved to /var/cache/conftool/dbconfig/20240410-213203-arnaudb.json
  • 21:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 21:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 21:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T360332)', diff saved to https://phabricator.wikimedia.org/P60301 and previous config saved to /var/cache/conftool/dbconfig/20240410-213140-arnaudb.json
  • 21:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P60300 and previous config saved to /var/cache/conftool/dbconfig/20240410-211632-arnaudb.json
  • 21:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P60298 and previous config saved to /var/cache/conftool/dbconfig/20240410-210125-arnaudb.json
  • 20:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T360332)', diff saved to https://phabricator.wikimedia.org/P60297 and previous config saved to /var/cache/conftool/dbconfig/20240410-204617-arnaudb.json
  • 20:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2116 (T360332)', diff saved to https://phabricator.wikimedia.org/P60296 and previous config saved to /var/cache/conftool/dbconfig/20240410-204316-arnaudb.json
  • 20:43 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 20:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 20:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112 (T360332)', diff saved to https://phabricator.wikimedia.org/P60295 and previous config saved to /var/cache/conftool/dbconfig/20240410-204253-arnaudb.json
  • 20:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P60294 and previous config saved to /var/cache/conftool/dbconfig/20240410-202745-arnaudb.json
  • 20:17 cjming: end of UTC late backport window
  • 20:15 cjming@deploy1002: Finished scap: Backport for LogStash: log HtmlOutputRendererHelper channel (T356157) (duration: 13m 51s)
  • 20:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P60293 and previous config saved to /var/cache/conftool/dbconfig/20240410-201237-arnaudb.json
  • 20:04 cjming@deploy1002: cjming and daniel: Continuing with sync
  • 20:04 cjming@deploy1002: cjming and daniel: Backport for LogStash: log HtmlOutputRendererHelper channel (T356157) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:01 cjming@deploy1002: Started scap: Backport for LogStash: log HtmlOutputRendererHelper channel (T356157)
  • 19:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112 (T360332)', diff saved to https://phabricator.wikimedia.org/P60292 and previous config saved to /var/cache/conftool/dbconfig/20240410-195730-arnaudb.json
  • 19:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2112 (T360332)', diff saved to https://phabricator.wikimedia.org/P60291 and previous config saved to /var/cache/conftool/dbconfig/20240410-195430-arnaudb.json
  • 19:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2112.codfw.wmnet with reason: Maintenance
  • 19:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2112.codfw.wmnet with reason: Maintenance
  • 19:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 19:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 19:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 19:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 19:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 19:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 19:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 19:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 19:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 19:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 19:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T360332)', diff saved to https://phabricator.wikimedia.org/P60290 and previous config saved to /var/cache/conftool/dbconfig/20240410-194909-arnaudb.json
  • 19:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P60289 and previous config saved to /var/cache/conftool/dbconfig/20240410-193402-arnaudb.json
  • 19:24 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3071.esams.wmnet,service=(cdn|ats-be)
  • 19:20 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3071.esams.wmnet with OS bullseye
  • 19:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P60288 and previous config saved to /var/cache/conftool/dbconfig/20240410-191854-arnaudb.json
  • 19:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T360332)', diff saved to https://phabricator.wikimedia.org/P60287 and previous config saved to /var/cache/conftool/dbconfig/20240410-190347-arnaudb.json
  • 18:54 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3071.esams.wmnet with reason: host reimage
  • 18:51 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3071.esams.wmnet with reason: host reimage
  • 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1235 (T360332)', diff saved to https://phabricator.wikimedia.org/P60285 and previous config saved to /var/cache/conftool/dbconfig/20240410-184656-arnaudb.json
  • 18:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1235.eqiad.wmnet with reason: Maintenance
  • 18:46 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1235.eqiad.wmnet with reason: Maintenance
  • 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T360332)', diff saved to https://phabricator.wikimedia.org/P60284 and previous config saved to /var/cache/conftool/dbconfig/20240410-184633-arnaudb.json
  • 18:34 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/echostore: apply
  • 18:34 eevans@deploy1002: helmfile [staging] START helmfile.d/services/echostore: apply
  • 18:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P60283 and previous config saved to /var/cache/conftool/dbconfig/20240410-183126-arnaudb.json
  • 18:30 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
  • 18:28 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp3071.esams.wmnet with OS bullseye
  • 18:26 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1115.eqiad.wmnet with OS bullseye
  • 18:24 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3071.esams.wmnet,service=(cdn|ats-be)
  • 18:17 eevans@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 18:16 eevans@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 18:16 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 18:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P60282 and previous config saved to /var/cache/conftool/dbconfig/20240410-181618-arnaudb.json
  • 18:15 eevans@deploy1002: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
  • 18:08 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1115.eqiad.wmnet with reason: host reimage
  • 18:05 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1115.eqiad.wmnet with reason: host reimage
  • 18:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T360332)', diff saved to https://phabricator.wikimedia.org/P60281 and previous config saved to /var/cache/conftool/dbconfig/20240410-180111-arnaudb.json
  • 17:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1234 (T360332)', diff saved to https://phabricator.wikimedia.org/P60280 and previous config saved to /var/cache/conftool/dbconfig/20240410-175816-arnaudb.json
  • 17:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 17:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 17:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T360332)', diff saved to https://phabricator.wikimedia.org/P60279 and previous config saved to /var/cache/conftool/dbconfig/20240410-175752-arnaudb.json
  • 17:48 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1115.eqiad.wmnet with OS bullseye
  • 17:48 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1115.eqiad.wmnet with OS bullseye
  • 17:46 swfrench-wmf: finished updating A:conf hosts to etcd-mirror 0.0.11-1 (T358636)
  • 17:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P60278 and previous config saved to /var/cache/conftool/dbconfig/20240410-174244-arnaudb.json
  • 17:37 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1115.eqiad.wmnet with OS bullseye
  • 17:37 swfrench-wmf: restarting etcd-mirror on conf2005.codfw.wmnet for T358636
  • 17:35 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:34 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P60277 and previous config saved to /var/cache/conftool/dbconfig/20240410-172736-arnaudb.json
  • 17:21 hashar@deploy1002: Finished scap: Backport for TitleLibrary: Don't register external titles as dependencies (T362222) (duration: 18m 53s)
  • 17:14 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T360332)', diff saved to https://phabricator.wikimedia.org/P60276 and previous config saved to /var/cache/conftool/dbconfig/20240410-171229-arnaudb.json
  • 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T360332)', diff saved to https://phabricator.wikimedia.org/P60275 and previous config saved to /var/cache/conftool/dbconfig/20240410-170930-arnaudb.json
  • 17:09 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 17:09 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T360332)', diff saved to https://phabricator.wikimedia.org/P60274 and previous config saved to /var/cache/conftool/dbconfig/20240410-170907-arnaudb.json
  • 17:07 hashar@deploy1002: hashar: Continuing with sync
  • 17:07 hashar@deploy1002: hashar: Backport for TitleLibrary: Don't register external titles as dependencies (T362222) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 17:06 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:06 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:05 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:05 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
  • 17:04 sukhe: depool cp1115 for firmware downgrade for PXE boot testing: T350179
  • 17:04 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 17:04 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:04 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 17:03 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:02 hnowlan: killing long-running videoscaler ffmpegs
  • 17:02 hashar@deploy1002: Started scap: Backport for TitleLibrary: Don't register external titles as dependencies (T362222)
  • 16:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P60272 and previous config saved to /var/cache/conftool/dbconfig/20240410-165359-arnaudb.json
  • 16:50 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:50 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:50 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:50 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P60270 and previous config saved to /var/cache/conftool/dbconfig/20240410-163851-arnaudb.json
  • 16:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T360332)', diff saved to https://phabricator.wikimedia.org/P60269 and previous config saved to /var/cache/conftool/dbconfig/20240410-162344-arnaudb.json
  • 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1228 (T360332)', diff saved to https://phabricator.wikimedia.org/P60268 and previous config saved to /var/cache/conftool/dbconfig/20240410-162101-arnaudb.json
  • 16:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 16:20 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T360332)', diff saved to https://phabricator.wikimedia.org/P60267 and previous config saved to /var/cache/conftool/dbconfig/20240410-162039-arnaudb.json
  • 16:19 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: sync
  • 16:19 elukey@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: sync
  • 16:16 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [eqiad] DONE helmfile.d/services/termbox: apply
  • 16:16 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [eqiad] START helmfile.d/services/termbox: apply
  • 16:15 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [codfw] DONE helmfile.d/services/termbox: apply
  • 16:15 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [codfw] START helmfile.d/services/termbox: apply
  • 16:14 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [staging] DONE helmfile.d/services/termbox: apply
  • 16:13 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [staging] START helmfile.d/services/termbox: apply
  • 16:12 swfrench-wmf: uploaded etcd-mirror 0.0.11-1 to apt.wikimedia.org (T358636)
  • 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P60265 and previous config saved to /var/cache/conftool/dbconfig/20240410-160531-arnaudb.json
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P60264 and previous config saved to /var/cache/conftool/dbconfig/20240410-155024-arnaudb.json
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T360332)', diff saved to https://phabricator.wikimedia.org/P60262 and previous config saved to /var/cache/conftool/dbconfig/20240410-153516-arnaudb.json
  • 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T360332)', diff saved to https://phabricator.wikimedia.org/P60261 and previous config saved to /var/cache/conftool/dbconfig/20240410-153229-arnaudb.json
  • 15:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 15:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T360332)', diff saved to https://phabricator.wikimedia.org/P60260 and previous config saved to /var/cache/conftool/dbconfig/20240410-153207-arnaudb.json
  • 15:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60259 and previous config saved to /var/cache/conftool/dbconfig/20240410-151659-arnaudb.json
  • 15:14 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 15:14 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 15:14 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 15:13 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 15:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1243 (T356166)', diff saved to https://phabricator.wikimedia.org/P60258 and previous config saved to /var/cache/conftool/dbconfig/20240410-150327-marostegui.json
  • 15:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
  • 15:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
  • 15:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60257 and previous config saved to /var/cache/conftool/dbconfig/20240410-150304-marostegui.json
  • 15:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60256 and previous config saved to /var/cache/conftool/dbconfig/20240410-150152-arnaudb.json
  • 14:58 moritzm: installing debian-archive-keyring updates on buster
  • 14:55 akosiaris: kill all ffmpegs on mw1437 and increase weight of mw1347 from 10 to 30 to direct most queries to it while the other 3 videoscalers serve the backlog
  • 14:54 akosiaris@cumin1002: conftool action : set/weight=30; selector: name=mw1437.*.wmnet,dc=eqiad
  • 14:51 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:51 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:50 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:50 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60255 and previous config saved to /var/cache/conftool/dbconfig/20240410-144757-marostegui.json
  • 14:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T360332)', diff saved to https://phabricator.wikimedia.org/P60254 and previous config saved to /var/cache/conftool/dbconfig/20240410-144644-arnaudb.json
  • 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T360332)', diff saved to https://phabricator.wikimedia.org/P60253 and previous config saved to /var/cache/conftool/dbconfig/20240410-144400-arnaudb.json
  • 14:43 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 14:43 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60252 and previous config saved to /var/cache/conftool/dbconfig/20240410-144336-arnaudb.json
  • 14:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60251 and previous config saved to /var/cache/conftool/dbconfig/20240410-143249-marostegui.json
  • 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60250 and previous config saved to /var/cache/conftool/dbconfig/20240410-142829-arnaudb.json
  • 14:21 sukhe@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:20 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60249 and previous config saved to /var/cache/conftool/dbconfig/20240410-141742-marostegui.json
  • 14:17 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1112.eqiad.wmnet,service=(cdn|ats-be)
  • 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60248 and previous config saved to /var/cache/conftool/dbconfig/20240410-141322-arnaudb.json
  • 14:07 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1112.eqiad.wmnet with OS bullseye
  • 13:58 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
  • 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60246 and previous config saved to /var/cache/conftool/dbconfig/20240410-135814-arnaudb.json
  • 13:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60245 and previous config saved to /var/cache/conftool/dbconfig/20240410-135525-arnaudb.json
  • 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T360332)', diff saved to https://phabricator.wikimedia.org/P60244 and previous config saved to /var/cache/conftool/dbconfig/20240410-135502-arnaudb.json
  • 13:54 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
  • 13:49 denisse: Delete unused Prometheus TLS certificates - T360414
  • 13:47 moritzm: installing unbound security updates
  • 13:46 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1112.eqiad.wmnet with reason: host reimage
  • 13:43 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1112.eqiad.wmnet with reason: host reimage
  • 13:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P60243 and previous config saved to /var/cache/conftool/dbconfig/20240410-133955-arnaudb.json
  • 13:39 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
  • 13:33 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 13:30 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on elastic2088.codfw.wmnet with reason: T361525
  • 13:30 bking@cumin2002: START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on elastic2088.codfw.wmnet with reason: T361525
  • 13:30 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 13:28 eevans@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: apply
  • 13:27 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1112.eqiad.wmnet with OS bullseye
  • 13:26 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1112.eqiad.wmnet with OS bullseye
  • 13:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P60242 and previous config saved to /var/cache/conftool/dbconfig/20240410-132447-arnaudb.json
  • 13:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60241 and previous config saved to /var/cache/conftool/dbconfig/20240410-131716-marostegui.json
  • 13:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 13:17 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1112.eqiad.wmnet with OS bullseye
  • 13:16 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 13:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T356166)', diff saved to https://phabricator.wikimedia.org/P60240 and previous config saved to /var/cache/conftool/dbconfig/20240410-131653-marostegui.json
  • 13:16 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1112.eqiad.wmnet,service=(cdn|ats-be)
  • 13:09 volans@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:09 volans@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test restoring dns entry - volans@cumin2002"
  • 13:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T360332)', diff saved to https://phabricator.wikimedia.org/P60239 and previous config saved to /var/cache/conftool/dbconfig/20240410-130940-arnaudb.json
  • 13:09 volans@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test restoring dns entry - volans@cumin2002"
  • 13:07 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 13:07 volans@cumin2002: START - Cookbook sre.dns.netbox
  • 13:07 sukhe: depool cp4052 for PXE boot issue testing
  • 13:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T360332)', diff saved to https://phabricator.wikimedia.org/P60238 and previous config saved to /var/cache/conftool/dbconfig/20240410-130650-arnaudb.json
  • 13:07 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
  • 13:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 13:06 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 13:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T360332)', diff saved to https://phabricator.wikimedia.org/P60237 and previous config saved to /var/cache/conftool/dbconfig/20240410-130626-arnaudb.json
  • 13:05 volans@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:05 volans@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test removing dns entry - volans@cumin2002"
  • 13:05 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns6001.wikimedia.org,service=authdns-update
  • 13:04 volans@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test removing dns entry - volans@cumin2002"
  • 13:02 volans@cumin2002: START - Cookbook sre.dns.netbox
  • 13:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P60236 and previous config saved to /var/cache/conftool/dbconfig/20240410-130145-marostegui.json
  • 12:59 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2003.wikimedia.org
  • 12:59 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:59 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp-test2003.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 12:56 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=dns6001.wikimedia.org,service=authdns-update
  • 12:56 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp-test2003.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 12:53 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
  • 12:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P60235 and previous config saved to /var/cache/conftool/dbconfig/20240410-125119-arnaudb.json
  • 12:48 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp-test2003.wikimedia.org
  • 12:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P60234 and previous config saved to /var/cache/conftool/dbconfig/20240410-124638-marostegui.json
  • 12:45 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "update for latest VMs - jmm@cumin2002"
  • 12:44 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "update for latest VMs - jmm@cumin2002"
  • 12:25 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60231 and previous config saved to /var/cache/conftool/dbconfig/20240410-122518-root.json
  • 12:25 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 12:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T360332)', diff saved to https://phabricator.wikimedia.org/P60230 and previous config saved to /var/cache/conftool/dbconfig/20240410-122104-arnaudb.json
  • 12:20 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on idp-test1002.wikimedia.org with reason: host reimage
  • 12:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1196 (T360332)', diff saved to https://phabricator.wikimedia.org/P60229 and previous config saved to /var/cache/conftool/dbconfig/20240410-121814-arnaudb.json
  • 12:18 slyngshede@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on idp-test1002.wikimedia.org with reason: host reimage
  • 12:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:18 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 12:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 12:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60228 and previous config saved to /var/cache/conftool/dbconfig/20240410-121743-arnaudb.json
  • 12:15 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P4496 --new-data-type external-id --summary 'T359297' # succeeded
  • 12:14 Lucas_WMDE: lucaswerkmeister-wmde@deploy1002 ~ $ mwscript-k8s extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P4496 --new-data-type external-id --summary 'T359297' # failed, will retry with non-k8s mwscript
  • 12:12 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw1421.eqiad.wmnet|mw1422.eqiad.wmnet|mw1491.eqiad.wmnet|mw1492.eqiad.wmnet|mw1493.eqiad.wmnet),cluster=kubernetes,service=kubesvc
  • 12:11 claime: Pooling and uncordoning mw1421.eqiad.wmnet,mw1422.eqiad.wmnet,mw1491.eqiad.wmnet,mw1492.eqiad.wmnet,mw1493.eqiad.wmnet - T351074
  • 12:10 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60227 and previous config saved to /var/cache/conftool/dbconfig/20240410-121012-root.json
  • 12:04 slyngshede@cumin1002: START - Cookbook sre.hosts.reimage for host idp-test1002.wikimedia.org with OS bookworm
  • 12:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60226 and previous config saved to /var/cache/conftool/dbconfig/20240410-120235-arnaudb.json
  • 12:01 claime: Running homer 'cr*eqiad*' commit 'T351074' and homer 'lsw1-e3-eqiad*' commit 'T351074'
  • 11:55 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60225 and previous config saved to /var/cache/conftool/dbconfig/20240410-115506-root.json
  • 11:54 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1492.eqiad.wmnet with OS bullseye
  • 11:53 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1491.eqiad.wmnet with OS bullseye
  • 11:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1422.eqiad.wmnet with OS bullseye
  • 11:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60224 and previous config saved to /var/cache/conftool/dbconfig/20240410-114728-arnaudb.json
  • 11:45 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1493.eqiad.wmnet with OS bullseye
  • 11:42 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1421.eqiad.wmnet with OS bullseye
  • 11:40 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60223 and previous config saved to /var/cache/conftool/dbconfig/20240410-114001-root.json
  • 11:38 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage
  • 11:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1491.eqiad.wmnet with reason: host reimage
  • 11:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60222 and previous config saved to /var/cache/conftool/dbconfig/20240410-113220-arnaudb.json
  • 11:31 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1422.eqiad.wmnet with reason: host reimage
  • 11:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60221 and previous config saved to /var/cache/conftool/dbconfig/20240410-112929-arnaudb.json
  • 11:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 11:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 11:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T360332)', diff saved to https://phabricator.wikimedia.org/P60220 and previous config saved to /var/cache/conftool/dbconfig/20240410-112907-arnaudb.json
  • 11:28 jiji@deploy1002: Finished scap: Deploy chart changes in gerrit:1015342 (duration: 08m 18s)
  • 11:27 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1493.eqiad.wmnet with reason: host reimage
  • 11:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60219 and previous config saved to /var/cache/conftool/dbconfig/20240410-112455-root.json
  • 11:24 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1421.eqiad.wmnet with reason: host reimage
  • 11:23 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1493.eqiad.wmnet with reason: host reimage
  • 11:22 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage
  • 11:22 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1491.eqiad.wmnet with reason: host reimage
  • 11:21 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1422.eqiad.wmnet with reason: host reimage
  • 11:21 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1421.eqiad.wmnet with reason: host reimage
  • 11:19 jiji@deploy1002: Started scap: Deploy chart changes in gerrit:1015342
  • 11:16 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
  • 11:15 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
  • 11:14 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
  • 11:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60218 and previous config saved to /var/cache/conftool/dbconfig/20240410-111400-arnaudb.json
  • 11:13 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
  • 11:12 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 11:12 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
  • 11:10 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1493.eqiad.wmnet with OS bullseye
  • 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60217 and previous config saved to /var/cache/conftool/dbconfig/20240410-110949-root.json
  • 11:09 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1492.eqiad.wmnet with OS bullseye
  • 11:09 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1491.eqiad.wmnet with OS bullseye
  • 11:08 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 11:08 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1422.eqiad.wmnet with OS bullseye
  • 11:08 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1421.eqiad.wmnet with OS bullseye
  • 11:07 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 11:07 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 11:07 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 11:03 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 11:02 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 11:02 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 11:02 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:59 claime: Depooling mw1421.eqiad.wmnet,mw1422.eqiad.wmnet,mw1491.eqiad.wmnet,mw1492.eqiad.wmnet,mw1493.eqiad.wmnet - T351074
  • 10:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60216 and previous config saved to /var/cache/conftool/dbconfig/20240410-105852-arnaudb.json
  • 10:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1175.eqiad.wmnet with OS bookworm
  • 10:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60215 and previous config saved to /var/cache/conftool/dbconfig/20240410-105444-root.json
  • 10:53 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 10:53 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T360332)', diff saved to https://phabricator.wikimedia.org/P60214 and previous config saved to /var/cache/conftool/dbconfig/20240410-104345-arnaudb.json
  • 10:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1169 (T360332)', diff saved to https://phabricator.wikimedia.org/P60213 and previous config saved to /var/cache/conftool/dbconfig/20240410-104053-arnaudb.json
  • 10:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 10:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 10:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T360332)', diff saved to https://phabricator.wikimedia.org/P60212 and previous config saved to /var/cache/conftool/dbconfig/20240410-104030-arnaudb.json
  • 10:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1175.eqiad.wmnet with reason: host reimage
  • 10:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1175.eqiad.wmnet with reason: host reimage
  • 10:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P60211 and previous config saved to /var/cache/conftool/dbconfig/20240410-102523-arnaudb.json
  • 10:21 claime: Enabling and running puppet on O:docker_registry_ha::registry - T360636
  • 10:19 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1175.eqiad.wmnet with OS bookworm
  • 10:18 claime: Enabling and running puppet on registry1003.eqiad.wmnet - T360636
  • 10:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1175 T362036', diff saved to https://phabricator.wikimedia.org/P60210 and previous config saved to /var/cache/conftool/dbconfig/20240410-101746-root.json
  • 10:16 claime: Disabling puppet on O:docker_registry_ha::registry - T360636
  • 10:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::sanitarium_master
  • 10:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P60209 and previous config saved to /var/cache/conftool/dbconfig/20240410-101015-arnaudb.json
  • 10:08 jiji@deploy1002: Finished scap: (no justification provided) (duration: 27m 59s)
  • 09:58 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::sanitarium_master
  • 09:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T360332)', diff saved to https://phabricator.wikimedia.org/P60208 and previous config saved to /var/cache/conftool/dbconfig/20240410-095508-arnaudb.json
  • 09:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T360332)', diff saved to https://phabricator.wikimedia.org/P60207 and previous config saved to /var/cache/conftool/dbconfig/20240410-095214-arnaudb.json
  • 09:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:42 effie: running scap sync-world to rebuild mw image and pick up gerrit:1015338
  • 09:40 jiji@deploy1002: Started scap: (no justification provided)
  • 08:49 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3070.esams.wmnet
  • 08:42 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3070.esams.wmnet with OS bullseye
  • 08:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60206 and previous config saved to /var/cache/conftool/dbconfig/20240410-083822-arnaudb.json
  • 08:35 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 08:34 gmodena@deploy1002: Finished deploy [airflow-dags/analytics@46818a3]: Deploying cassandra_load_pageview_top_articles changes MR#648 (duration: 00m 33s)
  • 08:34 hashar@deploy1002: Synchronized php: group1 wikis to 1.42.0-wmf.26 refs T360158 (duration: 13m 05s)
  • 08:34 gmodena@deploy1002: Started deploy [airflow-dags/analytics@46818a3]: Deploying cassandra_load_pageview_top_articles changes MR#648
  • 08:25 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 08:25 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 08:25 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 08:25 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 08:24 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 08:24 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 08:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60205 and previous config saved to /var/cache/conftool/dbconfig/20240410-082316-arnaudb.json
  • 08:21 hashar@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.26 refs T360158
  • 08:18 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3070.esams.wmnet with reason: host reimage
  • 08:15 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3070.esams.wmnet with reason: host reimage
  • 08:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60204 and previous config saved to /var/cache/conftool/dbconfig/20240410-080810-arnaudb.json
  • 07:56 moritzm: installing glibc security updates on bullseye
  • 07:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60203 and previous config saved to /var/cache/conftool/dbconfig/20240410-075304-arnaudb.json
  • 07:52 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3070.esams.wmnet with OS bullseye
  • 07:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 100%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60202 and previous config saved to /var/cache/conftool/dbconfig/20240410-075150-arnaudb.json
  • 07:50 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3070.esams.wmnet
  • 07:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60201 and previous config saved to /var/cache/conftool/dbconfig/20240410-073759-arnaudb.json
  • 07:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 75%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60200 and previous config saved to /var/cache/conftool/dbconfig/20240410-073644-arnaudb.json
  • 07:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: dumps::generation::server::spare
  • 07:29 akosiaris@deploy1002: Synchronized wmf-config/mc.php: Dummy sync for https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1018332 (duration: 14m 03s)
  • 07:25 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: dumps::generation::server::spare
  • 07:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60199 and previous config saved to /var/cache/conftool/dbconfig/20240410-072253-arnaudb.json
  • 07:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 50%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60198 and previous config saved to /var/cache/conftool/dbconfig/20240410-072137-arnaudb.json
  • 07:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60197 and previous config saved to /var/cache/conftool/dbconfig/20240410-070745-arnaudb.json
  • 07:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 25%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60196 and previous config saved to /var/cache/conftool/dbconfig/20240410-070631-arnaudb.json
  • 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60195 and previous config saved to /var/cache/conftool/dbconfig/20240410-065929-root.json
  • 06:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60194 and previous config saved to /var/cache/conftool/dbconfig/20240410-065239-arnaudb.json
  • 06:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 20%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60193 and previous config saved to /var/cache/conftool/dbconfig/20240410-065125-arnaudb.json
  • 06:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60192 and previous config saved to /var/cache/conftool/dbconfig/20240410-064423-root.json
  • 06:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60191 and previous config saved to /var/cache/conftool/dbconfig/20240410-063734-arnaudb.json
  • 06:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 10%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60190 and previous config saved to /var/cache/conftool/dbconfig/20240410-063620-arnaudb.json
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60189 and previous config saved to /var/cache/conftool/dbconfig/20240410-062917-root.json
  • 06:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 5%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60188 and previous config saved to /var/cache/conftool/dbconfig/20240410-062114-arnaudb.json
  • 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60187 and previous config saved to /var/cache/conftool/dbconfig/20240410-062003-root.json
  • 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60186 and previous config saved to /var/cache/conftool/dbconfig/20240410-061411-root.json
  • 06:04 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60185 and previous config saved to /var/cache/conftool/dbconfig/20240410-060457-root.json
  • 05:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60184 and previous config saved to /var/cache/conftool/dbconfig/20240410-055906-root.json
  • 05:49 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60183 and previous config saved to /var/cache/conftool/dbconfig/20240410-054952-root.json
  • 05:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60182 and previous config saved to /var/cache/conftool/dbconfig/20240410-054400-root.json
  • 05:34 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60181 and previous config saved to /var/cache/conftool/dbconfig/20240410-053445-root.json
  • 05:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1166.eqiad.wmnet with OS bookworm
  • 05:28 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60180 and previous config saved to /var/cache/conftool/dbconfig/20240410-052854-root.json
  • 05:19 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60179 and previous config saved to /var/cache/conftool/dbconfig/20240410-051939-root.json
  • 05:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1166.eqiad.wmnet with reason: host reimage
  • 05:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1166.eqiad.wmnet with reason: host reimage
  • 05:04 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60178 and previous config saved to /var/cache/conftool/dbconfig/20240410-050434-root.json
  • 04:58 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1166.eqiad.wmnet with OS bookworm
  • 04:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1166 T362134', diff saved to https://phabricator.wikimedia.org/P60177 and previous config saved to /var/cache/conftool/dbconfig/20240410-045710-marostegui.json
  • 04:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repool db1223', diff saved to https://phabricator.wikimedia.org/P60176 and previous config saved to /var/cache/conftool/dbconfig/20240410-045632-marostegui.json
  • 04:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1223 T362134', diff saved to https://phabricator.wikimedia.org/P60175 and previous config saved to /var/cache/conftool/dbconfig/20240410-045534-marostegui.json
  • 04:49 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60174 and previous config saved to /var/cache/conftool/dbconfig/20240410-044928-root.json
  • 04:46 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Kernel reboot
  • 04:46 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Kernel reboot
  • 04:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1241 (T356166)', diff saved to https://phabricator.wikimedia.org/P60173 and previous config saved to /var/cache/conftool/dbconfig/20240410-041604-marostegui.json
  • 04:15 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
  • 04:15 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
  • 04:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T356166)', diff saved to https://phabricator.wikimedia.org/P60172 and previous config saved to /var/cache/conftool/dbconfig/20240410-041541-marostegui.json
  • 04:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P60171 and previous config saved to /var/cache/conftool/dbconfig/20240410-040033-marostegui.json
  • 03:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P60170 and previous config saved to /var/cache/conftool/dbconfig/20240410-034526-marostegui.json
  • 03:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T356166)', diff saved to https://phabricator.wikimedia.org/P60169 and previous config saved to /var/cache/conftool/dbconfig/20240410-033019-marostegui.json

2024-04-09

  • 23:17 eileen: config revision changed from 7908b55e to 974afe9c
  • 23:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T360332)', diff saved to https://phabricator.wikimedia.org/P60168 and previous config saved to /var/cache/conftool/dbconfig/20240409-230828-arnaudb.json
  • 23:08 eileen: config revision changed from 064d18b0 to 7908b55e
  • 22:58 eileen: config revision changed from 8fb02f33 to 064d18b0
  • 22:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60167 and previous config saved to /var/cache/conftool/dbconfig/20240409-225321-arnaudb.json
  • 22:51 eileen: config revision changed from df416a50 to 8fb02f33
  • 22:48 eileen: config revision changed from cea14e30 to df416a50
  • 22:42 eileen: config revision changed from 075ddd44 to cea14e30
  • 22:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60166 and previous config saved to /var/cache/conftool/dbconfig/20240409-223813-arnaudb.json
  • 22:24 eileen: config revision changed from 3c1a0267 to 4638a4d2
  • 22:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T360332)', diff saved to https://phabricator.wikimedia.org/P60165 and previous config saved to /var/cache/conftool/dbconfig/20240409-222306-arnaudb.json
  • 22:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2209 (T360332)', diff saved to https://phabricator.wikimedia.org/P60164 and previous config saved to /var/cache/conftool/dbconfig/20240409-220755-arnaudb.json
  • 22:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
  • 22:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
  • 22:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T360332)', diff saved to https://phabricator.wikimedia.org/P60163 and previous config saved to /var/cache/conftool/dbconfig/20240409-220732-arnaudb.json
  • 22:03 eileen: civicrm upgraded from b05fd08f to a382a7b0
  • 21:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P60162 and previous config saved to /var/cache/conftool/dbconfig/20240409-215225-arnaudb.json
  • 21:38 eileen: civicrm upgraded from 8c7cc208 to b05fd08f
  • 21:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P60161 and previous config saved to /var/cache/conftool/dbconfig/20240409-213717-arnaudb.json
  • 21:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T360332)', diff saved to https://phabricator.wikimedia.org/P60160 and previous config saved to /var/cache/conftool/dbconfig/20240409-212210-arnaudb.json
  • 21:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2205 (T360332)', diff saved to https://phabricator.wikimedia.org/P60159 and previous config saved to /var/cache/conftool/dbconfig/20240409-210656-arnaudb.json
  • 21:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2205.codfw.wmnet with reason: Maintenance
  • 21:06 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2205.codfw.wmnet with reason: Maintenance
  • 21:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T360332)', diff saved to https://phabricator.wikimedia.org/P60158 and previous config saved to /var/cache/conftool/dbconfig/20240409-210633-arnaudb.json
  • 21:01 cjming: end of UTC late backport window
  • 21:00 cjming@deploy1002: Finished scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) (duration: 15m 27s)
  • 20:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60157 and previous config saved to /var/cache/conftool/dbconfig/20240409-205125-arnaudb.json
  • 20:48 cjming@deploy1002: jjmc89 and cjming: Continuing with sync
  • 20:47 cjming@deploy1002: jjmc89 and cjming: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:45 cdanis@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 20:45 cjming@deploy1002: Started scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188)
  • 20:44 cdanis@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 20:44 cjming@deploy1002: Finished scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) (duration: 16m 16s)
  • 20:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60156 and previous config saved to /var/cache/conftool/dbconfig/20240409-203617-arnaudb.json
  • 20:32 cjming@deploy1002: jjmc89 and cjming: Continuing with sync
  • 20:30 cjming@deploy1002: jjmc89 and cjming: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:28 cjming@deploy1002: Started scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188)
  • 20:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60155 and previous config saved to /var/cache/conftool/dbconfig/20240409-202410-arnaudb.json
  • 20:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T360332)', diff saved to https://phabricator.wikimedia.org/P60154 and previous config saved to /var/cache/conftool/dbconfig/20240409-202110-arnaudb.json
  • 20:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P60153 and previous config saved to /var/cache/conftool/dbconfig/20240409-200903-arnaudb.json
  • 20:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2194 (T360332)', diff saved to https://phabricator.wikimedia.org/P60152 and previous config saved to /var/cache/conftool/dbconfig/20240409-200556-arnaudb.json
  • 20:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance
  • 20:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance
  • 20:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T360332)', diff saved to https://phabricator.wikimedia.org/P60151 and previous config saved to /var/cache/conftool/dbconfig/20240409-200533-arnaudb.json
  • 20:02 cdanis@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 20:02 eileen: config revision changed from abccfdc0 to 3c1a0267
  • 20:02 cdanis@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 19:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P60150 and previous config saved to /var/cache/conftool/dbconfig/20240409-195355-arnaudb.json
  • 19:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P60149 and previous config saved to /var/cache/conftool/dbconfig/20240409-195024-arnaudb.json
  • 19:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60148 and previous config saved to /var/cache/conftool/dbconfig/20240409-193848-arnaudb.json
  • 19:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60147 and previous config saved to /var/cache/conftool/dbconfig/20240409-193611-arnaudb.json
  • 19:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2207.codfw.wmnet with reason: Maintenance
  • 19:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2207.codfw.wmnet with reason: Maintenance
  • 19:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P60146 and previous config saved to /var/cache/conftool/dbconfig/20240409-193517-arnaudb.json
  • 19:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 19:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 19:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60145 and previous config saved to /var/cache/conftool/dbconfig/20240409-193507-arnaudb.json
  • 19:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T360332)', diff saved to https://phabricator.wikimedia.org/P60143 and previous config saved to /var/cache/conftool/dbconfig/20240409-192010-arnaudb.json
  • 19:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P60142 and previous config saved to /var/cache/conftool/dbconfig/20240409-192000-arnaudb.json
  • 19:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2190 (T360332)', diff saved to https://phabricator.wikimedia.org/P60141 and previous config saved to /var/cache/conftool/dbconfig/20240409-190459-arnaudb.json
  • 19:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P60140 and previous config saved to /var/cache/conftool/dbconfig/20240409-190452-arnaudb.json
  • 19:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2190.codfw.wmnet with reason: Maintenance
  • 19:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2190.codfw.wmnet with reason: Maintenance
  • 19:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T360332)', diff saved to https://phabricator.wikimedia.org/P60139 and previous config saved to /var/cache/conftool/dbconfig/20240409-190436-arnaudb.json
  • 18:58 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1221 (T356166)', diff saved to https://phabricator.wikimedia.org/P60138 and previous config saved to /var/cache/conftool/dbconfig/20240409-185817-marostegui.json
  • 18:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 18:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 18:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
  • 18:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
  • 18:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T356166)', diff saved to https://phabricator.wikimedia.org/P60137 and previous config saved to /var/cache/conftool/dbconfig/20240409-185736-marostegui.json
  • 18:56 eoghan@cumin1002: END (PASS) - Cookbook sre.gitlab.failover (exit_code=0) Failover of gitlab from gitlab1003.wikimedia.org to gitlab1004.wikimedia.org
  • 18:54 eoghan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 'https://gitlab-replica.wikimedia.org/ https://gitlab-replica-old.wikimedia.org/' on all recursors
  • 18:54 eoghan@cumin1002: START - Cookbook sre.dns.wipe-cache 'https://gitlab-replica.wikimedia.org/ https://gitlab-replica-old.wikimedia.org/' on all recursors
  • 18:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60136 and previous config saved to /var/cache/conftool/dbconfig/20240409-184944-arnaudb.json
  • 18:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P60135 and previous config saved to /var/cache/conftool/dbconfig/20240409-184929-arnaudb.json
  • 18:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60134 and previous config saved to /var/cache/conftool/dbconfig/20240409-184702-arnaudb.json
  • 18:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2189.codfw.wmnet with reason: Maintenance
  • 18:46 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2189.codfw.wmnet with reason: Maintenance
  • 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60133 and previous config saved to /var/cache/conftool/dbconfig/20240409-184640-arnaudb.json
  • 18:43 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@875e0d2]: (no justification provided) (duration: 00m 26s)
  • 18:43 ebysans@deploy1002: Started deploy [airflow-dags/analytics@875e0d2]: (no justification provided)
  • 18:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P60132 and previous config saved to /var/cache/conftool/dbconfig/20240409-184228-marostegui.json
  • 18:41 SandraEbele_: deploying airflow dag to fix mediawiki_history_metrics_monthly dag
  • 18:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P60131 and previous config saved to /var/cache/conftool/dbconfig/20240409-183421-arnaudb.json
  • 18:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P60130 and previous config saved to /var/cache/conftool/dbconfig/20240409-183132-arnaudb.json
  • 18:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P60129 and previous config saved to /var/cache/conftool/dbconfig/20240409-182721-marostegui.json
  • 18:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T360332)', diff saved to https://phabricator.wikimedia.org/P60128 and previous config saved to /var/cache/conftool/dbconfig/20240409-181914-arnaudb.json
  • 18:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P60127 and previous config saved to /var/cache/conftool/dbconfig/20240409-181625-arnaudb.json
  • 18:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T356166)', diff saved to https://phabricator.wikimedia.org/P60126 and previous config saved to /var/cache/conftool/dbconfig/20240409-181213-marostegui.json
  • 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2177 (T360332)', diff saved to https://phabricator.wikimedia.org/P60125 and previous config saved to /var/cache/conftool/dbconfig/20240409-180306-arnaudb.json
  • 18:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 18:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 18:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60124 and previous config saved to /var/cache/conftool/dbconfig/20240409-180242-arnaudb.json
  • 18:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60123 and previous config saved to /var/cache/conftool/dbconfig/20240409-180117-arnaudb.json
  • 17:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60122 and previous config saved to /var/cache/conftool/dbconfig/20240409-175837-arnaudb.json
  • 17:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 17:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 17:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T360332)', diff saved to https://phabricator.wikimedia.org/P60121 and previous config saved to /var/cache/conftool/dbconfig/20240409-175813-arnaudb.json
  • 17:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P60120 and previous config saved to /var/cache/conftool/dbconfig/20240409-174734-arnaudb.json
  • 17:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P60119 and previous config saved to /var/cache/conftool/dbconfig/20240409-174306-arnaudb.json
  • 17:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P60118 and previous config saved to /var/cache/conftool/dbconfig/20240409-173226-arnaudb.json
  • 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P60117 and previous config saved to /var/cache/conftool/dbconfig/20240409-172758-arnaudb.json
  • 17:25 denisse: Delete unused grafana-labs TLS certificates - T360414
  • 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60116 and previous config saved to /var/cache/conftool/dbconfig/20240409-171719-arnaudb.json
  • 17:16 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1113.eqiad.wmnet,service=(cdn|ats-be)
  • 17:15 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1113.eqiad.wmnet with OS bullseye
  • 17:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T360332)', diff saved to https://phabricator.wikimedia.org/P60115 and previous config saved to /var/cache/conftool/dbconfig/20240409-171251-arnaudb.json
  • 17:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2148 (T360332)', diff saved to https://phabricator.wikimedia.org/P60114 and previous config saved to /var/cache/conftool/dbconfig/20240409-171009-arnaudb.json
  • 17:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 17:09 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60113 and previous config saved to /var/cache/conftool/dbconfig/20240409-170946-arnaudb.json
  • 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60112 and previous config saved to /var/cache/conftool/dbconfig/20240409-170234-arnaudb.json
  • 17:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 17:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 17:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 17:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 17:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60111 and previous config saved to /var/cache/conftool/dbconfig/20240409-170155-arnaudb.json
  • 16:56 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1113.eqiad.wmnet with reason: host reimage
  • 16:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60110 and previous config saved to /var/cache/conftool/dbconfig/20240409-165438-arnaudb.json
  • 16:54 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1113.eqiad.wmnet with reason: host reimage
  • 16:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60109 and previous config saved to /var/cache/conftool/dbconfig/20240409-164647-arnaudb.json
  • 16:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60108 and previous config saved to /var/cache/conftool/dbconfig/20240409-163931-arnaudb.json
  • 16:38 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1113.eqiad.wmnet with OS bullseye
  • 16:37 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1113.eqiad.wmnet with OS bullseye
  • 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60107 and previous config saved to /var/cache/conftool/dbconfig/20240409-163140-arnaudb.json
  • 16:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60106 and previous config saved to /var/cache/conftool/dbconfig/20240409-162423-arnaudb.json
  • 16:24 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1113.eqiad.wmnet with OS bullseye
  • 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60105 and previous config saved to /var/cache/conftool/dbconfig/20240409-162142-arnaudb.json
  • 16:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 16:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60104 and previous config saved to /var/cache/conftool/dbconfig/20240409-162119-arnaudb.json
  • 16:21 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1113.eqiad.wmnet
  • 16:20 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1113.eqiad.wmnet
  • 16:20 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1113.eqiad.wmnet with reason: NIC firmware upgrade and reimage
  • 16:19 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1113.eqiad.wmnet with reason: NIC firmware upgrade and reimage
  • 16:19 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1113.eqiad.wmnet,service=(cdn|ats-be)
  • 16:16 sukhe: depool cp1113 for PXE boot issue related testing T350179
  • 16:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60103 and previous config saved to /var/cache/conftool/dbconfig/20240409-161632-arnaudb.json
  • 16:13 denisse: Deleting unused webperf TLS certificates - T360414
  • 16:13 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dumpsdata1005.eqiad.wmnet with OS bullseye
  • 16:06 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
  • 16:06 sukhe: pool cp4052 after reimaging and new NIC firmware
  • 16:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60102 and previous config saved to /var/cache/conftool/dbconfig/20240409-160612-arnaudb.json
  • 16:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60101 and previous config saved to /var/cache/conftool/dbconfig/20240409-160225-arnaudb.json
  • 16:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:01 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host matomo1003.eqiad.wmnet with OS bookworm
  • 15:56 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dumpsdata1005.eqiad.wmnet with reason: host reimage
  • 15:54 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
  • 15:54 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dumpsdata1005.eqiad.wmnet with reason: host reimage
  • 15:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60100 and previous config saved to /var/cache/conftool/dbconfig/20240409-155104-arnaudb.json
  • 15:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 15:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 15:45 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:42 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:40 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1005.eqiad.wmnet with OS bullseye
  • 15:39 moritzm: installing python2.7 security updates
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60099 and previous config saved to /var/cache/conftool/dbconfig/20240409-153557-arnaudb.json
  • 15:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 15:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T360332)', diff saved to https://phabricator.wikimedia.org/P60098 and previous config saved to /var/cache/conftool/dbconfig/20240409-153512-arnaudb.json
  • 15:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60097 and previous config saved to /var/cache/conftool/dbconfig/20240409-153315-arnaudb.json
  • 15:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 15:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 15:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 15:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 15:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T360332)', diff saved to https://phabricator.wikimedia.org/P60096 and previous config saved to /var/cache/conftool/dbconfig/20240409-153257-arnaudb.json
  • 15:32 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 15:32 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 15:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and not P{cp[4037,4041,4045,4049,4052].ulsfo.wmnet} and A:cp
  • 15:26 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 15:24 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1005.eqiad.wmnet with OS bullseye
  • 15:22 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 15:22 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 15:20 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 15:20 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P60095 and previous config saved to /var/cache/conftool/dbconfig/20240409-152005-arnaudb.json
  • 15:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P60094 and previous config saved to /var/cache/conftool/dbconfig/20240409-151750-arnaudb.json
  • 15:16 eoghan@cumin1002: START - Cookbook sre.gitlab.failover Failover of gitlab from gitlab1003.wikimedia.org to gitlab1004.wikimedia.org
  • 15:06 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 15:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P60093 and previous config saved to /var/cache/conftool/dbconfig/20240409-150458-arnaudb.json
  • 15:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P60092 and previous config saved to /var/cache/conftool/dbconfig/20240409-150242-arnaudb.json
  • 15:02 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 15:01 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and not P{cp[4037,4041,4045,4049,4052].ulsfo.wmnet} and A:cp
  • 14:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T360332)', diff saved to https://phabricator.wikimedia.org/P60091 and previous config saved to /var/cache/conftool/dbconfig/20240409-144950-arnaudb.json
  • 14:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T360332)', diff saved to https://phabricator.wikimedia.org/P60090 and previous config saved to /var/cache/conftool/dbconfig/20240409-144735-arnaudb.json
  • 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2125 (T360332)', diff saved to https://phabricator.wikimedia.org/P60089 and previous config saved to /var/cache/conftool/dbconfig/20240409-144447-arnaudb.json
  • 14:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 14:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107 (T360332)', diff saved to https://phabricator.wikimedia.org/P60088 and previous config saved to /var/cache/conftool/dbconfig/20240409-144424-arnaudb.json
  • 14:35 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2105 (T360332)', diff saved to https://phabricator.wikimedia.org/P60087 and previous config saved to /var/cache/conftool/dbconfig/20240409-143445-arnaudb.json
  • 14:34 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 14:34 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 14:33 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:33 btullis@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host matomo1003.eqiad.wmnet
  • 14:33 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
  • 14:30 sukhe@cumin1002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:30 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 14:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 14:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P60086 and previous config saved to /var/cache/conftool/dbconfig/20240409-142916-arnaudb.json
  • 14:28 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
  • 14:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 14:24 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 14:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T360332)', diff saved to https://phabricator.wikimedia.org/P60085 and previous config saved to /var/cache/conftool/dbconfig/20240409-142414-arnaudb.json
  • 14:18 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Deploy new Truststore - elukey@cumin1002
  • 14:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P60084 and previous config saved to /var/cache/conftool/dbconfig/20240409-141410-arnaudb.json
  • 14:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60083 and previous config saved to /var/cache/conftool/dbconfig/20240409-140906-arnaudb.json
  • 14:07 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 14:07 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1005.eqiad.wmnet with OS bullseye
  • 14:04 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 14:03 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:02 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107 (T360332)', diff saved to https://phabricator.wikimedia.org/P60082 and previous config saved to /var/cache/conftool/dbconfig/20240409-135902-arnaudb.json
  • 13:57 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dumpsdata1004.eqiad.wmnet with OS bullseye
  • 13:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2112.codfw.wmnet onto db2212.codfw.wmnet
  • 13:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2107 (T360332)', diff saved to https://phabricator.wikimedia.org/P60081 and previous config saved to /var/cache/conftool/dbconfig/20240409-135621-arnaudb.json
  • 13:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 13:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 13:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 13:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 13:54 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1246.eqiad.wmnet with reason: Maintenance
  • 13:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1246.eqiad.wmnet with reason: Maintenance
  • 13:54 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60080 and previous config saved to /var/cache/conftool/dbconfig/20240409-135359-arnaudb.json
  • 13:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 13:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 13:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60079 and previous config saved to /var/cache/conftool/dbconfig/20240409-135335-arnaudb.json
  • 13:42 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 13:42 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye
  • 13:42 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dumpsdata1004.eqiad.wmnet with reason: host reimage
  • 13:38 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dumpsdata1004.eqiad.wmnet with reason: host reimage
  • 13:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T360332)', diff saved to https://phabricator.wikimedia.org/P60078 and previous config saved to /var/cache/conftool/dbconfig/20240409-133852-arnaudb.json
  • 13:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60077 and previous config saved to /var/cache/conftool/dbconfig/20240409-133827-arnaudb.json
  • 13:35 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 13:33 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 13:33 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4052.ulsfo.wmnet
  • 13:32 jforrester@deploy1002: Finished scap: Backport for zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184) (duration: 14m 31s)
  • 13:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1223 (T360332)', diff saved to https://phabricator.wikimedia.org/P60076 and previous config saved to /var/cache/conftool/dbconfig/20240409-133135-arnaudb.json
  • 13:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 13:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 13:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60075 and previous config saved to /var/cache/conftool/dbconfig/20240409-133112-arnaudb.json
  • 13:25 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1004.eqiad.wmnet with OS bullseye
  • 13:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60074 and previous config saved to /var/cache/conftool/dbconfig/20240409-132320-arnaudb.json
  • 13:21 jforrester@deploy1002: sdhehua and jforrester: Continuing with sync
  • 13:21 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:20 jforrester@deploy1002: sdhehua and jforrester: Backport for zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:18 sukhe@cumin1002: START - Cookbook sre.hosts.reboot-single for host cp4052.ulsfo.wmnet
  • 13:17 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: BIOS firmware upgrade
  • 13:17 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: BIOS firmware upgrade
  • 13:17 jforrester@deploy1002: Started scap: Backport for zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184)
  • 13:16 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 13:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60073 and previous config saved to /var/cache/conftool/dbconfig/20240409-131605-arnaudb.json
  • 13:16 sukhe: depool cp4052 for firmware upgrade
  • 13:16 jforrester@deploy1002: Finished scap: Backport for ExtensionDistributor: Add REL1_42 as a beta (T359844) (duration: 14m 09s)
  • 13:14 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on stat1010.eqiad.wmnet with reason: Connecting GPU power cable
  • 13:14 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on stat1010.eqiad.wmnet with reason: Connecting GPU power cable
  • 13:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60072 and previous config saved to /var/cache/conftool/dbconfig/20240409-130812-arnaudb.json
  • 13:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60071 and previous config saved to /var/cache/conftool/dbconfig/20240409-130543-arnaudb.json
  • 13:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1233.eqiad.wmnet with reason: Maintenance
  • 13:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1233.eqiad.wmnet with reason: Maintenance
  • 13:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60070 and previous config saved to /var/cache/conftool/dbconfig/20240409-130520-arnaudb.json
  • 13:04 jforrester@deploy1002: jforrester: Continuing with sync
  • 13:04 jforrester@deploy1002: jforrester: Backport for ExtensionDistributor: Add REL1_42 as a beta (T359844) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:01 jforrester@deploy1002: Started scap: Backport for ExtensionDistributor: Add REL1_42 as a beta (T359844)
  • 13:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60069 and previous config saved to /var/cache/conftool/dbconfig/20240409-130057-arnaudb.json
  • 12:53 vgutierrez: uploaded golang-github-gopacket-gopacket_1.2.0-2~wmf1 to apt.wm.o (bookworm)
  • 12:52 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: wmcs::openstack::codfw1dev::cinder_backups
  • 12:50 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 12:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60068 and previous config saved to /var/cache/conftool/dbconfig/20240409-125012-arnaudb.json
  • 12:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60067 and previous config saved to /var/cache/conftool/dbconfig/20240409-124550-arnaudb.json
  • 12:45 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 12:44 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Deploy new Truststore - elukey@cumin1002
  • 12:43 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: wmcs::openstack::codfw1dev::cinder_backups
  • 12:25 gmodena@deploy1002: Finished deploy [analytics/refinery@d45a15b]: Regular analytics weekly train [analytics/refinery@d45a15b6] (duration: 15m 41s)
  • 12:21 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudbackup1001-dev.eqiad.wmnet
  • 12:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60063 and previous config saved to /var/cache/conftool/dbconfig/20240409-122050-arnaudb.json
  • 12:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60062 and previous config saved to /var/cache/conftool/dbconfig/20240409-121958-arnaudb.json
  • 12:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60061 and previous config saved to /var/cache/conftool/dbconfig/20240409-121722-arnaudb.json
  • 12:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance
  • 12:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance
  • 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60060 and previous config saved to /var/cache/conftool/dbconfig/20240409-121622-arnaudb.json
  • 12:09 gmodena@deploy1002: Started deploy [analytics/refinery@d45a15b]: Regular analytics weekly train [analytics/refinery@d45a15b6]
  • 12:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60059 and previous config saved to /var/cache/conftool/dbconfig/20240409-120542-arnaudb.json
  • 12:02 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1004.eqiad.wmnet with OS bullseye
  • 12:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60058 and previous config saved to /var/cache/conftool/dbconfig/20240409-120115-arnaudb.json
  • 11:56 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM matomo1003.eqiad.wmnet - btullis@cumin1002"
  • 11:55 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM matomo1003.eqiad.wmnet - btullis@cumin1002"
  • 11:54 btullis@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) matomo1003.eqiad.wmnet on all recursors
  • 11:54 btullis@cumin1002: START - Cookbook sre.dns.wipe-cache matomo1003.eqiad.wmnet on all recursors
  • 11:54 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:54 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM matomo1003.eqiad.wmnet - btullis@cumin1002"
  • 11:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T360332)', diff saved to https://phabricator.wikimedia.org/P60057 and previous config saved to /var/cache/conftool/dbconfig/20240409-115035-arnaudb.json
  • 11:49 hnowlan@deploy1002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:48 hnowlan@deploy1002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:47 hnowlan@deploy1002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 11:47 hnowlan@deploy1002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
  • 11:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60056 and previous config saved to /var/cache/conftool/dbconfig/20240409-114607-arnaudb.json
  • 11:44 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2112.codfw.wmnet onto db2212.codfw.wmnet
  • 11:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1198 (T360332)', diff saved to https://phabricator.wikimedia.org/P60055 and previous config saved to /var/cache/conftool/dbconfig/20240409-114411-arnaudb.json
  • 11:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 11:43 hnowlan@deploy1002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:43 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60054 and previous config saved to /var/cache/conftool/dbconfig/20240409-114349-arnaudb.json
  • 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2112 in db2212 for T355422', diff saved to https://phabricator.wikimedia.org/P60053 and previous config saved to /var/cache/conftool/dbconfig/20240409-114302-arnaudb.json
  • 11:42 hnowlan@deploy1002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:42 hnowlan@deploy1002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
  • 11:42 hnowlan@deploy1002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 11:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
  • 11:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
  • 11:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2112.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
  • 11:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2112.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
  • 11:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60051 and previous config saved to /var/cache/conftool/dbconfig/20240409-113100-arnaudb.json
  • 11:29 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM matomo1003.eqiad.wmnet - btullis@cumin1002"
  • 11:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60050 and previous config saved to /var/cache/conftool/dbconfig/20240409-112841-arnaudb.json
  • 11:27 btullis@cumin1002: START - Cookbook sre.dns.netbox
  • 11:27 btullis@cumin1002: START - Cookbook sre.ganeti.makevm for new host matomo1003.eqiad.wmnet
  • 11:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60049 and previous config saved to /var/cache/conftool/dbconfig/20240409-112728-arnaudb.json
  • 11:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 11:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 11:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60048 and previous config saved to /var/cache/conftool/dbconfig/20240409-112705-arnaudb.json
  • 11:15 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1004.eqiad.wmnet with OS bullseye
  • 11:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60047 and previous config saved to /var/cache/conftool/dbconfig/20240409-111334-arnaudb.json
  • 11:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60046 and previous config saved to /var/cache/conftool/dbconfig/20240409-111157-arnaudb.json
  • 11:01 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host snapshot1015.eqiad.wmnet with OS bullseye
  • 10:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60045 and previous config saved to /var/cache/conftool/dbconfig/20240409-105827-arnaudb.json
  • 10:58 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60044 and previous config saved to /var/cache/conftool/dbconfig/20240409-105814-root.json
  • 10:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60043 and previous config saved to /var/cache/conftool/dbconfig/20240409-105650-arnaudb.json
  • 10:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60042 and previous config saved to /var/cache/conftool/dbconfig/20240409-104853-arnaudb.json
  • 10:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 10:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 10:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60041 and previous config saved to /var/cache/conftool/dbconfig/20240409-104830-arnaudb.json
  • 10:45 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 10:45 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 10:44 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 10:43 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60040 and previous config saved to /var/cache/conftool/dbconfig/20240409-104308-root.json
  • 10:42 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 10:41 jayme@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60039 and previous config saved to /var/cache/conftool/dbconfig/20240409-104143-arnaudb.json
  • 10:41 jayme@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60038 and previous config saved to /var/cache/conftool/dbconfig/20240409-103908-arnaudb.json
  • 10:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 10:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 10:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T360332)', diff saved to https://phabricator.wikimedia.org/P60037 and previous config saved to /var/cache/conftool/dbconfig/20240409-103845-arnaudb.json
  • 10:34 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on snapshot1015.eqiad.wmnet with reason: host reimage
  • 10:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60036 and previous config saved to /var/cache/conftool/dbconfig/20240409-103323-arnaudb.json
  • 10:32 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on db2097.codfw.wmnet with reason: host weirdness and possible decom
  • 10:31 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on db2097.codfw.wmnet with reason: host weirdness and possible decom
  • 10:31 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on snapshot1015.eqiad.wmnet with reason: host reimage
  • 10:28 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60035 and previous config saved to /var/cache/conftool/dbconfig/20240409-102803-root.json
  • 10:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P60034 and previous config saved to /var/cache/conftool/dbconfig/20240409-102337-arnaudb.json
  • 10:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 100%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60033 and previous config saved to /var/cache/conftool/dbconfig/20240409-102312-arnaudb.json
  • 10:19 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host snapshot1015.eqiad.wmnet with OS bullseye
  • 10:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60032 and previous config saved to /var/cache/conftool/dbconfig/20240409-101815-arnaudb.json
  • 10:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60031 and previous config saved to /var/cache/conftool/dbconfig/20240409-101257-root.json
  • 10:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P60030 and previous config saved to /var/cache/conftool/dbconfig/20240409-100830-arnaudb.json
  • 10:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 75%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60029 and previous config saved to /var/cache/conftool/dbconfig/20240409-100806-arnaudb.json
  • 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1199 (T356166)', diff saved to https://phabricator.wikimedia.org/P60028 and previous config saved to /var/cache/conftool/dbconfig/20240409-100635-marostegui.json
  • 10:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 10:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T356166)', diff saved to https://phabricator.wikimedia.org/P60027 and previous config saved to /var/cache/conftool/dbconfig/20240409-100612-marostegui.json
  • 10:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60026 and previous config saved to /var/cache/conftool/dbconfig/20240409-100308-arnaudb.json
  • 10:02 Emperor: puppet cert clean swift_eqiad T361844
  • 09:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60025 and previous config saved to /var/cache/conftool/dbconfig/20240409-095751-root.json
  • 09:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60024 and previous config saved to /var/cache/conftool/dbconfig/20240409-095642-arnaudb.json
  • 09:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 09:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 09:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T360332)', diff saved to https://phabricator.wikimedia.org/P60023 and previous config saved to /var/cache/conftool/dbconfig/20240409-095620-arnaudb.json
  • 09:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T360332)', diff saved to https://phabricator.wikimedia.org/P60022 and previous config saved to /var/cache/conftool/dbconfig/20240409-095323-arnaudb.json
  • 09:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 50%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60021 and previous config saved to /var/cache/conftool/dbconfig/20240409-095300-arnaudb.json
  • 09:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P60020 and previous config saved to /var/cache/conftool/dbconfig/20240409-095105-marostegui.json
  • 09:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1182 (T360332)', diff saved to https://phabricator.wikimedia.org/P60019 and previous config saved to /var/cache/conftool/dbconfig/20240409-095054-arnaudb.json
  • 09:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 09:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 09:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 09:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 09:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60018 and previous config saved to /var/cache/conftool/dbconfig/20240409-095004-arnaudb.json
  • 09:46 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 09:45 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60017 and previous config saved to /var/cache/conftool/dbconfig/20240409-094246-root.json
  • 09:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P60016 and previous config saved to /var/cache/conftool/dbconfig/20240409-094113-arnaudb.json
  • 09:40 hashar@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.26 refs T360158
  • 09:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 25%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60015 and previous config saved to /var/cache/conftool/dbconfig/20240409-093755-arnaudb.json
  • 09:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P60014 and previous config saved to /var/cache/conftool/dbconfig/20240409-093551-marostegui.json
  • 09:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P60013 and previous config saved to /var/cache/conftool/dbconfig/20240409-093457-arnaudb.json
  • 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60012 and previous config saved to /var/cache/conftool/dbconfig/20240409-092740-root.json
  • 09:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P60011 and previous config saved to /var/cache/conftool/dbconfig/20240409-092605-arnaudb.json
  • 09:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 16%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60010 and previous config saved to /var/cache/conftool/dbconfig/20240409-092249-arnaudb.json
  • 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T356166)', diff saved to https://phabricator.wikimedia.org/P60009 and previous config saved to /var/cache/conftool/dbconfig/20240409-092043-marostegui.json
  • 09:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P60008 and previous config saved to /var/cache/conftool/dbconfig/20240409-091949-arnaudb.json
  • 09:18 hashar@deploy1002: Finished scap: testwikis wikis to 1.42.0-wmf.26 refs T360158 (duration: 50m 11s)
  • 09:11 moritzm: installing postgresql-13 security updates
  • 09:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T360332)', diff saved to https://phabricator.wikimedia.org/P60007 and previous config saved to /var/cache/conftool/dbconfig/20240409-091057-arnaudb.json
  • 09:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 8%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60006 and previous config saved to /var/cache/conftool/dbconfig/20240409-090744-arnaudb.json
  • 09:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60005 and previous config saved to /var/cache/conftool/dbconfig/20240409-090442-arnaudb.json
  • 09:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1166 (T360332)', diff saved to https://phabricator.wikimedia.org/P60004 and previous config saved to /var/cache/conftool/dbconfig/20240409-090435-arnaudb.json
  • 09:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 09:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 09:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60003 and previous config saved to /var/cache/conftool/dbconfig/20240409-090210-arnaudb.json
  • 09:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 09:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 08:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 08:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 08:56 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1162.eqiad.wmnet onto db1246.eqiad.wmnet
  • 08:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 4%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60002 and previous config saved to /var/cache/conftool/dbconfig/20240409-085238-arnaudb.json
  • 08:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 2%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60001 and previous config saved to /var/cache/conftool/dbconfig/20240409-083733-arnaudb.json
  • 08:28 hashar@deploy1002: Started scap: testwikis wikis to 1.42.0-wmf.26 refs T360158
  • 08:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 1%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60000 and previous config saved to /var/cache/conftool/dbconfig/20240409-082227-arnaudb.json
  • 08:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2113.codfw.wmnet with OS bookworm
  • 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2113.codfw.wmnet with reason: host reimage
  • 07:54 Emperor: puppet cert clean swift_codfw T361844
  • 07:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2113.codfw.wmnet with reason: host reimage
  • 07:37 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2113.codfw.wmnet with OS bookworm
  • 07:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 depool for reimage T360116', diff saved to https://phabricator.wikimedia.org/P59999 and previous config saved to /var/cache/conftool/dbconfig/20240409-073406-arnaudb.json
  • 07:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2213.codfw.wmnet with reason: Silence for reimage
  • 07:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2213.codfw.wmnet with reason: Silence for reimage
  • 07:29 kartik@deploy1002: Finished scap: Backport for ContentTranslation: Limit publishing in zhwiki for extendedconfirmed users only (T349959) (duration: 25m 30s)
  • 07:16 kartik@deploy1002: kartik: Continuing with sync
  • 07:09 kartik@deploy1002: kartik: Backport for ContentTranslation: Limit publishing in zhwiki for extendedconfirmed users only (T349959) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 07:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 07:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T360332)', diff saved to https://phabricator.wikimedia.org/P59998 and previous config saved to /var/cache/conftool/dbconfig/20240409-070613-arnaudb.json
  • 07:04 kartik@deploy1002: Started scap: Backport for ContentTranslation: Limit publishing in zhwiki for extendedconfirmed users only (T349959)
  • 07:01 marostegui@cumin1002: START - Cookbook sre.mysql.clone of db1162.eqiad.wmnet onto db1246.eqiad.wmnet
  • 06:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P59997 and previous config saved to /var/cache/conftool/dbconfig/20240409-065105-arnaudb.json
  • 06:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P59996 and previous config saved to /var/cache/conftool/dbconfig/20240409-063558-arnaudb.json
  • 06:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T360332)', diff saved to https://phabricator.wikimedia.org/P59995 and previous config saved to /var/cache/conftool/dbconfig/20240409-062050-arnaudb.json
  • 06:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2219 (T360332)', diff saved to https://phabricator.wikimedia.org/P59994 and previous config saved to /var/cache/conftool/dbconfig/20240409-061830-arnaudb.json
  • 06:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2219.codfw.wmnet with reason: Maintenance
  • 06:18 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2219.codfw.wmnet with reason: Maintenance
  • 06:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59993 and previous config saved to /var/cache/conftool/dbconfig/20240409-061807-arnaudb.json
  • 06:15 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1246.eqiad.wmnet with OS bookworm
  • 06:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1162.eqiad.wmnet with OS bookworm
  • 06:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P59992 and previous config saved to /var/cache/conftool/dbconfig/20240409-060300-arnaudb.json
  • 05:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage
  • 05:51 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage
  • 05:48 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1162.eqiad.wmnet with reason: host reimage
  • 05:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P59991 and previous config saved to /var/cache/conftool/dbconfig/20240409-054752-arnaudb.json
  • 05:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1162.eqiad.wmnet with reason: host reimage
  • 05:39 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm
  • 05:33 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1162.eqiad.wmnet with OS bookworm
  • 05:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59990 and previous config saved to /var/cache/conftool/dbconfig/20240409-053245-arnaudb.json
  • 05:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59989 and previous config saved to /var/cache/conftool/dbconfig/20240409-053024-arnaudb.json
  • 05:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2210.codfw.wmnet with reason: Maintenance
  • 05:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1162 T362036', diff saved to https://phabricator.wikimedia.org/P59988 and previous config saved to /var/cache/conftool/dbconfig/20240409-053005-root.json
  • 05:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2210.codfw.wmnet with reason: Maintenance
  • 05:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T360332)', diff saved to https://phabricator.wikimedia.org/P59987 and previous config saved to /var/cache/conftool/dbconfig/20240409-053001-arnaudb.json
  • 05:28 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1222 to s2 primary and set section read-write T362036', diff saved to https://phabricator.wikimedia.org/P59986 and previous config saved to /var/cache/conftool/dbconfig/20240409-052855-marostegui.json
  • 05:28 marostegui@cumin1002: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - T362036', diff saved to https://phabricator.wikimedia.org/P59985 and previous config saved to /var/cache/conftool/dbconfig/20240409-052827-marostegui.json
  • 05:28 marostegui: Starting s2 eqiad failover from db1162 to db1222 - T362036
  • 05:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P59984 and previous config saved to /var/cache/conftool/dbconfig/20240409-051454-arnaudb.json
  • 05:10 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1222 with weight 0 T362036', diff saved to https://phabricator.wikimedia.org/P59983 and previous config saved to /var/cache/conftool/dbconfig/20240409-051027-marostegui.json
  • 05:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s2 T362036
  • 05:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s2 T362036
  • 04:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P59982 and previous config saved to /var/cache/conftool/dbconfig/20240409-045946-arnaudb.json
  • 04:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T360332)', diff saved to https://phabricator.wikimedia.org/P59981 and previous config saved to /var/cache/conftool/dbconfig/20240409-044438-arnaudb.json
  • 04:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2206 (T360332)', diff saved to https://phabricator.wikimedia.org/P59980 and previous config saved to /var/cache/conftool/dbconfig/20240409-044216-arnaudb.json
  • 04:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2206.codfw.wmnet with reason: Maintenance
  • 04:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2206.codfw.wmnet with reason: Maintenance
  • 04:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59979 and previous config saved to /var/cache/conftool/dbconfig/20240409-044204-arnaudb.json
  • 04:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P59978 and previous config saved to /var/cache/conftool/dbconfig/20240409-042657-arnaudb.json
  • 04:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P59977 and previous config saved to /var/cache/conftool/dbconfig/20240409-041149-arnaudb.json
  • 03:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59976 and previous config saved to /var/cache/conftool/dbconfig/20240409-035641-arnaudb.json
  • 03:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59975 and previous config saved to /var/cache/conftool/dbconfig/20240409-035422-arnaudb.json
  • 03:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 03:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 03:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T360332)', diff saved to https://phabricator.wikimedia.org/P59974 and previous config saved to /var/cache/conftool/dbconfig/20240409-035359-arnaudb.json
  • 03:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P59973 and previous config saved to /var/cache/conftool/dbconfig/20240409-033851-arnaudb.json
  • 03:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P59972 and previous config saved to /var/cache/conftool/dbconfig/20240409-032344-arnaudb.json
  • 03:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T360332)', diff saved to https://phabricator.wikimedia.org/P59971 and previous config saved to /var/cache/conftool/dbconfig/20240409-030836-arnaudb.json
  • 03:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2155 (T360332)', diff saved to https://phabricator.wikimedia.org/P59970 and previous config saved to /var/cache/conftool/dbconfig/20240409-030617-arnaudb.json
  • 03:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 03:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 03:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 03:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 03:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T360332)', diff saved to https://phabricator.wikimedia.org/P59969 and previous config saved to /var/cache/conftool/dbconfig/20240409-030537-arnaudb.json
  • 02:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P59968 and previous config saved to /var/cache/conftool/dbconfig/20240409-025030-arnaudb.json
  • 02:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P59967 and previous config saved to /var/cache/conftool/dbconfig/20240409-023522-arnaudb.json
  • 02:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T360332)', diff saved to https://phabricator.wikimedia.org/P59966 and previous config saved to /var/cache/conftool/dbconfig/20240409-022015-arnaudb.json
  • 02:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2147 (T360332)', diff saved to https://phabricator.wikimedia.org/P59965 and previous config saved to /var/cache/conftool/dbconfig/20240409-021755-arnaudb.json
  • 02:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 02:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 02:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140 (T360332)', diff saved to https://phabricator.wikimedia.org/P59964 and previous config saved to /var/cache/conftool/dbconfig/20240409-021731-arnaudb.json
  • 02:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P59963 and previous config saved to /var/cache/conftool/dbconfig/20240409-020223-arnaudb.json
  • 01:49 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@6bb821b]: (no justification provided) (duration: 00m 31s)
  • 01:49 ebysans@deploy1002: Started deploy [airflow-dags/analytics@6bb821b]: (no justification provided)
  • 01:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P59962 and previous config saved to /var/cache/conftool/dbconfig/20240409-014716-arnaudb.json
  • 01:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1190 (T356166)', diff saved to https://phabricator.wikimedia.org/P59961 and previous config saved to /var/cache/conftool/dbconfig/20240409-013231-marostegui.json
  • 01:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 01:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 01:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140 (T360332)', diff saved to https://phabricator.wikimedia.org/P59960 and previous config saved to /var/cache/conftool/dbconfig/20240409-013208-arnaudb.json
  • 01:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T356166)', diff saved to https://phabricator.wikimedia.org/P59959 and previous config saved to /var/cache/conftool/dbconfig/20240409-013208-marostegui.json
  • 01:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2140 (T360332)', diff saved to https://phabricator.wikimedia.org/P59958 and previous config saved to /var/cache/conftool/dbconfig/20240409-012949-arnaudb.json
  • 01:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 01:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 01:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 01:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 01:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137 (T360332)', diff saved to https://phabricator.wikimedia.org/P59957 and previous config saved to /var/cache/conftool/dbconfig/20240409-012858-arnaudb.json
  • 01:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P59956 and previous config saved to /var/cache/conftool/dbconfig/20240409-011700-marostegui.json
  • 01:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137', diff saved to https://phabricator.wikimedia.org/P59955 and previous config saved to /var/cache/conftool/dbconfig/20240409-011351-arnaudb.json
  • 01:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P59954 and previous config saved to /var/cache/conftool/dbconfig/20240409-010152-marostegui.json
  • 00:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137', diff saved to https://phabricator.wikimedia.org/P59953 and previous config saved to /var/cache/conftool/dbconfig/20240409-005843-arnaudb.json
  • 00:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T356166)', diff saved to https://phabricator.wikimedia.org/P59952 and previous config saved to /var/cache/conftool/dbconfig/20240409-004645-marostegui.json
  • 00:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137 (T360332)', diff saved to https://phabricator.wikimedia.org/P59951 and previous config saved to /var/cache/conftool/dbconfig/20240409-004336-arnaudb.json
  • 00:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2137 (T360332)', diff saved to https://phabricator.wikimedia.org/P59950 and previous config saved to /var/cache/conftool/dbconfig/20240409-004115-arnaudb.json
  • 00:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 00:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 00:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T360332)', diff saved to https://phabricator.wikimedia.org/P59949 and previous config saved to /var/cache/conftool/dbconfig/20240409-004052-arnaudb.json
  • 00:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P59948 and previous config saved to /var/cache/conftool/dbconfig/20240409-002545-arnaudb.json
  • 00:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P59947 and previous config saved to /var/cache/conftool/dbconfig/20240409-001037-arnaudb.json

2024-04-08

  • 23:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T360332)', diff saved to https://phabricator.wikimedia.org/P59946 and previous config saved to /var/cache/conftool/dbconfig/20240408-235530-arnaudb.json
  • 23:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2136 (T360332)', diff saved to https://phabricator.wikimedia.org/P59945 and previous config saved to /var/cache/conftool/dbconfig/20240408-235309-arnaudb.json
  • 23:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 23:52 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 23:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T360332)', diff saved to https://phabricator.wikimedia.org/P59944 and previous config saved to /var/cache/conftool/dbconfig/20240408-235247-arnaudb.json
  • 23:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P59943 and previous config saved to /var/cache/conftool/dbconfig/20240408-233739-arnaudb.json
  • 23:25 tstarling@deploy1002: Synchronized wmf-config/CommonSettings.php: stop writing to ipblocks table T355034 (duration: 12m 32s)
  • 23:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P59942 and previous config saved to /var/cache/conftool/dbconfig/20240408-232231-arnaudb.json
  • 23:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T360332)', diff saved to https://phabricator.wikimedia.org/P59941 and previous config saved to /var/cache/conftool/dbconfig/20240408-230723-arnaudb.json
  • 23:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2119 (T360332)', diff saved to https://phabricator.wikimedia.org/P59940 and previous config saved to /var/cache/conftool/dbconfig/20240408-230502-arnaudb.json
  • 23:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 23:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 23:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 23:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 23:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T360332)', diff saved to https://phabricator.wikimedia.org/P59939 and previous config saved to /var/cache/conftool/dbconfig/20240408-230410-arnaudb.json
  • 22:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P59938 and previous config saved to /var/cache/conftool/dbconfig/20240408-224903-arnaudb.json
  • 22:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P59937 and previous config saved to /var/cache/conftool/dbconfig/20240408-223355-arnaudb.json
  • 22:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T360332)', diff saved to https://phabricator.wikimedia.org/P59935 and previous config saved to /var/cache/conftool/dbconfig/20240408-221847-arnaudb.json
  • 22:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2106 (T360332)', diff saved to https://phabricator.wikimedia.org/P59934 and previous config saved to /var/cache/conftool/dbconfig/20240408-221626-arnaudb.json
  • 22:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 22:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 22:15 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 22:15 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 22:15 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 22:15 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 22:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T360332)', diff saved to https://phabricator.wikimedia.org/P59933 and previous config saved to /var/cache/conftool/dbconfig/20240408-221509-arnaudb.json
  • 22:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P59931 and previous config saved to /var/cache/conftool/dbconfig/20240408-220001-arnaudb.json
  • 21:55 eileen: config revision changed from fff12a6a to 3c1a0267
  • 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs1025.eqiad.wmnet
  • 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002"
  • 21:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P59930 and previous config saved to /var/cache/conftool/dbconfig/20240408-214454-arnaudb.json
  • 21:38 eileen: config revision changed from b08eb273 to fff12a6a
  • 21:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T360332)', diff saved to https://phabricator.wikimedia.org/P59929 and previous config saved to /var/cache/conftool/dbconfig/20240408-212946-arnaudb.json
  • 21:28 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002"
  • 21:27 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:27 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1249 (T360332)', diff saved to https://phabricator.wikimedia.org/P59928 and previous config saved to /var/cache/conftool/dbconfig/20240408-212628-arnaudb.json
  • 21:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
  • 21:26 ryankemper@cumin2002: START - Cookbook sre.dns.netbox
  • 21:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
  • 21:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T360332)', diff saved to https://phabricator.wikimedia.org/P59927 and previous config saved to /var/cache/conftool/dbconfig/20240408-212605-arnaudb.json
  • 21:20 ryankemper@cumin2002: START - Cookbook sre.hosts.decommission for hosts wdqs1025.eqiad.wmnet
  • 21:18 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:17 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P59926 and previous config saved to /var/cache/conftool/dbconfig/20240408-211056-arnaudb.json
  • 21:10 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:10 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:01 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:01 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:01 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P59925 and previous config saved to /var/cache/conftool/dbconfig/20240408-205548-arnaudb.json
  • 20:53 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:53 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:53 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:53 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:52 ebernhardson@deploy1002: Finished scap: Backport for cirrus: Restore traffic to codfw clusters (duration: 15m 25s)
  • 20:42 ebernhardson@deploy1002: ebernhardson: Continuing with sync
  • 20:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T360332)', diff saved to https://phabricator.wikimedia.org/P59924 and previous config saved to /var/cache/conftool/dbconfig/20240408-204041-arnaudb.json
  • 20:39 ebernhardson@deploy1002: ebernhardson: Backport for cirrus: Restore traffic to codfw clusters synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1248 (T360332)', diff saved to https://phabricator.wikimedia.org/P59923 and previous config saved to /var/cache/conftool/dbconfig/20240408-203723-arnaudb.json
  • 20:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
  • 20:37 ebernhardson@deploy1002: Started scap: Backport for cirrus: Restore traffic to codfw clusters
  • 20:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
  • 20:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T360332)', diff saved to https://phabricator.wikimedia.org/P59922 and previous config saved to /var/cache/conftool/dbconfig/20240408-203700-arnaudb.json
  • 20:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59921 and previous config saved to /var/cache/conftool/dbconfig/20240408-202955-arnaudb.json
  • 20:25 urbanecm@deploy1002: Finished scap: Backport for Revert "Mark all autoreviewed edits in PageSaveComplete hook" (T361918 T361940 T361960) (duration: 16m 25s)
  • 20:24 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:23 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P59920 and previous config saved to /var/cache/conftool/dbconfig/20240408-202153-arnaudb.json
  • 20:15 urbanecm@deploy1002: urbanecm and matmarex: Continuing with sync
  • 20:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P59919 and previous config saved to /var/cache/conftool/dbconfig/20240408-201447-arnaudb.json
  • 20:11 urbanecm@deploy1002: urbanecm and matmarex: Backport for Revert "Mark all autoreviewed edits in PageSaveComplete hook" (T361918 T361940 T361960) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:09 urbanecm@deploy1002: Started scap: Backport for Revert "Mark all autoreviewed edits in PageSaveComplete hook" (T361918 T361940 T361960)
  • 20:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P59918 and previous config saved to /var/cache/conftool/dbconfig/20240408-200645-arnaudb.json
  • 19:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P59917 and previous config saved to /var/cache/conftool/dbconfig/20240408-195940-arnaudb.json
  • 19:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T360332)', diff saved to https://phabricator.wikimedia.org/P59916 and previous config saved to /var/cache/conftool/dbconfig/20240408-195138-arnaudb.json
  • 19:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1247 (T360332)', diff saved to https://phabricator.wikimedia.org/P59915 and previous config saved to /var/cache/conftool/dbconfig/20240408-194919-arnaudb.json
  • 19:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
  • 19:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
  • 19:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 19:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 19:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P59914 and previous config saved to /var/cache/conftool/dbconfig/20240408-194843-arnaudb.json
  • 19:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59913 and previous config saved to /var/cache/conftool/dbconfig/20240408-194432-arnaudb.json
  • 19:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59912 and previous config saved to /var/cache/conftool/dbconfig/20240408-194113-arnaudb.json
  • 19:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
  • 19:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
  • 19:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59911 and previous config saved to /var/cache/conftool/dbconfig/20240408-194050-arnaudb.json
  • 19:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P59910 and previous config saved to /var/cache/conftool/dbconfig/20240408-193336-arnaudb.json
  • 19:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P59909 and previous config saved to /var/cache/conftool/dbconfig/20240408-192543-arnaudb.json
  • 19:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P59908 and previous config saved to /var/cache/conftool/dbconfig/20240408-191828-arnaudb.json
  • 19:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P59907 and previous config saved to /var/cache/conftool/dbconfig/20240408-191035-arnaudb.json
  • 19:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P59905 and previous config saved to /var/cache/conftool/dbconfig/20240408-190319-arnaudb.json
  • 18:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59904 and previous config saved to /var/cache/conftool/dbconfig/20240408-185528-arnaudb.json
  • 18:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59903 and previous config saved to /var/cache/conftool/dbconfig/20240408-185309-arnaudb.json
  • 18:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2211.codfw.wmnet with reason: Maintenance
  • 18:52 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2211.codfw.wmnet with reason: Maintenance
  • 18:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59902 and previous config saved to /var/cache/conftool/dbconfig/20240408-185247-arnaudb.json
  • 18:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P59901 and previous config saved to /var/cache/conftool/dbconfig/20240408-183739-arnaudb.json
  • 18:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P59900 and previous config saved to /var/cache/conftool/dbconfig/20240408-182232-arnaudb.json
  • 18:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59899 and previous config saved to /var/cache/conftool/dbconfig/20240408-180724-arnaudb.json
  • 18:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59898 and previous config saved to /var/cache/conftool/dbconfig/20240408-180406-arnaudb.json
  • 18:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2192.codfw.wmnet with reason: Maintenance
  • 18:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2192.codfw.wmnet with reason: Maintenance
  • 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59897 and previous config saved to /var/cache/conftool/dbconfig/20240408-180343-arnaudb.json
  • 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P59896 and previous config saved to /var/cache/conftool/dbconfig/20240408-180253-arnaudb.json
  • 18:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 18:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 18:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T360332)', diff saved to https://phabricator.wikimedia.org/P59895 and previous config saved to /var/cache/conftool/dbconfig/20240408-180231-arnaudb.json
  • 17:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P59894 and previous config saved to /var/cache/conftool/dbconfig/20240408-174835-arnaudb.json
  • 17:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P59893 and previous config saved to /var/cache/conftool/dbconfig/20240408-174723-arnaudb.json
  • 17:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P59892 and previous config saved to /var/cache/conftool/dbconfig/20240408-173327-arnaudb.json
  • 17:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P59891 and previous config saved to /var/cache/conftool/dbconfig/20240408-173215-arnaudb.json
  • 17:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59890 and previous config saved to /var/cache/conftool/dbconfig/20240408-171819-arnaudb.json
  • 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T360332)', diff saved to https://phabricator.wikimedia.org/P59889 and previous config saved to /var/cache/conftool/dbconfig/20240408-171707-arnaudb.json
  • 17:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59888 and previous config saved to /var/cache/conftool/dbconfig/20240408-171502-arnaudb.json
  • 17:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1243 (T360332)', diff saved to https://phabricator.wikimedia.org/P59887 and previous config saved to /var/cache/conftool/dbconfig/20240408-171448-arnaudb.json
  • 17:14 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T360332)', diff saved to https://phabricator.wikimedia.org/P59886 and previous config saved to /var/cache/conftool/dbconfig/20240408-171439-arnaudb.json
  • 17:14 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T360332)', diff saved to https://phabricator.wikimedia.org/P59885 and previous config saved to /var/cache/conftool/dbconfig/20240408-171425-arnaudb.json
  • 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P59884 and previous config saved to /var/cache/conftool/dbconfig/20240408-165931-arnaudb.json
  • 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P59883 and previous config saved to /var/cache/conftool/dbconfig/20240408-165917-arnaudb.json
  • 16:57 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs20[08-12]*: Deploy new Truststore - elukey@cumin1002
  • 16:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1160 (T356166)', diff saved to https://phabricator.wikimedia.org/P59882 and previous config saved to /var/cache/conftool/dbconfig/20240408-164524-marostegui.json
  • 16:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 16:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P59881 and previous config saved to /var/cache/conftool/dbconfig/20240408-164424-arnaudb.json
  • 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P59880 and previous config saved to /var/cache/conftool/dbconfig/20240408-164410-arnaudb.json
  • 16:32 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:32 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:32 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:32 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T360332)', diff saved to https://phabricator.wikimedia.org/P59879 and previous config saved to /var/cache/conftool/dbconfig/20240408-162916-arnaudb.json
  • 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T360332)', diff saved to https://phabricator.wikimedia.org/P59878 and previous config saved to /var/cache/conftool/dbconfig/20240408-162902-arnaudb.json
  • 16:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2171 (T360332)', diff saved to https://phabricator.wikimedia.org/P59877 and previous config saved to /var/cache/conftool/dbconfig/20240408-162655-arnaudb.json
  • 16:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1242 (T360332)', diff saved to https://phabricator.wikimedia.org/P59876 and previous config saved to /var/cache/conftool/dbconfig/20240408-162645-arnaudb.json
  • 16:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T360332)', diff saved to https://phabricator.wikimedia.org/P59875 and previous config saved to /var/cache/conftool/dbconfig/20240408-162633-arnaudb.json
  • 16:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T360332)', diff saved to https://phabricator.wikimedia.org/P59874 and previous config saved to /var/cache/conftool/dbconfig/20240408-162621-arnaudb.json
  • 16:19 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs20[08-12]*: Deploy new Truststore - elukey@cumin1002
  • 16:15 elukey: manually dran + restart cassandra-a on aqs2007 - cookbook failed
  • 16:15 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dumpsdata1002.eqiad.wmnet
  • 16:14 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:14 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P59873 and previous config saved to /var/cache/conftool/dbconfig/20240408-161125-arnaudb.json
  • 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P59872 and previous config saved to /var/cache/conftool/dbconfig/20240408-161114-arnaudb.json
  • 16:07 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 16:06 elukey@cumin1002: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching A:aqs-codfw: Deploy new Truststore - elukey@cumin1002
  • 16:01 btullis@cumin1002: START - Cookbook sre.dns.netbox
  • 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P59871 and previous config saved to /var/cache/conftool/dbconfig/20240408-155618-arnaudb.json
  • 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P59870 and previous config saved to /var/cache/conftool/dbconfig/20240408-155606-arnaudb.json
  • 15:48 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:47 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:43 btullis@cumin1002: START - Cookbook sre.hosts.decommission for hosts dumpsdata1002.eqiad.wmnet
  • 15:42 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dumpsdata1001.eqiad.wmnet
  • 15:42 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:42 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T360332)', diff saved to https://phabricator.wikimedia.org/P59869 and previous config saved to /var/cache/conftool/dbconfig/20240408-154110-arnaudb.json
  • 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T360332)', diff saved to https://phabricator.wikimedia.org/P59868 and previous config saved to /var/cache/conftool/dbconfig/20240408-154059-arnaudb.json
  • 15:39 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3069.esams.wmnet,service=(cdn|ats-be)
  • 15:38 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 15:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1241 (T360332)', diff saved to https://phabricator.wikimedia.org/P59867 and previous config saved to /var/cache/conftool/dbconfig/20240408-153842-arnaudb.json
  • 15:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
  • 15:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
  • 15:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T360332)', diff saved to https://phabricator.wikimedia.org/P59866 and previous config saved to /var/cache/conftool/dbconfig/20240408-153819-arnaudb.json
  • 15:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2157 (T360332)', diff saved to https://phabricator.wikimedia.org/P59865 and previous config saved to /var/cache/conftool/dbconfig/20240408-153753-arnaudb.json
  • 15:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 15:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 15:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T360332)', diff saved to https://phabricator.wikimedia.org/P59864 and previous config saved to /var/cache/conftool/dbconfig/20240408-153730-arnaudb.json
  • 15:31 btullis@cumin1002: START - Cookbook sre.dns.netbox
  • 15:25 btullis@cumin1002: START - Cookbook sre.hosts.decommission for hosts dumpsdata1001.eqiad.wmnet
  • 15:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P59863 and previous config saved to /var/cache/conftool/dbconfig/20240408-152311-arnaudb.json
  • 15:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P59862 and previous config saved to /var/cache/conftool/dbconfig/20240408-152221-arnaudb.json
  • 15:20 vgutierrez: Uploaded golang-gitlab-wikimedia-sre-qemutest-dev 0.1.0 to apt.wm.o (bookworm)
  • 15:17 dancy@deploy1002: Finished deploy [restbase/deploy@c4d19d7]: testing T361608 (duration: 13m 59s)
  • 15:15 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3069.esams.wmnet with OS bullseye
  • 15:15 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Deploy new Truststore - elukey@cumin1002
  • 15:12 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:11 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:10 elukey: drain and restart cassandra-a on aqs1011 to test the new truststore
  • 15:10 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:10 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:10 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:09 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P59860 and previous config saved to /var/cache/conftool/dbconfig/20240408-150803-arnaudb.json
  • 15:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P59859 and previous config saved to /var/cache/conftool/dbconfig/20240408-150713-arnaudb.json
  • 15:04 godog: kill -9 thanos-store on titan1001
  • 15:03 dancy@deploy1002: Started deploy [restbase/deploy@c4d19d7]: testing T361608
  • 14:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T360332)', diff saved to https://phabricator.wikimedia.org/P59858 and previous config saved to /var/cache/conftool/dbconfig/20240408-145256-arnaudb.json
  • 14:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T360332)', diff saved to https://phabricator.wikimedia.org/P59857 and previous config saved to /var/cache/conftool/dbconfig/20240408-145205-arnaudb.json
  • 14:51 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3069.esams.wmnet with reason: host reimage
  • 14:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2128 (T360332)', diff saved to https://phabricator.wikimedia.org/P59856 and previous config saved to /var/cache/conftool/dbconfig/20240408-144847-arnaudb.json
  • 14:48 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3069.esams.wmnet with reason: host reimage
  • 14:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 14:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 14:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 14:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 14:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113 (T360332)', diff saved to https://phabricator.wikimedia.org/P59855 and previous config saved to /var/cache/conftool/dbconfig/20240408-144808-arnaudb.json
  • 14:47 jayme@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1221 (T360332)', diff saved to https://phabricator.wikimedia.org/P59854 and previous config saved to /var/cache/conftool/dbconfig/20240408-144738-arnaudb.json
  • 14:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 14:47 jayme@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:47 jayme@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 14:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
  • 14:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
  • 14:47 jayme@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T360332)', diff saved to https://phabricator.wikimedia.org/P59853 and previous config saved to /var/cache/conftool/dbconfig/20240408-144657-arnaudb.json
  • 14:44 jayme@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:43 jayme@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 14:43 jayme@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 14:42 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 14:42 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 14:42 jayme@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 14:41 jayme@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 14:40 jayme@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 14:39 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 14:38 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 14:37 godog: bounce thanos-query and thanos-store on titan1002 - stuck on high CPU
  • 14:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113', diff saved to https://phabricator.wikimedia.org/P59852 and previous config saved to /var/cache/conftool/dbconfig/20240408-143301-arnaudb.json
  • 14:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P59851 and previous config saved to /var/cache/conftool/dbconfig/20240408-143149-arnaudb.json
  • 14:24 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp3069.esams.wmnet with OS bullseye
  • 14:20 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 14:19 sukhe: depool cp3069 to prepare for reimaging: T360430
  • 14:19 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 14:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113', diff saved to https://phabricator.wikimedia.org/P59850 and previous config saved to /var/cache/conftool/dbconfig/20240408-141753-arnaudb.json
  • 14:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P59849 and previous config saved to /var/cache/conftool/dbconfig/20240408-141641-arnaudb.json
  • 14:06 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 14:04 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 14:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113 (T360332)', diff saved to https://phabricator.wikimedia.org/P59847 and previous config saved to /var/cache/conftool/dbconfig/20240408-140246-arnaudb.json
  • 14:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T360332)', diff saved to https://phabricator.wikimedia.org/P59846 and previous config saved to /var/cache/conftool/dbconfig/20240408-140132-arnaudb.json
  • 14:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2113 (T360332)', diff saved to https://phabricator.wikimedia.org/P59845 and previous config saved to /var/cache/conftool/dbconfig/20240408-135926-arnaudb.json
  • 13:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1199 (T360332)', diff saved to https://phabricator.wikimedia.org/P59844 and previous config saved to /var/cache/conftool/dbconfig/20240408-135915-arnaudb.json
  • 13:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 13:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 13:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 13:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T360332)', diff saved to https://phabricator.wikimedia.org/P59843 and previous config saved to /var/cache/conftool/dbconfig/20240408-135852-arnaudb.json
  • 13:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 13:57 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 13:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 13:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 13:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T360332)', diff saved to https://phabricator.wikimedia.org/P59842 and previous config saved to /var/cache/conftool/dbconfig/20240408-135444-arnaudb.json
  • 13:45 Lucas_WMDE: UTC afternoon backport+config window done
  • 13:44 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1207.eqiad.wmnet
  • 13:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P59841 and previous config saved to /var/cache/conftool/dbconfig/20240408-134345-arnaudb.json
  • 13:43 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Enable abusefilter block at bnwiki (T361852) (duration: 28m 46s)
  • 13:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59840 and previous config saved to /var/cache/conftool/dbconfig/20240408-133936-arnaudb.json
  • 13:37 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1207.eqiad.wmnet
  • 13:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1229.eqiad.wmnet
  • 13:33 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[4037,4041,4045,4049].ulsfo.wmnet} and A:cp
  • 13:32 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and yahya: Continuing with sync
  • 13:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P59839 and previous config saved to /var/cache/conftool/dbconfig/20240408-132838-arnaudb.json
  • 13:28 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1229.eqiad.wmnet
  • 13:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1175.eqiad.wmnet
  • 13:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2212.codfw.wmnet with reason: Silence for clone
  • 13:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2212.codfw.wmnet with reason: Silence for clone
  • 13:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59838 and previous config saved to /var/cache/conftool/dbconfig/20240408-132429-arnaudb.json
  • 13:23 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[4037,4041,4045,4049].ulsfo.wmnet} and A:cp
  • 13:21 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1175.eqiad.wmnet
  • 13:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1241.eqiad.wmnet
  • 13:17 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and yahya: Backport for Enable abusefilter block at bnwiki (T361852) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:15 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1241.eqiad.wmnet
  • 13:15 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 13:14 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 13:14 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Enable abusefilter block at bnwiki (T361852)
  • 13:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T360332)', diff saved to https://phabricator.wikimedia.org/P59837 and previous config saved to /var/cache/conftool/dbconfig/20240408-131331-arnaudb.json
  • 13:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1200.eqiad.wmnet
  • 13:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1190 (T360332)', diff saved to https://phabricator.wikimedia.org/P59836 and previous config saved to /var/cache/conftool/dbconfig/20240408-131113-arnaudb.json
  • 13:11 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 13:10 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 13:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T360332)', diff saved to https://phabricator.wikimedia.org/P59835 and previous config saved to /var/cache/conftool/dbconfig/20240408-131051-arnaudb.json
  • 13:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T360332)', diff saved to https://phabricator.wikimedia.org/P59834 and previous config saved to /var/cache/conftool/dbconfig/20240408-130921-arnaudb.json
  • 13:05 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1200.eqiad.wmnet
  • 13:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1230 (T360332)', diff saved to https://phabricator.wikimedia.org/P59833 and previous config saved to /var/cache/conftool/dbconfig/20240408-130543-arnaudb.json
  • 13:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 13:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 13:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 13:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 13:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59832 and previous config saved to /var/cache/conftool/dbconfig/20240408-130443-arnaudb.json
  • 12:56 elukey: nodetool-a drain + restart of cassandra instances on aqs1010 to pick up the new truststore
  • 12:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P59831 and previous config saved to /var/cache/conftool/dbconfig/20240408-125543-arnaudb.json
  • 12:55 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on aqs1010.eqiad.wmnet with reason: Replace Java Truststore
  • 12:55 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on aqs1010.eqiad.wmnet with reason: Replace Java Truststore
  • 12:53 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1180.eqiad.wmnet
  • 12:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P59830 and previous config saved to /var/cache/conftool/dbconfig/20240408-124935-arnaudb.json
  • 12:47 jayme@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:46 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1180.eqiad.wmnet
  • 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1194.eqiad.wmnet
  • 12:46 jayme@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 12:46 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:44 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1237.eqiad.wmnet
  • 12:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T360332)', diff saved to https://phabricator.wikimedia.org/P59827 and previous config saved to /var/cache/conftool/dbconfig/20240408-122527-arnaudb.json
  • 12:22 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1160 (T360332)', diff saved to https://phabricator.wikimedia.org/P59826 and previous config saved to /var/cache/conftool/dbconfig/20240408-122209-arnaudb.json
  • 12:22 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 12:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 12:19 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59825 and previous config saved to /var/cache/conftool/dbconfig/20240408-121920-arnaudb.json
  • 12:17 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1237.eqiad.wmnet
  • 12:17 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1022.eqiad.wmnet
  • 12:17 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59824 and previous config saved to /var/cache/conftool/dbconfig/20240408-121642-arnaudb.json
  • 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1213.eqiad.wmnet with reason: Maintenance
  • 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1213.eqiad.wmnet with reason: Maintenance
  • 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59823 and previous config saved to /var/cache/conftool/dbconfig/20240408-121609-arnaudb.json
  • 12:15 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:14 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:11 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:09 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1022.eqiad.wmnet
  • 12:07 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1031.eqiad.wmnet
  • 12:04 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:02 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host stat1011.eqiad.wmnet
  • 12:02 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host stat1011.eqiad.wmnet
  • 12:01 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59822 and previous config saved to /var/cache/conftool/dbconfig/20240408-120101-arnaudb.json
  • 11:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1031.eqiad.wmnet
  • 11:57 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1030.eqiad.wmnet
  • 11:49 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1030.eqiad.wmnet
  • 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1029.eqiad.wmnet
  • 11:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59821 and previous config saved to /var/cache/conftool/dbconfig/20240408-114552-arnaudb.json
  • 11:42 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1029.eqiad.wmnet
  • 11:35 moritzm: installing glibc security updates on bullseye
  • 11:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59820 and previous config saved to /var/cache/conftool/dbconfig/20240408-113045-arnaudb.json
  • 11:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59819 and previous config saved to /var/cache/conftool/dbconfig/20240408-112807-arnaudb.json
  • 11:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 11:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 11:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T360332)', diff saved to https://phabricator.wikimedia.org/P59818 and previous config saved to /var/cache/conftool/dbconfig/20240408-112744-arnaudb.json
  • 11:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59817 and previous config saved to /var/cache/conftool/dbconfig/20240408-112052-arnaudb.json
  • 11:13 hnowlan@deploy1002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:12 hnowlan@deploy1002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59816 and previous config saved to /var/cache/conftool/dbconfig/20240408-111236-arnaudb.json
  • 11:12 hnowlan@deploy1002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 11:12 hnowlan@deploy1002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
  • 11:11 hnowlan@deploy1002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:10 hnowlan@deploy1002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:09 hnowlan@deploy1002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 11:09 hnowlan@deploy1002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
  • 11:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59815 and previous config saved to /var/cache/conftool/dbconfig/20240408-110545-arnaudb.json
  • 11:03 btullis: started manual wikidata dump on snapshot1009 for T252396
  • 10:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59814 and previous config saved to /var/cache/conftool/dbconfig/20240408-105729-arnaudb.json
  • 10:51 Dreamy_Jazz: Starting scan on dewiki for MediaModeration to catch-up on monthly limits - https://wikitech.wikimedia.org/wiki/MediaModeration
  • 10:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59813 and previous config saved to /var/cache/conftool/dbconfig/20240408-105036-arnaudb.json
  • 10:49 moritzm: installing postgresql-13 security updates
  • 10:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T360332)', diff saved to https://phabricator.wikimedia.org/P59812 and previous config saved to /var/cache/conftool/dbconfig/20240408-104221-arnaudb.json
  • 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1200 (T360332)', diff saved to https://phabricator.wikimedia.org/P59811 and previous config saved to /var/cache/conftool/dbconfig/20240408-103945-arnaudb.json
  • 10:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 10:39 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T360332)', diff saved to https://phabricator.wikimedia.org/P59810 and previous config saved to /var/cache/conftool/dbconfig/20240408-103922-arnaudb.json
  • 10:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59809 and previous config saved to /var/cache/conftool/dbconfig/20240408-103529-arnaudb.json
  • 10:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59808 and previous config saved to /var/cache/conftool/dbconfig/20240408-103313-arnaudb.json
  • 10:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 10:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 10:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59807 and previous config saved to /var/cache/conftool/dbconfig/20240408-103249-arnaudb.json
  • 10:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59806 and previous config saved to /var/cache/conftool/dbconfig/20240408-102414-arnaudb.json
  • 10:24 Dreamy_Jazz: Starting MediaModeration scanning script (stopped over the weekend due to server instability) - https://wikitech.wikimedia.org/wiki/MediaModeration
  • 10:18 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2128.codfw.wmnet
  • 10:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59805 and previous config saved to /var/cache/conftool/dbconfig/20240408-101741-arnaudb.json
  • 10:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59804 and previous config saved to /var/cache/conftool/dbconfig/20240408-100906-arnaudb.json
  • 10:07 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2128.codfw.wmnet
  • 10:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59803 and previous config saved to /var/cache/conftool/dbconfig/20240408-100233-arnaudb.json
  • 09:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T360332)', diff saved to https://phabricator.wikimedia.org/P59802 and previous config saved to /var/cache/conftool/dbconfig/20240408-095359-arnaudb.json
  • 09:53 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::proxy::master
  • 09:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1185 (T360332)', diff saved to https://phabricator.wikimedia.org/P59801 and previous config saved to /var/cache/conftool/dbconfig/20240408-095123-arnaudb.json
  • 09:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 09:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 09:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59800 and previous config saved to /var/cache/conftool/dbconfig/20240408-095100-arnaudb.json
  • 09:49 jgiannelos@deploy1002: Finished deploy [restbase/deploy@c4d19d7]: (no justification provided) (duration: 03m 49s)
  • 09:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59799 and previous config saved to /var/cache/conftool/dbconfig/20240408-094726-arnaudb.json
  • 09:46 jgiannelos@deploy1002: Started deploy [restbase/deploy@c4d19d7]: (no justification provided)
  • 09:45 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/services/blubberoid: apply
  • 09:45 jayme@deploy1002: helmfile [eqiad] START helmfile.d/services/blubberoid: apply
  • 09:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59798 and previous config saved to /var/cache/conftool/dbconfig/20240408-094510-arnaudb.json
  • 09:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 09:45 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/services/blubberoid: apply
  • 09:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 09:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59797 and previous config saved to /var/cache/conftool/dbconfig/20240408-094447-arnaudb.json
  • 09:44 jayme@deploy1002: helmfile [codfw] START helmfile.d/services/blubberoid: apply
  • 09:44 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
  • 09:44 jayme@deploy1002: helmfile [eqiad] START helmfile.d/services/apertium: apply
  • 09:43 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/services/apertium: apply
  • 09:43 jayme@deploy1002: helmfile [codfw] START helmfile.d/services/apertium: apply
  • 09:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 100%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59796 and previous config saved to /var/cache/conftool/dbconfig/20240408-094102-arnaudb.json
  • 09:37 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::proxy::master
  • 09:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59795 and previous config saved to /var/cache/conftool/dbconfig/20240408-093552-arnaudb.json
  • 09:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59794 and previous config saved to /var/cache/conftool/dbconfig/20240408-092939-arnaudb.json
  • 09:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 75%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59793 and previous config saved to /var/cache/conftool/dbconfig/20240408-092557-arnaudb.json
  • 09:22 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/apertium: apply
  • 09:21 jayme@deploy1002: helmfile [staging] START helmfile.d/services/apertium: apply
  • 09:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59792 and previous config saved to /var/cache/conftool/dbconfig/20240408-092045-arnaudb.json
  • 09:20 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/blubberoid: apply
  • 09:17 jayme@deploy1002: helmfile [staging] START helmfile.d/services/blubberoid: apply
  • 09:17 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 09:16 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 09:16 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:16 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 09:14 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 15830
  • 09:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59791 and previous config saved to /var/cache/conftool/dbconfig/20240408-091432-arnaudb.json
  • 09:10 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 15830
  • 09:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 50%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59790 and previous config saved to /var/cache/conftool/dbconfig/20240408-091051-arnaudb.json
  • 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::sanitarium_multiinstance
  • 09:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59789 and previous config saved to /var/cache/conftool/dbconfig/20240408-090535-arnaudb.json
  • 09:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59788 and previous config saved to /var/cache/conftool/dbconfig/20240408-090258-arnaudb.json
  • 09:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 09:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 08:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59787 and previous config saved to /var/cache/conftool/dbconfig/20240408-085924-arnaudb.json
  • 08:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 08:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 08:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59786 and previous config saved to /var/cache/conftool/dbconfig/20240408-085708-arnaudb.json
  • 08:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 08:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 08:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 25%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59785 and previous config saved to /var/cache/conftool/dbconfig/20240408-085545-arnaudb.json
  • 08:44 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::sanitarium_multiinstance
  • 08:41 godog: grafana upgrade to 9.5.18 - T361830
  • 08:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2114.codfw.wmnet onto db2214.codfw.wmnet
  • 08:29 dcausse: restarting blazegraph on wdqs1020 (BlazegraphFreeAllocatorsDecreasingRapidly)
  • 08:26 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
  • 08:25 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
  • 08:24 kartik@deploy1002: Finished scap: Backport for Enable the unified dashboard on the test instance for all languages (T360607) (duration: 15m 47s)
  • 08:24 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
  • 08:23 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
  • 08:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Bump db2112 weight T361786', diff saved to https://phabricator.wikimedia.org/P59784 and previous config saved to /var/cache/conftool/dbconfig/20240408-081320-arnaudb.json
  • 08:12 kartik@deploy1002: kartik: Continuing with sync
  • 08:12 volans: restarted stashbot that had died few minutes ago
  • 08:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2203 to s1 primary T361786', diff saved to https://phabricator.wikimedia.org/P59783 and previous config saved to /var/cache/conftool/dbconfig/20240408-080910-arnaudb.json
  • 08:08 arnaudb: Starting s1 codfw failover from db2112 to db2203 - T361786
  • 08:08 kartik@deploy1002: Started scap: Backport for Enable the unified dashboard on the test instance for all languages (T360607)
  • 07:57 filippo@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 07:57 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 07:56 filippo@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 07:56 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 07:56 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 07:55 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 07:48 kartik@deploy1002: Finished scap: Backport for Add Kartographer Parsoid support to hewikivoyage (T342871 T361025) (duration: 35m 43s)
  • 07:47 moritzm: installing util-linux security updates on bullseye/bookworm
  • 07:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59782 and previous config saved to /var/cache/conftool/dbconfig/20240408-074448-root.json
  • 07:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2203 with weight 0 T361786', diff saved to https://phabricator.wikimedia.org/P59781 and previous config saved to /var/cache/conftool/dbconfig/20240408-074006-arnaudb.json
  • 07:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 37 hosts with reason: Primary switchover s1 T361786
  • 07:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 37 hosts with reason: Primary switchover s1 T361786
  • 07:35 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2114.codfw.wmnet onto db2214.codfw.wmnet
  • 07:35 kartik@deploy1002: kartik and ihurbain: Continuing with sync
  • 07:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2114 in db2214 for T355422', diff saved to https://phabricator.wikimedia.org/P59780 and previous config saved to /var/cache/conftool/dbconfig/20240408-073239-arnaudb.json
  • 07:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
  • 07:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
  • 07:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
  • 07:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
  • 07:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59779 and previous config saved to /var/cache/conftool/dbconfig/20240408-072942-root.json
  • 07:25 kartik@deploy1002: kartik and ihurbain: Backport for Add Kartographer Parsoid support to hewikivoyage (T342871 T361025) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59778 and previous config saved to /var/cache/conftool/dbconfig/20240408-071436-root.json
  • 07:12 kartik@deploy1002: Started scap: Backport for Add Kartographer Parsoid support to hewikivoyage (T342871 T361025)
  • 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59777 and previous config saved to /var/cache/conftool/dbconfig/20240408-065931-root.json
  • 06:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59776 and previous config saved to /var/cache/conftool/dbconfig/20240408-064424-root.json
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59775 and previous config saved to /var/cache/conftool/dbconfig/20240408-062919-root.json
  • 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59774 and previous config saved to /var/cache/conftool/dbconfig/20240408-061413-root.json
  • 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1156', diff saved to https://phabricator.wikimedia.org/P59773 and previous config saved to /var/cache/conftool/dbconfig/20240408-060554-root.json
  • 04:02 denisse: Cleaning Prometheus and Thanos-BE log gzips older than 45 days on centrallog2002
  • 04:01 denisse: Cleaning Prometheus and Thanos-BE log gzips older than 45 days on centrallog1002

2024-04-06

  • 15:33 jhathaway@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM mx-out2001.wikimedia.org
  • 15:33 jhathaway@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM mx-out2001.wikimedia.org
  • 03:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 03:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 03:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T360332)', diff saved to https://phabricator.wikimedia.org/P59763 and previous config saved to /var/cache/conftool/dbconfig/20240406-034152-arnaudb.json
  • 03:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P59762 and previous config saved to /var/cache/conftool/dbconfig/20240406-032644-arnaudb.json
  • 03:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P59761 and previous config saved to /var/cache/conftool/dbconfig/20240406-031136-arnaudb.json
  • 02:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T360332)', diff saved to https://phabricator.wikimedia.org/P59760 and previous config saved to /var/cache/conftool/dbconfig/20240406-025629-arnaudb.json
  • 02:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2195 (T360332)', diff saved to https://phabricator.wikimedia.org/P59759 and previous config saved to /var/cache/conftool/dbconfig/20240406-025411-arnaudb.json
  • 02:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2195.codfw.wmnet with reason: Maintenance
  • 02:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2195.codfw.wmnet with reason: Maintenance
  • 02:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T360332)', diff saved to https://phabricator.wikimedia.org/P59758 and previous config saved to /var/cache/conftool/dbconfig/20240406-025348-arnaudb.json
  • 02:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P59757 and previous config saved to /var/cache/conftool/dbconfig/20240406-023841-arnaudb.json
  • 02:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P59756 and previous config saved to /var/cache/conftool/dbconfig/20240406-022333-arnaudb.json
  • 02:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T360332)', diff saved to https://phabricator.wikimedia.org/P59755 and previous config saved to /var/cache/conftool/dbconfig/20240406-020826-arnaudb.json
  • 02:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2181 (T360332)', diff saved to https://phabricator.wikimedia.org/P59754 and previous config saved to /var/cache/conftool/dbconfig/20240406-020608-arnaudb.json
  • 02:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2181.codfw.wmnet with reason: Maintenance
  • 02:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2181.codfw.wmnet with reason: Maintenance
  • 02:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59753 and previous config saved to /var/cache/conftool/dbconfig/20240406-020545-arnaudb.json
  • 01:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P59752 and previous config saved to /var/cache/conftool/dbconfig/20240406-015037-arnaudb.json
  • 01:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P59751 and previous config saved to /var/cache/conftool/dbconfig/20240406-013530-arnaudb.json
  • 01:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59750 and previous config saved to /var/cache/conftool/dbconfig/20240406-012021-arnaudb.json
  • 01:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59749 and previous config saved to /var/cache/conftool/dbconfig/20240406-011803-arnaudb.json
  • 01:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 01:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 01:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T360332)', diff saved to https://phabricator.wikimedia.org/P59748 and previous config saved to /var/cache/conftool/dbconfig/20240406-011740-arnaudb.json
  • 01:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P59747 and previous config saved to /var/cache/conftool/dbconfig/20240406-010231-arnaudb.json
  • 00:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P59746 and previous config saved to /var/cache/conftool/dbconfig/20240406-004724-arnaudb.json
  • 00:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T360332)', diff saved to https://phabricator.wikimedia.org/P59745 and previous config saved to /var/cache/conftool/dbconfig/20240406-003216-arnaudb.json
  • 00:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2166 (T360332)', diff saved to https://phabricator.wikimedia.org/P59744 and previous config saved to /var/cache/conftool/dbconfig/20240406-002958-arnaudb.json
  • 00:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2166.codfw.wmnet with reason: Maintenance
  • 00:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2166.codfw.wmnet with reason: Maintenance
  • 00:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T360332)', diff saved to https://phabricator.wikimedia.org/P59743 and previous config saved to /var/cache/conftool/dbconfig/20240406-002935-arnaudb.json
  • 00:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P59742 and previous config saved to /var/cache/conftool/dbconfig/20240406-001428-arnaudb.json

2024-04-05

  • 23:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P59741 and previous config saved to /var/cache/conftool/dbconfig/20240405-235920-arnaudb.json
  • 23:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T360332)', diff saved to https://phabricator.wikimedia.org/P59740 and previous config saved to /var/cache/conftool/dbconfig/20240405-234413-arnaudb.json
  • 23:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2164 (T360332)', diff saved to https://phabricator.wikimedia.org/P59739 and previous config saved to /var/cache/conftool/dbconfig/20240405-234156-arnaudb.json
  • 23:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 23:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 23:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2164.codfw.wmnet with reason: Maintenance
  • 23:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2164.codfw.wmnet with reason: Maintenance
  • 23:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T360332)', diff saved to https://phabricator.wikimedia.org/P59738 and previous config saved to /var/cache/conftool/dbconfig/20240405-234117-arnaudb.json
  • 23:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P59737 and previous config saved to /var/cache/conftool/dbconfig/20240405-232609-arnaudb.json
  • 23:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P59736 and previous config saved to /var/cache/conftool/dbconfig/20240405-231102-arnaudb.json
  • 22:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T360332)', diff saved to https://phabricator.wikimedia.org/P59735 and previous config saved to /var/cache/conftool/dbconfig/20240405-225554-arnaudb.json
  • 22:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2163 (T360332)', diff saved to https://phabricator.wikimedia.org/P59734 and previous config saved to /var/cache/conftool/dbconfig/20240405-225336-arnaudb.json
  • 22:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance
  • 22:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance
  • 22:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T360332)', diff saved to https://phabricator.wikimedia.org/P59733 and previous config saved to /var/cache/conftool/dbconfig/20240405-225313-arnaudb.json
  • 22:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P59732 and previous config saved to /var/cache/conftool/dbconfig/20240405-223806-arnaudb.json
  • 22:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P59731 and previous config saved to /var/cache/conftool/dbconfig/20240405-222259-arnaudb.json
  • 22:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T360332)', diff saved to https://phabricator.wikimedia.org/P59730 and previous config saved to /var/cache/conftool/dbconfig/20240405-220751-arnaudb.json
  • 22:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2162 (T360332)', diff saved to https://phabricator.wikimedia.org/P59729 and previous config saved to /var/cache/conftool/dbconfig/20240405-220533-arnaudb.json
  • 22:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2162.codfw.wmnet with reason: Maintenance
  • 22:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2162.codfw.wmnet with reason: Maintenance
  • 22:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59728 and previous config saved to /var/cache/conftool/dbconfig/20240405-220510-arnaudb.json
  • 21:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P59727 and previous config saved to /var/cache/conftool/dbconfig/20240405-215001-arnaudb.json
  • 21:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P59725 and previous config saved to /var/cache/conftool/dbconfig/20240405-213454-arnaudb.json
  • 21:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59724 and previous config saved to /var/cache/conftool/dbconfig/20240405-211946-arnaudb.json
  • 21:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59723 and previous config saved to /var/cache/conftool/dbconfig/20240405-211728-arnaudb.json
  • 21:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2161.codfw.wmnet with reason: Maintenance
  • 21:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2161.codfw.wmnet with reason: Maintenance
  • 21:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T360332)', diff saved to https://phabricator.wikimedia.org/P59722 and previous config saved to /var/cache/conftool/dbconfig/20240405-211705-arnaudb.json
  • 21:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P59721 and previous config saved to /var/cache/conftool/dbconfig/20240405-210157-arnaudb.json
  • 20:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P59720 and previous config saved to /var/cache/conftool/dbconfig/20240405-204650-arnaudb.json
  • 20:40 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx-out2001.wikimedia.org with reason: host reimage
  • 20:37 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on mx-out2001.wikimedia.org with reason: host reimage
  • 20:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T360332)', diff saved to https://phabricator.wikimedia.org/P59719 and previous config saved to /var/cache/conftool/dbconfig/20240405-203143-arnaudb.json
  • 20:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2154 (T360332)', diff saved to https://phabricator.wikimedia.org/P59718 and previous config saved to /var/cache/conftool/dbconfig/20240405-202925-arnaudb.json
  • 20:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2154.codfw.wmnet with reason: Maintenance
  • 20:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2154.codfw.wmnet with reason: Maintenance
  • 20:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T360332)', diff saved to https://phabricator.wikimedia.org/P59717 and previous config saved to /var/cache/conftool/dbconfig/20240405-202901-arnaudb.json
  • 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host mx-out2001.wikimedia.org with OS bookworm
  • 20:19 jhathaway@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
  • 20:18 jhathaway@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
  • 20:18 jhathaway@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mx-out2001.wikimedia.org on all recursors
  • 20:18 jhathaway@cumin2002: START - Cookbook sre.dns.wipe-cache mx-out2001.wikimedia.org on all recursors
  • 20:18 jhathaway@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:18 jhathaway@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
  • 20:16 jhathaway@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
  • 20:15 jhathaway@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mx-out1001.wikimedia.org with OS bookworm
  • 20:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P59716 and previous config saved to /var/cache/conftool/dbconfig/20240405-201354-arnaudb.json
  • 20:13 jhathaway@cumin2002: START - Cookbook sre.dns.netbox
  • 20:13 jhathaway@cumin2002: START - Cookbook sre.ganeti.makevm for new host mx-out2001.wikimedia.org
  • 20:02 jhathaway@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
  • 19:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P59715 and previous config saved to /var/cache/conftool/dbconfig/20240405-195847-arnaudb.json
  • 19:57 jhathaway@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
  • 19:45 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host mx-out1001.wikimedia.org with OS bookworm
  • 19:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T360332)', diff saved to https://phabricator.wikimedia.org/P59714 and previous config saved to /var/cache/conftool/dbconfig/20240405-194339-arnaudb.json
  • 19:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2152 (T360332)', diff saved to https://phabricator.wikimedia.org/P59713 and previous config saved to /var/cache/conftool/dbconfig/20240405-194221-arnaudb.json
  • 19:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2152.codfw.wmnet with reason: Maintenance
  • 19:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2152.codfw.wmnet with reason: Maintenance
  • 19:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 19:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 19:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T360332)', diff saved to https://phabricator.wikimedia.org/P59712 and previous config saved to /var/cache/conftool/dbconfig/20240405-194057-arnaudb.json
  • 19:40 jhathaway@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host mx-out1001.wikimedia.org
  • 19:40 jhathaway@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host mx-out1001.wikimedia.org with OS bookworm
  • 19:27 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: Host down
  • 19:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: Host down
  • 19:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59711 and previous config saved to /var/cache/conftool/dbconfig/20240405-192549-arnaudb.json
  • 19:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59710 and previous config saved to /var/cache/conftool/dbconfig/20240405-191042-arnaudb.json
  • 19:02 mutante: codesearch - puppet trying to restart hound-search after deploying gerrit:1017179 and gerrit:1016480
  • 18:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T360332)', diff saved to https://phabricator.wikimedia.org/P59709 and previous config saved to /var/cache/conftool/dbconfig/20240405-185533-arnaudb.json
  • 18:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T360332)', diff saved to https://phabricator.wikimedia.org/P59708 and previous config saved to /var/cache/conftool/dbconfig/20240405-185216-arnaudb.json
  • 18:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 18:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 18:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 18:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 18:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59707 and previous config saved to /var/cache/conftool/dbconfig/20240405-185131-arnaudb.json
  • 18:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59706 and previous config saved to /var/cache/conftool/dbconfig/20240405-183623-arnaudb.json
  • 18:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59705 and previous config saved to /var/cache/conftool/dbconfig/20240405-182115-arnaudb.json
  • 18:13 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1214.eqiad.wmnet
  • 18:13 sukhe@cumin2002: START - Cookbook sre.hosts.remove-downtime for db1214.eqiad.wmnet
  • 18:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59704 and previous config saved to /var/cache/conftool/dbconfig/20240405-180608-arnaudb.json
  • 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59703 and previous config saved to /var/cache/conftool/dbconfig/20240405-180352-arnaudb.json
  • 18:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 18:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59702 and previous config saved to /var/cache/conftool/dbconfig/20240405-180330-arnaudb.json
  • 18:03 dzahn@cumin2002: dbctl commit (dc=all): 'depool db1246', diff saved to https://phabricator.wikimedia.org/P59701 and previous config saved to /var/cache/conftool/dbconfig/20240405-180319-dzahn.json
  • 18:01 mutante: depooling db1246 which went down and paged
  • 17:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59700 and previous config saved to /var/cache/conftool/dbconfig/20240405-174735-arnaudb.json
  • 17:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59699 and previous config saved to /var/cache/conftool/dbconfig/20240405-173227-arnaudb.json
  • 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59698 and previous config saved to /var/cache/conftool/dbconfig/20240405-171719-arnaudb.json
  • 17:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59697 and previous config saved to /var/cache/conftool/dbconfig/20240405-171502-arnaudb.json
  • 17:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T360332)', diff saved to https://phabricator.wikimedia.org/P59696 and previous config saved to /var/cache/conftool/dbconfig/20240405-171439-arnaudb.json
  • 17:14 jhathaway@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
  • 17:11 jhathaway@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
  • 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59695 and previous config saved to /var/cache/conftool/dbconfig/20240405-165931-arnaudb.json
  • 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59694 and previous config saved to /var/cache/conftool/dbconfig/20240405-164424-arnaudb.json
  • 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T360332)', diff saved to https://phabricator.wikimedia.org/P59693 and previous config saved to /var/cache/conftool/dbconfig/20240405-162916-arnaudb.json
  • 16:27 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host mx-out1001.wikimedia.org with OS bookworm
  • 16:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T360332)', diff saved to https://phabricator.wikimedia.org/P59692 and previous config saved to /var/cache/conftool/dbconfig/20240405-162700-arnaudb.json
  • 16:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59691 and previous config saved to /var/cache/conftool/dbconfig/20240405-162637-arnaudb.json
  • 16:25 jhathaway@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
  • 16:24 jhathaway@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
  • 16:24 jhathaway@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mx-out1001.wikimedia.org on all recursors
  • 16:24 jhathaway@cumin1002: START - Cookbook sre.dns.wipe-cache mx-out1001.wikimedia.org on all recursors
  • 16:24 jhathaway@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:24 jhathaway@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
  • 16:18 bking@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:18 bking@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59689 and previous config saved to /var/cache/conftool/dbconfig/20240405-161130-arnaudb.json
  • 16:07 jhathaway@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
  • 16:03 jhathaway@cumin1002: START - Cookbook sre.dns.netbox
  • 16:03 jhathaway@cumin1002: START - Cookbook sre.ganeti.makevm for new host mx-out1001.wikimedia.org
  • 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59688 and previous config saved to /var/cache/conftool/dbconfig/20240405-155622-arnaudb.json
  • 15:45 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
  • 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59687 and previous config saved to /var/cache/conftool/dbconfig/20240405-154115-arnaudb.json
  • 15:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59686 and previous config saved to /var/cache/conftool/dbconfig/20240405-153759-arnaudb.json
  • 15:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 15:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 15:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59685 and previous config saved to /var/cache/conftool/dbconfig/20240405-153736-arnaudb.json
  • 15:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59684 and previous config saved to /var/cache/conftool/dbconfig/20240405-152228-arnaudb.json
  • 15:20 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
  • 15:17 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
  • 15:14 aikochou@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 15:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59683 and previous config saved to /var/cache/conftool/dbconfig/20240405-150721-arnaudb.json
  • 15:03 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
  • 14:56 dancy@deploy1002: Installation of scap version "4.75.0" completed for 353 hosts
  • 14:55 dancy@deploy1002: Installing scap version "4.75.0" for 353 hosts
  • 14:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59682 and previous config saved to /var/cache/conftool/dbconfig/20240405-145213-arnaudb.json
  • 14:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59681 and previous config saved to /var/cache/conftool/dbconfig/20240405-144957-arnaudb.json
  • 14:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 14:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 14:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59680 and previous config saved to /var/cache/conftool/dbconfig/20240405-144934-arnaudb.json
  • 14:36 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1016.eqiad.wmnet
  • 14:36 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1016.eqiad.wmnet
  • 14:35 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet
  • 14:35 aikochou@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 14:34 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet
  • 14:34 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet
  • 14:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59679 and previous config saved to /var/cache/conftool/dbconfig/20240405-143427-arnaudb.json
  • 14:33 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1014.eqiad.wmnet
  • 14:33 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet
  • 14:29 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet
  • 14:26 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1020.eqiad.wmnet
  • 14:25 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1020.eqiad.wmnet
  • 14:25 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1019.eqiad.wmnet
  • 14:22 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet
  • 14:22 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1018.eqiad.wmnet
  • 14:20 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1018.eqiad.wmnet
  • 14:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59678 and previous config saved to /var/cache/conftool/dbconfig/20240405-141919-arnaudb.json
  • 14:18 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet
  • 14:15 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet
  • 14:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59677 and previous config saved to /var/cache/conftool/dbconfig/20240405-140412-arnaudb.json
  • 13:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59676 and previous config saved to /var/cache/conftool/dbconfig/20240405-130347-arnaudb.json
  • 13:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 13:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 13:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59675 and previous config saved to /var/cache/conftool/dbconfig/20240405-130324-arnaudb.json
  • 12:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59674 and previous config saved to /var/cache/conftool/dbconfig/20240405-124816-arnaudb.json
  • 12:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59673 and previous config saved to /var/cache/conftool/dbconfig/20240405-123309-arnaudb.json
  • 12:32 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet
  • 12:32 fabfur: repool cp4037 (T361845)
  • 12:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59672 and previous config saved to /var/cache/conftool/dbconfig/20240405-121801-arnaudb.json
  • 12:10 ayounsi@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts testvm2006.codfw.wmnet
  • 12:10 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:10 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2006.codfw.wmnet decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 11:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59671 and previous config saved to /var/cache/conftool/dbconfig/20240405-111736-arnaudb.json
  • 11:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 11:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 11:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59670 and previous config saved to /var/cache/conftool/dbconfig/20240405-111713-arnaudb.json
  • 11:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59669 and previous config saved to /var/cache/conftool/dbconfig/20240405-110204-arnaudb.json
  • 10:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59668 and previous config saved to /var/cache/conftool/dbconfig/20240405-104657-arnaudb.json
  • 10:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59667 and previous config saved to /var/cache/conftool/dbconfig/20240405-103149-arnaudb.json
  • 09:51 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2006.codfw.wmnet decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 09:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 09:37 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 09:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T356166)', diff saved to https://phabricator.wikimedia.org/P59666 and previous config saved to /var/cache/conftool/dbconfig/20240405-093745-marostegui.json
  • 09:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59665 and previous config saved to /var/cache/conftool/dbconfig/20240405-093124-arnaudb.json
  • 09:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 09:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 09:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 09:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 09:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59664 and previous config saved to /var/cache/conftool/dbconfig/20240405-093038-arnaudb.json
  • 09:24 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 09:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P59663 and previous config saved to /var/cache/conftool/dbconfig/20240405-092237-marostegui.json
  • 09:19 ayounsi@cumin1002: START - Cookbook sre.hosts.decommission for hosts testvm2006.codfw.wmnet
  • 09:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59662 and previous config saved to /var/cache/conftool/dbconfig/20240405-091531-arnaudb.json
  • 09:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P59661 and previous config saved to /var/cache/conftool/dbconfig/20240405-090730-marostegui.json
  • 09:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59660 and previous config saved to /var/cache/conftool/dbconfig/20240405-090023-arnaudb.json
  • 08:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T356166)', diff saved to https://phabricator.wikimedia.org/P59659 and previous config saved to /var/cache/conftool/dbconfig/20240405-085222-marostegui.json
  • 08:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59658 and previous config saved to /var/cache/conftool/dbconfig/20240405-084515-arnaudb.json
  • 07:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P59655 and previous config saved to /var/cache/conftool/dbconfig/20240405-075831-marostegui.json
  • 07:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59654 and previous config saved to /var/cache/conftool/dbconfig/20240405-075646-arnaudb.json
  • 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 07:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 07:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 07:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P59653 and previous config saved to /var/cache/conftool/dbconfig/20240405-074323-marostegui.json
  • 07:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T356166)', diff saved to https://phabricator.wikimedia.org/P59652 and previous config saved to /var/cache/conftool/dbconfig/20240405-072816-marostegui.json
  • 06:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 108
  • 06:35 ayounsi@cumin1002: START - Cookbook sre.network.debug for Netbox circuit ID 108
  • 04:40 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1234 (T356166)', diff saved to https://phabricator.wikimedia.org/P59651 and previous config saved to /var/cache/conftool/dbconfig/20240405-044048-marostegui.json
  • 04:40 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 04:40 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 04:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T356166)', diff saved to https://phabricator.wikimedia.org/P59650 and previous config saved to /var/cache/conftool/dbconfig/20240405-044025-marostegui.json
  • 04:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P59649 and previous config saved to /var/cache/conftool/dbconfig/20240405-042517-marostegui.json
  • 04:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P59648 and previous config saved to /var/cache/conftool/dbconfig/20240405-041010-marostegui.json
  • 03:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T360332)', diff saved to https://phabricator.wikimedia.org/P59647 and previous config saved to /var/cache/conftool/dbconfig/20240405-035829-arnaudb.json
  • 03:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T356166)', diff saved to https://phabricator.wikimedia.org/P59646 and previous config saved to /var/cache/conftool/dbconfig/20240405-035503-marostegui.json
  • 03:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T356166)', diff saved to https://phabricator.wikimedia.org/P59645 and previous config saved to /var/cache/conftool/dbconfig/20240405-035353-marostegui.json
  • 03:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 03:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 03:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T356166)', diff saved to https://phabricator.wikimedia.org/P59644 and previous config saved to /var/cache/conftool/dbconfig/20240405-035331-marostegui.json
  • 03:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P59643 and previous config saved to /var/cache/conftool/dbconfig/20240405-034322-arnaudb.json
  • 03:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P59642 and previous config saved to /var/cache/conftool/dbconfig/20240405-033823-marostegui.json
  • 03:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P59641 and previous config saved to /var/cache/conftool/dbconfig/20240405-032814-arnaudb.json
  • 03:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P59640 and previous config saved to /var/cache/conftool/dbconfig/20240405-032316-marostegui.json
  • 03:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T360332)', diff saved to https://phabricator.wikimedia.org/P59639 and previous config saved to /var/cache/conftool/dbconfig/20240405-031307-arnaudb.json
  • 03:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2220 (T360332)', diff saved to https://phabricator.wikimedia.org/P59638 and previous config saved to /var/cache/conftool/dbconfig/20240405-031028-arnaudb.json
  • 03:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2220.codfw.wmnet with reason: Maintenance
  • 03:10 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2220.codfw.wmnet with reason: Maintenance
  • 03:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T360332)', diff saved to https://phabricator.wikimedia.org/P59637 and previous config saved to /var/cache/conftool/dbconfig/20240405-031005-arnaudb.json
  • 03:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T356166)', diff saved to https://phabricator.wikimedia.org/P59636 and previous config saved to /var/cache/conftool/dbconfig/20240405-030809-marostegui.json
  • 02:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P59635 and previous config saved to /var/cache/conftool/dbconfig/20240405-025458-arnaudb.json
  • 02:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P59634 and previous config saved to /var/cache/conftool/dbconfig/20240405-023949-arnaudb.json
  • 02:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T360332)', diff saved to https://phabricator.wikimedia.org/P59633 and previous config saved to /var/cache/conftool/dbconfig/20240405-022442-arnaudb.json
  • 02:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2218 (T360332)', diff saved to https://phabricator.wikimedia.org/P59632 and previous config saved to /var/cache/conftool/dbconfig/20240405-022201-arnaudb.json
  • 02:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2218.codfw.wmnet with reason: Maintenance
  • 02:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2218.codfw.wmnet with reason: Maintenance
  • 02:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T360332)', diff saved to https://phabricator.wikimedia.org/P59631 and previous config saved to /var/cache/conftool/dbconfig/20240405-022138-arnaudb.json
  • 02:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P59630 and previous config saved to /var/cache/conftool/dbconfig/20240405-020630-arnaudb.json
  • 01:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P59629 and previous config saved to /var/cache/conftool/dbconfig/20240405-015123-arnaudb.json
  • 01:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T360332)', diff saved to https://phabricator.wikimedia.org/P59628 and previous config saved to /var/cache/conftool/dbconfig/20240405-013615-arnaudb.json
  • 01:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2208 (T360332)', diff saved to https://phabricator.wikimedia.org/P59627 and previous config saved to /var/cache/conftool/dbconfig/20240405-013336-arnaudb.json
  • 01:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2208.codfw.wmnet with reason: Maintenance
  • 01:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2208.codfw.wmnet with reason: Maintenance
  • 01:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 01:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 01:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T360332)', diff saved to https://phabricator.wikimedia.org/P59626 and previous config saved to /var/cache/conftool/dbconfig/20240405-013227-arnaudb.json
  • 01:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P59625 and previous config saved to /var/cache/conftool/dbconfig/20240405-011720-arnaudb.json
  • 01:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P59623 and previous config saved to /var/cache/conftool/dbconfig/20240405-010212-arnaudb.json
  • 00:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1228 (T356166)', diff saved to https://phabricator.wikimedia.org/P59622 and previous config saved to /var/cache/conftool/dbconfig/20240405-005341-marostegui.json
  • 00:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 00:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 00:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T356166)', diff saved to https://phabricator.wikimedia.org/P59621 and previous config saved to /var/cache/conftool/dbconfig/20240405-005318-marostegui.json
  • 00:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T360332)', diff saved to https://phabricator.wikimedia.org/P59620 and previous config saved to /var/cache/conftool/dbconfig/20240405-004705-arnaudb.json
  • 00:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2182 (T360332)', diff saved to https://phabricator.wikimedia.org/P59619 and previous config saved to /var/cache/conftool/dbconfig/20240405-004428-arnaudb.json
  • 00:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 00:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 00:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59618 and previous config saved to /var/cache/conftool/dbconfig/20240405-004405-arnaudb.json
  • 00:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P59617 and previous config saved to /var/cache/conftool/dbconfig/20240405-003810-marostegui.json
  • 00:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P59616 and previous config saved to /var/cache/conftool/dbconfig/20240405-002857-arnaudb.json
  • 00:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P59615 and previous config saved to /var/cache/conftool/dbconfig/20240405-002303-marostegui.json
  • 00:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P59614 and previous config saved to /var/cache/conftool/dbconfig/20240405-001350-arnaudb.json
  • 00:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T356166)', diff saved to https://phabricator.wikimedia.org/P59613 and previous config saved to /var/cache/conftool/dbconfig/20240405-000755-marostegui.json

2024-04-04

  • 23:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59612 and previous config saved to /var/cache/conftool/dbconfig/20240404-235843-arnaudb.json
  • 23:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59611 and previous config saved to /var/cache/conftool/dbconfig/20240404-235606-arnaudb.json
  • 23:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 23:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 23:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T360332)', diff saved to https://phabricator.wikimedia.org/P59610 and previous config saved to /var/cache/conftool/dbconfig/20240404-235543-arnaudb.json
  • 23:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P59609 and previous config saved to /var/cache/conftool/dbconfig/20240404-234035-arnaudb.json
  • 23:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P59608 and previous config saved to /var/cache/conftool/dbconfig/20240404-232528-arnaudb.json
  • 23:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T360332)', diff saved to https://phabricator.wikimedia.org/P59606 and previous config saved to /var/cache/conftool/dbconfig/20240404-231020-arnaudb.json
  • 23:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2159 (T360332)', diff saved to https://phabricator.wikimedia.org/P59605 and previous config saved to /var/cache/conftool/dbconfig/20240404-230743-arnaudb.json
  • 23:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 23:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 23:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 23:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 23:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T360332)', diff saved to https://phabricator.wikimedia.org/P59604 and previous config saved to /var/cache/conftool/dbconfig/20240404-230704-arnaudb.json
  • 22:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P59603 and previous config saved to /var/cache/conftool/dbconfig/20240404-225156-arnaudb.json
  • 22:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 22:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 22:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T355609)', diff saved to https://phabricator.wikimedia.org/P59602 and previous config saved to /var/cache/conftool/dbconfig/20240404-224119-marostegui.json
  • 22:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P59601 and previous config saved to /var/cache/conftool/dbconfig/20240404-223649-arnaudb.json
  • 22:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59600 and previous config saved to /var/cache/conftool/dbconfig/20240404-222612-marostegui.json
  • 22:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T360332)', diff saved to https://phabricator.wikimedia.org/P59599 and previous config saved to /var/cache/conftool/dbconfig/20240404-222141-arnaudb.json
  • 22:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2150 (T360332)', diff saved to https://phabricator.wikimedia.org/P59598 and previous config saved to /var/cache/conftool/dbconfig/20240404-221903-arnaudb.json
  • 22:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 22:18 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 22:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T360332)', diff saved to https://phabricator.wikimedia.org/P59597 and previous config saved to /var/cache/conftool/dbconfig/20240404-221839-arnaudb.json
  • 22:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59596 and previous config saved to /var/cache/conftool/dbconfig/20240404-221104-marostegui.json
  • 22:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P59595 and previous config saved to /var/cache/conftool/dbconfig/20240404-220331-arnaudb.json
  • 21:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T355609)', diff saved to https://phabricator.wikimedia.org/P59594 and previous config saved to /var/cache/conftool/dbconfig/20240404-215557-marostegui.json
  • 21:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P59593 and previous config saved to /var/cache/conftool/dbconfig/20240404-214824-arnaudb.json
  • 21:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T356166)', diff saved to https://phabricator.wikimedia.org/P59592 and previous config saved to /var/cache/conftool/dbconfig/20240404-214817-marostegui.json
  • 21:48 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 21:48 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 21:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T356166)', diff saved to https://phabricator.wikimedia.org/P59591 and previous config saved to /var/cache/conftool/dbconfig/20240404-214753-marostegui.json
  • 21:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T360332)', diff saved to https://phabricator.wikimedia.org/P59590 and previous config saved to /var/cache/conftool/dbconfig/20240404-213317-arnaudb.json
  • 21:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P59589 and previous config saved to /var/cache/conftool/dbconfig/20240404-213245-marostegui.json
  • 21:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2122 (T360332)', diff saved to https://phabricator.wikimedia.org/P59588 and previous config saved to /var/cache/conftool/dbconfig/20240404-213031-arnaudb.json
  • 21:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 21:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 21:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T360332)', diff saved to https://phabricator.wikimedia.org/P59587 and previous config saved to /var/cache/conftool/dbconfig/20240404-213008-arnaudb.json
  • 21:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P59586 and previous config saved to /var/cache/conftool/dbconfig/20240404-211738-marostegui.json
  • 21:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P59585 and previous config saved to /var/cache/conftool/dbconfig/20240404-211501-arnaudb.json
  • 21:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T355609)', diff saved to https://phabricator.wikimedia.org/P59584 and previous config saved to /var/cache/conftool/dbconfig/20240404-211248-marostegui.json
  • 21:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 21:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 21:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T356166)', diff saved to https://phabricator.wikimedia.org/P59583 and previous config saved to /var/cache/conftool/dbconfig/20240404-210230-marostegui.json
  • 20:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P59582 and previous config saved to /var/cache/conftool/dbconfig/20240404-205953-arnaudb.json
  • 20:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T360332)', diff saved to https://phabricator.wikimedia.org/P59581 and previous config saved to /var/cache/conftool/dbconfig/20240404-204446-arnaudb.json
  • 20:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2120 (T360332)', diff saved to https://phabricator.wikimedia.org/P59580 and previous config saved to /var/cache/conftool/dbconfig/20240404-204204-arnaudb.json
  • 20:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 20:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 20:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T360332)', diff saved to https://phabricator.wikimedia.org/P59579 and previous config saved to /var/cache/conftool/dbconfig/20240404-204141-arnaudb.json
  • 20:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 20:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 20:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T355609)', diff saved to https://phabricator.wikimedia.org/P59578 and previous config saved to /var/cache/conftool/dbconfig/20240404-203302-marostegui.json
  • 20:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P59577 and previous config saved to /var/cache/conftool/dbconfig/20240404-202634-arnaudb.json
  • 20:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59576 and previous config saved to /var/cache/conftool/dbconfig/20240404-201755-marostegui.json
  • 20:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P59575 and previous config saved to /var/cache/conftool/dbconfig/20240404-201126-arnaudb.json
  • 20:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59574 and previous config saved to /var/cache/conftool/dbconfig/20240404-200247-marostegui.json
  • 19:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T360332)', diff saved to https://phabricator.wikimedia.org/P59573 and previous config saved to /var/cache/conftool/dbconfig/20240404-195615-arnaudb.json
  • 19:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2108 (T360332)', diff saved to https://phabricator.wikimedia.org/P59572 and previous config saved to /var/cache/conftool/dbconfig/20240404-195333-arnaudb.json
  • 19:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 19:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 19:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:52 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 19:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 19:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T360332)', diff saved to https://phabricator.wikimedia.org/P59571 and previous config saved to /var/cache/conftool/dbconfig/20240404-195138-arnaudb.json
  • 19:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T355609)', diff saved to https://phabricator.wikimedia.org/P59570 and previous config saved to /var/cache/conftool/dbconfig/20240404-194739-marostegui.json
  • 19:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P59569 and previous config saved to /var/cache/conftool/dbconfig/20240404-193631-arnaudb.json
  • 19:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P59568 and previous config saved to /var/cache/conftool/dbconfig/20240404-192123-arnaudb.json
  • 19:11 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
  • 19:10 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
  • 19:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T360332)', diff saved to https://phabricator.wikimedia.org/P59567 and previous config saved to /var/cache/conftool/dbconfig/20240404-190616-arnaudb.json
  • 19:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1236 (T360332)', diff saved to https://phabricator.wikimedia.org/P59566 and previous config saved to /var/cache/conftool/dbconfig/20240404-190146-arnaudb.json
  • 19:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance
  • 19:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance
  • 19:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T360332)', diff saved to https://phabricator.wikimedia.org/P59565 and previous config saved to /var/cache/conftool/dbconfig/20240404-190123-arnaudb.json
  • 18:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T355609)', diff saved to https://phabricator.wikimedia.org/P59564 and previous config saved to /var/cache/conftool/dbconfig/20240404-185458-marostegui.json
  • 18:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 18:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 18:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T355609)', diff saved to https://phabricator.wikimedia.org/P59563 and previous config saved to /var/cache/conftool/dbconfig/20240404-185436-marostegui.json
  • 18:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T356166)', diff saved to https://phabricator.wikimedia.org/P59562 and previous config saved to /var/cache/conftool/dbconfig/20240404-185319-marostegui.json
  • 18:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 18:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 18:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T356166)', diff saved to https://phabricator.wikimedia.org/P59561 and previous config saved to /var/cache/conftool/dbconfig/20240404-185256-marostegui.json
  • 18:49 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 18:46 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P59560 and previous config saved to /var/cache/conftool/dbconfig/20240404-184616-arnaudb.json
  • 18:41 denisse@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on webperf2003.codfw.wmnet,webperf1003.eqiad.wmnet with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
  • 18:40 denisse@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on webperf2003.codfw.wmnet,webperf1003.eqiad.wmnet with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
  • 18:39 denisse@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:30:00 on performance.wikimedia.org with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
  • 18:39 denisse@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on performance.wikimedia.org with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
  • 18:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59559 and previous config saved to /var/cache/conftool/dbconfig/20240404-183928-marostegui.json
  • 18:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P59558 and previous config saved to /var/cache/conftool/dbconfig/20240404-183748-marostegui.json
  • 18:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P59557 and previous config saved to /var/cache/conftool/dbconfig/20240404-183108-arnaudb.json
  • 18:31 denisse: Disabling Puppet on the webperf hosts part of the cergen to CFSSL migration - T360414
  • 18:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59556 and previous config saved to /var/cache/conftool/dbconfig/20240404-182421-marostegui.json
  • 18:24 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 18:23 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye
  • 18:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P59555 and previous config saved to /var/cache/conftool/dbconfig/20240404-182241-marostegui.json
  • 18:19 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
  • 18:16 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 18:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T360332)', diff saved to https://phabricator.wikimedia.org/P59554 and previous config saved to /var/cache/conftool/dbconfig/20240404-181601-arnaudb.json
  • 18:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1227 (T360332)', diff saved to https://phabricator.wikimedia.org/P59553 and previous config saved to /var/cache/conftool/dbconfig/20240404-181326-arnaudb.json
  • 18:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
  • 18:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
  • 18:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T360332)', diff saved to https://phabricator.wikimedia.org/P59552 and previous config saved to /var/cache/conftool/dbconfig/20240404-181303-arnaudb.json
  • 18:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T355609)', diff saved to https://phabricator.wikimedia.org/P59551 and previous config saved to /var/cache/conftool/dbconfig/20240404-180913-marostegui.json
  • 18:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T356166)', diff saved to https://phabricator.wikimedia.org/P59550 and previous config saved to /var/cache/conftool/dbconfig/20240404-180733-marostegui.json
  • 18:04 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
  • 18:01 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
  • 17:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P59549 and previous config saved to /var/cache/conftool/dbconfig/20240404-175756-arnaudb.json
  • 17:49 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
  • 17:48 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet with OS bookworm
  • 17:48 sukhe: depool cp4052 to prepare for reimaging
  • 17:46 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 17:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P59548 and previous config saved to /var/cache/conftool/dbconfig/20240404-174246-arnaudb.json
  • 17:37 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 17:36 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 17:35 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:29 moritzm: installing isl bugfix updates from Bookworm point release
  • 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T360332)', diff saved to https://phabricator.wikimedia.org/P59547 and previous config saved to /var/cache/conftool/dbconfig/20240404-172739-arnaudb.json
  • 17:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1202 (T360332)', diff saved to https://phabricator.wikimedia.org/P59546 and previous config saved to /var/cache/conftool/dbconfig/20240404-172408-arnaudb.json
  • 17:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 17:23 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 17:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T360332)', diff saved to https://phabricator.wikimedia.org/P59545 and previous config saved to /var/cache/conftool/dbconfig/20240404-172343-arnaudb.json
  • 17:22 moritzm: installing qemu security updates on bookworm
  • 17:19 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 17:17 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 17:17 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 17:15 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 17:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T355609)', diff saved to https://phabricator.wikimedia.org/P59544 and previous config saved to /var/cache/conftool/dbconfig/20240404-171347-marostegui.json
  • 17:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 17:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 17:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59543 and previous config saved to /var/cache/conftool/dbconfig/20240404-171324-marostegui.json
  • 17:13 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 17:12 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 17:09 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3068.esams.wmnet
  • 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P59542 and previous config saved to /var/cache/conftool/dbconfig/20240404-170836-arnaudb.json
  • 16:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59541 and previous config saved to /var/cache/conftool/dbconfig/20240404-165816-marostegui.json
  • 16:58 pmiazga: T355281 executed “mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki --skipclusters=main,echo,growth,mediamoderation,extstore en wikipedia test2wiki test2.wikipedia.beta.wmcloud.org” on deployment-deploy03.deployment-prep
  • 16:55 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3068.esams.wmnet with OS bullseye
  • 16:54 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1001-dev.eqiad.wmnet with reason: host reimage
  • 16:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P59540 and previous config saved to /var/cache/conftool/dbconfig/20240404-165328-arnaudb.json
  • 16:51 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1001-dev.eqiad.wmnet with reason: host reimage
  • 16:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59539 and previous config saved to /var/cache/conftool/dbconfig/20240404-164309-marostegui.json
  • 16:42 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudbackup1001-dev.eqiad.wmnet with OS bookworm
  • 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T360332)', diff saved to https://phabricator.wikimedia.org/P59538 and previous config saved to /var/cache/conftool/dbconfig/20240404-163819-arnaudb.json
  • 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1194 (T360332)', diff saved to https://phabricator.wikimedia.org/P59537 and previous config saved to /var/cache/conftool/dbconfig/20240404-163549-arnaudb.json
  • 16:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 16:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59536 and previous config saved to /var/cache/conftool/dbconfig/20240404-163526-arnaudb.json
  • 16:32 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3068.esams.wmnet with reason: host reimage
  • 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59535 and previous config saved to /var/cache/conftool/dbconfig/20240404-163107-arnaudb.json
  • 16:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59534 and previous config saved to /var/cache/conftool/dbconfig/20240404-163053-arnaudb.json
  • 16:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59533 and previous config saved to /var/cache/conftool/dbconfig/20240404-162801-marostegui.json
  • 16:27 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3068.esams.wmnet with reason: host reimage
  • 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P59532 and previous config saved to /var/cache/conftool/dbconfig/20240404-162019-arnaudb.json
  • 16:16 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59531 and previous config saved to /var/cache/conftool/dbconfig/20240404-161601-arnaudb.json
  • 16:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59530 and previous config saved to /var/cache/conftool/dbconfig/20240404-161547-arnaudb.json
  • 16:10 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 16:09 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P59529 and previous config saved to /var/cache/conftool/dbconfig/20240404-160508-arnaudb.json
  • 16:04 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 16:04 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3068.esams.wmnet with OS bullseye
  • 16:03 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 16:03 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 16:02 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 16:01 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 16:01 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59528 and previous config saved to /var/cache/conftool/dbconfig/20240404-160055-arnaudb.json
  • 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59527 and previous config saved to /var/cache/conftool/dbconfig/20240404-160041-arnaudb.json
  • 15:55 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3068.esams.wmnet
  • 15:54 fabfur: depooling cp3068 for reimage (T360430)
  • 15:54 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1207 (T356166)', diff saved to https://phabricator.wikimedia.org/P59526 and previous config saved to /var/cache/conftool/dbconfig/20240404-155420-marostegui.json
  • 15:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 15:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 15:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T356166)', diff saved to https://phabricator.wikimedia.org/P59525 and previous config saved to /var/cache/conftool/dbconfig/20240404-155357-marostegui.json
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59524 and previous config saved to /var/cache/conftool/dbconfig/20240404-155000-arnaudb.json
  • 15:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59523 and previous config saved to /var/cache/conftool/dbconfig/20240404-154730-arnaudb.json
  • 15:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 15:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 15:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T360332)', diff saved to https://phabricator.wikimedia.org/P59522 and previous config saved to /var/cache/conftool/dbconfig/20240404-154707-arnaudb.json
  • 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59521 and previous config saved to /var/cache/conftool/dbconfig/20240404-154549-arnaudb.json
  • 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59520 and previous config saved to /var/cache/conftool/dbconfig/20240404-154535-arnaudb.json
  • 15:45 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 108
  • 15:45 ayounsi@cumin1002: START - Cookbook sre.network.debug for Netbox circuit ID 108
  • 15:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P59519 and previous config saved to /var/cache/conftool/dbconfig/20240404-153850-marostegui.json
  • 15:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59518 and previous config saved to /var/cache/conftool/dbconfig/20240404-153626-marostegui.json
  • 15:36 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2214.codfw.wmnet with reason: depooled, see T361851
  • 15:36 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 15:36 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db2214.codfw.wmnet with reason: depooled, see T361851
  • 15:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 15:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T355609)', diff saved to https://phabricator.wikimedia.org/P59517 and previous config saved to /var/cache/conftool/dbconfig/20240404-153603-marostegui.json
  • 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P59516 and previous config saved to /var/cache/conftool/dbconfig/20240404-153200-arnaudb.json
  • 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59515 and previous config saved to /var/cache/conftool/dbconfig/20240404-153043-arnaudb.json
  • 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59514 and previous config saved to /var/cache/conftool/dbconfig/20240404-153030-arnaudb.json
  • 15:30 sukhe@cumin2002: dbctl commit (dc=all): 'depool db2214', diff saved to https://phabricator.wikimedia.org/P59513 and previous config saved to /var/cache/conftool/dbconfig/20240404-153023-sukhe.json
  • 15:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P59512 and previous config saved to /var/cache/conftool/dbconfig/20240404-152341-marostegui.json
  • 15:23 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 15:22 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 15:22 logmsgbot: dreamyjazz Deployed security patch for T361296
  • 15:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59511 and previous config saved to /var/cache/conftool/dbconfig/20240404-152056-marostegui.json
  • 15:18 jgiannelos@deploy1002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 15:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P59510 and previous config saved to /var/cache/conftool/dbconfig/20240404-151653-arnaudb.json
  • 15:16 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 15:16 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 15:16 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 15:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 100%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59509 and previous config saved to /var/cache/conftool/dbconfig/20240404-151547-arnaudb.json
  • 15:15 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 15:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59508 and previous config saved to /var/cache/conftool/dbconfig/20240404-151537-arnaudb.json
  • 15:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59507 and previous config saved to /var/cache/conftool/dbconfig/20240404-151524-arnaudb.json
  • 15:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 100%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59506 and previous config saved to /var/cache/conftool/dbconfig/20240404-151123-arnaudb.json
  • 15:10 herron: beginning rolling hardware upgrades on titan200[12] T361229
  • 15:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T356166)', diff saved to https://phabricator.wikimedia.org/P59505 and previous config saved to /var/cache/conftool/dbconfig/20240404-150833-marostegui.json
  • 15:08 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 15:08 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 15:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59504 and previous config saved to /var/cache/conftool/dbconfig/20240404-150549-marostegui.json
  • 15:03 logmsgbot: dreamyjazz Deployed security patch for T361295
  • 15:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T360332)', diff saved to https://phabricator.wikimedia.org/P59503 and previous config saved to /var/cache/conftool/dbconfig/20240404-150145-arnaudb.json
  • 15:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 70%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59502 and previous config saved to /var/cache/conftool/dbconfig/20240404-150041-arnaudb.json
  • 15:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59501 and previous config saved to /var/cache/conftool/dbconfig/20240404-150032-arnaudb.json
  • 15:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59500 and previous config saved to /var/cache/conftool/dbconfig/20240404-150017-arnaudb.json
  • 14:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1174 (T360332)', diff saved to https://phabricator.wikimedia.org/P59499 and previous config saved to /var/cache/conftool/dbconfig/20240404-145714-arnaudb.json
  • 14:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 14:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 14:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 70%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59498 and previous config saved to /var/cache/conftool/dbconfig/20240404-145617-arnaudb.json
  • 14:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T360332)', diff saved to https://phabricator.wikimedia.org/P59497 and previous config saved to /var/cache/conftool/dbconfig/20240404-145613-arnaudb.json
  • 14:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T355609)', diff saved to https://phabricator.wikimedia.org/P59496 and previous config saved to /var/cache/conftool/dbconfig/20240404-145041-marostegui.json
  • 14:48 dreamyjazz@deploy1002: Finished scap: Backport for Remove sk translation of centralauth-rightslog-name (T361695) (duration: 14m 49s)
  • 14:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 50%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59495 and previous config saved to /var/cache/conftool/dbconfig/20240404-144536-arnaudb.json
  • 14:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59494 and previous config saved to /var/cache/conftool/dbconfig/20240404-144526-arnaudb.json
  • 14:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59493 and previous config saved to /var/cache/conftool/dbconfig/20240404-144511-arnaudb.json
  • 14:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 50%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59492 and previous config saved to /var/cache/conftool/dbconfig/20240404-144111-arnaudb.json
  • 14:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P59491 and previous config saved to /var/cache/conftool/dbconfig/20240404-144105-arnaudb.json
  • 14:36 dreamyjazz@deploy1002: dreamyjazz: Continuing with sync
  • 14:36 dreamyjazz@deploy1002: dreamyjazz: Backport for Remove sk translation of centralauth-rightslog-name (T361695) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:33 dreamyjazz@deploy1002: Started scap: Backport for Remove sk translation of centralauth-rightslog-name (T361695)
  • 14:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 30%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59490 and previous config saved to /var/cache/conftool/dbconfig/20240404-143030-arnaudb.json
  • 14:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59489 and previous config saved to /var/cache/conftool/dbconfig/20240404-143020-arnaudb.json
  • 14:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59488 and previous config saved to /var/cache/conftool/dbconfig/20240404-143006-arnaudb.json
  • 14:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 30%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59487 and previous config saved to /var/cache/conftool/dbconfig/20240404-142606-arnaudb.json
  • 14:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P59486 and previous config saved to /var/cache/conftool/dbconfig/20240404-142558-arnaudb.json
  • 14:22 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Enable wgVisualEditorAllowExternalLinkPaste at collabwiki (duration: 18m 43s)
  • 14:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 20%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59485 and previous config saved to /var/cache/conftool/dbconfig/20240404-141517-arnaudb.json
  • 14:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 20%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59484 and previous config saved to /var/cache/conftool/dbconfig/20240404-141100-arnaudb.json
  • 14:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T360332)', diff saved to https://phabricator.wikimedia.org/P59483 and previous config saved to /var/cache/conftool/dbconfig/20240404-141051-arnaudb.json
  • 14:09 logmsgbot: lucaswerkmeister-wmde@deploy1002 esanders and lucaswerkmeister-wmde: Continuing with sync
  • 14:05 logmsgbot: lucaswerkmeister-wmde@deploy1002 esanders and lucaswerkmeister-wmde: Backport for Enable wgVisualEditorAllowExternalLinkPaste at collabwiki synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:03 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Enable wgVisualEditorAllowExternalLinkPaste at collabwiki
  • 14:01 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for End EditCheck add-a-reference A/B test (T361727) (duration: 20m 05s)
  • 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T355609)', diff saved to https://phabricator.wikimedia.org/P59482 and previous config saved to /var/cache/conftool/dbconfig/20240404-140050-marostegui.json
  • 14:00 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 14:00 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T355609)', diff saved to https://phabricator.wikimedia.org/P59481 and previous config saved to /var/cache/conftool/dbconfig/20240404-140027-marostegui.json
  • 14:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 10%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59480 and previous config saved to /var/cache/conftool/dbconfig/20240404-140011-arnaudb.json
  • 13:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 10%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59479 and previous config saved to /var/cache/conftool/dbconfig/20240404-135547-arnaudb.json
  • 13:48 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Continuing with sync
  • 13:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1170 (T360332)', diff saved to https://phabricator.wikimedia.org/P59478 and previous config saved to /var/cache/conftool/dbconfig/20240404-134607-arnaudb.json
  • 13:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:45 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59477 and previous config saved to /var/cache/conftool/dbconfig/20240404-134544-arnaudb.json
  • 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59476 and previous config saved to /var/cache/conftool/dbconfig/20240404-134519-marostegui.json
  • 13:43 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Backport for End EditCheck add-a-reference A/B test (T361727) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:41 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for End EditCheck add-a-reference A/B test (T361727)
  • 13:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2107.codfw.wmnet onto db2207.codfw.wmnet
  • 13:39 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for DiscussionTools: Remove no-op config (duration: 15m 10s)
  • 13:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P59475 and previous config saved to /var/cache/conftool/dbconfig/20240404-133037-arnaudb.json
  • 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59474 and previous config saved to /var/cache/conftool/dbconfig/20240404-133012-marostegui.json
  • 13:26 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Continuing with sync
  • 13:26 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Backport for DiscussionTools: Remove no-op config synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:24 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for DiscussionTools: Remove no-op config
  • 13:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P59473 and previous config saved to /var/cache/conftool/dbconfig/20240404-131529-arnaudb.json
  • 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T355609)', diff saved to https://phabricator.wikimedia.org/P59472 and previous config saved to /var/cache/conftool/dbconfig/20240404-131504-marostegui.json
  • 13:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2113.codfw.wmnet onto db2213.codfw.wmnet
  • 13:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59471 and previous config saved to /var/cache/conftool/dbconfig/20240404-130022-arnaudb.json
  • 12:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T356166)', diff saved to https://phabricator.wikimedia.org/P59470 and previous config saved to /var/cache/conftool/dbconfig/20240404-124257-marostegui.json
  • 12:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 12:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 12:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T356166)', diff saved to https://phabricator.wikimedia.org/P59469 and previous config saved to /var/cache/conftool/dbconfig/20240404-124235-marostegui.json
  • 12:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59468 and previous config saved to /var/cache/conftool/dbconfig/20240404-123645-arnaudb.json
  • 12:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 12:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 12:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P59467 and previous config saved to /var/cache/conftool/dbconfig/20240404-122727-marostegui.json
  • 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T355609)', diff saved to https://phabricator.wikimedia.org/P59466 and previous config saved to /var/cache/conftool/dbconfig/20240404-122557-marostegui.json
  • 12:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 12:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T355609)', diff saved to https://phabricator.wikimedia.org/P59465 and previous config saved to /var/cache/conftool/dbconfig/20240404-122535-marostegui.json
  • 12:18 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2113.codfw.wmnet onto db2213.codfw.wmnet
  • 12:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2113 in db2213 for T355422', diff saved to https://phabricator.wikimedia.org/P59463 and previous config saved to /var/cache/conftool/dbconfig/20240404-121722-arnaudb.json
  • 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
  • 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
  • 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2113.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
  • 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2113.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
  • 12:12 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2107.codfw.wmnet onto db2207.codfw.wmnet
  • 12:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P59462 and previous config saved to /var/cache/conftool/dbconfig/20240404-121218-marostegui.json
  • 12:11 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 12:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59461 and previous config saved to /var/cache/conftool/dbconfig/20240404-121027-marostegui.json
  • 12:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2107 in db2207 for T355422', diff saved to https://phabricator.wikimedia.org/P59460 and previous config saved to /var/cache/conftool/dbconfig/20240404-121008-arnaudb.json
  • 12:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
  • 12:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
  • 12:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2107.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
  • 12:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2107.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
  • 11:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T356166)', diff saved to https://phabricator.wikimedia.org/P59459 and previous config saved to /var/cache/conftool/dbconfig/20240404-115709-marostegui.json
  • 11:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59458 and previous config saved to /var/cache/conftool/dbconfig/20240404-115520-marostegui.json
  • 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 108
  • 11:49 ayounsi@cumin1002: START - Cookbook sre.network.debug for Netbox circuit ID 108
  • 11:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T355609)', diff saved to https://phabricator.wikimedia.org/P59457 and previous config saved to /var/cache/conftool/dbconfig/20240404-114012-marostegui.json
  • 10:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T355609)', diff saved to https://phabricator.wikimedia.org/P59456 and previous config saved to /var/cache/conftool/dbconfig/20240404-105158-marostegui.json
  • 10:51 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 10:51 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 10:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T355609)', diff saved to https://phabricator.wikimedia.org/P59455 and previous config saved to /var/cache/conftool/dbconfig/20240404-105135-marostegui.json
  • 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59453 and previous config saved to /var/cache/conftool/dbconfig/20240404-103628-marostegui.json
  • 10:34 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:34 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 10:33 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:33 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59451 and previous config saved to /var/cache/conftool/dbconfig/20240404-102120-marostegui.json
  • 10:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::objectstash
  • 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T355609)', diff saved to https://phabricator.wikimedia.org/P59450 and previous config saved to /var/cache/conftool/dbconfig/20240404-100612-marostegui.json
  • 10:05 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:05 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 09:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::objectstash
  • 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::misc
  • 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2104.codfw.wmnet
  • 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2104.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
  • 09:48 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2104.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
  • 09:46 marostegui@cumin1002: START - Cookbook sre.dns.netbox
  • 09:46 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1196 (T356166)', diff saved to https://phabricator.wikimedia.org/P59449 and previous config saved to /var/cache/conftool/dbconfig/20240404-094608-marostegui.json
  • 09:46 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T356166)', diff saved to https://phabricator.wikimedia.org/P59448 and previous config saved to /var/cache/conftool/dbconfig/20240404-094536-marostegui.json
  • 09:41 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2104.codfw.wmnet
  • 09:31 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::misc
  • 09:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P59446 and previous config saved to /var/cache/conftool/dbconfig/20240404-093028-marostegui.json
  • 09:18 arnaudb@cumin1002: dbctl commit (dc=all): 'bump db2113 weight', diff saved to https://phabricator.wikimedia.org/P59445 and previous config saved to /var/cache/conftool/dbconfig/20240404-091858-arnaudb.json
  • 09:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T355609)', diff saved to https://phabricator.wikimedia.org/P59444 and previous config saved to /var/cache/conftool/dbconfig/20240404-091732-marostegui.json
  • 09:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 09:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 09:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T355609)', diff saved to https://phabricator.wikimedia.org/P59443 and previous config saved to /var/cache/conftool/dbconfig/20240404-091709-marostegui.json
  • 09:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P59442 and previous config saved to /var/cache/conftool/dbconfig/20240404-091521-marostegui.json
  • 09:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2123 to s5 primary T361789', diff saved to https://phabricator.wikimedia.org/P59441 and previous config saved to /var/cache/conftool/dbconfig/20240404-091512-arnaudb.json
  • 09:14 arnaudb: Starting s5 codfw failover from db2113 to db2123 - T361789
  • 09:11 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::backup_source
  • 09:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59440 and previous config saved to /var/cache/conftool/dbconfig/20240404-090202-marostegui.json
  • 09:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T356166)', diff saved to https://phabricator.wikimedia.org/P59439 and previous config saved to /var/cache/conftool/dbconfig/20240404-090007-marostegui.json
  • 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1186 (T356166)', diff saved to https://phabricator.wikimedia.org/P59438 and previous config saved to /var/cache/conftool/dbconfig/20240404-085856-marostegui.json
  • 08:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 08:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T356166)', diff saved to https://phabricator.wikimedia.org/P59437 and previous config saved to /var/cache/conftool/dbconfig/20240404-085834-marostegui.json
  • 08:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2123 with weight 0 T361789', diff saved to https://phabricator.wikimedia.org/P59436 and previous config saved to /var/cache/conftool/dbconfig/20240404-085606-arnaudb.json
  • 08:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s5 T361789
  • 08:55 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::backup_source
  • 08:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s5 T361789
  • 08:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59435 and previous config saved to /var/cache/conftool/dbconfig/20240404-084655-marostegui.json
  • 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P59434 and previous config saved to /var/cache/conftool/dbconfig/20240404-084327-marostegui.json
  • 08:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T355609)', diff saved to https://phabricator.wikimedia.org/P59433 and previous config saved to /var/cache/conftool/dbconfig/20240404-083147-marostegui.json
  • 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P59432 and previous config saved to /var/cache/conftool/dbconfig/20240404-082819-marostegui.json
  • 08:25 arnaudb@cumin1002: dbctl commit (dc=all): 'bump db2107 weight', diff saved to https://phabricator.wikimedia.org/P59431 and previous config saved to /var/cache/conftool/dbconfig/20240404-082547-root.json
  • 08:24 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 08:23 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 08:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2204 to s2 primary T361682', diff saved to https://phabricator.wikimedia.org/P59430 and previous config saved to /var/cache/conftool/dbconfig/20240404-082200-arnaudb.json
  • 08:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2204 with weight 0 T361682', diff saved to https://phabricator.wikimedia.org/P59427 and previous config saved to /var/cache/conftool/dbconfig/20240404-080408-arnaudb.json
  • 08:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 29 hosts with reason: Primary switchover s2 T361682
  • 08:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 29 hosts with reason: Primary switchover s2 T361682
  • 07:43 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T355609)', diff saved to https://phabricator.wikimedia.org/P59426 and previous config saved to /var/cache/conftool/dbconfig/20240404-074313-marostegui.json
  • 07:43 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 07:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 07:36 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2104 from dbctl T361779', diff saved to https://phabricator.wikimedia.org/P59425 and previous config saved to /var/cache/conftool/dbconfig/20240404-073600-root.json
  • 07:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59424 and previous config saved to /var/cache/conftool/dbconfig/20240404-072928-root.json
  • 07:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59423 and previous config saved to /var/cache/conftool/dbconfig/20240404-071423-root.json
  • 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59422 and previous config saved to /var/cache/conftool/dbconfig/20240404-065917-root.json
  • 06:59 moritzm: installing util-linux security updates
  • 06:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 06:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 06:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T355609)', diff saved to https://phabricator.wikimedia.org/P59421 and previous config saved to /var/cache/conftool/dbconfig/20240404-065758-marostegui.json
  • 06:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59420 and previous config saved to /var/cache/conftool/dbconfig/20240404-064411-root.json
  • 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59419 and previous config saved to /var/cache/conftool/dbconfig/20240404-064250-marostegui.json
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59418 and previous config saved to /var/cache/conftool/dbconfig/20240404-062905-root.json
  • 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59417 and previous config saved to /var/cache/conftool/dbconfig/20240404-062743-marostegui.json
  • 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59416 and previous config saved to /var/cache/conftool/dbconfig/20240404-061400-root.json
  • 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T355609)', diff saved to https://phabricator.wikimedia.org/P59415 and previous config saved to /var/cache/conftool/dbconfig/20240404-061234-marostegui.json
  • 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1169 (T356166)', diff saved to https://phabricator.wikimedia.org/P59414 and previous config saved to /var/cache/conftool/dbconfig/20240404-060524-marostegui.json
  • 06:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 06:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T356166)', diff saved to https://phabricator.wikimedia.org/P59413 and previous config saved to /var/cache/conftool/dbconfig/20240404-060501-marostegui.json
  • 06:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2126.codfw.wmnet with OS bookworm
  • 05:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59412 and previous config saved to /var/cache/conftool/dbconfig/20240404-055854-root.json
  • 05:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P59411 and previous config saved to /var/cache/conftool/dbconfig/20240404-054953-marostegui.json
  • 05:39 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2126.codfw.wmnet with reason: host reimage
  • 05:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2126.codfw.wmnet with reason: host reimage
  • 05:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P59410 and previous config saved to /var/cache/conftool/dbconfig/20240404-053446-marostegui.json
  • 05:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T355609)', diff saved to https://phabricator.wikimedia.org/P59409 and previous config saved to /var/cache/conftool/dbconfig/20240404-052338-marostegui.json
  • 05:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 05:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 05:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 05:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 05:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T356166)', diff saved to https://phabricator.wikimedia.org/P59408 and previous config saved to /var/cache/conftool/dbconfig/20240404-051938-marostegui.json
  • 05:19 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2126.codfw.wmnet with OS bookworm
  • 05:18 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2126 T361543', diff saved to https://phabricator.wikimedia.org/P59407 and previous config saved to /var/cache/conftool/dbconfig/20240404-051758-root.json
  • 05:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T356166)', diff saved to https://phabricator.wikimedia.org/P59406 and previous config saved to /var/cache/conftool/dbconfig/20240404-051728-marostegui.json
  • 05:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 05:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 02:48 TimStarling: ran maintain-views on clouddb1013-1019 (T355034)
  • 02:29 eileen: civicrm upgraded from a0fb57d3 to 8c7cc208
  • 02:28 eileen: config revision changed from 3ed18c47 to abccfdc0
  • 02:11 eileen: config revision changed from 3ed18c47 to abccfdc0
  • 01:07 TimStarling: on clouddb1020 running maintain-views --all-databases --replace-all --auto-depool (T355034)
  • 00:56 TimStarling: on clouddb1021 ran maintain-views for all databases

2024-04-03

  • 23:33 TimStarling: on clouddb1021 ran maintain-views for enwiki
  • 22:35 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:34 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:34 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:34 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T360332)', diff saved to https://phabricator.wikimedia.org/P59405 and previous config saved to /var/cache/conftool/dbconfig/20240403-222610-arnaudb.json
  • 22:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P59404 and previous config saved to /var/cache/conftool/dbconfig/20240403-221103-arnaudb.json
  • 22:09 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:09 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:09 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:06 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:06 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:06 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:06 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:06 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:05 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:00 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:00 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:00 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:59 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:58 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:58 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:58 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P59403 and previous config saved to /var/cache/conftool/dbconfig/20240403-215555-arnaudb.json
  • 21:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T360332)', diff saved to https://phabricator.wikimedia.org/P59402 and previous config saved to /var/cache/conftool/dbconfig/20240403-214048-arnaudb.json
  • 21:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2217 (T360332)', diff saved to https://phabricator.wikimedia.org/P59401 and previous config saved to /var/cache/conftool/dbconfig/20240403-213825-arnaudb.json
  • 21:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2217.codfw.wmnet with reason: Maintenance
  • 21:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2217.codfw.wmnet with reason: Maintenance
  • 21:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59400 and previous config saved to /var/cache/conftool/dbconfig/20240403-213802-arnaudb.json
  • 21:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P59399 and previous config saved to /var/cache/conftool/dbconfig/20240403-212255-arnaudb.json
  • 21:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P59398 and previous config saved to /var/cache/conftool/dbconfig/20240403-210747-arnaudb.json
  • 21:02 jforrester@deploy1002: Finished scap: Backport for component: Add SandboxLink to Portuguese Wikiquote (T361447) (duration: 14m 18s)
  • 20:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59397 and previous config saved to /var/cache/conftool/dbconfig/20240403-205240-arnaudb.json
  • 20:51 jforrester@deploy1002: ederporto and jforrester: Continuing with sync
  • 20:50 jforrester@deploy1002: ederporto and jforrester: Backport for component: Add SandboxLink to Portuguese Wikiquote (T361447) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:50 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:50 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59396 and previous config saved to /var/cache/conftool/dbconfig/20240403-205014-arnaudb.json
  • 20:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2214.codfw.wmnet with reason: Maintenance
  • 20:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2214.codfw.wmnet with reason: Maintenance
  • 20:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59395 and previous config saved to /var/cache/conftool/dbconfig/20240403-204949-arnaudb.json
  • 20:48 jforrester@deploy1002: Started scap: Backport for component: Add SandboxLink to Portuguese Wikiquote (T361447)
  • 20:45 jforrester@deploy1002: Finished scap: Backport for Update the WikiLambda instrumentation to use core interaction events (T350497) (duration: 19m 03s)
  • 20:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P59394 and previous config saved to /var/cache/conftool/dbconfig/20240403-203440-arnaudb.json
  • 20:34 jforrester@deploy1002: sfaci and jforrester: Continuing with sync
  • 20:28 jforrester@deploy1002: sfaci and jforrester: Backport for Update the WikiLambda instrumentation to use core interaction events (T350497) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:26 jforrester@deploy1002: Started scap: Backport for Update the WikiLambda instrumentation to use core interaction events (T350497)
  • 20:23 jforrester@deploy1002: Finished scap: Backport for Centralize API calls in api.js mixin and fix error handling (T361598 T315432) (duration: 14m 58s)
  • 20:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P59393 and previous config saved to /var/cache/conftool/dbconfig/20240403-201933-arnaudb.json
  • 20:12 jforrester@deploy1002: jforrester: Continuing with sync
  • 20:11 jforrester@deploy1002: jforrester: Backport for Centralize API calls in api.js mixin and fix error handling (T361598 T315432) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:10 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:10 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:08 jforrester@deploy1002: Started scap: Backport for Centralize API calls in api.js mixin and fix error handling (T361598 T315432)
  • 20:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59392 and previous config saved to /var/cache/conftool/dbconfig/20240403-200425-arnaudb.json
  • 20:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59391 and previous config saved to /var/cache/conftool/dbconfig/20240403-200201-arnaudb.json
  • 20:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2193.codfw.wmnet with reason: Maintenance
  • 20:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2193.codfw.wmnet with reason: Maintenance
  • 20:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59390 and previous config saved to /var/cache/conftool/dbconfig/20240403-200137-arnaudb.json
  • 20:00 mutante: stewards1001 - rebooting to switch from iptables to nftables
  • 19:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P59388 and previous config saved to /var/cache/conftool/dbconfig/20240403-194630-arnaudb.json
  • 19:38 mutante: stewards2001 - reboot to switch from iptables to nftables
  • 19:31 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:31 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P59387 and previous config saved to /var/cache/conftool/dbconfig/20240403-193122-arnaudb.json
  • 19:29 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:29 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:16 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:16 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59386 and previous config saved to /var/cache/conftool/dbconfig/20240403-191615-arnaudb.json
  • 19:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59385 and previous config saved to /var/cache/conftool/dbconfig/20240403-191351-arnaudb.json
  • 19:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 19:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 19:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59384 and previous config saved to /var/cache/conftool/dbconfig/20240403-191328-arnaudb.json
  • 19:06 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:06 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:04 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
  • 19:03 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
  • 19:03 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
  • 19:02 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
  • 19:02 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
  • 19:01 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
  • 18:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P59383 and previous config saved to /var/cache/conftool/dbconfig/20240403-185821-arnaudb.json
  • 18:57 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 18:57 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 18:51 eileen: config revision changed from 821f145b to 3ed18c47
  • 18:49 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 18:49 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 18:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P59382 and previous config saved to /var/cache/conftool/dbconfig/20240403-184313-arnaudb.json
  • 18:37 eevans@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts restbase[1019-1027].eqiad.wmnet
  • 18:37 eevans@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:37 eevans@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: restbase[1019-1027].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1002"
  • 18:35 eevans@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: restbase[1019-1027].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1002"
  • 18:34 eevans@cumin1002: START - Cookbook sre.dns.netbox
  • 18:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59381 and previous config saved to /var/cache/conftool/dbconfig/20240403-182806-arnaudb.json
  • 18:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59380 and previous config saved to /var/cache/conftool/dbconfig/20240403-182543-arnaudb.json
  • 18:25 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 18:25 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 18:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59379 and previous config saved to /var/cache/conftool/dbconfig/20240403-182520-arnaudb.json
  • 18:13 logmsgbot: dreamyjazz Deployed security patch for T361479
  • 18:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P59378 and previous config saved to /var/cache/conftool/dbconfig/20240403-181013-arnaudb.json
  • 18:09 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
  • 18:08 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
  • 18:07 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
  • 18:07 eevans@cumin1002: START - Cookbook sre.hosts.decommission for hosts restbase[1019-1027].eqiad.wmnet
  • 18:06 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
  • 18:05 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 18:04 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
  • 18:03 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59377 and previous config saved to /var/cache/conftool/dbconfig/20240403-180323-root.json
  • 17:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P59375 and previous config saved to /var/cache/conftool/dbconfig/20240403-175505-arnaudb.json
  • 17:48 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59374 and previous config saved to /var/cache/conftool/dbconfig/20240403-174817-root.json
  • 17:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59373 and previous config saved to /var/cache/conftool/dbconfig/20240403-173958-arnaudb.json
  • 17:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59372 and previous config saved to /var/cache/conftool/dbconfig/20240403-173835-arnaudb.json
  • 17:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 17:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 17:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 17:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 17:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59371 and previous config saved to /var/cache/conftool/dbconfig/20240403-173756-arnaudb.json
  • 17:33 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59370 and previous config saved to /var/cache/conftool/dbconfig/20240403-173312-root.json
  • 17:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P59369 and previous config saved to /var/cache/conftool/dbconfig/20240403-172249-arnaudb.json
  • 17:18 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59368 and previous config saved to /var/cache/conftool/dbconfig/20240403-171806-root.json
  • 17:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P59367 and previous config saved to /var/cache/conftool/dbconfig/20240403-170741-arnaudb.json
  • 17:04 herron: performing rolling memory upgrades on prometheus100[56] T360687
  • 17:03 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59366 and previous config saved to /var/cache/conftool/dbconfig/20240403-170300-root.json
  • 16:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:59 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:54 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:54 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59365 and previous config saved to /var/cache/conftool/dbconfig/20240403-165234-arnaudb.json
  • 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59364 and previous config saved to /var/cache/conftool/dbconfig/20240403-165011-arnaudb.json
  • 16:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance
  • 16:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance
  • 16:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T360332)', diff saved to https://phabricator.wikimedia.org/P59363 and previous config saved to /var/cache/conftool/dbconfig/20240403-164948-arnaudb.json
  • 16:47 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59362 and previous config saved to /var/cache/conftool/dbconfig/20240403-164754-root.json
  • 16:38 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot-master (exit_code=0) rolling restart_daemons on A:maps-master
  • 16:36 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot-master rolling restart_daemons on A:maps-master
  • 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P59361 and previous config saved to /var/cache/conftool/dbconfig/20240403-163440-arnaudb.json
  • 16:32 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59360 and previous config saved to /var/cache/conftool/dbconfig/20240403-163249-root.json
  • 16:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59359 and previous config saved to /var/cache/conftool/dbconfig/20240403-163004-marostegui.json
  • 16:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 16:29 jayme@deploy1002: Finished scap: (no justification provided) (duration: 03m 34s)
  • 16:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 16:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 16:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 16:26 effie: pooling back mw-web-ro in eqiad
  • 16:26 jayme@deploy1002: Started scap: (no justification provided)
  • 16:26 jiji@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad
  • 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P59358 and previous config saved to /var/cache/conftool/dbconfig/20240403-161933-arnaudb.json
  • 16:14 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 16:12 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 16:08 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
  • 16:07 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: apply
  • 16:05 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
  • 16:05 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: apply
  • 16:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T360332)', diff saved to https://phabricator.wikimedia.org/P59357 and previous config saved to /var/cache/conftool/dbconfig/20240403-160425-arnaudb.json
  • 16:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2124 (T360332)', diff saved to https://phabricator.wikimedia.org/P59356 and previous config saved to /var/cache/conftool/dbconfig/20240403-160159-arnaudb.json
  • 16:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 16:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 16:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59355 and previous config saved to /var/cache/conftool/dbconfig/20240403-160136-arnaudb.json
  • 15:53 jiji@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=mw-web-ro,name=eqiad
  • 15:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P59354 and previous config saved to /var/cache/conftool/dbconfig/20240403-154628-arnaudb.json
  • 15:33 jiji@cumin1002: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None
  • 15:33 jiji@cumin1002: START - Cookbook sre.discovery.datacenter status all services in all: None - None
  • 15:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P59353 and previous config saved to /var/cache/conftool/dbconfig/20240403-153121-arnaudb.json
  • 15:22 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:22 Dreamy_Jazz: Starting MediaModeration scanning script again - It crashed due to the outage
  • 15:22 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59352 and previous config saved to /var/cache/conftool/dbconfig/20240403-151614-arnaudb.json
  • 15:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59351 and previous config saved to /var/cache/conftool/dbconfig/20240403-151349-arnaudb.json
  • 15:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 15:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 15:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 15:12 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 15:12 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 15:12 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 15:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59350 and previous config saved to /var/cache/conftool/dbconfig/20240403-151233-arnaudb.json
  • 15:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2098.codfw.wmnet with reason: restart of mysqld
  • 15:03 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2098.codfw.wmnet with reason: restart of mysqld
  • 15:02 dreamyjazz@deploy1002: Finished scap: (no justification provided) (duration: 18m 54s)
  • 15:01 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: sync
  • 15:01 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1040.eqiad.wmnet with OS bookworm
  • 15:01 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifeeds: sync
  • 14:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59349 and previous config saved to /var/cache/conftool/dbconfig/20240403-145725-arnaudb.json
  • 14:44 dreamyjazz@deploy1002: Started scap: (no justification provided)
  • 14:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59347 and previous config saved to /var/cache/conftool/dbconfig/20240403-144217-arnaudb.json
  • 14:34 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: host reimage
  • 14:31 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: host reimage
  • 14:27 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 14:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59346 and previous config saved to /var/cache/conftool/dbconfig/20240403-142709-arnaudb.json
  • 14:27 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 14:26 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 14:26 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 14:24 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1040
  • 14:24 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1040
  • 14:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 14:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 14:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T356166)', diff saved to https://phabricator.wikimedia.org/P59345 and previous config saved to /var/cache/conftool/dbconfig/20240403-142142-marostegui.json
  • 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 14:16 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1040.eqiad.wmnet with OS bookworm
  • 14:11 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-eqiad
  • 14:11 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 14:09 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 14:09 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 14:08 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 2527
  • 14:07 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 2527
  • 14:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59344 and previous config saved to /var/cache/conftool/dbconfig/20240403-140634-marostegui.json
  • 14:06 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-codfw
  • 13:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59343 and previous config saved to /var/cache/conftool/dbconfig/20240403-135126-marostegui.json
  • 13:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59342 and previous config saved to /var/cache/conftool/dbconfig/20240403-134136-arnaudb.json
  • 13:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 13:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 13:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 13:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 13:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T360332)', diff saved to https://phabricator.wikimedia.org/P59341 and previous config saved to /var/cache/conftool/dbconfig/20240403-134044-arnaudb.json
  • 13:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T356166)', diff saved to https://phabricator.wikimedia.org/P59340 and previous config saved to /var/cache/conftool/dbconfig/20240403-133619-marostegui.json
  • 13:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T356166)', diff saved to https://phabricator.wikimedia.org/P59339 and previous config saved to /var/cache/conftool/dbconfig/20240403-133200-marostegui.json
  • 13:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 13:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 13:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 13:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 13:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59336 and previous config saved to /var/cache/conftool/dbconfig/20240403-131606-marostegui.json
  • 13:15 moritzm: installing tiff security updates
  • 13:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P59335 and previous config saved to /var/cache/conftool/dbconfig/20240403-131029-arnaudb.json
  • 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59334 and previous config saved to /var/cache/conftool/dbconfig/20240403-130058-marostegui.json
  • 12:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T360332)', diff saved to https://phabricator.wikimedia.org/P59333 and previous config saved to /var/cache/conftool/dbconfig/20240403-125521-arnaudb.json
  • 12:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T356166)', diff saved to https://phabricator.wikimedia.org/P59332 and previous config saved to /var/cache/conftool/dbconfig/20240403-124550-marostegui.json
  • 12:42 hashar: Upgrading CI Jenkins # T360759
  • 12:34 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1039.eqiad.wmnet with OS bookworm
  • 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T356166)', diff saved to https://phabricator.wikimedia.org/P59330 and previous config saved to /var/cache/conftool/dbconfig/20240403-123329-marostegui.json
  • 12:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 12:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T356166)', diff saved to https://phabricator.wikimedia.org/P59329 and previous config saved to /var/cache/conftool/dbconfig/20240403-123306-marostegui.json
  • 12:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1224 (T360332)', diff saved to https://phabricator.wikimedia.org/P59328 and previous config saved to /var/cache/conftool/dbconfig/20240403-123156-arnaudb.json
  • 12:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1224.eqiad.wmnet with reason: Maintenance
  • 12:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1224.eqiad.wmnet with reason: Maintenance
  • 12:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T360332)', diff saved to https://phabricator.wikimedia.org/P59327 and previous config saved to /var/cache/conftool/dbconfig/20240403-123133-arnaudb.json
  • 12:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59326 and previous config saved to /var/cache/conftool/dbconfig/20240403-121759-marostegui.json
  • 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59325 and previous config saved to /var/cache/conftool/dbconfig/20240403-121626-arnaudb.json
  • 12:11 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1039.eqiad.wmnet with reason: host reimage
  • 12:08 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1039.eqiad.wmnet with reason: host reimage
  • 12:07 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1039
  • 12:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59324 and previous config saved to /var/cache/conftool/dbconfig/20240403-120251-marostegui.json
  • 12:02 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 12:02 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 12:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59323 and previous config saved to /var/cache/conftool/dbconfig/20240403-120118-arnaudb.json
  • 11:58 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1039
  • 11:52 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 11:52 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 11:50 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1039.eqiad.wmnet with OS bookworm
  • 11:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T356166)', diff saved to https://phabricator.wikimedia.org/P59322 and previous config saved to /var/cache/conftool/dbconfig/20240403-114743-marostegui.json
  • 11:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T360332)', diff saved to https://phabricator.wikimedia.org/P59321 and previous config saved to /var/cache/conftool/dbconfig/20240403-114611-arnaudb.json
  • 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T356166)', diff saved to https://phabricator.wikimedia.org/P59320 and previous config saved to /var/cache/conftool/dbconfig/20240403-114525-marostegui.json
  • 11:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 11:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T356166)', diff saved to https://phabricator.wikimedia.org/P59319 and previous config saved to /var/cache/conftool/dbconfig/20240403-114502-marostegui.json
  • 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1201 (T360332)', diff saved to https://phabricator.wikimedia.org/P59318 and previous config saved to /var/cache/conftool/dbconfig/20240403-114350-arnaudb.json
  • 11:43 moritzm: installing imagemagick security updates
  • 11:43 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 11:43 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T360332)', diff saved to https://phabricator.wikimedia.org/P59317 and previous config saved to /var/cache/conftool/dbconfig/20240403-114327-arnaudb.json
  • 11:33 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1038.eqiad.wmnet with OS bookworm
  • 11:30 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
  • 11:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59315 and previous config saved to /var/cache/conftool/dbconfig/20240403-112955-marostegui.json
  • 11:29 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
  • 11:28 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
  • 11:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59314 and previous config saved to /var/cache/conftool/dbconfig/20240403-112819-arnaudb.json
  • 11:28 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
  • 11:27 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 11:27 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
  • 11:25 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 11:24 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 11:23 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 11:23 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 11:16 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1038
  • 11:16 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1038
  • 11:16 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
  • 11:15 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
  • 11:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59313 and previous config saved to /var/cache/conftool/dbconfig/20240403-111447-marostegui.json
  • 11:14 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
  • 11:13 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
  • 11:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59312 and previous config saved to /var/cache/conftool/dbconfig/20240403-111312-arnaudb.json
  • 11:11 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
  • 11:11 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
  • 11:10 fab@deploy1002: Finished deploy [airflow-dags/research@75163c7]: (no justification provided) (duration: 00m 32s)
  • 11:09 fab@deploy1002: Started deploy [airflow-dags/research@75163c7]: (no justification provided)
  • 11:08 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
  • 11:08 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
  • 11:07 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1038.eqiad.wmnet with reason: host reimage
  • 11:05 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
  • 11:05 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
  • 11:04 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1038.eqiad.wmnet with reason: host reimage
  • 10:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T356166)', diff saved to https://phabricator.wikimedia.org/P59311 and previous config saved to /var/cache/conftool/dbconfig/20240403-105940-marostegui.json
  • 10:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T360332)', diff saved to https://phabricator.wikimedia.org/P59310 and previous config saved to /var/cache/conftool/dbconfig/20240403-105804-arnaudb.json
  • 10:57 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T356166)', diff saved to https://phabricator.wikimedia.org/P59309 and previous config saved to /var/cache/conftool/dbconfig/20240403-105722-marostegui.json
  • 10:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 10:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 10:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T356166)', diff saved to https://phabricator.wikimedia.org/P59308 and previous config saved to /var/cache/conftool/dbconfig/20240403-105659-marostegui.json
  • 10:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1187 (T360332)', diff saved to https://phabricator.wikimedia.org/P59307 and previous config saved to /var/cache/conftool/dbconfig/20240403-105545-arnaudb.json
  • 10:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 10:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 10:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59306 and previous config saved to /var/cache/conftool/dbconfig/20240403-105522-arnaudb.json
  • 10:47 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 10:47 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1038.eqiad.wmnet with OS bookworm
  • 10:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59305 and previous config saved to /var/cache/conftool/dbconfig/20240403-104152-marostegui.json
  • 10:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59304 and previous config saved to /var/cache/conftool/dbconfig/20240403-104014-arnaudb.json
  • 10:29 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:29 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 10:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59303 and previous config saved to /var/cache/conftool/dbconfig/20240403-102644-marostegui.json
  • 10:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59302 and previous config saved to /var/cache/conftool/dbconfig/20240403-102507-arnaudb.json
  • 10:20 logmsgbot: dreamyjazz Deployed security patch for T361293
  • 10:17 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1037.eqiad.wmnet with OS bookworm
  • 10:14 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 10:14 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 10:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T356166)', diff saved to https://phabricator.wikimedia.org/P59301 and previous config saved to /var/cache/conftool/dbconfig/20240403-101137-marostegui.json
  • 10:10 marostegui: Restart sanitarium db1154 T361673
  • 10:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59300 and previous config saved to /var/cache/conftool/dbconfig/20240403-100959-arnaudb.json
  • 10:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T356166)', diff saved to https://phabricator.wikimedia.org/P59299 and previous config saved to /var/cache/conftool/dbconfig/20240403-100919-marostegui.json
  • 10:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 10:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 10:09 moritzm: installing util-linux security updates
  • 10:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T356166)', diff saved to https://phabricator.wikimedia.org/P59298 and previous config saved to /var/cache/conftool/dbconfig/20240403-100857-marostegui.json
  • 10:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59297 and previous config saved to /var/cache/conftool/dbconfig/20240403-100735-arnaudb.json
  • 10:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 10:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 10:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59296 and previous config saved to /var/cache/conftool/dbconfig/20240403-100712-arnaudb.json
  • 10:06 logmsgbot: dreamyjazz Deployed security patch for T361293
  • 09:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59295 and previous config saved to /var/cache/conftool/dbconfig/20240403-095349-marostegui.json
  • 09:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59294 and previous config saved to /var/cache/conftool/dbconfig/20240403-095204-arnaudb.json
  • 09:48 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1037.eqiad.wmnet with reason: host reimage
  • 09:45 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1037.eqiad.wmnet with reason: host reimage
  • 09:45 Dreamy_Jazz: Doing security deploy for T361293
  • 09:43 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1037
  • 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59293 and previous config saved to /var/cache/conftool/dbconfig/20240403-094241-root.json
  • 09:42 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1037
  • 09:39 godog: roll-restart prometheus/k8s in codfw/eqiad to apply new retention settings - T360537
  • 09:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59292 and previous config saved to /var/cache/conftool/dbconfig/20240403-093842-marostegui.json
  • 09:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59291 and previous config saved to /var/cache/conftool/dbconfig/20240403-093657-arnaudb.json
  • 09:31 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 09:31 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 09:27 marostegui: Restart sanitarium db1155 T361673
  • 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59290 and previous config saved to /var/cache/conftool/dbconfig/20240403-092735-root.json
  • 09:27 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1037.eqiad.wmnet with OS bookworm
  • 09:24 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3067.esams.wmnet
  • 09:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T356166)', diff saved to https://phabricator.wikimedia.org/P59289 and previous config saved to /var/cache/conftool/dbconfig/20240403-092334-marostegui.json
  • 09:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59288 and previous config saved to /var/cache/conftool/dbconfig/20240403-092149-arnaudb.json
  • 09:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T356166)', diff saved to https://phabricator.wikimedia.org/P59287 and previous config saved to /var/cache/conftool/dbconfig/20240403-092116-marostegui.json
  • 09:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 09:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T356166)', diff saved to https://phabricator.wikimedia.org/P59286 and previous config saved to /var/cache/conftool/dbconfig/20240403-092053-marostegui.json
  • 09:19 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3067.esams.wmnet with OS bullseye
  • 09:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59285 and previous config saved to /var/cache/conftool/dbconfig/20240403-091929-arnaudb.json
  • 09:19 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 09:19 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 09:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P59284 and previous config saved to /var/cache/conftool/dbconfig/20240403-091906-arnaudb.json
  • 09:19 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 09:18 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 09:12 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59283 and previous config saved to /var/cache/conftool/dbconfig/20240403-091229-root.json
  • 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59282 and previous config saved to /var/cache/conftool/dbconfig/20240403-090545-marostegui.json
  • 09:04 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 09:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59281 and previous config saved to /var/cache/conftool/dbconfig/20240403-090358-arnaudb.json
  • 09:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 09:00 slyngs: Upgraded Bitu / idm.wikimedia.org to version 0.0.6-2
  • 08:57 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59280 and previous config saved to /var/cache/conftool/dbconfig/20240403-085723-root.json
  • 08:55 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3067.esams.wmnet with reason: host reimage
  • 08:52 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3067.esams.wmnet with reason: host reimage
  • 08:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59279 and previous config saved to /var/cache/conftool/dbconfig/20240403-085037-marostegui.json
  • 08:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59278 and previous config saved to /var/cache/conftool/dbconfig/20240403-084851-arnaudb.json
  • 08:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59276 and previous config saved to /var/cache/conftool/dbconfig/20240403-084218-root.json
  • 08:36 marostegui: stop sanitarium codfw hosts T361673
  • 08:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T356166)', diff saved to https://phabricator.wikimedia.org/P59275 and previous config saved to /var/cache/conftool/dbconfig/20240403-083530-marostegui.json
  • 08:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P59274 and previous config saved to /var/cache/conftool/dbconfig/20240403-083343-arnaudb.json
  • 08:33 jnuche@deploy1002: Synchronized php: group1 wikis to 1.42.0-wmf.25 refs T360157 (duration: 13m 00s)
  • 08:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T356166)', diff saved to https://phabricator.wikimedia.org/P59273 and previous config saved to /var/cache/conftool/dbconfig/20240403-083313-marostegui.json
  • 08:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts puppetmaster1002.eqiad.wmnet
  • 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: puppetmaster1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 08:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 08:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T356166)', diff saved to https://phabricator.wikimedia.org/P59272 and previous config saved to /var/cache/conftool/dbconfig/20240403-083249-marostegui.json
  • 08:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P59271 and previous config saved to /var/cache/conftool/dbconfig/20240403-083123-arnaudb.json
  • 08:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 08:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 08:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 08:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 08:29 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3067.esams.wmnet with OS bullseye
  • 08:28 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: puppetmaster1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 08:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59270 and previous config saved to /var/cache/conftool/dbconfig/20240403-082712-root.json
  • 08:27 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3067.esams.wmnet
  • 08:24 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 08:24 fabfur: depool cp3067 for reimage (T360430)
  • 08:09 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2100.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 08:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59267 and previous config saved to /var/cache/conftool/dbconfig/20240403-080235-marostegui.json
  • 08:01 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 07:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2125.codfw.wmnet with OS bookworm
  • 07:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59266 and previous config saved to /var/cache/conftool/dbconfig/20240403-075510-root.json
  • 07:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2202.codfw.wmnet with OS bookworm
  • 07:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T356166)', diff saved to https://phabricator.wikimedia.org/P59265 and previous config saved to /var/cache/conftool/dbconfig/20240403-074727-marostegui.json
  • 07:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T356166)', diff saved to https://phabricator.wikimedia.org/P59264 and previous config saved to /var/cache/conftool/dbconfig/20240403-074509-marostegui.json
  • 07:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 07:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 07:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T356166)', diff saved to https://phabricator.wikimedia.org/P59263 and previous config saved to /var/cache/conftool/dbconfig/20240403-074446-marostegui.json
  • 07:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59262 and previous config saved to /var/cache/conftool/dbconfig/20240403-074004-root.json
  • 07:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2125.codfw.wmnet with reason: host reimage
  • 07:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: host reimage
  • 07:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59261 and previous config saved to /var/cache/conftool/dbconfig/20240403-072938-marostegui.json
  • 07:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2125.codfw.wmnet with reason: host reimage
  • 07:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2202.codfw.wmnet with reason: host reimage
  • 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59260 and previous config saved to /var/cache/conftool/dbconfig/20240403-072459-root.json
  • 07:18 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2100.codfw.wmnet
  • 07:17 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59259 and previous config saved to /var/cache/conftool/dbconfig/20240403-071718-root.json
  • 07:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59258 and previous config saved to /var/cache/conftool/dbconfig/20240403-071431-marostegui.json
  • 07:11 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2202.codfw.wmnet with OS bookworm
  • 07:11 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2125.codfw.wmnet with OS bookworm
  • 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59257 and previous config saved to /var/cache/conftool/dbconfig/20240403-070953-root.json
  • 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2125 T361543', diff saved to https://phabricator.wikimedia.org/P59256 and previous config saved to /var/cache/conftool/dbconfig/20240403-070946-root.json
  • 07:02 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59255 and previous config saved to /var/cache/conftool/dbconfig/20240403-070212-root.json
  • 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T356166)', diff saved to https://phabricator.wikimedia.org/P59254 and previous config saved to /var/cache/conftool/dbconfig/20240403-065923-marostegui.json
  • 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T356166)', diff saved to https://phabricator.wikimedia.org/P59253 and previous config saved to /var/cache/conftool/dbconfig/20240403-065706-marostegui.json
  • 06:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59252 and previous config saved to /var/cache/conftool/dbconfig/20240403-065617-marostegui.json
  • 06:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59251 and previous config saved to /var/cache/conftool/dbconfig/20240403-065447-root.json
  • 06:47 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59250 and previous config saved to /var/cache/conftool/dbconfig/20240403-064704-root.json
  • 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59249 and previous config saved to /var/cache/conftool/dbconfig/20240403-064110-marostegui.json
  • 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59248 and previous config saved to /var/cache/conftool/dbconfig/20240403-063941-root.json
  • 06:31 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59247 and previous config saved to /var/cache/conftool/dbconfig/20240403-063159-root.json
  • 06:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59246 and previous config saved to /var/cache/conftool/dbconfig/20240403-062602-marostegui.json
  • 06:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2148.codfw.wmnet with OS bookworm
  • 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59245 and previous config saved to /var/cache/conftool/dbconfig/20240403-062436-root.json
  • 06:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 06:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 06:16 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59244 and previous config saved to /var/cache/conftool/dbconfig/20240403-061653-root.json
  • 06:13 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 06:13 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 06:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59243 and previous config saved to /var/cache/conftool/dbconfig/20240403-061055-marostegui.json
  • 06:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2148.codfw.wmnet with reason: host reimage
  • 06:01 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59242 and previous config saved to /var/cache/conftool/dbconfig/20240403-060147-root.json
  • 06:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2148.codfw.wmnet with reason: host reimage
  • 05:51 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 05:50 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 05:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS bookworm
  • 05:46 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59241 and previous config saved to /var/cache/conftool/dbconfig/20240403-054641-root.json
  • 05:44 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2148.codfw.wmnet with OS bookworm
  • 05:43 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2148 T361543', diff saved to https://phabricator.wikimedia.org/P59240 and previous config saved to /var/cache/conftool/dbconfig/20240403-054310-root.json
  • 05:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage
  • 05:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage
  • 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS bookworm
  • 05:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1222 T361543', diff saved to https://phabricator.wikimedia.org/P59239 and previous config saved to /var/cache/conftool/dbconfig/20240403-051149-root.json
  • 05:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59238 and previous config saved to /var/cache/conftool/dbconfig/20240403-051029-marostegui.json
  • 05:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 05:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 05:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 05:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 01:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 01:58 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 01:51 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 01:51 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 01:40 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2002.codfw.wmnet with OS bookworm
  • 01:19 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2002.codfw.wmnet with reason: host reimage
  • 01:15 cwhite@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2002.codfw.wmnet with reason: host reimage
  • 01:06 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 01:06 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 01:00 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 01:00 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:46 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2002.codfw.wmnet with OS bookworm
  • 00:44 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:44 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:43 cwhite@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logging-hd2002.codfw.wmnet with OS bookworm
  • 00:37 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:36 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:30 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:30 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:25 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2003.codfw.wmnet with OS bookworm
  • 00:25 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:25 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:17 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:17 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:13 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:13 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:07 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:07 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:03 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2003.codfw.wmnet with reason: host reimage

2024-04-02

  • 23:59 cwhite@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2003.codfw.wmnet with reason: host reimage
  • 23:59 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:59 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:50 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2002.codfw.wmnet with OS bookworm
  • 23:48 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:48 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:42 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:42 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:30 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2003.codfw.wmnet with OS bookworm
  • 23:29 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:28 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:28 cwhite@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logging-hd2003.codfw.wmnet with OS bookworm
  • 23:22 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:22 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:15 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:15 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:09 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:09 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:03 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:57 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:52 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:52 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:46 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:46 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:42 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:42 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:40 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:40 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:38 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:38 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:35 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:35 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:33 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:33 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:31 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:31 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:31 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2003.codfw.wmnet with OS bookworm
  • 22:29 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:29 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:28 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2001.codfw.wmnet with OS bookworm
  • 22:27 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:27 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:25 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:25 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:21 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:21 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:19 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:19 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:17 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:17 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:15 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:15 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:13 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:13 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:11 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:11 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:09 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:09 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:08 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:08 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:07 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:07 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:07 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:07 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:05 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:05 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:05 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2001.codfw.wmnet with reason: host reimage
  • 22:03 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:02 cwhite@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2001.codfw.wmnet with reason: host reimage
  • 22:01 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:01 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:58 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:56 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:56 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:54 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:54 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:52 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:52 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:50 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:50 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:50 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:49 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:49 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:48 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:48 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:46 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:46 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:44 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:44 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:42 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:42 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:40 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:40 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:38 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:38 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:38 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:38 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:36 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:36 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:36 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:34 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:34 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:34 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:32 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:32 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:32 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2001.codfw.wmnet with OS bookworm
  • 21:31 cwhite@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host logging-hd2001.codfw.wmnet with OS bookworm
  • 21:30 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:30 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:28 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:28 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:26 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:26 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:24 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:24 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:21 bking@cumin2002: END (PASS) - Cookbook sre.ganeti.resource-report (exit_code=0)
  • 21:21 bking@cumin2002: START - Cookbook sre.ganeti.resource-report
  • 21:21 bking@cumin2002: END (PASS) - Cookbook sre.ganeti.resource-report (exit_code=0)
  • 21:21 bking@cumin2002: START - Cookbook sre.ganeti.resource-report
  • 21:16 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2001.codfw.wmnet with OS bookworm
  • 20:46 mutante: DNS - added new project language 'igl' - Igala is a Yoruboid language, spoken by the Igala ethnic group of Nigeria (800,000 speakers) T361644
  • 20:13 sfaci@deploy1002: Finished deploy [airflow-dags/analytics@75163c7]: (no justification provided) (duration: 00m 16s)
  • 20:13 sfaci@deploy1002: Started deploy [airflow-dags/analytics@75163c7]: (no justification provided)
  • 20:07 sfaci@deploy1002: Finished deploy [airflow-dags/analytics@75163c7]: (no justification provided) (duration: 00m 46s)
  • 20:06 sfaci@deploy1002: Started deploy [airflow-dags/analytics@75163c7]: (no justification provided)
  • 20:06 sfaci@deploy1002: Finished deploy [airflow-dags/analytics@75163c7]: (no justification provided) (duration: 00m 54s)
  • 20:05 sfaci@deploy1002: Started deploy [airflow-dags/analytics@75163c7]: (no justification provided)
  • 18:48 cstone: civicrm upgraded from ed776060 to 5b7f9e06
  • 17:26 cwhite@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logging-hd2001.codfw.wmnet with OS bookworm
  • 17:24 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 17:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 17:19 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 17:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 17:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T356166)', diff saved to https://phabricator.wikimedia.org/P59232 and previous config saved to /var/cache/conftool/dbconfig/20240402-171935-marostegui.json
  • 17:13 Dreamy_Jazz: Creating cu_useragent table on WMF wikis - T359312
  • 17:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59230 and previous config saved to /var/cache/conftool/dbconfig/20240402-170625-arnaudb.json
  • 17:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59229 and previous config saved to /var/cache/conftool/dbconfig/20240402-170610-arnaudb.json
  • 17:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59228 and previous config saved to /var/cache/conftool/dbconfig/20240402-170603-arnaudb.json
  • 17:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59227 and previous config saved to /var/cache/conftool/dbconfig/20240402-170555-arnaudb.json
  • 17:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P59226 and previous config saved to /var/cache/conftool/dbconfig/20240402-170427-marostegui.json
  • 16:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:59 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 100%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59225 and previous config saved to /var/cache/conftool/dbconfig/20240402-165601-arnaudb.json
  • 16:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59224 and previous config saved to /var/cache/conftool/dbconfig/20240402-165119-arnaudb.json
  • 16:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59223 and previous config saved to /var/cache/conftool/dbconfig/20240402-165105-arnaudb.json
  • 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59222 and previous config saved to /var/cache/conftool/dbconfig/20240402-165058-arnaudb.json
  • 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59221 and previous config saved to /var/cache/conftool/dbconfig/20240402-165049-arnaudb.json
  • 16:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P59220 and previous config saved to /var/cache/conftool/dbconfig/20240402-164920-marostegui.json
  • 16:45 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 16:44 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 16:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 75%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59219 and previous config saved to /var/cache/conftool/dbconfig/20240402-164055-arnaudb.json
  • 16:40 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 16:38 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:38 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59218 and previous config saved to /var/cache/conftool/dbconfig/20240402-163613-arnaudb.json
  • 16:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59217 and previous config saved to /var/cache/conftool/dbconfig/20240402-163559-arnaudb.json
  • 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59216 and previous config saved to /var/cache/conftool/dbconfig/20240402-163552-arnaudb.json
  • 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59215 and previous config saved to /var/cache/conftool/dbconfig/20240402-163544-arnaudb.json
  • 16:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T356166)', diff saved to https://phabricator.wikimedia.org/P59214 and previous config saved to /var/cache/conftool/dbconfig/20240402-163413-marostegui.json
  • 16:28 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:27 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 50%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59213 and previous config saved to /var/cache/conftool/dbconfig/20240402-162550-arnaudb.json
  • 16:22 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1036.eqiad.wmnet with OS bookworm
  • 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59212 and previous config saved to /var/cache/conftool/dbconfig/20240402-162107-arnaudb.json
  • 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59211 and previous config saved to /var/cache/conftool/dbconfig/20240402-162053-arnaudb.json
  • 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59210 and previous config saved to /var/cache/conftool/dbconfig/20240402-162046-arnaudb.json
  • 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59209 and previous config saved to /var/cache/conftool/dbconfig/20240402-162038-arnaudb.json
  • 16:13 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
  • 16:11 vriley@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
  • 16:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 25%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59208 and previous config saved to /var/cache/conftool/dbconfig/20240402-161044-arnaudb.json
  • 16:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59207 and previous config saved to /var/cache/conftool/dbconfig/20240402-160602-arnaudb.json
  • 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59206 and previous config saved to /var/cache/conftool/dbconfig/20240402-160547-arnaudb.json
  • 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59205 and previous config saved to /var/cache/conftool/dbconfig/20240402-160540-arnaudb.json
  • 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59204 and previous config saved to /var/cache/conftool/dbconfig/20240402-160532-arnaudb.json
  • 16:04 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:04 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:02 jnuche@deploy1002: Finished scap: Backport for rest: add default null to nullable typed prop (T361577) (duration: 14m 39s)
  • 16:02 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2001.codfw.wmnet with OS bookworm
  • 15:57 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:57 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:57 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:57 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
  • 15:55 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1036
  • 15:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 15%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59203 and previous config saved to /var/cache/conftool/dbconfig/20240402-155538-arnaudb.json
  • 15:55 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1036
  • 15:54 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:54 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:54 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:51 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1036.eqiad.wmnet with reason: host reimage
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59202 and previous config saved to /var/cache/conftool/dbconfig/20240402-155056-arnaudb.json
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59201 and previous config saved to /var/cache/conftool/dbconfig/20240402-155042-arnaudb.json
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59200 and previous config saved to /var/cache/conftool/dbconfig/20240402-155035-arnaudb.json
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59199 and previous config saved to /var/cache/conftool/dbconfig/20240402-155026-arnaudb.json
  • 15:50 jnuche@deploy1002: samtar and jnuche: Continuing with sync
  • 15:50 jnuche@deploy1002: samtar and jnuche: Backport for rest: add default null to nullable typed prop (T361577) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 15:49 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1036.eqiad.wmnet with reason: host reimage
  • 15:47 jnuche@deploy1002: Started scap: Backport for rest: add default null to nullable typed prop (T361577)
  • 15:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 10%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59198 and previous config saved to /var/cache/conftool/dbconfig/20240402-154033-arnaudb.json
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59197 and previous config saved to /var/cache/conftool/dbconfig/20240402-153550-arnaudb.json
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59196 and previous config saved to /var/cache/conftool/dbconfig/20240402-153536-arnaudb.json
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59195 and previous config saved to /var/cache/conftool/dbconfig/20240402-153529-arnaudb.json
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59194 and previous config saved to /var/cache/conftool/dbconfig/20240402-153521-arnaudb.json
  • 15:31 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1036.eqiad.wmnet with OS bookworm
  • 15:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 5%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59193 and previous config saved to /var/cache/conftool/dbconfig/20240402-152527-arnaudb.json
  • 15:23 jgiannelos@deploy1002: Finished deploy [restbase/deploy@c4d19d7]: (no justification provided) (duration: 00m 16s)
  • 15:23 jgiannelos@deploy1002: Started deploy [restbase/deploy@c4d19d7]: (no justification provided)
  • 15:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1230.eqiad.wmnet with OS bookworm
  • 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59192 and previous config saved to /var/cache/conftool/dbconfig/20240402-152044-arnaudb.json
  • 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59191 and previous config saved to /var/cache/conftool/dbconfig/20240402-152031-arnaudb.json
  • 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59190 and previous config saved to /var/cache/conftool/dbconfig/20240402-152023-arnaudb.json
  • 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59189 and previous config saved to /var/cache/conftool/dbconfig/20240402-152015-arnaudb.json
  • 15:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1223 (T356166)', diff saved to https://phabricator.wikimedia.org/P59188 and previous config saved to /var/cache/conftool/dbconfig/20240402-151235-marostegui.json
  • 15:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 15:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 15:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T356166)', diff saved to https://phabricator.wikimedia.org/P59187 and previous config saved to /var/cache/conftool/dbconfig/20240402-151213-marostegui.json
  • 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59186 and previous config saved to /var/cache/conftool/dbconfig/20240402-150538-arnaudb.json
  • 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59185 and previous config saved to /var/cache/conftool/dbconfig/20240402-150525-arnaudb.json
  • 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59184 and previous config saved to /var/cache/conftool/dbconfig/20240402-150516-arnaudb.json
  • 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59183 and previous config saved to /var/cache/conftool/dbconfig/20240402-150509-arnaudb.json
  • 15:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1230.eqiad.wmnet with reason: host reimage
  • 14:57 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1230.eqiad.wmnet with reason: host reimage
  • 14:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P59182 and previous config saved to /var/cache/conftool/dbconfig/20240402-145705-marostegui.json
  • 14:56 moritzm: installing mariadb security updates (as packaged in Debian, unrelated to the wmf-mariadb packages)
  • 14:45 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1230.eqiad.wmnet with OS bookworm
  • 14:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1230.eqiad.wmnet with reason: Silence for reimaging
  • 14:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1230.eqiad.wmnet with reason: Silence for reimaging
  • 14:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool to reimage db1230', diff saved to https://phabricator.wikimedia.org/P59181 and previous config saved to /var/cache/conftool/dbconfig/20240402-144221-arnaudb.json
  • 14:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P59180 and previous config saved to /var/cache/conftool/dbconfig/20240402-144158-marostegui.json
  • 14:38 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3066.esams.wmnet
  • 14:38 fabfur: repooling cp3066 after reimage (T360430)
  • 14:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 100%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59179 and previous config saved to /var/cache/conftool/dbconfig/20240402-142859-arnaudb.json
  • 14:28 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3066.esams.wmnet with OS bullseye
  • 14:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T356166)', diff saved to https://phabricator.wikimedia.org/P59178 and previous config saved to /var/cache/conftool/dbconfig/20240402-142650-marostegui.json
  • 14:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59177 and previous config saved to /var/cache/conftool/dbconfig/20240402-141541-root.json
  • 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 75%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59176 and previous config saved to /var/cache/conftool/dbconfig/20240402-141353-arnaudb.json
  • 14:09 moritzm: imported jenkins 2.440.2 to thirdparty/ci for buster-wikimedia T360759
  • 14:05 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3066.esams.wmnet with reason: host reimage
  • 14:02 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3066.esams.wmnet with reason: host reimage
  • 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59174 and previous config saved to /var/cache/conftool/dbconfig/20240402-140035-root.json
  • 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 50%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59173 and previous config saved to /var/cache/conftool/dbconfig/20240402-135847-arnaudb.json
  • 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59172 and previous config saved to /var/cache/conftool/dbconfig/20240402-134528-root.json
  • 13:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 25%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59171 and previous config saved to /var/cache/conftool/dbconfig/20240402-134342-arnaudb.json
  • 13:38 TheresNoTime: closing UTC afternoon backport window
  • 13:38 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3066.esams.wmnet with OS bullseye
  • 13:37 samtar@deploy1002: Finished scap: Backport for InitialiseSettings: Enable Edit Recovery on all projects (T355548) (duration: 16m 26s)
  • 13:33 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3066.esams.wmnet
  • 13:32 fabfur: depool cp3066 for reimage (T360430)
  • 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59170 and previous config saved to /var/cache/conftool/dbconfig/20240402-133023-root.json
  • 13:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 15%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59169 and previous config saved to /var/cache/conftool/dbconfig/20240402-132836-arnaudb.json
  • 13:25 samtar@deploy1002: samtar: Continuing with sync
  • 13:23 samtar@deploy1002: samtar: Backport for InitialiseSettings: Enable Edit Recovery on all projects (T355548) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:21 samtar@deploy1002: Started scap: Backport for InitialiseSettings: Enable Edit Recovery on all projects (T355548)
  • 13:18 dreamyjazz@deploy1002: Finished scap: Backport for Deploy partial action blocks everywhere (T353496) (duration: 15m 33s)
  • 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59168 and previous config saved to /var/cache/conftool/dbconfig/20240402-131517-root.json
  • 13:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 10%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59167 and previous config saved to /var/cache/conftool/dbconfig/20240402-131330-arnaudb.json
  • 13:05 dreamyjazz@deploy1002: dreamyjazz and tchanders: Continuing with sync
  • 13:05 dreamyjazz@deploy1002: dreamyjazz and tchanders: Backport for Deploy partial action blocks everywhere (T353496) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1212 (T356166)', diff saved to https://phabricator.wikimedia.org/P59166 and previous config saved to /var/cache/conftool/dbconfig/20240402-130423-marostegui.json
  • 13:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 13:04 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1212.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1212.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T356166)', diff saved to https://phabricator.wikimedia.org/P59165 and previous config saved to /var/cache/conftool/dbconfig/20240402-130341-marostegui.json
  • 13:02 dreamyjazz@deploy1002: Started scap: Backport for Deploy partial action blocks everywhere (T353496)
  • 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59164 and previous config saved to /var/cache/conftool/dbconfig/20240402-130012-root.json
  • 12:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 5%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59163 and previous config saved to /var/cache/conftool/dbconfig/20240402-125825-arnaudb.json
  • 12:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P59162 and previous config saved to /var/cache/conftool/dbconfig/20240402-124834-marostegui.json
  • 12:45 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59161 and previous config saved to /var/cache/conftool/dbconfig/20240402-124506-root.json
  • 12:44 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1188.eqiad.wmnet with OS bookworm
  • 12:39 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1035.eqiad.wmnet with OS bookworm
  • 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P59160 and previous config saved to /var/cache/conftool/dbconfig/20240402-123326-marostegui.json
  • 12:28 taavi: taavi@deploy1002 ~ $ sudo systemctl kill train-presync.service # T361580
  • 12:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1188.eqiad.wmnet with reason: host reimage
  • 12:20 hnowlan@deploy2002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 12:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1188.eqiad.wmnet with reason: host reimage
  • 12:19 hnowlan@deploy2002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 12:18 hnowlan@deploy2002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
  • 12:18 hnowlan@deploy2002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 12:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T356166)', diff saved to https://phabricator.wikimedia.org/P59159 and previous config saved to /var/cache/conftool/dbconfig/20240402-121819-marostegui.json
  • 12:13 moritzm: installing pillow security updates
  • 12:13 hnowlan@deploy2002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 12:12 hnowlan@deploy2002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 12:11 hnowlan@deploy2002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
  • 12:11 hnowlan@deploy2002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 12:09 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1035.eqiad.wmnet with reason: host reimage
  • 12:07 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1188.eqiad.wmnet with OS bookworm
  • 12:06 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1035.eqiad.wmnet with reason: host reimage
  • 12:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1188 T361543', diff saved to https://phabricator.wikimedia.org/P59158 and previous config saved to /var/cache/conftool/dbconfig/20240402-120455-root.json
  • 11:58 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1035
  • 11:58 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1035
  • 11:49 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1035.eqiad.wmnet with OS bookworm
  • 11:33 sfaci@deploy1002: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
  • 11:32 sfaci@deploy1002: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
  • 11:32 sfaci@deploy1002: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
  • 11:32 sfaci@deploy1002: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
  • 11:31 sfaci@deploy1002: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
  • 11:31 sfaci@deploy1002: helmfile [staging] START helmfile.d/services/editor-analytics: apply
  • 11:29 sfaci@deploy1002: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
  • 11:29 sfaci@deploy1002: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
  • 11:29 sfaci@deploy1002: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
  • 11:28 sfaci@deploy1002: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
  • 11:25 sfaci@deploy1002: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
  • 11:25 sfaci@deploy1002: helmfile [staging] START helmfile.d/services/edit-analytics: apply
  • 11:01 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1198 (T356166)', diff saved to https://phabricator.wikimedia.org/P59157 and previous config saved to /var/cache/conftool/dbconfig/20240402-110122-marostegui.json
  • 11:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 11:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 11:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T356166)', diff saved to https://phabricator.wikimedia.org/P59156 and previous config saved to /var/cache/conftool/dbconfig/20240402-110100-marostegui.json
  • 10:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P59155 and previous config saved to /var/cache/conftool/dbconfig/20240402-104552-marostegui.json
  • 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P59154 and previous config saved to /var/cache/conftool/dbconfig/20240402-103045-marostegui.json
  • 10:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59153 and previous config saved to /var/cache/conftool/dbconfig/20240402-102951-root.json
  • 10:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T356166)', diff saved to https://phabricator.wikimedia.org/P59152 and previous config saved to /var/cache/conftool/dbconfig/20240402-101538-marostegui.json
  • 10:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59151 and previous config saved to /var/cache/conftool/dbconfig/20240402-101445-root.json
  • 09:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59150 and previous config saved to /var/cache/conftool/dbconfig/20240402-095939-root.json
  • 09:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1185.eqiad.wmnet with OS bookworm
  • 09:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2115.codfw.wmnet onto db2215.codfw.wmnet
  • 09:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59149 and previous config saved to /var/cache/conftool/dbconfig/20240402-094433-root.json
  • 09:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1185.eqiad.wmnet with reason: host reimage
  • 09:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59148 and previous config saved to /var/cache/conftool/dbconfig/20240402-092928-root.json
  • 09:28 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1185.eqiad.wmnet with reason: host reimage
  • 09:22 jnuche@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.25 refs T360157
  • 09:15 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1185.eqiad.wmnet with OS bookworm
  • 09:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59147 and previous config saved to /var/cache/conftool/dbconfig/20240402-091422-root.json
  • 09:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1185.eqiad.wmnet with reason: Silence for reimaging
  • 09:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1185.eqiad.wmnet with reason: Silence for reimaging
  • 09:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool to reimage db1185', diff saved to https://phabricator.wikimedia.org/P59146 and previous config saved to /var/cache/conftool/dbconfig/20240402-091303-arnaudb.json
  • 09:02 jnuche@deploy1002: Finished scap: testwikis wikis to 1.42.0-wmf.25 refs T360157 (duration: 51m 03s)
  • 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59145 and previous config saved to /var/cache/conftool/dbconfig/20240402-085917-root.json
  • 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1189 (T356166)', diff saved to https://phabricator.wikimedia.org/P59144 and previous config saved to /var/cache/conftool/dbconfig/20240402-085814-marostegui.json
  • 08:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T356166)', diff saved to https://phabricator.wikimedia.org/P59143 and previous config saved to /var/cache/conftool/dbconfig/20240402-085752-marostegui.json
  • 08:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1197.eqiad.wmnet with OS bookworm
  • 08:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P59142 and previous config saved to /var/cache/conftool/dbconfig/20240402-084244-marostegui.json
  • 08:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1197.eqiad.wmnet with reason: host reimage
  • 08:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1197.eqiad.wmnet with reason: host reimage
  • 08:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P59141 and previous config saved to /var/cache/conftool/dbconfig/20240402-082737-marostegui.json
  • 08:06 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59137 and previous config saved to /var/cache/conftool/dbconfig/20240402-080642-root.json
  • 08:02 godog: restore SRE business hours routing/escalation after the holidays - T350192
  • 07:51 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 38278
  • 07:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59134 and previous config saved to /var/cache/conftool/dbconfig/20240402-075136-root.json
  • 07:49 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 38278
  • 07:47 hashar: UTC morning backport window completed
  • 07:46 hashar@deploy1002: Finished scap: Backport for webrequest: disable canary events. (T314956 T351117) (duration: 34m 03s)
  • 07:39 taavi: update firewall policy on cr-eqiad, cr-codfw T361537
  • 07:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59133 and previous config saved to /var/cache/conftool/dbconfig/20240402-073631-root.json
  • 07:30 hashar@deploy1002: gmodena and hashar: Continuing with sync
  • 07:28 hashar@deploy1002: gmodena and hashar: Backport for webrequest: disable canary events. (T314956 T351117) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:21 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59132 and previous config saved to /var/cache/conftool/dbconfig/20240402-072125-root.json
  • 07:12 hashar@deploy1002: Started scap: Backport for webrequest: disable canary events. (T314956 T351117)
  • 07:06 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59131 and previous config saved to /var/cache/conftool/dbconfig/20240402-070619-root.json
  • 07:03 moritzm: installing util-linux security updates
  • 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1175 (T356166)', diff saved to https://phabricator.wikimedia.org/P59130 and previous config saved to /var/cache/conftool/dbconfig/20240402-065751-marostegui.json
  • 06:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 06:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T356166)', diff saved to https://phabricator.wikimedia.org/P59129 and previous config saved to /var/cache/conftool/dbconfig/20240402-065728-marostegui.json
  • 06:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59128 and previous config saved to /var/cache/conftool/dbconfig/20240402-065113-root.json
  • 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P59127 and previous config saved to /var/cache/conftool/dbconfig/20240402-064221-marostegui.json
  • 06:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59126 and previous config saved to /var/cache/conftool/dbconfig/20240402-063607-root.json
  • 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59125 and previous config saved to /var/cache/conftool/dbconfig/20240402-063510-root.json
  • 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P59124 and previous config saved to /var/cache/conftool/dbconfig/20240402-062713-marostegui.json
  • 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59123 and previous config saved to /var/cache/conftool/dbconfig/20240402-062427-root.json
  • 06:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1229.eqiad.wmnet with OS bookworm
  • 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59122 and previous config saved to /var/cache/conftool/dbconfig/20240402-062004-root.json
  • 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T356166)', diff saved to https://phabricator.wikimedia.org/P59121 and previous config saved to /var/cache/conftool/dbconfig/20240402-061206-marostegui.json
  • 06:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59120 and previous config saved to /var/cache/conftool/dbconfig/20240402-060922-root.json
  • 06:04 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59119 and previous config saved to /var/cache/conftool/dbconfig/20240402-060459-root.json
  • 06:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1229.eqiad.wmnet with reason: host reimage
  • 05:59 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1229.eqiad.wmnet with reason: host reimage
  • 05:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59118 and previous config saved to /var/cache/conftool/dbconfig/20240402-055416-root.json
  • 05:49 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59117 and previous config saved to /var/cache/conftool/dbconfig/20240402-054953-root.json
  • 05:46 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1229.eqiad.wmnet with OS bookworm
  • 05:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1229 T361543', diff saved to https://phabricator.wikimedia.org/P59116 and previous config saved to /var/cache/conftool/dbconfig/20240402-054408-root.json
  • 05:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59115 and previous config saved to /var/cache/conftool/dbconfig/20240402-053910-root.json
  • 05:34 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59114 and previous config saved to /var/cache/conftool/dbconfig/20240402-053447-root.json
  • 05:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59113 and previous config saved to /var/cache/conftool/dbconfig/20240402-052404-root.json
  • 05:19 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59112 and previous config saved to /var/cache/conftool/dbconfig/20240402-051942-root.json
  • 05:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59111 and previous config saved to /var/cache/conftool/dbconfig/20240402-050859-root.json
  • 05:04 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59110 and previous config saved to /var/cache/conftool/dbconfig/20240402-050436-root.json
  • 04:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1166 (T356166)', diff saved to https://phabricator.wikimedia.org/P59109 and previous config saved to /var/cache/conftool/dbconfig/20240402-045716-marostegui.json
  • 04:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 04:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 04:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1024 T358746', diff saved to https://phabricator.wikimedia.org/P59108 and previous config saved to /var/cache/conftool/dbconfig/20240402-045559-root.json
  • 04:53 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59107 and previous config saved to /var/cache/conftool/dbconfig/20240402-045353-root.json
  • 04:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 04:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 03:03 mwpresync@deploy1002: Pruned MediaWiki: 1.42.0-wmf.22 (duration: 03m 20s)

2024-04-01

  • 23:14 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1006.eqiad.wmnet with OS bullseye
  • 22:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 22:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 22:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 22:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 22:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T356166)', diff saved to https://phabricator.wikimedia.org/P59105 and previous config saved to /var/cache/conftool/dbconfig/20240401-223055-marostegui.json
  • 22:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59104 and previous config saved to /var/cache/conftool/dbconfig/20240401-221548-marostegui.json
  • 22:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59103 and previous config saved to /var/cache/conftool/dbconfig/20240401-220040-marostegui.json
  • 21:53 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
  • 21:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T356166)', diff saved to https://phabricator.wikimedia.org/P59102 and previous config saved to /var/cache/conftool/dbconfig/20240401-214532-marostegui.json
  • 21:38 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1230 (T356166)', diff saved to https://phabricator.wikimedia.org/P59101 and previous config saved to /var/cache/conftool/dbconfig/20240401-213834-marostegui.json
  • 21:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 21:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 21:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 21:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T356166)', diff saved to https://phabricator.wikimedia.org/P59100 and previous config saved to /var/cache/conftool/dbconfig/20240401-212751-marostegui.json
  • 21:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P59099 and previous config saved to /var/cache/conftool/dbconfig/20240401-211244-marostegui.json
  • 21:03 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1006.eqiad.wmnet with OS bullseye
  • 20:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P59098 and previous config saved to /var/cache/conftool/dbconfig/20240401-205736-marostegui.json
  • 20:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T356166)', diff saved to https://phabricator.wikimedia.org/P59097 and previous config saved to /var/cache/conftool/dbconfig/20240401-204229-marostegui.json
  • 20:36 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2088.codfw.wmnet with OS bullseye
  • 20:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1213 (T356166)', diff saved to https://phabricator.wikimedia.org/P59096 and previous config saved to /var/cache/conftool/dbconfig/20240401-203254-marostegui.json
  • 20:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1213.eqiad.wmnet with reason: Maintenance
  • 20:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1213.eqiad.wmnet with reason: Maintenance
  • 20:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T356166)', diff saved to https://phabricator.wikimedia.org/P59095 and previous config saved to /var/cache/conftool/dbconfig/20240401-203232-marostegui.json
  • 20:19 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2088.codfw.wmnet with reason: host reimage
  • 20:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59094 and previous config saved to /var/cache/conftool/dbconfig/20240401-201725-marostegui.json
  • 20:17 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2088.codfw.wmnet with reason: host reimage
  • 20:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59093 and previous config saved to /var/cache/conftool/dbconfig/20240401-200217-marostegui.json
  • 20:00 bking@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2088.codfw.wmnet with OS bullseye
  • 19:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T356166)', diff saved to https://phabricator.wikimedia.org/P59092 and previous config saved to /var/cache/conftool/dbconfig/20240401-194709-marostegui.json
  • 19:42 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
  • 19:37 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1210 (T356166)', diff saved to https://phabricator.wikimedia.org/P59091 and previous config saved to /var/cache/conftool/dbconfig/20240401-193713-marostegui.json
  • 19:37 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 19:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 19:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T356166)', diff saved to https://phabricator.wikimedia.org/P59090 and previous config saved to /var/cache/conftool/dbconfig/20240401-193650-marostegui.json
  • 19:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59089 and previous config saved to /var/cache/conftool/dbconfig/20240401-192143-marostegui.json
  • 19:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59088 and previous config saved to /var/cache/conftool/dbconfig/20240401-190635-marostegui.json
  • 18:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T356166)', diff saved to https://phabricator.wikimedia.org/P59087 and previous config saved to /var/cache/conftool/dbconfig/20240401-185128-marostegui.json
  • 18:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1200 (T356166)', diff saved to https://phabricator.wikimedia.org/P59086 and previous config saved to /var/cache/conftool/dbconfig/20240401-184455-marostegui.json
  • 18:44 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 18:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 18:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T356166)', diff saved to https://phabricator.wikimedia.org/P59085 and previous config saved to /var/cache/conftool/dbconfig/20240401-184432-marostegui.json
  • 18:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59084 and previous config saved to /var/cache/conftool/dbconfig/20240401-182924-marostegui.json
  • 18:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59083 and previous config saved to /var/cache/conftool/dbconfig/20240401-181417-marostegui.json
  • 17:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T356166)', diff saved to https://phabricator.wikimedia.org/P59082 and previous config saved to /var/cache/conftool/dbconfig/20240401-175910-marostegui.json
  • 17:58 mutante: LDAP - removed uid migr from groups nda and wmde (T361266)
  • 17:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1185 (T356166)', diff saved to https://phabricator.wikimedia.org/P59081 and previous config saved to /var/cache/conftool/dbconfig/20240401-175300-marostegui.json
  • 17:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 17:52 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 17:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T356166)', diff saved to https://phabricator.wikimedia.org/P59080 and previous config saved to /var/cache/conftool/dbconfig/20240401-175237-marostegui.json
  • 17:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59079 and previous config saved to /var/cache/conftool/dbconfig/20240401-173729-marostegui.json
  • 17:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59078 and previous config saved to /var/cache/conftool/dbconfig/20240401-172221-marostegui.json
  • 17:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T356166)', diff saved to https://phabricator.wikimedia.org/P59077 and previous config saved to /var/cache/conftool/dbconfig/20240401-170713-marostegui.json
  • 16:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1161 (T356166)', diff saved to https://phabricator.wikimedia.org/P59076 and previous config saved to /var/cache/conftool/dbconfig/20240401-165559-marostegui.json
  • 16:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 16:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 16:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 16:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 16:43 eevans@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@42653e6]: (no justification provided) (duration: 00m 33s)
  • 16:42 eevans@deploy1002: Started deploy [cassandra/logstash-logback-encoder@42653e6]: (no justification provided)
  • 16:07 Dreamy_Jazz: Restarting MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration
  • 16:05 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host elastic2088.codfw.wmnet
  • 15:50 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host elastic2088.codfw.wmnet
  • 15:36 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1005.eqiad.wmnet with OS bullseye
  • 15:36 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 15:35 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 15:33 urandom: cassandra (restbase): re-enable blocking read-repair — T354561
  • 15:30 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
  • 15:14 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1005.eqiad.wmnet with reason: host reimage
  • 15:13 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
  • 15:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 15:11 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1005.eqiad.wmnet with reason: host reimage
  • 15:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 15:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T356166)', diff saved to https://phabricator.wikimedia.org/P59075 and previous config saved to /var/cache/conftool/dbconfig/20240401-151114-marostegui.json
  • 14:58 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1005.eqiad.wmnet with OS bullseye
  • 14:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59074 and previous config saved to /var/cache/conftool/dbconfig/20240401-145606-marostegui.json
  • 14:52 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 14:52 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2088']
  • 14:51 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2088']
  • 14:49 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 14:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59073 and previous config saved to /var/cache/conftool/dbconfig/20240401-144059-marostegui.json
  • 14:32 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['elastic2088']
  • 14:32 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2088']
  • 14:28 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1005.eqiad.wmnet with OS bullseye
  • 14:27 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 14:26 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye
  • 14:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T356166)', diff saved to https://phabricator.wikimedia.org/P59072 and previous config saved to /var/cache/conftool/dbconfig/20240401-142552-marostegui.json
  • 14:06 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 14:06 sukhe: reimage cp4052 back to bullseye
  • 13:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1231 (T356166)', diff saved to https://phabricator.wikimedia.org/P59071 and previous config saved to /var/cache/conftool/dbconfig/20240401-135204-marostegui.json
  • 13:51 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 13:11 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1005.eqiad.wmnet with OS bullseye
  • 12:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T356166)', diff saved to https://phabricator.wikimedia.org/P59070 and previous config saved to /var/cache/conftool/dbconfig/20240401-125524-marostegui.json
  • 12:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P59069 and previous config saved to /var/cache/conftool/dbconfig/20240401-124017-marostegui.json
  • 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P59068 and previous config saved to /var/cache/conftool/dbconfig/20240401-122510-marostegui.json
  • 12:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T356166)', diff saved to https://phabricator.wikimedia.org/P59067 and previous config saved to /var/cache/conftool/dbconfig/20240401-121002-marostegui.json
  • 11:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1224 (T356166)', diff saved to https://phabricator.wikimedia.org/P59066 and previous config saved to /var/cache/conftool/dbconfig/20240401-114105-marostegui.json
  • 11:40 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1224.eqiad.wmnet with reason: Maintenance
  • 11:40 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1224.eqiad.wmnet with reason: Maintenance
  • 11:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T356166)', diff saved to https://phabricator.wikimedia.org/P59065 and previous config saved to /var/cache/conftool/dbconfig/20240401-114043-marostegui.json
  • 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59064 and previous config saved to /var/cache/conftool/dbconfig/20240401-112536-marostegui.json
  • 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59063 and previous config saved to /var/cache/conftool/dbconfig/20240401-111028-marostegui.json
  • 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T356166)', diff saved to https://phabricator.wikimedia.org/P59062 and previous config saved to /var/cache/conftool/dbconfig/20240401-105521-marostegui.json
  • 10:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1201 (T356166)', diff saved to https://phabricator.wikimedia.org/P59061 and previous config saved to /var/cache/conftool/dbconfig/20240401-102328-marostegui.json
  • 10:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T356166)', diff saved to https://phabricator.wikimedia.org/P59060 and previous config saved to /var/cache/conftool/dbconfig/20240401-102306-marostegui.json
  • 10:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59059 and previous config saved to /var/cache/conftool/dbconfig/20240401-100758-marostegui.json
  • 09:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59058 and previous config saved to /var/cache/conftool/dbconfig/20240401-095251-marostegui.json
  • 09:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T356166)', diff saved to https://phabricator.wikimedia.org/P59057 and previous config saved to /var/cache/conftool/dbconfig/20240401-093744-marostegui.json
  • 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1187 (T356166)', diff saved to https://phabricator.wikimedia.org/P59056 and previous config saved to /var/cache/conftool/dbconfig/20240401-090527-marostegui.json
  • 09:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 09:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T356166)', diff saved to https://phabricator.wikimedia.org/P59055 and previous config saved to /var/cache/conftool/dbconfig/20240401-090503-marostegui.json
  • 08:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59054 and previous config saved to /var/cache/conftool/dbconfig/20240401-084956-marostegui.json
  • 08:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59053 and previous config saved to /var/cache/conftool/dbconfig/20240401-083449-marostegui.json
  • 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1180 (T356166)', diff saved to https://phabricator.wikimedia.org/P59051 and previous config saved to /var/cache/conftool/dbconfig/20240401-074221-marostegui.json
  • 07:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 07:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T356166)', diff saved to https://phabricator.wikimedia.org/P59050 and previous config saved to /var/cache/conftool/dbconfig/20240401-074158-marostegui.json
  • 07:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59049 and previous config saved to /var/cache/conftool/dbconfig/20240401-072650-marostegui.json
  • 07:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59048 and previous config saved to /var/cache/conftool/dbconfig/20240401-071143-marostegui.json
  • 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T356166)', diff saved to https://phabricator.wikimedia.org/P59047 and previous config saved to /var/cache/conftool/dbconfig/20240401-065635-marostegui.json
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1168 (T356166)', diff saved to https://phabricator.wikimedia.org/P59046 and previous config saved to /var/cache/conftool/dbconfig/20240401-062954-marostegui.json
  • 06:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 06:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T356166)', diff saved to https://phabricator.wikimedia.org/P59045 and previous config saved to /var/cache/conftool/dbconfig/20240401-062932-marostegui.json
  • 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59044 and previous config saved to /var/cache/conftool/dbconfig/20240401-061423-marostegui.json
  • 05:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59043 and previous config saved to /var/cache/conftool/dbconfig/20240401-055915-marostegui.json
  • 05:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T356166)', diff saved to https://phabricator.wikimedia.org/P59042 and previous config saved to /var/cache/conftool/dbconfig/20240401-054408-marostegui.json
  • 05:14 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T356166)', diff saved to https://phabricator.wikimedia.org/P59041 and previous config saved to /var/cache/conftool/dbconfig/20240401-051402-marostegui.json
  • 05:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 05:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 05:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 9 hosts with reason: Fixing intermediate master
  • 05:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on 9 hosts with reason: Fixing intermediate master

Archives

See Server Admin Log/Archives.