Jump to content

Server Admin Log

From Wikitech

2024-12-09

  • 15:56 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1073.eqiad.wmnet with OS bookworm
  • 15:55 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1073.eqiad.wmnet with OS bookworm
  • 15:45 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2090.codfw.wmnet with OS bookworm
  • 15:44 hnowlan@deploy2002: Started scap sync-world: Rebuild and deploy to pick up new php8.1 base
  • 15:43 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2089.codfw.wmnet with OS bookworm
  • 15:34 Emperor: depool/restart swift/repool ms-fe1012
  • 15:34 mszabo@deploy2002: Finished scap sync-world: Backport for dialog: Fix wrong title on Types of unacceptable behavior step (T381529), dialog: Fix spacing between buttons in the dialog footer (T381530), Prep IRS config for testwiki (duration: 13m 39s)
  • 15:33 Emperor: depool/restart swift/repool ms-fe1010
  • 15:28 mszabo@deploy2002: mszabo: Continuing with sync
  • 15:25 mszabo@deploy2002: mszabo: Backport for dialog: Fix wrong title on Types of unacceptable behavior step (T381529), dialog: Fix spacing between buttons in the dialog footer (T381530), Prep IRS config for testwiki synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 15:25 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2090.codfw.wmnet with reason: host reimage
  • 15:22 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2089.codfw.wmnet with reason: host reimage
  • 15:21 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1075.eqiad.wmnet with OS bookworm
  • 15:20 mszabo@deploy2002: Started scap sync-world: Backport for dialog: Fix wrong title on Types of unacceptable behavior step (T381529), dialog: Fix spacing between buttons in the dialog footer (T381530), Prep IRS config for testwiki
  • 15:20 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2090.codfw.wmnet with reason: host reimage
  • 15:18 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1074.eqiad.wmnet with OS bookworm
  • 15:18 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2089.codfw.wmnet with reason: host reimage
  • 15:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1072.eqiad.wmnet with OS bookworm
  • 15:04 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1075.eqiad.wmnet with reason: host reimage
  • 15:01 Lucas_WMDE: UTC afternoon backport+config window done
  • 15:00 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1074.eqiad.wmnet with reason: host reimage
  • 15:00 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2090
  • 15:00 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2090
  • 15:00 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2090.codfw.wmnet with OS bookworm
  • 15:00 bking@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1025.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:59 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:59 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for idwikivoyage: add logo, wordmark (T381080) (duration: 11m 44s)
  • 14:59 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2089
  • 14:59 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2089
  • 14:58 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2089.codfw.wmnet with OS bookworm
  • 14:58 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:56 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1072.eqiad.wmnet with reason: host reimage
  • 14:54 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1075.eqiad.wmnet with reason: host reimage
  • 14:53 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1074.eqiad.wmnet with reason: host reimage
  • 14:53 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, anzx: Continuing with sync
  • 14:53 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1072.eqiad.wmnet with reason: host reimage
  • 14:51 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, anzx: Backport for idwikivoyage: add logo, wordmark (T381080) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:47 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for idwikivoyage: add logo, wordmark (T381080)
  • 14:46 bking@cumin2002: START - Cookbook sre.hosts.provision for host wdqs1025.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:44 jayme@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:44 jayme@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:39 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for Translate: Enable message group subscription for 6 wikis (T372386) (duration: 14m 34s)
  • 14:36 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1075.eqiad.wmnet with OS bookworm
  • 14:35 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1074.eqiad.wmnet with OS bookworm
  • 14:35 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1073.eqiad.wmnet with OS bookworm
  • 14:35 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1072.eqiad.wmnet with OS bookworm
  • 14:34 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker[2089-2090].codfw.wmnet with reason: reimage
  • 14:34 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wikikube-worker[2089-2090].codfw.wmnet with reason: reimage
  • 14:34 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2089-2090].codfw.wmnet
  • 14:33 lucaswerkmeister-wmde@deploy2002: abi, lucaswerkmeister-wmde: Continuing with sync
  • 14:33 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2089-2090].codfw.wmnet
  • 14:29 lucaswerkmeister-wmde@deploy2002: abi, lucaswerkmeister-wmde: Backport for Translate: Enable message group subscription for 6 wikis (T372386) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:25 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1072.eqiad.wmnet wikikube-worker1073.eqiad.wmnet wikikube-worker1074.eqiad.wmnet wikikube-worker1075.eqiad.wmnet on all recursors
  • 14:25 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1072.eqiad.wmnet wikikube-worker1073.eqiad.wmnet wikikube-worker1074.eqiad.wmnet wikikube-worker1075.eqiad.wmnet on all recursors
  • 14:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1050 to wikikube-worker1075
  • 14:24 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1075
  • 14:24 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for Translate: Enable message group subscription for 6 wikis (T372386)
  • 14:24 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1075
  • 14:24 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:24 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1050 to wikikube-worker1075 - jelto@cumin1002"
  • 14:23 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for Add Metrics Platform stream configuration for translate_extension (T364460) (duration: 17m 12s)
  • 14:23 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1050 to wikikube-worker1075 - jelto@cumin1002"
  • 14:19 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 14:18 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1050 to wikikube-worker1075
  • 14:17 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1049 to wikikube-worker1074
  • 14:17 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1074
  • 14:16 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, wangombe: Continuing with sync
  • 14:16 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1074
  • 14:16 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:16 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1049 to wikikube-worker1074 - jelto@cumin1002"
  • 14:15 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1049 to wikikube-worker1074 - jelto@cumin1002"
  • 14:12 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 14:12 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1049 to wikikube-worker1074
  • 14:11 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, wangombe: Backport for Add Metrics Platform stream configuration for translate_extension (T364460) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1048 to wikikube-worker1073
  • 14:10 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1073
  • 14:10 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1073
  • 14:10 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:10 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1048 to wikikube-worker1073 - jelto@cumin1002"
  • 14:09 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1048 to wikikube-worker1073 - jelto@cumin1002"
  • 14:06 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 14:06 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for Add Metrics Platform stream configuration for translate_extension (T364460)
  • 14:05 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1048 to wikikube-worker1073
  • 14:04 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1047 to wikikube-worker1072
  • 14:03 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1072
  • 14:03 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1072
  • 14:03 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:03 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1047 to wikikube-worker1072 - jelto@cumin1002"
  • 14:02 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1047 to wikikube-worker1072 - jelto@cumin1002"
  • 14:00 Lucas_WMDE: 'Updated the Wikidata property suggester with data from 20241125’s JSON dump: mwscript-k8s --attach -- extensions/PropertySuggester/maintenance/UpdateTable.php --wiki wikidatawiki --file php://stdin < wbs_propertypairs.csv # T377986, T376604'
  • 13:58 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 13:57 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1047 to wikikube-worker1072
  • 13:54 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1047-1050].eqiad.wmnet
  • 13:49 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1047-1050].eqiad.wmnet
  • 13:46 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2103-2106].codfw.wmnet
  • 13:46 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2103-2106].codfw.wmnet
  • 13:30 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1068,1070-1071].eqiad.wmnet
  • 13:30 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1068,1070-1071].eqiad.wmnet
  • 13:16 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 12:57 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1069.eqiad.wmnet with OS bookworm
  • 12:26 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2106.codfw.wmnet with OS bookworm
  • 12:22 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2105.codfw.wmnet with OS bookworm
  • 12:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1068.eqiad.wmnet with OS bookworm
  • 12:12 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2104.codfw.wmnet with OS bookworm
  • 12:08 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2103.codfw.wmnet with OS bookworm
  • 12:07 moritzm: installing reportbug bugfix updates
  • 12:06 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2106.codfw.wmnet with reason: host reimage
  • 12:05 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1069.eqiad.wmnet with OS bookworm
  • 12:04 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1069.eqiad.wmnet with OS bookworm
  • 12:02 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2105.codfw.wmnet with reason: host reimage
  • 11:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1071.eqiad.wmnet with OS bookworm
  • 11:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1068.eqiad.wmnet with reason: host reimage
  • 11:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1070.eqiad.wmnet with OS bookworm
  • 11:51 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2104.codfw.wmnet with reason: host reimage
  • 11:49 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1068.eqiad.wmnet with reason: host reimage
  • 11:48 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2103.codfw.wmnet with reason: host reimage
  • 11:46 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2106.codfw.wmnet with reason: host reimage
  • 11:45 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2105.codfw.wmnet with reason: host reimage
  • 11:45 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2104.codfw.wmnet with reason: host reimage
  • 11:45 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2103.codfw.wmnet with reason: host reimage
  • 11:42 root@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1210.eqiad.wmnet onto db1159.eqiad.wmnet
  • 11:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1071.eqiad.wmnet with reason: host reimage
  • 11:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1070.eqiad.wmnet with reason: host reimage
  • 11:32 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1068.eqiad.wmnet with OS bookworm
  • 11:30 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1071.eqiad.wmnet with reason: host reimage
  • 11:30 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1070.eqiad.wmnet with reason: host reimage
  • 11:27 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1068.eqiad.wmnet with OS bookworm
  • 11:27 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2106
  • 11:27 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2106
  • 11:26 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2106.codfw.wmnet with OS bookworm
  • 11:26 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2105
  • 11:26 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2105
  • 11:26 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2105.codfw.wmnet with OS bookworm
  • 11:25 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2104
  • 11:25 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2104
  • 11:25 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2104.codfw.wmnet with OS bookworm
  • 11:25 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2103
  • 11:25 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2103
  • 11:25 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2103.codfw.wmnet with OS bookworm
  • 11:21 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:21 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:21 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:21 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:12 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1071.eqiad.wmnet with OS bookworm
  • 11:12 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1070.eqiad.wmnet with OS bookworm
  • 11:11 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1069.eqiad.wmnet with OS bookworm
  • 11:11 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1068.eqiad.wmnet with OS bookworm
  • 11:05 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1068.eqiad.wmnet wikikube-worker1069.eqiad.wmnet wikikube-worker1070.eqiad.wmnet wikikube-worker1071.eqiad.wmnet on all recursors
  • 11:05 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1068.eqiad.wmnet wikikube-worker1069.eqiad.wmnet wikikube-worker1070.eqiad.wmnet wikikube-worker1071.eqiad.wmnet on all recursors
  • 11:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1046 to wikikube-worker1071
  • 11:04 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1071
  • 11:03 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1071
  • 11:03 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:03 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1046 to wikikube-worker1071 - jelto@cumin1002"
  • 11:03 root@cumin1002: START - Cookbook sre.mysql.clone of db1210.eqiad.wmnet onto db1159.eqiad.wmnet
  • 11:03 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1046 to wikikube-worker1071 - jelto@cumin1002"
  • 11:02 jayme@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:01 jayme@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:01 jayme@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:01 jayme@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: cloning
  • 11:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1210.eqiad.wmnet with reason: cloning
  • 11:00 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: cloning
  • 11:00 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: cloning
  • 10:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1210 to clone db1159 T381550', diff saved to https://phabricator.wikimedia.org/P71640 and previous config saved to /var/cache/conftool/dbconfig/20241209-105941-marostegui.json
  • 10:59 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 10:59 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1046 to wikikube-worker1071
  • 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1045 to wikikube-worker1070
  • 10:57 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1070
  • 10:56 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1070
  • 10:56 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:56 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1045 to wikikube-worker1070 - jelto@cumin1002"
  • 10:56 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1045 to wikikube-worker1070 - jelto@cumin1002"
  • 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2024 to clone es2045', diff saved to https://phabricator.wikimedia.org/P71639 and previous config saved to /var/cache/conftool/dbconfig/20241209-105508-marostegui.json
  • 10:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es2024.codfw.wmnet with reason: cloning
  • 10:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es2024.codfw.wmnet with reason: cloning
  • 10:52 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 10:52 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1045 to wikikube-worker1070
  • 10:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1044 to wikikube-worker1069
  • 10:51 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1069
  • 10:50 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1069
  • 10:50 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:50 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1044 to wikikube-worker1069 - jelto@cumin1002"
  • 10:49 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1044 to wikikube-worker1069 - jelto@cumin1002"
  • 10:45 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 10:45 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1044 to wikikube-worker1069
  • 10:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1043 to wikikube-worker1068
  • 10:44 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1068
  • 10:42 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1068
  • 10:42 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:42 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1043 to wikikube-worker1068 - jelto@cumin1002"
  • 10:42 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1043 to wikikube-worker1068 - jelto@cumin1002"
  • 10:41 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2103-2106].codfw.wmnet
  • 10:39 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2103-2106].codfw.wmnet
  • 10:38 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 10:38 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker[2103-2106].codfw.wmnet with reason: reimage
  • 10:38 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1043 to wikikube-worker1068
  • 10:38 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wikikube-worker[2103-2106].codfw.wmnet with reason: reimage
  • 10:36 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1043-1046].eqiad.wmnet
  • 10:35 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2074-2075,2091,2124].codfw.wmnet
  • 10:35 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2074-2075,2091,2124].codfw.wmnet
  • 10:34 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1043-1046].eqiad.wmnet
  • 10:30 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2091.codfw.wmnet with OS bookworm
  • 10:25 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2075.codfw.wmnet with OS bookworm
  • 10:23 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2074.codfw.wmnet with OS bookworm
  • 10:20 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1064-1067].eqiad.wmnet
  • 10:20 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1064-1067].eqiad.wmnet
  • 10:10 moritzm: rebalance Ganeti cluster in codfw/A following server refresh T376594
  • 10:10 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2091.codfw.wmnet with reason: host reimage
  • 10:06 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 10:06 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2075.codfw.wmnet with reason: host reimage
  • 10:04 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1065.eqiad.wmnet with OS bookworm
  • 10:03 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2074.codfw.wmnet with reason: host reimage
  • 10:02 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2124.codfw.wmnet with OS bookworm
  • 10:00 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2091.codfw.wmnet with reason: host reimage
  • 09:59 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2075.codfw.wmnet with reason: host reimage
  • 09:59 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2074.codfw.wmnet with reason: host reimage
  • 09:56 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1067.eqiad.wmnet with OS bookworm
  • 09:54 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1066.eqiad.wmnet with OS bookworm
  • 09:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1064.eqiad.wmnet with OS bookworm
  • 09:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1065.eqiad.wmnet with reason: host reimage
  • 09:42 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2124.codfw.wmnet with reason: host reimage
  • 09:40 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2075.codfw.wmnet with OS bookworm
  • 09:40 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2074.codfw.wmnet with OS bookworm
  • 09:40 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2091.codfw.wmnet with OS bookworm
  • 09:39 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:39 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:39 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:38 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2124.codfw.wmnet with reason: host reimage
  • 09:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1067.eqiad.wmnet with reason: host reimage
  • 09:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1066.eqiad.wmnet with reason: host reimage
  • 09:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1064.eqiad.wmnet with reason: host reimage
  • 09:30 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1067.eqiad.wmnet with reason: host reimage
  • 09:29 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1066.eqiad.wmnet with reason: host reimage
  • 09:28 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1065.eqiad.wmnet with reason: host reimage
  • 09:28 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1064.eqiad.wmnet with reason: host reimage
  • 09:18 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2124.codfw.wmnet with OS bookworm
  • 09:16 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2124.codfw.wmnet with OS bookworm
  • 09:16 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2124.codfw.wmnet with OS bookworm
  • 09:14 jayme@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:13 jayme@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:13 jayme@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:12 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1067.eqiad.wmnet with OS bookworm
  • 09:12 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1066.eqiad.wmnet with OS bookworm
  • 09:10 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1065.eqiad.wmnet with OS bookworm
  • 09:10 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1064.eqiad.wmnet with OS bookworm
  • 09:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1064.eqiad.wmnet wikikube-worker1065.eqiad.wmnet wikikube-worker1066.eqiad.wmnet wikikube-worker1067.eqiad.wmnet on all recursors
  • 09:07 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1064.eqiad.wmnet wikikube-worker1065.eqiad.wmnet wikikube-worker1066.eqiad.wmnet wikikube-worker1067.eqiad.wmnet on all recursors
  • 09:07 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1042 to wikikube-worker1067
  • 09:06 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1067
  • 09:05 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1067
  • 09:04 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:04 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1042 to wikikube-worker1067 - jelto@cumin1002"
  • 09:04 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1042 to wikikube-worker1067 - jelto@cumin1002"
  • 09:04 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2074-2075,2091,2124].codfw.wmnet
  • 09:02 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2074-2075,2091,2124].codfw.wmnet
  • 09:01 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker[2074-2075,2091,2124].codfw.wmnet with reason: reimage
  • 09:01 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 09:00 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wikikube-worker[2074-2075,2091,2124].codfw.wmnet with reason: reimage
  • 09:00 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1042 to wikikube-worker1067
  • 08:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1041 to wikikube-worker1066
  • 08:59 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1066
  • 08:58 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1066
  • 08:58 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:58 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1041 to wikikube-worker1066 - jelto@cumin1002"
  • 08:57 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1041 to wikikube-worker1066 - jelto@cumin1002"
  • 08:54 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 08:53 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1041 to wikikube-worker1066
  • 08:53 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1040 to wikikube-worker1065
  • 08:52 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1065
  • 08:50 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1065
  • 08:50 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:50 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1040 to wikikube-worker1065 - jelto@cumin1002"
  • 08:50 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1040 to wikikube-worker1065 - jelto@cumin1002"
  • 08:46 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 08:45 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1040 to wikikube-worker1065
  • 08:43 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1039 to wikikube-worker1064
  • 08:42 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1064
  • 08:41 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:40 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1064
  • 08:40 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:40 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1039 to wikikube-worker1064 - jelto@cumin1002"
  • 08:39 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1039 to wikikube-worker1064 - jelto@cumin1002"
  • 08:36 elukey@cumin2002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:35 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 08:35 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1039 to wikikube-worker1064
  • 08:35 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:34 elukey@cumin2002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1039-1042].eqiad.wmnet
  • 08:29 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1039-1042].eqiad.wmnet
  • 07:34 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1056.eqiad.wmnet
  • 07:34 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1056.eqiad.wmnet
  • 07:18 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 06:29 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 06:28 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 05:54 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 05:53 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 05:41 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 05:40 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 05:39 tstarling@deploy2002: Finished deploy [restbase/deploy@8184836]: also deploy to restbase2036-9 T380726 T377896 (duration: 16m 06s)
  • 05:23 tstarling@deploy2002: Started deploy [restbase/deploy@8184836]: also deploy to restbase2036-9 T380726 T377896
  • 04:45 tstarling@deploy2002: Finished deploy [restbase/deploy@0531d4e]: try again after removing decom servers T380790 T380726 (duration: 14m 36s)
  • 04:31 tstarling@deploy2002: Started deploy [restbase/deploy@0531d4e]: try again after removing decom servers T380790 T380726
  • 04:29 tstarling@deploy2002: Finished deploy [restbase/deploy@27f4a8e]: try again, seems like restbase2026 at least was skipped T380726 (duration: 09m 00s)
  • 04:20 tstarling@deploy2002: Started deploy [restbase/deploy@27f4a8e]: try again, seems like restbase2026 at least was skipped T380726
  • 04:08 tstarling@deploy2002: Finished deploy [restbase/deploy@27f4a8e]: add 3 wikis T380726 (duration: 10m 46s)
  • 03:58 tstarling@deploy2002: Started deploy [restbase/deploy@27f4a8e]: add 3 wikis T380726
  • 03:55 tstarling@deploy2002: Finished deploy [restbase/deploy@6d0b97e]: no-op test deploy (duration: 11m 22s)
  • 03:44 tstarling@deploy2002: Started deploy [restbase/deploy@6d0b97e]: no-op test deploy
  • 03:30 tstarling@deploy2002: Finished scap sync-world: Backport for Prepare for migration of the Interwiki extension to core (T33951) (duration: 31m 17s)
  • 03:20 tstarling@deploy2002: tstarling: Continuing with sync
  • 03:10 tstarling@deploy2002: tstarling: Backport for Prepare for migration of the Interwiki extension to core (T33951) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 02:59 tstarling@deploy2002: Started scap sync-world: Backport for Prepare for migration of the Interwiki extension to core (T33951)

2024-12-08

  • 19:25 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 19:24 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply

2024-12-07

  • 00:33 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:32 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply

2024-12-06

  • 23:18 mutante: clouddumps1001/clouddumps1002: rm /srv/dumps/xmldatadumps/public/other/misc/phabricator_public.dump - an uncompressed old file from Sep 2023 - normal dumps are gzipped and current
  • 22:33 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 22:33 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 20:29 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1025.eqiad.wmnet with OS bullseye
  • 20:12 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1025.eqiad.wmnet with reason: host reimage
  • 20:08 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1025.eqiad.wmnet with reason: host reimage
  • 19:40 bking@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs1025.eqiad.wmnet with OS bullseye
  • 19:05 bking@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1025.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:00 bking@cumin2002: START - Cookbook sre.hosts.provision for host wdqs1025.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 17:31 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1057.eqiad.wmnet with OS bookworm
  • 17:29 topranks: splitting codfw -> eqsin traffic over path via ulsfo as direct link is saturated
  • 17:08 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1058-1063].eqiad.wmnet
  • 17:08 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1058-1063].eqiad.wmnet
  • 17:08 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1059.eqiad.wmnet with OS bookworm
  • 16:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1059.eqiad.wmnet with reason: host reimage
  • 16:45 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1059.eqiad.wmnet with reason: host reimage
  • 16:30 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1061.eqiad.wmnet with OS bookworm
  • 16:29 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1059.eqiad.wmnet with OS bookworm
  • 16:29 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1059.eqiad.wmnet with OS bookworm
  • 16:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1063.eqiad.wmnet with OS bookworm
  • 16:24 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1062.eqiad.wmnet with OS bookworm
  • 16:20 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1060.eqiad.wmnet with OS bookworm
  • 16:17 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1058.eqiad.wmnet with OS bookworm
  • 16:12 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1061.eqiad.wmnet with reason: host reimage
  • 16:11 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1057.eqiad.wmnet with OS bookworm
  • 16:10 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1057.eqiad.wmnet with OS bookworm
  • 16:09 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1063.eqiad.wmnet with reason: host reimage
  • 16:05 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1062.eqiad.wmnet with reason: host reimage
  • 16:01 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1060.eqiad.wmnet with reason: host reimage
  • 15:59 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1063.eqiad.wmnet with reason: host reimage
  • 15:58 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1062.eqiad.wmnet with reason: host reimage
  • 15:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1058.eqiad.wmnet with reason: host reimage
  • 15:58 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1061.eqiad.wmnet with reason: host reimage
  • 15:57 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1060.eqiad.wmnet with reason: host reimage
  • 15:54 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1058.eqiad.wmnet with reason: host reimage
  • 15:43 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1063.eqiad.wmnet with OS bookworm
  • 15:43 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1062.eqiad.wmnet with OS bookworm
  • 15:42 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1061.eqiad.wmnet with OS bookworm
  • 15:41 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1060.eqiad.wmnet with OS bookworm
  • 15:41 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1059.eqiad.wmnet with OS bookworm
  • 15:39 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1058.eqiad.wmnet with OS bookworm
  • 15:36 kamila@cumin1002: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1058.eqiad.wmnet
  • 15:36 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1058.eqiad.wmnet with OS bullseye
  • 15:36 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1058.eqiad.wmnet with OS bullseye
  • 15:36 kamila@cumin1002: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1058.eqiad.wmnet
  • 15:34 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1058.eqiad.wmnet wikikube-worker1059.eqiad.wmnet wikikube-worker1060.eqiad.wmnet wikikube-worker1061.eqiad.wmnet wikikube-worker1062.eqiad.wmnet wikikube-worker1063.eqiad.wmnet on all recursors
  • 15:34 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1058.eqiad.wmnet wikikube-worker1059.eqiad.wmnet wikikube-worker1060.eqiad.wmnet wikikube-worker1061.eqiad.wmnet wikikube-worker1062.eqiad.wmnet wikikube-worker1063.eqiad.wmnet on all recursors
  • 15:33 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1434 to wikikube-worker1062
  • 15:33 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1062
  • 15:32 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1062
  • 15:32 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:31 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1435 to wikikube-worker1063
  • 15:30 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1063
  • 15:29 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 15:29 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1063
  • 15:29 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1433 to wikikube-worker1061
  • 15:28 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1061
  • 15:27 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 15:27 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1061
  • 15:26 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:26 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1433 to wikikube-worker1061 - kamila@cumin1002"
  • 15:26 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1433 to wikikube-worker1061 - kamila@cumin1002"
  • 15:24 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1432 to wikikube-worker1060
  • 15:23 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1060
  • 15:22 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 15:22 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1060
  • 15:22 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:22 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1432 to wikikube-worker1060 - kamila@cumin1002"
  • 15:22 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1432 to wikikube-worker1060 - kamila@cumin1002"
  • 15:20 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1435 to wikikube-worker1063
  • 15:20 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1434 to wikikube-worker1062
  • 15:20 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1433 to wikikube-worker1061
  • 15:19 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1431 to wikikube-worker1059
  • 15:19 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1059
  • 15:18 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 15:18 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1059
  • 15:18 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:18 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1431 to wikikube-worker1059 - kamila@cumin1002"
  • 15:18 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1432 to wikikube-worker1060
  • 15:18 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1431 to wikikube-worker1059 - kamila@cumin1002"
  • 15:15 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1430 to wikikube-worker1058
  • 15:14 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1058
  • 15:14 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 15:13 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1058
  • 15:13 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:13 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1430 to wikikube-worker1058 - kamila@cumin1002"
  • 15:13 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1430 to wikikube-worker1058 - kamila@cumin1002"
  • 15:10 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1431 to wikikube-worker1059
  • 15:08 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 15:08 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1430 to wikikube-worker1058
  • 15:05 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1430-1435].eqiad.wmnet
  • 15:02 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1430-1435].eqiad.wmnet
  • 14:50 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1057.eqiad.wmnet with OS bookworm
  • 14:49 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1057.eqiad.wmnet with OS bookworm
  • 14:27 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: ganeti1009.eqiad.wmnet
  • 14:27 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: ganeti1009.eqiad.wmnet
  • 14:20 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1020.eqiad.wmnet
  • 14:20 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:20 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1020.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 14:19 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1020.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 14:11 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 14:10 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1056.eqiad.wmnet with OS bookworm
  • 13:56 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1020.eqiad.wmnet
  • 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1018.eqiad.wmnet
  • 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1018.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 13:50 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1056.eqiad.wmnet with reason: host reimage
  • 13:47 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1018.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 13:46 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1056.eqiad.wmnet with reason: host reimage
  • 13:43 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 13:36 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1018.eqiad.wmnet
  • 13:35 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1017.eqiad.wmnet
  • 13:35 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1017.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 13:35 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1017.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 13:31 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 13:29 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1057.eqiad.wmnet with OS bookworm
  • 13:29 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1056.eqiad.wmnet with OS bookworm
  • 13:25 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1056.eqiad.wmnet wikikube-worker1057.eqiad.wmnet on all recursors
  • 13:25 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1056.eqiad.wmnet wikikube-worker1057.eqiad.wmnet on all recursors
  • 13:21 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1017.eqiad.wmnet
  • 13:19 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1038 to wikikube-worker1057
  • 13:19 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1057
  • 13:18 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1057
  • 13:18 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:17 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1038 to wikikube-worker1057 - jelto@cumin1002"
  • 13:17 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1038 to wikikube-worker1057 - jelto@cumin1002"
  • 13:13 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 13:13 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1038 to wikikube-worker1057
  • 13:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1037 to wikikube-worker1056
  • 13:11 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1016.eqiad.wmnet
  • 13:11 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:11 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 13:11 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1056
  • 13:11 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 13:10 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1056
  • 13:10 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:10 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1037 to wikikube-worker1056 - jelto@cumin1002"
  • 13:07 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 13:01 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1037 to wikikube-worker1056 - jelto@cumin1002"
  • 12:58 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1016.eqiad.wmnet
  • 12:56 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 12:56 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1037 to wikikube-worker1056
  • 12:48 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1037-1038].eqiad.wmnet
  • 12:47 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1037-1038].eqiad.wmnet
  • 12:40 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts ganeti1009.eqiad.wmnet
  • 12:40 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:40 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1009.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 12:39 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1009.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 12:36 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 12:21 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1009.eqiad.wmnet
  • 12:15 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1054-1055].eqiad.wmnet
  • 12:15 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1054-1055].eqiad.wmnet
  • 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host build2002.codfw.wmnet with OS bookworm
  • 11:58 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 11:56 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1055.eqiad.wmnet with OS bookworm
  • 11:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1054.eqiad.wmnet with OS bookworm
  • 11:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on build2002.codfw.wmnet with reason: host reimage
  • 11:48 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on build2002.codfw.wmnet with reason: host reimage
  • 11:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1055.eqiad.wmnet with reason: host reimage
  • 11:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1054.eqiad.wmnet with reason: host reimage
  • 11:32 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1055.eqiad.wmnet with reason: host reimage
  • 11:30 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1054.eqiad.wmnet with reason: host reimage
  • 11:30 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host build2002.codfw.wmnet with OS bookworm
  • 11:05 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1055.eqiad.wmnet with OS bookworm
  • 11:05 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1054.eqiad.wmnet with OS bookworm
  • 11:00 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1054.eqiad.wmnet wikikube-worker1055.eqiad.wmnet on all recursors
  • 11:00 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1054.eqiad.wmnet wikikube-worker1055.eqiad.wmnet on all recursors
  • 10:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1036 to wikikube-worker1055
  • 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1055
  • 10:57 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1055
  • 10:57 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:57 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1036 to wikikube-worker1055 - jelto@cumin1002"
  • 10:57 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1036 to wikikube-worker1055 - jelto@cumin1002"
  • 10:53 jmm@cumin2002: END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2042.codfw.wmnet to cluster codfw and group D
  • 10:53 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 10:53 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1036 to wikikube-worker1055
  • 10:52 jmm@cumin2002: START - Cookbook sre.ganeti.addnode for new host ganeti2042.codfw.wmnet to cluster codfw and group D
  • 10:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1035 to wikikube-worker1054
  • 10:48 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1054
  • 10:47 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1054
  • 10:47 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:47 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1035 to wikikube-worker1054 - jelto@cumin1002"
  • 10:47 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1035 to wikikube-worker1054 - jelto@cumin1002"
  • 10:43 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 10:43 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1035 to wikikube-worker1054
  • 10:41 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1035-1036].eqiad.wmnet
  • 10:39 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1035-1036].eqiad.wmnet
  • 10:27 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1052-1053].eqiad.wmnet
  • 10:27 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1052-1053].eqiad.wmnet
  • 10:11 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 10:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1053.eqiad.wmnet with OS bookworm
  • 10:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1052.eqiad.wmnet with OS bookworm
  • 09:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1053.eqiad.wmnet with reason: host reimage
  • 09:48 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1052.eqiad.wmnet with reason: host reimage
  • 09:46 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1053.eqiad.wmnet with reason: host reimage
  • 09:45 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1052.eqiad.wmnet with reason: host reimage
  • 09:28 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1053.eqiad.wmnet with OS bookworm
  • 09:28 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1052.eqiad.wmnet with OS bookworm
  • 09:24 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1052.eqiad.wmnet wikikube-worker1053.eqiad.wmnet on all recursors
  • 09:24 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1052.eqiad.wmnet wikikube-worker1053.eqiad.wmnet on all recursors
  • 09:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1034 to wikikube-worker1053
  • 09:22 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1053
  • 09:21 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1053
  • 09:21 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:21 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1034 to wikikube-worker1053 - jelto@cumin1002"
  • 09:20 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1034 to wikikube-worker1053 - jelto@cumin1002"
  • 09:16 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 09:16 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1034 to wikikube-worker1053
  • 09:16 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1033 to wikikube-worker1052
  • 09:15 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1052
  • 09:14 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1052
  • 09:13 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:13 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1033 to wikikube-worker1052 - jelto@cumin1002"
  • 09:13 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1033 to wikikube-worker1052 - jelto@cumin1002"
  • 09:09 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 09:09 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1033 to wikikube-worker1052
  • 09:03 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1033-1034].eqiad.wmnet
  • 09:02 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1033-1034].eqiad.wmnet
  • 09:00 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:55 elukey@cumin2002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:43 elukey@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:33 moritzm: uploaded ruby-sys-filesystem 1.4.3-1~wmf11u1 to component/puppet7 for Bullseye (needed by the mountpoints fact in facter 4) T381538
  • 08:33 elukey@cumin2002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:30 elukey@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:30 elukey@cumin2002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:26 hashar@deploy2002: Finished deploy [gerrit/gerrit@ac50ebe]: Reinstate the banner for the developer survey (duration: 00m 11s)
  • 08:26 hashar@deploy2002: Started deploy [gerrit/gerrit@ac50ebe]: Reinstate the banner for the developer survey
  • 08:18 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:18 elukey@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:17 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:17 elukey@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:17 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:16 elukey@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 07:21 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 100%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71633 and previous config saved to /var/cache/conftool/dbconfig/20241206-072120-root.json
  • 07:20 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 07:19 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 07:07 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 07:06 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 07:06 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 75%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71632 and previous config saved to /var/cache/conftool/dbconfig/20241206-070614-root.json
  • 07:05 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 07:04 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 06:36 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 25%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71630 and previous config saved to /var/cache/conftool/dbconfig/20241206-063603-root.json
  • 06:25 marostegui@cumin1002: dbctl commit (dc=all): 'es2023 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P71629 and previous config saved to /var/cache/conftool/dbconfig/20241206-062527-root.json
  • 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 10%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71628 and previous config saved to /var/cache/conftool/dbconfig/20241206-062058-root.json
  • 06:10 marostegui@cumin1002: dbctl commit (dc=all): 'es2023 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P71627 and previous config saved to /var/cache/conftool/dbconfig/20241206-061021-root.json
  • 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 5%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71626 and previous config saved to /var/cache/conftool/dbconfig/20241206-060552-root.json
  • 05:55 marostegui@cumin1002: dbctl commit (dc=all): 'es2023 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P71625 and previous config saved to /var/cache/conftool/dbconfig/20241206-055516-root.json
  • 05:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1154.eqiad.wmnet with reason: Alter table
  • 05:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1154.eqiad.wmnet with reason: Alter table
  • 05:50 marostegui@cumin1002: dbctl commit (dc=all): 'es2044 (re)pooling @ 1%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71624 and previous config saved to /var/cache/conftool/dbconfig/20241206-055047-root.json
  • 05:44 marostegui@cumin1002: dbctl commit (dc=all): 'Add es2044 to dbctl depooled T381259', diff saved to https://phabricator.wikimedia.org/P71623 and previous config saved to /var/cache/conftool/dbconfig/20241206-054457-marostegui.json
  • 05:40 marostegui@cumin1002: dbctl commit (dc=all): 'es2023 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P71622 and previous config saved to /var/cache/conftool/dbconfig/20241206-054010-root.json
  • 01:50 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:49 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:49 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:48 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:47 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:47 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:47 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:47 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:32 TimStarling: on mwmaint2002: deleting MediaWiki:Sitesupport-url pages per T379205
  • 01:16 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:15 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:19 urbanecm: mwmaint2002: foreachwikiindblist growthexperiments extensions/GrowthExperiments/maintenance/revalidateLinkRecommendations.php --all --verbose # T380455
  • 00:19 urbanecm: Delete previously-started mwscript-k8s instances of revalidateLinkRecommendations.php (T380455)

2024-12-05

  • 23:26 jhathaway: looking at puppet failures on an-workers
  • 23:23 urbanecm: Start revalidateLinkRecommendations.php for Add Link-enabled wikis via mwscript-k8s (T380455)
  • 22:53 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2020.codfw.wmnet w/ force delete existing files, repooling both afterwards
  • 22:00 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2020.codfw.wmnet w/ force delete existing files, repooling both afterwards
  • 21:24 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
  • 21:23 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
  • 21:21 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
  • 21:21 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
  • 21:20 cdanis@deploy2002: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
  • 21:19 cdanis@deploy2002: helmfile [staging] START helmfile.d/services/chart-renderer: apply
  • 20:27 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2019.codfw.wmnet w/ force delete existing files, repooling both afterwards
  • 19:34 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2019.codfw.wmnet w/ force delete existing files, repooling both afterwards
  • 19:28 jhuneidi@deploy2002: rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.6 refs T375665
  • 18:44 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1043.eqiad.wmnet with OS bookworm
  • 18:13 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1048-1049,1051].eqiad.wmnet
  • 18:13 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1048-1049,1051].eqiad.wmnet
  • 18:00 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS bookworm
  • 18:00 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es1043.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 17:59 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 17:57 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1051.eqiad.wmnet with OS bookworm
  • 17:53 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host es1043.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 17:53 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1050.eqiad.wmnet with OS bookworm
  • 17:41 pt1979@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:40 pt1979@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1051.eqiad.wmnet with reason: host reimage
  • 17:36 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1051.eqiad.wmnet with reason: host reimage
  • 17:36 jhathaway: upgrading facter on bullseye puppet nodes
  • 17:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1050.eqiad.wmnet with reason: host reimage
  • 17:30 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1050.eqiad.wmnet with reason: host reimage
  • 17:25 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1125.eqiad.wmnet with reason: Test setup should not alert
  • 17:25 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1125.eqiad.wmnet with reason: Test setup should not alert
  • 17:21 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:21 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:19 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 17:19 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1051.eqiad.wmnet with OS bookworm
  • 17:19 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 17:12 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1050.eqiad.wmnet with OS bookworm
  • 17:11 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1051.eqiad.wmnet with OS bookworm
  • 17:11 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1050.eqiad.wmnet with OS bookworm
  • 17:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1049.eqiad.wmnet with OS bookworm
  • 17:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1048.eqiad.wmnet with OS bookworm
  • 17:04 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:03 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:03 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:03 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:01 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:01 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:00 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 17:00 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1049.eqiad.wmnet with reason: host reimage
  • 16:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1048.eqiad.wmnet with reason: host reimage
  • 16:46 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudelastic1012.eqiad.wmnet with OS bullseye
  • 16:45 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudelastic1011.eqiad.wmnet with OS bullseye
  • 16:43 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1049.eqiad.wmnet with reason: host reimage
  • 16:43 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1048.eqiad.wmnet with reason: host reimage
  • 16:36 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:36 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:31 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:31 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:30 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply
  • 16:29 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply
  • 16:29 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/benthos-cache-invalidator: apply
  • 16:29 jiji@deploy2002: helmfile [staging] START helmfile.d/services/benthos-cache-invalidator: apply
  • 16:29 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
  • 16:28 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/changeprop: apply
  • 16:28 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/changeprop: apply
  • 16:28 jiji@deploy2002: helmfile [staging] START helmfile.d/services/changeprop: apply
  • 16:27 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:27 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:26 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1051.eqiad.wmnet with OS bookworm
  • 16:26 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1050.eqiad.wmnet with OS bookworm
  • 16:26 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1049.eqiad.wmnet with OS bookworm
  • 16:25 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1048.eqiad.wmnet with OS bookworm
  • 16:23 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1048.eqiad.wmnet wikikube-worker1049.eqiad.wmnet wikikube-worker1050.eqiad.wmnet wikikube-worker1051.eqiad.wmnet on all recursors
  • 16:23 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1048.eqiad.wmnet wikikube-worker1049.eqiad.wmnet wikikube-worker1050.eqiad.wmnet wikikube-worker1051.eqiad.wmnet on all recursors
  • 16:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1032 to wikikube-worker1051
  • 16:22 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:22 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:22 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1051
  • 16:21 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1051
  • 16:21 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:21 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1032 to wikikube-worker1051 - jelto@cumin1002"
  • 16:21 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1032 to wikikube-worker1051 - jelto@cumin1002"
  • 16:20 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply
  • 16:20 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:20 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:20 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply
  • 16:20 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/eventgate-main: apply
  • 16:19 jiji@deploy2002: helmfile [staging] START helmfile.d/services/eventgate-main: apply
  • 16:19 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
  • 16:18 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
  • 16:18 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
  • 16:18 jiji@deploy2002: helmfile [staging] START helmfile.d/services/eventstreams: apply
  • 16:17 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:17 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:17 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1011.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:17 jclark@cumin1002: START - Cookbook sre.hosts.provision for host cloudelastic1012.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:15 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 16:15 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:15 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for cloudelastic - jclark@cumin1002"
  • 16:14 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for cloudelastic - jclark@cumin1002"
  • 16:14 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 16:14 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 16:13 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1032 to wikikube-worker1051
  • 16:13 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 16:12 jiji@deploy2002: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
  • 16:11 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply
  • 16:11 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2018.codfw.wmnet w/ force delete existing files, repooling source-only afterwards
  • 16:11 jiji@deploy2002: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply
  • 16:11 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/benthos-cache-invalidator: apply
  • 16:11 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply
  • 16:11 jiji@deploy2002: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply
  • 16:11 jclark@cumin1002: START - Cookbook sre.dns.netbox
  • 16:11 jiji@deploy2002: helmfile [staging] START helmfile.d/services/benthos-cache-invalidator: apply
  • 16:08 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1010.eqiad.wmnet
  • 16:08 jiji@cumin1002: START - Cookbook sre.hosts.remove-downtime for kafka-main1010.eqiad.wmnet
  • 16:08 jiji@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 16:07 jiji@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 16:07 jiji@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 16:07 jiji@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 16:07 jiji@deploy2002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 16:07 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1026.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 16:06 jiji@deploy2002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 16:06 jiji@deploy2002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 16:06 jiji@deploy2002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 16:06 jiji@deploy2002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 16:06 jiji@deploy2002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 16:05 jiji@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 16:05 jiji@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 16:05 jiji@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 16:05 jiji@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 16:05 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 16:04 jiji@deploy2002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 16:04 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 16:04 jiji@deploy2002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 16:02 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal scholarly tier) xfer scholarly_articles from wdqs1023.eqiad.wmnet -> wdqs1027.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 15:58 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1031 to wikikube-worker1050
  • 15:58 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1050
  • 15:56 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1050
  • 15:56 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:56 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1031 to wikikube-worker1050 - jelto@cumin1002"
  • 15:55 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1031 to wikikube-worker1050 - jelto@cumin1002"
  • 15:55 moritzm: installing nghttp2 security updates
  • 15:52 jiji@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad
  • 15:52 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 15:51 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1031 to wikikube-worker1050
  • 15:41 jiji@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad
  • 15:29 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1030 to wikikube-worker1049
  • 15:28 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1049
  • 15:27 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1049
  • 15:27 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:27 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1030 to wikikube-worker1049 - jelto@cumin1002"
  • 15:26 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1030 to wikikube-worker1049 - jelto@cumin1002"
  • 15:24 moritzm: installing postgresql security updates
  • 15:22 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 15:22 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1030 to wikikube-worker1049
  • 15:20 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1029 to wikikube-worker1048
  • 15:20 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1048
  • 15:18 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1048
  • 15:18 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:18 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1029 to wikikube-worker1048 - jelto@cumin1002"
  • 15:18 dbrant@deploy2002: Finished scap sync-world: Backport for Enable Parsoid Fragment mode on Chart pilot wikis (T381436 T381312 T380758) (duration: 19m 51s)
  • 15:18 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1029 to wikikube-worker1048 - jelto@cumin1002"
  • 15:15 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1026.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 15:15 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal scholarly tier) xfer scholarly_articles from wdqs1023.eqiad.wmnet -> wdqs1027.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 15:15 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2018.codfw.wmnet w/ force delete existing files, repooling source-only afterwards
  • 15:13 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 15:12 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1029 to wikikube-worker1048
  • 15:10 dbrant@deploy2002: dbrant, cscott: Continuing with sync
  • 15:10 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1029-1032].eqiad.wmnet
  • 15:10 bking@cumin2002: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2018.codfw.wmnet w/ force delete existing files, repooling source-only afterwards
  • 15:10 bking@cumin2002: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T376150, initialize wdqs internal scholarly tier) xfer scholarly_articles from wdqs1023.eqiad.wmnet -> wdqs1027.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 15:10 bking@cumin2002: END (ERROR) - Cookbook sre.wdqs.data-transfer (exit_code=97) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1026.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 15:10 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1026.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 15:09 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal scholarly tier) xfer scholarly_articles from wdqs1023.eqiad.wmnet -> wdqs1027.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 15:09 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2018.codfw.wmnet w/ force delete existing files, repooling source-only afterwards
  • 15:08 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1029-1032].eqiad.wmnet
  • 15:07 dbrant@deploy2002: dbrant, cscott: Backport for Enable Parsoid Fragment mode on Chart pilot wikis (T381436 T381312 T380758) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 15:00 jhancock@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1043.eqiad.wmnet with OS bookworm
  • 14:58 dbrant@deploy2002: Started scap sync-world: Backport for Enable Parsoid Fragment mode on Chart pilot wikis (T381436 T381312 T380758)
  • 14:55 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1045.eqiad.wmnet with OS bookworm
  • 14:55 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
  • 14:53 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
  • 14:37 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1045.eqiad.wmnet with reason: host reimage
  • 14:36 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1046-1047].eqiad.wmnet
  • 14:36 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1046-1047].eqiad.wmnet
  • 14:34 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on es1045.eqiad.wmnet with reason: host reimage
  • 14:29 dbrant@deploy2002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 14:28 dbrant@deploy2002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 14:28 dbrant@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 14:27 dbrant@deploy2002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 14:20 dbrant@deploy2002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 14:20 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 14:20 dbrant@deploy2002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 14:20 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1047.eqiad.wmnet with OS bookworm
  • 14:18 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host es1045.eqiad.wmnet with OS bookworm
  • 14:17 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS bookworm
  • 14:16 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1046.eqiad.wmnet with OS bookworm
  • 14:04 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es1043.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 14:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1047.eqiad.wmnet with reason: host reimage
  • 13:59 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es1045.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 13:57 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1046.eqiad.wmnet with reason: host reimage
  • 13:54 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1047.eqiad.wmnet with reason: host reimage
  • 13:54 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1046.eqiad.wmnet with reason: host reimage
  • 13:53 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host es1043.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 13:53 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host es1045.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 13:42 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2178.codfw.wmnet
  • 13:42 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2178.codfw.wmnet
  • 13:36 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1047.eqiad.wmnet with OS bookworm
  • 13:36 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1046.eqiad.wmnet with OS bookworm
  • 13:32 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1046.eqiad.wmnet wikikube-worker1047.eqiad.wmnet on all recursors
  • 13:32 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1046.eqiad.wmnet wikikube-worker1047.eqiad.wmnet on all recursors
  • 13:23 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2178.codfw.wmnet with OS bookworm
  • 13:08 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on kafka-main[1005,1010].eqiad.wmnet with reason: Hardware refresh
  • 13:08 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on kafka-main[1005,1010].eqiad.wmnet with reason: Hardware refresh
  • 13:03 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2178.codfw.wmnet with reason: host reimage
  • 12:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1028 to wikikube-worker1047
  • 12:58 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1047
  • 12:57 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1047
  • 12:57 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:57 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1028 to wikikube-worker1047 - jelto@cumin1002"
  • 12:57 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1028 to wikikube-worker1047 - jelto@cumin1002"
  • 12:56 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2178.codfw.wmnet with reason: host reimage
  • 12:53 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 12:53 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1028 to wikikube-worker1047
  • 12:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1027 to wikikube-worker1046
  • 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1046
  • 12:50 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1046
  • 12:50 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:50 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1027 to wikikube-worker1046 - jelto@cumin1002"
  • 12:50 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1027 to wikikube-worker1046 - jelto@cumin1002"
  • 12:46 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 12:45 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1027 to wikikube-worker1046
  • 12:42 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2176-2177,2179].codfw.wmnet
  • 12:42 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2176-2177,2179].codfw.wmnet
  • 12:39 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1027-1028].eqiad.wmnet
  • 12:37 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2178
  • 12:37 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2178
  • 12:36 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2178
  • 12:36 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2178.codfw.wmnet 185.48.192.10.in-addr.arpa 5.8.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 12:36 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2178.codfw.wmnet 185.48.192.10.in-addr.arpa 5.8.1.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 12:36 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:36 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2178 - jayme@cumin2002"
  • 12:36 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2178 - jayme@cumin2002"
  • 12:36 jgleeson: payments updated from 119448ca to ab7e70ec
  • 12:35 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1027-1028].eqiad.wmnet
  • 12:26 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 12:16 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 12:16 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 12:16 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T371742)', diff saved to https://phabricator.wikimedia.org/P71620 and previous config saved to /var/cache/conftool/dbconfig/20241205-121609-ladsgroup.json
  • 12:15 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1038-1043].eqiad.wmnet
  • 12:15 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1038-1043].eqiad.wmnet
  • 12:06 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2176.codfw.wmnet with OS bookworm
  • 12:02 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2177.codfw.wmnet with OS bookworm
  • 12:01 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P71617 and previous config saved to /var/cache/conftool/dbconfig/20241205-120102-ladsgroup.json
  • 11:58 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2179.codfw.wmnet with OS bookworm
  • 11:46 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2176.codfw.wmnet with reason: host reimage
  • 11:45 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P71616 and previous config saved to /var/cache/conftool/dbconfig/20241205-114555-ladsgroup.json
  • 11:43 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2176.codfw.wmnet with reason: host reimage
  • 11:42 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2177.codfw.wmnet with reason: host reimage
  • 11:38 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2177.codfw.wmnet with reason: host reimage
  • 11:38 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2179.codfw.wmnet with reason: host reimage
  • 11:34 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2179.codfw.wmnet with reason: host reimage
  • 11:30 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T371742)', diff saved to https://phabricator.wikimedia.org/P71615 and previous config saved to /var/cache/conftool/dbconfig/20241205-113048-ladsgroup.json
  • 11:26 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2178
  • 11:25 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2176
  • 11:25 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2176
  • 11:25 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2176
  • 11:25 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2176.codfw.wmnet 81.48.192.10.in-addr.arpa 1.8.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:25 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2176.codfw.wmnet 81.48.192.10.in-addr.arpa 1.8.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:25 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:25 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2176 - jayme@cumin2002"
  • 11:24 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2176 - jayme@cumin2002"
  • 11:20 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 11:19 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2176
  • 11:19 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2177
  • 11:19 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2177
  • 11:19 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2177
  • 11:19 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2177.codfw.wmnet 83.48.192.10.in-addr.arpa 3.8.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:19 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2177.codfw.wmnet 83.48.192.10.in-addr.arpa 3.8.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:19 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:19 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2177 - jayme@cumin2002"
  • 11:19 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2177 - jayme@cumin2002"
  • 11:15 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2176.codfw.wmnet with OS bookworm
  • 11:15 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 11:15 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2176.codfw.wmnet with OS bookworm
  • 11:15 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2177
  • 11:15 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2179
  • 11:15 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2179
  • 11:15 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2179
  • 11:15 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2179.codfw.wmnet 207.48.192.10.in-addr.arpa 7.0.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:15 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2179.codfw.wmnet 207.48.192.10.in-addr.arpa 7.0.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:15 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:15 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2179 - jayme@cumin2002"
  • 11:15 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2179 - jayme@cumin2002"
  • 11:14 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2176.codfw.wmnet with OS bookworm
  • 11:13 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2177.codfw.wmnet with OS bookworm
  • 11:13 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2449 to wikikube-worker2177
  • 11:12 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2177
  • 11:12 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2178.codfw.wmnet with OS bookworm
  • 11:11 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2177
  • 11:11 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:11 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2449 to wikikube-worker2177 - jayme@cumin2002"
  • 11:11 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 11:11 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2179
  • 11:11 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2179.codfw.wmnet with OS bookworm
  • 11:10 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2179.codfw.wmnet with OS bookworm
  • 11:09 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2449 to wikikube-worker2177 - jayme@cumin2002"
  • 11:09 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2179.codfw.wmnet with OS bookworm
  • 11:05 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 11:05 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2449 to wikikube-worker2177
  • 10:54 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2451 to wikikube-worker2179
  • 10:54 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2179
  • 10:52 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2179
  • 10:52 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:51 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2450 to wikikube-worker2178
  • 10:50 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2178
  • 10:50 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:50 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2178
  • 10:50 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:50 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2450 to wikikube-worker2178 - jayme@cumin2002"
  • 10:50 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2450 to wikikube-worker2178 - jayme@cumin2002"
  • 10:47 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2448 to wikikube-worker2176
  • 10:46 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:46 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2176
  • 10:46 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2176
  • 10:46 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:46 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2448 to wikikube-worker2176 - jayme@cumin2002"
  • 10:45 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2448 to wikikube-worker2176 - jayme@cumin2002"
  • 10:43 dcausse: reindexed all wikidata entity schemas (T376252)
  • 10:42 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2451 to wikikube-worker2179
  • 10:42 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from mw2449 to wikikube-worker2177
  • 10:42 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2450 to wikikube-worker2178
  • 10:42 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2449 to wikikube-worker2177
  • 10:42 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:41 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2448 to wikikube-worker2176
  • 10:37 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2451.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:37 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2450.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:37 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2449.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:37 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2448.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:33 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1044-1045].eqiad.wmnet
  • 10:33 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1044-1045].eqiad.wmnet
  • 10:28 jelto: homer 'lsw1-f3-eqiad*' commit 'T377876'
  • 10:27 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1045.eqiad.wmnet with OS bookworm
  • 10:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1045.eqiad.wmnet with reason: host reimage
  • 10:04 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1045.eqiad.wmnet with reason: host reimage
  • 09:56 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2451.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:55 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T371742)', diff saved to https://phabricator.wikimedia.org/P71614 and previous config saved to /var/cache/conftool/dbconfig/20241205-095554-ladsgroup.json
  • 09:55 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 09:55 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2450.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:55 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 09:55 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2449.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:54 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2448.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:48 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1045.eqiad.wmnet with OS bookworm
  • 09:47 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1044.eqiad.wmnet with OS bookworm
  • 09:40 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2450-2451].codfw.wmnet
  • 09:39 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2450-2451].codfw.wmnet
  • 09:39 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mw[2448-2451].codfw.wmnet with reason: reimage
  • 09:38 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on mw[2448-2451].codfw.wmnet with reason: reimage
  • 09:38 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2448-2449].codfw.wmnet
  • 09:37 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2448-2449].codfw.wmnet
  • 09:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1044.eqiad.wmnet with reason: host reimage
  • 09:25 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1044.eqiad.wmnet with reason: host reimage
  • 09:20 jayme: destroyed unused expiring puppet certs - T381474
  • 09:15 fabfur: deploying haproxykafka also on magru and drmrs (T378578)
  • 09:09 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1044.eqiad.wmnet with OS bookworm
  • 09:06 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1044.eqiad.wmnet wikikube-worker1045.eqiad.wmnet on all recursors
  • 09:06 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1044.eqiad.wmnet wikikube-worker1045.eqiad.wmnet on all recursors
  • 09:04 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1026 to wikikube-worker1045
  • 09:04 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1045
  • 09:03 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1045
  • 09:03 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:03 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1026 to wikikube-worker1045 - jelto@cumin1002"
  • 09:03 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1026 to wikikube-worker1045 - jelto@cumin1002"
  • 08:58 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 08:58 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1026 to wikikube-worker1045
  • 08:58 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1025 to wikikube-worker1044
  • 08:57 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1044
  • 08:55 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1044
  • 08:55 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:55 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1025 to wikikube-worker1044 - jelto@cumin1002"
  • 08:54 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1025 to wikikube-worker1044 - jelto@cumin1002"
  • 08:49 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 08:49 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1025 to wikikube-worker1044
  • 08:47 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 08:46 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 08:46 moritzm: rebalance Ganeti eqiad/D following server refreshes
  • 08:08 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 08:07 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 08:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T371742)', diff saved to https://phabricator.wikimedia.org/P71611 and previous config saved to /var/cache/conftool/dbconfig/20241205-080745-ladsgroup.json
  • 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P71610 and previous config saved to /var/cache/conftool/dbconfig/20241205-075237-ladsgroup.json
  • 07:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P71609 and previous config saved to /var/cache/conftool/dbconfig/20241205-073730-ladsgroup.json
  • 07:36 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1025-1026].eqiad.wmnet
  • 07:32 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1025-1026].eqiad.wmnet
  • 07:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T371742)', diff saved to https://phabricator.wikimedia.org/P71608 and previous config saved to /var/cache/conftool/dbconfig/20241205-072223-ladsgroup.json
  • 07:16 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 06:31 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 100%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71607 and previous config saved to /var/cache/conftool/dbconfig/20241205-063132-root.json
  • 06:16 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 75%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71606 and previous config saved to /var/cache/conftool/dbconfig/20241205-061626-root.json
  • 06:06 marostegui@cumin1002: dbctl commit (dc=all): 'es2024 (re)pooling @ 100%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71605 and previous config saved to /var/cache/conftool/dbconfig/20241205-060631-root.json
  • 06:06 marostegui@cumin1002: dbctl commit (dc=all): 'es2022 (re)pooling @ 100%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71604 and previous config saved to /var/cache/conftool/dbconfig/20241205-060612-root.json
  • 06:01 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 50%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71603 and previous config saved to /var/cache/conftool/dbconfig/20241205-060121-root.json
  • 05:58 eileen: civicrm upgraded from 74c059a4 to f9c89e50
  • 05:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T371742)', diff saved to https://phabricator.wikimedia.org/P71602 and previous config saved to /var/cache/conftool/dbconfig/20241205-055442-ladsgroup.json
  • 05:54 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 05:54 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 05:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T371742)', diff saved to https://phabricator.wikimedia.org/P71601 and previous config saved to /var/cache/conftool/dbconfig/20241205-055420-ladsgroup.json
  • 05:51 marostegui@cumin1002: dbctl commit (dc=all): 'es2024 (re)pooling @ 75%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71600 and previous config saved to /var/cache/conftool/dbconfig/20241205-055125-root.json
  • 05:51 marostegui@cumin1002: dbctl commit (dc=all): 'es2022 (re)pooling @ 75%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71599 and previous config saved to /var/cache/conftool/dbconfig/20241205-055106-root.json
  • 05:46 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 25%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71598 and previous config saved to /var/cache/conftool/dbconfig/20241205-054615-root.json
  • 05:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es2023.codfw.wmnet with reason: cloning
  • 05:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es2023.codfw.wmnet with reason: cloning
  • 05:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2023 to clone es2044', diff saved to https://phabricator.wikimedia.org/P71597 and previous config saved to /var/cache/conftool/dbconfig/20241205-054200-marostegui.json
  • 05:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es2025.codfw.wmnet with reason: cloning
  • 05:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es2025.codfw.wmnet with reason: cloning
  • 05:41 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es2025 to es5 master T381259', diff saved to https://phabricator.wikimedia.org/P71596 and previous config saved to /var/cache/conftool/dbconfig/20241205-054114-marostegui.json
  • 05:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P71595 and previous config saved to /var/cache/conftool/dbconfig/20241205-053912-ladsgroup.json
  • 05:36 marostegui@cumin1002: dbctl commit (dc=all): 'es2024 (re)pooling @ 50%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71593 and previous config saved to /var/cache/conftool/dbconfig/20241205-053620-root.json
  • 05:36 marostegui@cumin1002: dbctl commit (dc=all): 'es2022 (re)pooling @ 50%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71592 and previous config saved to /var/cache/conftool/dbconfig/20241205-053601-root.json
  • 05:31 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 10%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71591 and previous config saved to /var/cache/conftool/dbconfig/20241205-053109-root.json
  • 05:28 marostegui: Failover m3 from db1159 to db1213 - T381365
  • 05:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P71590 and previous config saved to /var/cache/conftool/dbconfig/20241205-052405-ladsgroup.json
  • 05:21 marostegui@cumin1002: dbctl commit (dc=all): 'es2024 (re)pooling @ 25%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71589 and previous config saved to /var/cache/conftool/dbconfig/20241205-052114-root.json
  • 05:20 marostegui@cumin1002: dbctl commit (dc=all): 'es2022 (re)pooling @ 25%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71588 and previous config saved to /var/cache/conftool/dbconfig/20241205-052056-root.json
  • 05:20 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2134,2160].codfw.wmnet,db[1159,1213,1217].eqiad.wmnet with reason: m3 master switchover T381365
  • 05:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db[2134,2160].codfw.wmnet,db[1159,1213,1217].eqiad.wmnet with reason: m3 master switchover T381365
  • 05:16 marostegui@cumin1002: dbctl commit (dc=all): 'es2043 (re)pooling @ 1%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71587 and previous config saved to /var/cache/conftool/dbconfig/20241205-051604-root.json
  • 05:15 marostegui@cumin1002: dbctl commit (dc=all): 'Add es2043 depooled T381259', diff saved to https://phabricator.wikimedia.org/P71586 and previous config saved to /var/cache/conftool/dbconfig/20241205-051545-marostegui.json
  • 05:09 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T371742)', diff saved to https://phabricator.wikimedia.org/P71585 and previous config saved to /var/cache/conftool/dbconfig/20241205-050858-ladsgroup.json
  • 05:06 marostegui@cumin1002: dbctl commit (dc=all): 'es2024 (re)pooling @ 10%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71584 and previous config saved to /var/cache/conftool/dbconfig/20241205-050609-root.json
  • 05:05 marostegui@cumin1002: dbctl commit (dc=all): 'es2022 (re)pooling @ 10%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71583 and previous config saved to /var/cache/conftool/dbconfig/20241205-050550-root.json
  • 03:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T371742)', diff saved to https://phabricator.wikimedia.org/P71578 and previous config saved to /var/cache/conftool/dbconfig/20241205-033803-ladsgroup.json
  • 03:37 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 03:37 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 03:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209 (T371742)', diff saved to https://phabricator.wikimedia.org/P71577 and previous config saved to /var/cache/conftool/dbconfig/20241205-033751-ladsgroup.json
  • 03:34 eileen: tools upgraded from b230f718 to c7b53ecd
  • 03:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P71576 and previous config saved to /var/cache/conftool/dbconfig/20241205-032245-ladsgroup.json
  • 03:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P71575 and previous config saved to /var/cache/conftool/dbconfig/20241205-030737-ladsgroup.json
  • 02:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1209 (T371742)', diff saved to https://phabricator.wikimedia.org/P71574 and previous config saved to /var/cache/conftool/dbconfig/20241205-025230-ladsgroup.json
  • 02:45 eileen: civicrm upgraded from 6361a578 to 74c059a4
  • 01:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1209 (T371742)', diff saved to https://phabricator.wikimedia.org/P71573 and previous config saved to /var/cache/conftool/dbconfig/20241205-012108-ladsgroup.json
  • 01:21 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1209.eqiad.wmnet with reason: Maintenance
  • 01:20 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1209.eqiad.wmnet with reason: Maintenance
  • 01:20 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T371742)', diff saved to https://phabricator.wikimedia.org/P71572 and previous config saved to /var/cache/conftool/dbconfig/20241205-012046-ladsgroup.json
  • 01:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P71571 and previous config saved to /var/cache/conftool/dbconfig/20241205-010539-ladsgroup.json
  • 01:03 sukhe: re-enabling puppet on A:lvs [post-wdqs merge]
  • 00:58 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:57 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:50 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P71570 and previous config saved to /var/cache/conftool/dbconfig/20241205-005031-ladsgroup.json
  • 00:49 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:49 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:35 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T371742)', diff saved to https://phabricator.wikimedia.org/P71569 and previous config saved to /var/cache/conftool/dbconfig/20241205-003524-ladsgroup.json
  • 00:30 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1043.eqiad.wmnet with OS bookworm
  • 00:15 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1085.eqiad.wmnet with OS bullseye
  • 00:15 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002"
  • 00:15 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002"
  • 00:00 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 23:00:00 on 8 hosts with reason: T376150 non-prod hosts

2024-12-04

  • 23:59 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 23:00:00 on 8 hosts with reason: T376150 non-prod hosts
  • 23:57 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1085.eqiad.wmnet with reason: host reimage
  • 23:54 vriley@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1085.eqiad.wmnet with reason: host reimage
  • 23:47 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS bookworm
  • 23:43 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1085.eqiad.wmnet with OS bullseye
  • 23:42 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1044.eqiad.wmnet with OS bookworm
  • 23:42 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 23:40 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 23:39 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 23:35 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 23:35 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 23:32 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 23:32 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 23:26 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1085.eqiad.wmnet with OS bullseye
  • 23:21 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 23:20 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 23:16 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1043.eqiad.wmnet with OS bookworm
  • 23:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1042.eqiad.wmnet with OS bookworm
  • 23:10 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1043.eqiad.wmnet with OS bookworm
  • 22:57 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1043.eqiad.wmnet with reason: host reimage
  • 22:56 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 22:55 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T371742)', diff saved to https://phabricator.wikimedia.org/P71567 and previous config saved to /var/cache/conftool/dbconfig/20241204-225545-ladsgroup.json
  • 22:55 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 22:55 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T371742)', diff saved to https://phabricator.wikimedia.org/P71566 and previous config saved to /var/cache/conftool/dbconfig/20241204-225523-ladsgroup.json
  • 22:54 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1041.eqiad.wmnet with OS bookworm
  • 22:54 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1042.eqiad.wmnet with reason: host reimage
  • 22:51 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1040.eqiad.wmnet with OS bookworm
  • 22:50 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1043.eqiad.wmnet with reason: host reimage
  • 22:50 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1042.eqiad.wmnet with reason: host reimage
  • 22:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P71565 and previous config saved to /var/cache/conftool/dbconfig/20241204-224016-ladsgroup.json
  • 22:38 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1044.eqiad.wmnet with reason: host reimage
  • 22:37 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1045.eqiad.wmnet with OS bookworm
  • 22:35 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1041.eqiad.wmnet with reason: host reimage
  • 22:34 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1043.eqiad.wmnet with OS bookworm
  • 22:34 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1042.eqiad.wmnet with OS bookworm
  • 22:33 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on es1044.eqiad.wmnet with reason: host reimage
  • 22:32 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1040.eqiad.wmnet with reason: host reimage
  • 22:30 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1041.eqiad.wmnet with reason: host reimage
  • 22:29 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1040.eqiad.wmnet with reason: host reimage
  • 22:26 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS bookworm
  • 22:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P71564 and previous config saved to /var/cache/conftool/dbconfig/20241204-222509-ladsgroup.json
  • 22:18 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1085.eqiad.wmnet with OS bullseye
  • 22:16 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1044.eqiad.wmnet with OS bookworm
  • 22:13 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1041.eqiad.wmnet with OS bookworm
  • 22:13 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1040.eqiad.wmnet with OS bookworm
  • 22:12 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1086.eqiad.wmnet with OS bullseye
  • 22:12 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 22:12 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 22:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T371742)', diff saved to https://phabricator.wikimedia.org/P71563 and previous config saved to /var/cache/conftool/dbconfig/20241204-221001-ladsgroup.json
  • 21:59 jhuneidi@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.6 refs T375665
  • 21:57 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1043.eqiad.wmnet with OS bookworm
  • 21:57 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1044.eqiad.wmnet with OS bookworm
  • 21:49 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1086.eqiad.wmnet with reason: host reimage
  • 21:46 cjming: end of UTC late backport window
  • 21:46 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host es1045
  • 21:45 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1086.eqiad.wmnet with reason: host reimage
  • 21:45 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host es1045
  • 21:43 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:43 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for ms-be - jclark@cumin1002"
  • 21:43 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for ms-be - jclark@cumin1002"
  • 21:43 cjming@deploy2002: Finished scap sync-world: Backport for CSP for banner preview: allow remind me later SMS host (T380232) (duration: 17m 39s)
  • 21:40 jclark@cumin1002: START - Cookbook sre.dns.netbox
  • 21:37 cjming@deploy2002: cjming, gjg: Continuing with sync
  • 21:34 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1086.eqiad.wmnet with OS bullseye
  • 21:34 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 21:34 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 21:32 cjming@deploy2002: cjming, gjg: Backport for CSP for banner preview: allow remind me later SMS host (T380232) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 21:26 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs-internal-scholarly.discovery.wmnet on all recursors
  • 21:26 sukhe@cumin1002: START - Cookbook sre.dns.wipe-cache wdqs-internal-scholarly.discovery.wmnet on all recursors
  • 21:26 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs-internal-main.discovery.wmnet on all recursors
  • 21:26 sukhe@cumin1002: START - Cookbook sre.dns.wipe-cache wdqs-internal-main.discovery.wmnet on all recursors
  • 21:25 cjming@deploy2002: Started scap sync-world: Backport for CSP for banner preview: allow remind me later SMS host (T380232)
  • 21:25 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs2013.codfw.wmnet
  • 21:24 ryankemper: T379334 `ryankemper@dns1004:~$ sudo -i authdns-update` completed
  • 21:23 cjming@deploy2002: Finished scap sync-world: Backport for Enable Chart extension on several pilot wikis (T381436 T381312) (duration: 17m 29s)
  • 21:22 sukhe@cumin1002: START - Cookbook sre.hosts.reboot-single for host lvs2013.codfw.wmnet
  • 21:21 ryankemper: T379334 Final step (step 9) of spinning up these new services; merged https://gerrit.wikimedia.org/r/c/operations/dns/+/1100165/, next up is the authdns update
  • 21:18 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-internal-scholarly
  • 21:18 ryankemper@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs-internal-main
  • 21:17 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1045.eqiad.wmnet with OS bookworm
  • 21:14 cjming@deploy2002: cjming, bvibber: Continuing with sync
  • 21:13 cjming@deploy2002: cjming, bvibber: Backport for Enable Chart extension on several pilot wikis (T381436 T381312) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 21:09 ryankemper: T380555 Rolling out prod change => `ryankemper@cumin2002:~$ sudo cumin -b 8 'A:dnsbox' 'run-puppet-agent'`
  • 21:05 ryankemper: T380555 Moving `wdqs-internal-[main,scholarly]` services into prod by merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1094074
  • 21:05 cjming@deploy2002: Started scap sync-world: Backport for Enable Chart extension on several pilot wikis (T381436 T381312)
  • 21:03 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs2013.codfw.wmnet with reason: rebooting shortly
  • 21:03 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs2013.codfw.wmnet with reason: rebooting shortly
  • 21:01 joal@deploy2002: Finished deploy [analytics/refinery@7ba91e1] (hadoop-test): Regular analytics weekly train TEST - HOTFIX 2 [analytics/refinery@7ba91e13] (duration: 00m 29s)
  • 21:00 joal@deploy2002: Started deploy [analytics/refinery@7ba91e1] (hadoop-test): Regular analytics weekly train TEST - HOTFIX 2 [analytics/refinery@7ba91e13]
  • 21:00 joal@deploy2002: Finished deploy [analytics/refinery@7ba91e1] (thin): Regular analytics weekly train THIN - HOTFIX 2 [analytics/refinery@7ba91e13] (duration: 00m 31s)
  • 20:59 joal@deploy2002: Started deploy [analytics/refinery@7ba91e1] (thin): Regular analytics weekly train THIN - HOTFIX 2 [analytics/refinery@7ba91e13]
  • 20:59 cjming@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
  • 20:59 cjming@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
  • 20:59 joal@deploy2002: Finished deploy [analytics/refinery@7ba91e1]: Regular analytics weekly train - HOTFIX 2 [analytics/refinery@7ba91e13] (duration: 01m 48s)
  • 20:57 joal@deploy2002: Started deploy [analytics/refinery@7ba91e1]: Regular analytics weekly train - HOTFIX 2 [analytics/refinery@7ba91e13]
  • 20:51 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1039.eqiad.wmnet with OS bookworm
  • 20:50 cjming@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
  • 20:49 cjming@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
  • 20:47 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1038.eqiad.wmnet with OS bookworm
  • 20:37 ryankemper: T380555 hosts happily pooled (except that `lvs2013` aka `A:lvs-low-traffic-codfw` cannot talk to `wdqs2026`) and `sudo ipvsadm -L -n` shows `10.2.1.93` and `10.2.1.94` as expected, codfw all done
  • 20:33 sukhe@cumin1002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic-eqiad and A:lvs
  • 20:32 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1039.eqiad.wmnet with reason: host reimage
  • 20:32 sukhe@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-eqiad and A:lvs
  • 20:30 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T371742)', diff saved to https://phabricator.wikimedia.org/P71562 and previous config saved to /var/cache/conftool/dbconfig/20241204-203043-ladsgroup.json
  • 20:30 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 20:30 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 20:30 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T371742)', diff saved to https://phabricator.wikimedia.org/P71561 and previous config saved to /var/cache/conftool/dbconfig/20241204-203021-ladsgroup.json
  • 20:29 ryankemper@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic-codfw and A:lvs
  • 20:28 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1038.eqiad.wmnet with reason: host reimage
  • 20:28 ryankemper: T380555 `sudo cookbook sre.loadbalancer.restart-pybal --query 'A:lvs-low-traffic-codfw' --reason 'rolling out new wdqs-internal-[main,scholarly] services' restart_daemons`
  • 20:28 ryankemper@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-codfw and A:lvs
  • 20:28 ryankemper: T380555 ran `sudo -E cumin 'A:lvs-low-traffic-codfw' 'run-puppet-agent --force'`
  • 20:28 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for lvs1020.eqiad.wmnet
  • 20:28 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for lvs1020.eqiad.wmnet
  • 20:26 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1039.eqiad.wmnet with reason: host reimage
  • 20:25 sukhe@cumin1002: END (ERROR) - Cookbook sre.loadbalancer.restart-pybal (exit_code=97) rolling-restart of pybal on A:lvs-secondary-eqiad and A:lvs
  • 20:25 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1038.eqiad.wmnet with reason: host reimage
  • 20:24 ryankemper: T380555 hosts happily pooled and `sudo ipvsadm -L -n` shows `10.2.1.93` and `10.2.1.94` as expected), proceeding to `A:lvs-low-traffic-codfw`
  • 20:23 sukhe@cumin1002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad and A:lvs
  • 20:23 dbrant@deploy2002: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
  • 20:22 dbrant@deploy2002: helmfile [codfw] START helmfile.d/services/push-notifications: apply
  • 20:22 dbrant@deploy2002: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
  • 20:21 ryankemper@cumin2002: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-codfw and A:lvs
  • 20:21 ryankemper: T380555 `sudo cookbook sre.loadbalancer.restart-pybal --query 'A:lvs-secondary-codfw' --reason 'rolling out new wdqs-internal-[main,scholarly] services' restart_daemons`
  • 20:21 dbrant@deploy2002: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
  • 20:20 ryankemper@cumin2002: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-codfw and A:lvs
  • 20:20 dbrant@deploy2002: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
  • 20:20 dbrant@deploy2002: helmfile [staging] START helmfile.d/services/push-notifications: apply
  • 20:18 ryankemper: T380555 `sudo cookbook sre.loadbalancer.restart-pybal 'A:lvs-secondary-codfw' --reason 'rolling out new wdqs-internal-[main,scholarly] services'`
  • 20:17 ryankemper: T380555 `sudo -E cumin 'A:lvs-secondary-codfw' 'run-puppet-agent --force'`
  • 20:17 ryankemper: T380555 Beginning lvs rolling restarts. first up `A:lvs-secondary-codfw`
  • 20:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P71560 and previous config saved to /var/cache/conftool/dbconfig/20241204-201513-ladsgroup.json
  • 20:12 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
  • 20:12 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
  • 20:12 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
  • 20:12 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
  • 20:10 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1039.eqiad.wmnet with OS bookworm
  • 20:09 cdanis@deploy2002: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
  • 20:09 cdanis@deploy2002: helmfile [staging] START helmfile.d/services/chart-renderer: apply
  • 20:08 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1038.eqiad.wmnet with OS bookworm
  • 20:08 ryankemper: T380555 ran `ryankemper@cumin2002:~$ sudo -E cumin 'lvs*' 'disable-puppet T380555'`
  • 20:07 ryankemper: T380555 Disabling puppet on lvs hosts in preparation for merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1094070 which will move `wdqs-internal-[main,scholarly]` from `service_setup` to `lvs_setup`
  • 20:04 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "T377876 - kamila@cumin1002"
  • 20:04 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "T377876 - kamila@cumin1002"
  • 20:00 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P71559 and previous config saved to /var/cache/conftool/dbconfig/20241204-200006-ladsgroup.json
  • 19:57 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1038.eqiad.wmnet wikikube-worker1039.eqiad.wmnet wikikube-worker1040.eqiad.wmnet wikikube-worker1041.eqiad.wmnet wikikube-worker1042.eqiad.wmnet wikikube-worker1043.eqiad.wmnet on all recursors
  • 19:57 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1038.eqiad.wmnet wikikube-worker1039.eqiad.wmnet wikikube-worker1040.eqiad.wmnet wikikube-worker1041.eqiad.wmnet wikikube-worker1042.eqiad.wmnet wikikube-worker1043.eqiad.wmnet on all recursors
  • 19:55 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1038.eqiad.wmnet on all recursors
  • 19:55 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1038.eqiad.wmnet on all recursors
  • 19:55 ryankemper: T380555 Running puppet on `wdqs2018`
  • 19:55 ryankemper: T380555 Proceeding to step 5 of new lvs service process. Merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1094069 to enable lvs::realserver functionality
  • 19:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1496 to wikikube-worker1043
  • 19:53 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
  • 19:53 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1043
  • 19:53 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
  • 19:53 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1043
  • 19:53 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:53 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1496 to wikikube-worker1043 - kamila@cumin1002"
  • 19:52 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1496 to wikikube-worker1043 - kamila@cumin1002"
  • 19:52 sukhe: sudo cumin "O:config_master" "run-puppet-agent"
  • 19:50 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1495 to wikikube-worker1042
  • 19:49 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1042
  • 19:49 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 19:49 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1042
  • 19:49 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:49 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1495 to wikikube-worker1042 - kamila@cumin1002"
  • 19:49 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1495 to wikikube-worker1042 - kamila@cumin1002"
  • 19:45 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1496 to wikikube-worker1043
  • 19:45 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T371742)', diff saved to https://phabricator.wikimedia.org/P71558 and previous config saved to /var/cache/conftool/dbconfig/20241204-194459-ladsgroup.json
  • 19:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1494 to wikikube-worker1041
  • 19:43 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 19:43 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1041
  • 19:43 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1041
  • 19:43 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:43 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1494 to wikikube-worker1041 - kamila@cumin1002"
  • 19:42 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1494 to wikikube-worker1041 - kamila@cumin1002"
  • 19:40 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
  • 19:40 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1495 to wikikube-worker1042
  • 19:40 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
  • 19:39 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1493 to wikikube-worker1040
  • 19:39 joal@deploy2002: Finished deploy [airflow-dags/analytics@df2cac9]: Regular analytics weekly train [airflow-dags/analytics@df2cac98] (duration: 03m 55s)
  • 19:38 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1040
  • 19:38 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 19:38 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1040
  • 19:38 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:38 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1493 to wikikube-worker1040 - kamila@cumin1002"
  • 19:38 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1493 to wikikube-worker1040 - kamila@cumin1002"
  • 19:37 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1494 to wikikube-worker1041
  • 19:36 cdanis@deploy2002: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
  • 19:35 cdanis@deploy2002: helmfile [staging] START helmfile.d/services/chart-renderer: apply
  • 19:35 joal@deploy2002: Started deploy [airflow-dags/analytics@df2cac9]: Regular analytics weekly train [airflow-dags/analytics@df2cac98]
  • 19:35 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1492 to wikikube-worker1039
  • 19:34 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 19:34 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1039
  • 19:34 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1039
  • 19:34 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:34 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1492 to wikikube-worker1039 - kamila@cumin1002"
  • 19:33 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1492 to wikikube-worker1039 - kamila@cumin1002"
  • 19:30 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1493 to wikikube-worker1040
  • 19:30 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1491 to wikikube-worker1038
  • 19:29 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1038
  • 19:29 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 19:29 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1038
  • 19:29 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:29 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1491 to wikikube-worker1038 - kamila@cumin1002"
  • 19:29 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1491 to wikikube-worker1038 - kamila@cumin1002"
  • 19:26 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1492 to wikikube-worker1039
  • 19:25 kamila@cumin1002: START - Cookbook sre.dns.netbox
  • 19:25 joal@deploy2002: Finished deploy [analytics/refinery@1f94312] (hadoop-test): Regular analytics weekly train TEST - HOTFIX [analytics/refinery@1f94312a] (duration: 00m 26s)
  • 19:25 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1491 to wikikube-worker1038
  • 19:24 joal@deploy2002: Started deploy [analytics/refinery@1f94312] (hadoop-test): Regular analytics weekly train TEST - HOTFIX [analytics/refinery@1f94312a]
  • 19:24 joal@deploy2002: Finished deploy [analytics/refinery@1f94312] (thin): Regular analytics weekly train THIN - HOTFIX [analytics/refinery@1f94312a] (duration: 00m 30s)
  • 19:23 joal@deploy2002: Started deploy [analytics/refinery@1f94312] (thin): Regular analytics weekly train THIN - HOTFIX [analytics/refinery@1f94312a]
  • 19:23 joal@deploy2002: Finished deploy [analytics/refinery@1f94312]: Regular analytics weekly train - HOTFIX [analytics/refinery@1f94312a] (duration: 03m 17s)
  • 19:22 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1491-1496].eqiad.wmnet
  • 19:20 joal@deploy2002: Started deploy [analytics/refinery@1f94312]: Regular analytics weekly train - HOTFIX [analytics/refinery@1f94312a]
  • 19:19 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1491-1496].eqiad.wmnet
  • 19:16 ryankemper: T380555 Merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1094069 to enable `lvs::realserver`
  • 19:09 ryankemper: T379333 Merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1097542 to establish envoy on `A:wdqs-internal-main` and `A:wdqs-internal-scholarly`; running puppet on `wdqs2018` to test change
  • 19:03 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 19:02 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 19:02 ryankemper: T380555 Merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1094061 to establish initial service definitions for `wdqs-internal-main` and `wdqs-internal-scholarly`
  • 18:58 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 18:58 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 18:58 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 18:58 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 18:57 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs-internal-main.svc.eqiad.wmnet on all recursors
  • 18:57 sukhe@cumin1002: START - Cookbook sre.dns.wipe-cache wdqs-internal-main.svc.eqiad.wmnet on all recursors
  • 18:55 ryankemper: T379334 Successfully ran `sudo authdns-update` on `dns1004`
  • 18:52 ryankemper: T379334 Creating A and PTR records for `wdqs-internal-main` and `wdqs-internal-scholarly` VIPs [merging https://gerrit.wikimedia.org/r/c/operations/dns/+/1100010/ & running authdns update after]
  • 18:48 cjming@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
  • 18:47 cjming@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
  • 18:47 ryankemper: T379330 `wdqs-internal-main` and `wdqs-internal-scholarly` pools created
  • 18:46 ryankemper@cumin2002: conftool action : set/pooled=yes:weight=10; selector: cluster=wdqs-internal-main,service=wdqs-main
  • 18:46 ryankemper@cumin2002: conftool action : set/pooled=yes:weight=10; selector: cluster=wdqs-internal-scholarly,service=wdqs-scholarly
  • 18:35 dbrant@deploy2002: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
  • 18:35 dbrant@deploy2002: helmfile [codfw] START helmfile.d/services/push-notifications: apply
  • 18:34 dbrant@deploy2002: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
  • 18:33 dbrant@deploy2002: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
  • 18:30 dbrant@deploy2002: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
  • 18:30 dbrant@deploy2002: helmfile [staging] START helmfile.d/services/push-notifications: apply
  • 18:13 swfrench@deploy2002: Finished scap sync-world: Deployment to clear noop chart diff from 1081449 - T377040 (duration: 02m 07s)
  • 18:11 swfrench@deploy2002: Started scap sync-world: Deployment to clear noop chart diff from 1081449 - T377040
  • 18:04 cjming@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
  • 18:04 cjming@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
  • 18:01 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T371742)', diff saved to https://phabricator.wikimedia.org/P71556 and previous config saved to /var/cache/conftool/dbconfig/20241204-180114-ladsgroup.json
  • 18:01 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 18:00 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 18:00 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T371742)', diff saved to https://phabricator.wikimedia.org/P71555 and previous config saved to /var/cache/conftool/dbconfig/20241204-180052-ladsgroup.json
  • 17:55 joal@deploy2002: Finished deploy [analytics/refinery@6e3ee14] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@6e3ee14b] (duration: 00m 31s)
  • 17:54 joal@deploy2002: Started deploy [analytics/refinery@6e3ee14] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@6e3ee14b]
  • 17:54 joal@deploy2002: Finished deploy [analytics/refinery@6e3ee14] (thin): Regular analytics weekly train THIN [analytics/refinery@6e3ee14b] (duration: 00m 37s)
  • 17:54 joal@deploy2002: Started deploy [analytics/refinery@6e3ee14] (thin): Regular analytics weekly train THIN [analytics/refinery@6e3ee14b]
  • 17:52 joal@deploy2002: Finished deploy [analytics/refinery@6e3ee14]: Regular analytics weekly train [analytics/refinery@6e3ee14b] (duration: 02m 05s)
  • 17:50 bd808: Moved SAL fediverse posts to https://wikimedia.social/@sal. Many thanks to botsin.space for providing hosting for so long.
  • 17:50 joal@deploy2002: Started deploy [analytics/refinery@6e3ee14]: Regular analytics weekly train [analytics/refinery@6e3ee14b]
  • 17:45 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P71554 and previous config saved to /var/cache/conftool/dbconfig/20241204-174544-ladsgroup.json
  • 17:30 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P71553 and previous config saved to /var/cache/conftool/dbconfig/20241204-173037-ladsgroup.json
  • 17:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T371742)', diff saved to https://phabricator.wikimedia.org/P71551 and previous config saved to /var/cache/conftool/dbconfig/20241204-171530-ladsgroup.json
  • 17:10 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephmon1003.eqiad.wmnet
  • 17:10 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:10 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcephmon1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 17:08 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcephmon1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 17:04 andrew@cumin1002: START - Cookbook sre.dns.netbox
  • 17:00 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudcephmon1003.eqiad.wmnet
  • 16:59 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephmon1002.eqiad.wmnet
  • 16:59 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:59 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcephmon1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 16:59 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcephmon1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 16:56 andrew@cumin1002: START - Cookbook sre.dns.netbox
  • 16:52 jgleeson: smashpig-listeners updated from 79b463b4 to 17ac74f2
  • 16:51 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudcephmon1002.eqiad.wmnet
  • 16:46 isaranto@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' .
  • 16:45 isaranto@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
  • 16:38 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephmon1001.eqiad.wmnet
  • 16:38 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:38 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcephmon1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 16:38 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcephmon1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 16:37 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
  • 16:37 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/changeprop: apply
  • 16:37 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/changeprop: apply
  • 16:37 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
  • 16:36 jiji@deploy2002: helmfile [staging] START helmfile.d/services/changeprop: apply
  • 16:36 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/mw-page-content-change-enrich: apply
  • 16:36 jiji@deploy2002: helmfile [staging] START helmfile.d/services/mw-page-content-change-enrich: apply
  • 16:36 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
  • 16:36 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
  • 16:36 jiji@deploy2002: helmfile [staging] START helmfile.d/services/eventstreams: apply
  • 16:35 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2173-2175].codfw.wmnet
  • 16:35 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2173-2175].codfw.wmnet
  • 16:34 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 16:34 andrew@cumin1002: START - Cookbook sre.dns.netbox
  • 16:34 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply
  • 16:34 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply
  • 16:34 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/eventgate-main: apply
  • 16:33 jiji@deploy2002: helmfile [staging] START helmfile.d/services/eventgate-main: apply
  • 16:33 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 16:33 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 16:33 jiji@deploy2002: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
  • 16:33 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/benthos-cache-invalidator: apply
  • 16:32 jiji@deploy2002: helmfile [staging] START helmfile.d/services/benthos-cache-invalidator: apply
  • 16:32 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2175.codfw.wmnet with OS bookworm
  • 16:27 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudcephmon1001.eqiad.wmnet
  • 16:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Schema change
  • 16:22 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Schema change
  • 16:21 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2174.codfw.wmnet with OS bookworm
  • 16:21 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 100%: After schema change', diff saved to https://phabricator.wikimedia.org/P71550 and previous config saved to /var/cache/conftool/dbconfig/20241204-162127-root.json
  • 16:19 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1009.eqiad.wmnet
  • 16:19 jiji@cumin1002: START - Cookbook sre.hosts.remove-downtime for kafka-main1009.eqiad.wmnet
  • 16:18 jiji@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 16:17 jiji@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 16:17 jiji@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 16:17 jiji@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 16:17 jiji@deploy2002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 16:16 jiji@deploy2002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 16:16 jiji@deploy2002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 16:16 jiji@deploy2002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 16:16 jiji@deploy2002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 16:15 jiji@deploy2002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 16:15 jiji@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 16:15 jiji@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 16:15 jiji@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 16:15 jiji@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 16:14 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 16:14 jiji@deploy2002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 16:14 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 16:14 jiji@deploy2002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 16:13 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2173.codfw.wmnet with OS bookworm
  • 16:13 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2175.codfw.wmnet with reason: host reimage
  • 16:12 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1025.eqiad.wmnet with OS bullseye
  • 16:12 bking@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - bking@cumin2002"
  • 16:09 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2175.codfw.wmnet with reason: host reimage
  • 16:06 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 75%: After schema change', diff saved to https://phabricator.wikimedia.org/P71549 and previous config saved to /var/cache/conftool/dbconfig/20241204-160622-root.json
  • 16:06 hnowlan@deploy2002: Finished scap sync-world: Rebuild and deploy to pick up new php8.1 base (duration: 42m 17s)
  • 16:01 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2174.codfw.wmnet with reason: host reimage
  • 15:55 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2174.codfw.wmnet with reason: host reimage
  • 15:54 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2173.codfw.wmnet with reason: host reimage
  • 15:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 50%: After schema change', diff saved to https://phabricator.wikimedia.org/P71548 and previous config saved to /var/cache/conftool/dbconfig/20241204-155116-root.json
  • 15:51 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2175
  • 15:51 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2175
  • 15:50 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2173.codfw.wmnet with reason: host reimage
  • 15:46 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2175
  • 15:46 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2175.codfw.wmnet 80.48.192.10.in-addr.arpa 0.8.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 15:46 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2175.codfw.wmnet 80.48.192.10.in-addr.arpa 0.8.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 15:46 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:45 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2175 - jayme@cumin2002"
  • 15:45 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2175 - jayme@cumin2002"
  • 15:45 vgutierrez: restarting purged on cp1115
  • 15:41 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1036-1037].eqiad.wmnet
  • 15:41 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1036-1037].eqiad.wmnet
  • 15:39 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 15:37 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2175
  • 15:36 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2174
  • 15:36 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2174
  • 15:36 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2174
  • 15:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P71546 and previous config saved to /var/cache/conftool/dbconfig/20241204-153611-root.json
  • 15:36 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2174.codfw.wmnet 79.48.192.10.in-addr.arpa 9.7.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 15:36 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2174.codfw.wmnet 79.48.192.10.in-addr.arpa 9.7.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 15:36 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:36 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2174 - jayme@cumin2002"
  • 15:36 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2174 - jayme@cumin2002"
  • 15:32 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T371742)', diff saved to https://phabricator.wikimedia.org/P71545 and previous config saved to /var/cache/conftool/dbconfig/20241204-153234-ladsgroup.json
  • 15:32 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 15:32 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 15:32 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T371742)', diff saved to https://phabricator.wikimedia.org/P71544 and previous config saved to /var/cache/conftool/dbconfig/20241204-153212-ladsgroup.json
  • 15:31 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 15:31 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2174
  • 15:31 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2173
  • 15:31 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2173
  • 15:30 jiji@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad
  • 15:30 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2173
  • 15:30 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2173.codfw.wmnet 78.48.192.10.in-addr.arpa 8.7.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 15:30 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2173.codfw.wmnet 78.48.192.10.in-addr.arpa 8.7.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 15:30 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:30 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2173 - jayme@cumin2002"
  • 15:30 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2173 - jayme@cumin2002"
  • 15:28 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2175.codfw.wmnet with OS bookworm
  • 15:27 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2174.codfw.wmnet with OS bookworm
  • 15:27 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 15:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1037.eqiad.wmnet with OS bookworm
  • 15:26 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 15:26 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2173.codfw.wmnet wikikube-worker2174.codfw.wmnet wikikube-worker2175.codfw.wmnet on all recursors
  • 15:26 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2173.codfw.wmnet wikikube-worker2174.codfw.wmnet wikikube-worker2175.codfw.wmnet on all recursors
  • 15:26 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2173
  • 15:26 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2173.codfw.wmnet with OS bookworm
  • 15:25 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2447 to wikikube-worker2175
  • 15:24 hnowlan@deploy2002: Started scap sync-world: Rebuild and deploy to pick up new php8.1 base
  • 15:24 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2175
  • 15:24 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2175
  • 15:24 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:22 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2446 to wikikube-worker2174
  • 15:22 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2174
  • 15:22 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 15:21 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2174
  • 15:21 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:21 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2446 to wikikube-worker2174 - jayme@cumin2002"
  • 15:21 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2446 to wikikube-worker2174 - jayme@cumin2002"
  • 15:21 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1045.eqiad.wmnet with OS bookworm
  • 15:21 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2445 to wikikube-worker2173
  • 15:21 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 10%: After schema change', diff saved to https://phabricator.wikimedia.org/P71543 and previous config saved to /var/cache/conftool/dbconfig/20241204-152105-root.json
  • 15:20 jiji@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad
  • 15:20 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2173
  • 15:18 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2173
  • 15:18 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:18 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2445 to wikikube-worker2173 - jayme@cumin2002"
  • 15:18 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2445 to wikikube-worker2173 - jayme@cumin2002"
  • 15:17 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P71542 and previous config saved to /var/cache/conftool/dbconfig/20241204-151705-ladsgroup.json
  • 15:16 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 15:10 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2447 to wikikube-worker2175
  • 15:10 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2446 to wikikube-worker2174
  • 15:10 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 15:10 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2445 to wikikube-worker2173
  • 15:06 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - bking@cumin2002"
  • 15:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1037.eqiad.wmnet with reason: host reimage
  • 15:03 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1037.eqiad.wmnet with reason: host reimage
  • 15:01 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P71541 and previous config saved to /var/cache/conftool/dbconfig/20241204-150157-ladsgroup.json
  • 15:01 TheresNoTime: '[samtar@deploy2002 ~]$ mwscript-k8s --comment="T373634" -f -- namespaceDupes.php --wiki hsbwiktionary --fix' for T373634
  • 14:59 samtar@deploy2002: Finished scap sync-world: Backport for Add new namespaces to hsb wiktionary (T373634) (duration: 10m 16s)
  • 14:54 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet
  • 14:54 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet
  • 14:52 samtar@deploy2002: samtar, srishakatux: Continuing with sync
  • 14:51 samtar@deploy2002: samtar, srishakatux: Backport for Add new namespaces to hsb wiktionary (T373634) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:50 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2446.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:50 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2447.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:50 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2445.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:49 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1025.eqiad.wmnet with reason: host reimage
  • 14:48 samtar@deploy2002: Started scap sync-world: Backport for Add new namespaces to hsb wiktionary (T373634)
  • 14:47 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1037.eqiad.wmnet with OS bookworm
  • 14:46 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T371742)', diff saved to https://phabricator.wikimedia.org/P71540 and previous config saved to /var/cache/conftool/dbconfig/20241204-144651-ladsgroup.json
  • 14:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1036.eqiad.wmnet with OS bookworm
  • 14:46 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1025.eqiad.wmnet with reason: host reimage
  • 14:31 Lucas_WMDE: UTC afternoon backport+config window done
  • 14:29 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for Translate: Enable message group subscription for 6 wikis (T372386) (duration: 18m 12s)
  • 14:27 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1036.eqiad.wmnet with reason: host reimage
  • 14:23 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1036.eqiad.wmnet with reason: host reimage
  • 14:22 lucaswerkmeister-wmde@deploy2002: abi, lucaswerkmeister-wmde: Continuing with sync
  • 14:19 bking@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs1025.eqiad.wmnet with OS bullseye
  • 14:18 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2015.codfw.wmnet with OS bookworm
  • 14:17 lucaswerkmeister-wmde@deploy2002: abi, lucaswerkmeister-wmde: Backport for Translate: Enable message group subscription for 6 wikis (T372386) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:11 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for Translate: Enable message group subscription for 6 wikis (T372386)
  • 14:07 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1036.eqiad.wmnet with OS bookworm
  • 14:05 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1036.eqiad.wmnet wikikube-worker1037.eqiad.wmnet on all recursors
  • 14:05 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1036.eqiad.wmnet wikikube-worker1037.eqiad.wmnet on all recursors
  • 14:01 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2446.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:01 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2447.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:01 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2445.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 14:00 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1045.eqiad.wmnet with OS bookworm
  • 14:00 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1044.eqiad.wmnet with OS bookworm
  • 14:00 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS bookworm
  • 14:00 dreamyjazz@deploy2002: Finished scap sync-world: Backport for Ensure IP reveal buttons are not shown on Special:MassGlobalBlock (T124607) (duration: 13m 08s)
  • 13:58 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2015.codfw.wmnet with reason: host reimage
  • 13:58 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on mw[2445-2447].codfw.wmnet with reason: reimage
  • 13:57 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 4:00:00 on mw[2445-2447].codfw.wmnet with reason: reimage
  • 13:56 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1024 to wikikube-worker1037
  • 13:55 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1037
  • 13:54 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2015.codfw.wmnet with reason: host reimage
  • 13:54 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1037
  • 13:54 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:54 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1024 to wikikube-worker1037 - jelto@cumin1002"
  • 13:54 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1024 to wikikube-worker1037 - jelto@cumin1002"
  • 13:53 dreamyjazz@deploy2002: tchanders, dreamyjazz: Continuing with sync
  • 13:53 dreamyjazz@deploy2002: tchanders, dreamyjazz: Backport for Ensure IP reveal buttons are not shown on Special:MassGlobalBlock (T124607) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:51 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2445-2447].codfw.wmnet
  • 13:50 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 13:50 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2445-2447].codfw.wmnet
  • 13:50 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1024 to wikikube-worker1037
  • 13:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1023 to wikikube-worker1036
  • 13:48 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1036
  • 13:47 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1036
  • 13:47 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:47 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1023 to wikikube-worker1036 - jelto@cumin1002"
  • 13:47 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1023 to wikikube-worker1036 - jelto@cumin1002"
  • 13:47 dreamyjazz@deploy2002: Started scap sync-world: Backport for Ensure IP reveal buttons are not shown on Special:MassGlobalBlock (T124607)
  • {{safesubst:SAL entry|1=13:42 dreamyjazz@deploy2002: Finished scap sync-world: Backport for Stats: Move StatsFactory flush into emitBufferedStats (T380609), Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169), Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169), [[gerrit:1100449|Revert "Stats: Move StatsFactory flush into emitBufferedSt}}
  • 13:41 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2016,2171-2172].codfw.wmnet
  • 13:41 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2016,2171-2172].codfw.wmnet
  • 13:39 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 13:39 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1023 to wikikube-worker1036
  • 13:35 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2015.codfw.wmnet with OS bookworm
  • 13:35 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2015.codfw.wmnet with OS bookworm
  • 13:33 dreamyjazz@deploy2002: dreamyjazz: Continuing with sync
  • 13:33 dreamyjazz@deploy2002: dreamyjazz: Backport for Stats: Move StatsFactory flush into emitBufferedStats (T380609), Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169), Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169), Revert "Stats: Move StatsFactory flush into emitBufferedStats" synced
  • 13:31 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1023-1024].eqiad.wmnet
  • 13:30 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1023-1024].eqiad.wmnet
  • {{safesubst:SAL entry|1=13:28 dreamyjazz@deploy2002: Started scap sync-world: Backport for Stats: Move StatsFactory flush into emitBufferedStats (T380609), Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169), Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169), [[gerrit:1100449|Revert "Stats: Move StatsFactory flush into emitBufferedSta}}
  • 13:20 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Alter table
  • 13:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet with reason: Alter table
  • 13:19 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on clouddb1020.eqiad.wmnet with reason: Alter table
  • 13:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on clouddb1020.eqiad.wmnet with reason: Alter table
  • 13:19 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on clouddb1016.eqiad.wmnet with reason: Alter table
  • 13:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on clouddb1016.eqiad.wmnet with reason: Alter table
  • 13:19 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1154.eqiad.wmnet with reason: Alter table
  • 13:18 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1154.eqiad.wmnet with reason: Alter table
  • 13:12 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 13:12 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 13:12 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 13:12 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 13:12 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 13:12 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 13:09 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 13:09 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 13:09 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 13:09 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 13:09 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 13:09 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 13:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T371742)', diff saved to https://phabricator.wikimedia.org/P71537 and previous config saved to /var/cache/conftool/dbconfig/20241204-130614-ladsgroup.json
  • 13:06 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 13:06 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 13:05 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 13:05 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 13:05 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 13:05 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 13:05 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 13:05 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 13:05 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 13:04 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 13:04 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 13:04 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 13:04 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 13:04 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 13:00 dreamyjazz@deploy2002: Started scap sync-world: Backport for Stats: Move StatsFactory flush into emitBufferedStats (T380609), Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169), Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)
  • 12:59 dreamyjazz@deploy2002: Started scap sync-world: Backport for Stats: Move StatsFactory flush into emitBufferedStats (T380609), Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169), Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)
  • 12:57 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 12:56 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/citoid: apply
  • 12:55 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 12:55 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/citoid: apply
  • 12:54 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on kafka-main[1004,1009].eqiad.wmnet with reason: Hardware refresh
  • 12:54 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on kafka-main[1004,1009].eqiad.wmnet with reason: Hardware refresh
  • 12:52 stevemunene@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: sync on production
  • 12:47 stevemunene@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply on production
  • 12:47 moritzm: uploaded mailman3 3.3.8-2~deb12u2+wmf1 T377045
  • 12:42 stevemunene@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: sync on staging
  • 12:40 hnowlan: imported debs for mercurius_1.0.2
  • 12:38 stevemunene@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply on staging
  • 12:33 mvolz@deploy2002: helmfile [codfw] START helmfile.d/services/citoid: apply
  • 12:32 moritzm: installing glib2.0 security updates
  • 12:22 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2172.codfw.wmnet with OS bookworm
  • 12:06 dreamyjazz@deploy2002: Finished scap sync-world: Backport for Create a DB list for wikis with continuous MediaModeration scans (T355169) (duration: 13m 02s)
  • 12:02 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2172.codfw.wmnet with reason: host reimage
  • 12:01 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
  • 12:01 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
  • 11:59 dreamyjazz@deploy2002: dreamyjazz: Continuing with sync
  • 11:59 dreamyjazz@deploy2002: dreamyjazz: Backport for Create a DB list for wikis with continuous MediaModeration scans (T355169) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 11:58 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2172.codfw.wmnet with reason: host reimage
  • 11:53 dreamyjazz@deploy2002: Started scap sync-world: Backport for Create a DB list for wikis with continuous MediaModeration scans (T355169)
  • 11:52 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2016.codfw.wmnet with OS bookworm
  • 11:49 vgutierrez: re-enabling outbound bandwidth limits enforced by haproxy on the upload cluster
  • 11:39 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2172.codfw.wmnet with OS bookworm
  • 11:38 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2172.codfw.wmnet with OS bookworm
  • 11:36 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2171.codfw.wmnet with OS bookworm
  • 11:35 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 11:35 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 11:32 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2016.codfw.wmnet with reason: host reimage
  • 11:26 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2016.codfw.wmnet with reason: host reimage
  • 11:24 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 11:24 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 11:14 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2171.codfw.wmnet with reason: host reimage
  • 11:13 vgutierrez: disabling outbound bandwidth limits enforced by haproxy on the upload cluster (we are getting haproxy crashes)
  • 11:11 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2171.codfw.wmnet with reason: host reimage
  • 11:07 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2016
  • 11:07 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2016
  • 11:07 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2016
  • 11:07 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2016.codfw.wmnet 151.32.192.10.in-addr.arpa 1.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:07 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2016.codfw.wmnet 151.32.192.10.in-addr.arpa 1.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:07 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:07 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2016 - jayme@cumin2002"
  • 11:07 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2016 - jayme@cumin2002"
  • 11:03 vgutierrez: restarting haproxy on cp1107
  • 10:58 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:58 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2016
  • 10:57 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2015
  • 10:57 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2015
  • 10:57 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2015.codfw.wmnet wikikube-worker2016.codfw.wmnet wikikube-worker2171.codfw.wmnet wikikube-worker2172.codfw.wmnet on all recursors
  • 10:57 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2015.codfw.wmnet wikikube-worker2016.codfw.wmnet wikikube-worker2171.codfw.wmnet wikikube-worker2172.codfw.wmnet on all recursors
  • 10:57 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2015
  • 10:57 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2015.codfw.wmnet 149.32.192.10.in-addr.arpa 9.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 10:57 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2015.codfw.wmnet 149.32.192.10.in-addr.arpa 9.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 10:57 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:57 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2015 - jayme@cumin2002"
  • 10:57 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2015 - jayme@cumin2002"
  • 10:54 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2016.codfw.wmnet with OS bookworm
  • 10:53 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:53 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2015
  • 10:52 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2171
  • 10:52 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2171
  • 10:52 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2171
  • 10:52 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2171.codfw.wmnet 152.32.192.10.in-addr.arpa 2.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 10:52 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2171.codfw.wmnet 152.32.192.10.in-addr.arpa 2.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 10:52 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:50 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:49 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2016.codfw.wmnet with OS bookworm
  • 10:49 brouberol@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-presto1005.eqiad.wmnet
  • 10:49 brouberol@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:48 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2016.codfw.wmnet with OS bookworm
  • 10:47 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2442 to wikikube-worker2016
  • 10:47 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2016
  • 10:46 brouberol@cumin2002: START - Cookbook sre.dns.netbox
  • 10:46 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2016
  • 10:46 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:46 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2442 to wikikube-worker2016 - jayme@cumin2002"
  • 10:46 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2442 to wikikube-worker2016 - jayme@cumin2002"
  • 10:41 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:41 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2172
  • 10:41 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2172
  • 10:41 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2172
  • 10:41 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2172.codfw.wmnet 77.48.192.10.in-addr.arpa 7.7.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 10:41 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2172.codfw.wmnet 77.48.192.10.in-addr.arpa 7.7.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 10:41 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:41 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2172 - jayme@cumin2002"
  • 10:40 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2172 - jayme@cumin2002"
  • 10:39 brouberol@cumin2002: START - Cookbook sre.hosts.decommission for hosts an-presto1005.eqiad.wmnet
  • 10:38 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2015.codfw.wmnet with OS bookworm
  • 10:38 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2440 to wikikube-worker2015
  • 10:37 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2442 to wikikube-worker2016
  • 10:37 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2015
  • 10:37 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2171
  • 10:36 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2171.codfw.wmnet with OS bookworm
  • 10:36 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:36 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2015
  • 10:36 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:36 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2172
  • 10:36 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2172.codfw.wmnet with OS bookworm
  • 10:35 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2443 to wikikube-worker2171
  • 10:35 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2171
  • 10:34 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:33 brouberol@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-presto1004.eqiad.wmnet
  • 10:33 moritzm: removing ganeti2018 from active Ganeti nodes T376594
  • 10:33 brouberol@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:30 brouberol@cumin2002: START - Cookbook sre.dns.netbox
  • 10:30 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2444 to wikikube-worker2172
  • 10:30 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2171
  • 10:30 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:30 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2172
  • 10:29 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2172
  • 10:29 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:29 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2444 to wikikube-worker2172 - jayme@cumin2002"
  • 10:28 vgutierrez: enabling outbound bandwidth limits enforced by haproxy on the upload cluster
  • 10:28 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:27 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from mw2442 to wikikube-worker2016
  • 10:27 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2444 to wikikube-worker2172 - jayme@cumin2002"
  • 10:27 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2442 to wikikube-worker2016
  • 10:23 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from mw2442 to wikikube-worker2016
  • 10:23 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2442 to wikikube-worker2016
  • 10:23 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:23 jayme@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 10:22 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2444 to wikikube-worker2172
  • 10:22 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2443 to wikikube-worker2171
  • 10:22 jayme@cumin2002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from mw2442 to wikikube-worker20160
  • 10:22 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 10:22 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2442 to wikikube-worker20160
  • 10:22 brouberol@cumin2002: START - Cookbook sre.hosts.decommission for hosts an-presto1004.eqiad.wmnet
  • 10:21 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 10:21 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:20 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2440 to wikikube-worker2015
  • 10:19 brouberol@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-presto1003.eqiad.wmnet
  • 10:19 brouberol@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:19 brouberol@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-presto1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin2002"
  • 10:19 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 10:19 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 10:19 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 10:19 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 10:13 brouberol@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-presto1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin2002"
  • 10:10 brouberol@cumin2002: START - Cookbook sre.dns.netbox
  • 10:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 8 hosts with reason: Rebooting
  • 10:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on 8 hosts with reason: Rebooting
  • 10:04 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2018.codfw.wmnet
  • 10:04 brouberol@cumin2002: START - Cookbook sre.hosts.decommission for hosts an-presto1003.eqiad.wmnet
  • 10:03 brouberol@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-presto1002.eqiad.wmnet
  • 10:03 brouberol@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:03 brouberol@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-presto1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin2002"
  • 10:02 brouberol@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-presto1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin2002"
  • 09:58 brouberol@cumin2002: START - Cookbook sre.dns.netbox
  • 09:56 godog: bump space for prometheus k8s-mlserve in eqiad
  • 09:50 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2444.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:50 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2443.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:46 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2440.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:39 brouberol@cumin2002: START - Cookbook sre.hosts.decommission for hosts an-presto1002.eqiad.wmnet
  • 09:36 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es2024.codfw.wmnet with reason: cloning
  • 09:35 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es2024.codfw.wmnet with reason: cloning
  • 09:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2024 to clone es2045', diff saved to https://phabricator.wikimedia.org/P71535 and previous config saved to /var/cache/conftool/dbconfig/20241204-093541-marostegui.json
  • 09:35 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es2023 to es5 master T381259', diff saved to https://phabricator.wikimedia.org/P71534 and previous config saved to /var/cache/conftool/dbconfig/20241204-093519-marostegui.json
  • 09:35 brouberol@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-presto1001.eqiad.wmnet
  • 09:35 brouberol@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:35 brouberol@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-presto1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin2002"
  • 09:34 brouberol@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-presto1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - brouberol@cumin2002"
  • 09:33 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2444.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:33 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2443.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:32 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2442.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:30 brouberol@cumin2002: START - Cookbook sre.dns.netbox
  • 09:21 brouberol@cumin2002: START - Cookbook sre.hosts.decommission for hosts an-presto1001.eqiad.wmnet
  • 09:15 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2442.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:14 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2440.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:12 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mw[2440,2442-2444].codfw.wmnet with reason: T377877
  • 09:12 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 100%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71533 and previous config saved to /var/cache/conftool/dbconfig/20241204-091229-root.json
  • 09:12 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on mw[2440,2442-2444].codfw.wmnet with reason: T377877
  • 09:07 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2440,2442-2444].codfw.wmnet
  • 09:05 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2440,2442-2444].codfw.wmnet
  • 08:57 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 75%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71532 and previous config saved to /var/cache/conftool/dbconfig/20241204-085724-root.json
  • 08:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es2022.codfw.wmnet with reason: cloning
  • 08:51 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es2022.codfw.wmnet with reason: cloning
  • 08:51 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2022 to clone es2043', diff saved to https://phabricator.wikimedia.org/P71531 and previous config saved to /var/cache/conftool/dbconfig/20241204-085143-marostegui.json
  • 08:51 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es2020 to es4 master T381259', diff saved to https://phabricator.wikimedia.org/P71530 and previous config saved to /var/cache/conftool/dbconfig/20241204-085124-marostegui.json
  • 08:46 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 100%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71529 and previous config saved to /var/cache/conftool/dbconfig/20241204-084650-root.json
  • 08:42 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 50%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71528 and previous config saved to /var/cache/conftool/dbconfig/20241204-084219-root.json
  • 08:35 moritzm: rebalance Ganeti eqiad/C following server refreshes
  • 08:31 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 75%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71527 and previous config saved to /var/cache/conftool/dbconfig/20241204-083145-root.json
  • 08:27 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 25%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71526 and previous config saved to /var/cache/conftool/dbconfig/20241204-082714-root.json
  • 08:25 kharlan@deploy2002: Finished scap sync-world: Backport for dialog: Don't duplicate the footer in the behaviour list template (T381189) (duration: 12m 08s)
  • 08:23 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2018.codfw.wmnet
  • 08:20 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2018.codfw.wmnet
  • 08:18 kharlan@deploy2002: kharlan: Continuing with sync
  • 08:18 kharlan@deploy2002: kharlan: Backport for dialog: Don't duplicate the footer in the behaviour list template (T381189) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 08:16 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 50%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71525 and previous config saved to /var/cache/conftool/dbconfig/20241204-081640-root.json
  • 08:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2018.codfw.wmnet
  • 08:13 kharlan@deploy2002: Started scap sync-world: Backport for dialog: Don't duplicate the footer in the behaviour list template (T381189)
  • 08:12 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 10%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71524 and previous config saved to /var/cache/conftool/dbconfig/20241204-081208-root.json
  • 08:01 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 25%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71522 and previous config saved to /var/cache/conftool/dbconfig/20241204-080134-root.json
  • 07:57 marostegui@cumin1002: dbctl commit (dc=all): 'es2046 (re)pooling @ 1%: Pooling in es5', diff saved to https://phabricator.wikimedia.org/P71520 and previous config saved to /var/cache/conftool/dbconfig/20241204-075703-root.json
  • 07:54 marostegui@cumin1002: dbctl commit (dc=all): 'Add es2046 to es5 depooled T381259', diff saved to https://phabricator.wikimedia.org/P71519 and previous config saved to /var/cache/conftool/dbconfig/20241204-075427-marostegui.json
  • 07:46 marostegui@cumin1002: dbctl commit (dc=all): 'es2042 (re)pooling @ 10%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71518 and previous config saved to /var/cache/conftool/dbconfig/20241204-074629-root.json
  • 07:08 marostegui@cumin1002: dbctl commit (dc=all): 'es2025 (re)pooling @ 100%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71517 and previous config saved to /var/cache/conftool/dbconfig/20241204-070855-root.json
  • 07:08 marostegui@cumin1002: dbctl commit (dc=all): 'es2021 (re)pooling @ 100%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71516 and previous config saved to /var/cache/conftool/dbconfig/20241204-070829-root.json
  • 06:53 marostegui@cumin1002: dbctl commit (dc=all): 'es2025 (re)pooling @ 75%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71515 and previous config saved to /var/cache/conftool/dbconfig/20241204-065349-root.json
  • 06:53 marostegui@cumin1002: dbctl commit (dc=all): 'es2021 (re)pooling @ 75%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71514 and previous config saved to /var/cache/conftool/dbconfig/20241204-065324-root.json
  • 06:38 marostegui@cumin1002: dbctl commit (dc=all): 'es2025 (re)pooling @ 50%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71513 and previous config saved to /var/cache/conftool/dbconfig/20241204-063844-root.json
  • 06:38 marostegui@cumin1002: dbctl commit (dc=all): 'es2021 (re)pooling @ 50%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71512 and previous config saved to /var/cache/conftool/dbconfig/20241204-063819-root.json
  • 06:23 marostegui@cumin1002: dbctl commit (dc=all): 'es2025 (re)pooling @ 25%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71511 and previous config saved to /var/cache/conftool/dbconfig/20241204-062339-root.json
  • 06:23 marostegui@cumin1002: dbctl commit (dc=all): 'es2021 (re)pooling @ 25%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71510 and previous config saved to /var/cache/conftool/dbconfig/20241204-062313-root.json
  • 06:18 marostegui@cumin1002: dbctl commit (dc=all): 'Add es2042 to dbctl depooled T381259', diff saved to https://phabricator.wikimedia.org/P71509 and previous config saved to /var/cache/conftool/dbconfig/20241204-061821-marostegui.json
  • 06:08 marostegui@cumin1002: dbctl commit (dc=all): 'es2025 (re)pooling @ 10%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71508 and previous config saved to /var/cache/conftool/dbconfig/20241204-060834-root.json
  • 06:08 marostegui@cumin1002: dbctl commit (dc=all): 'es2021 (re)pooling @ 10%: Repooling cloning', diff saved to https://phabricator.wikimedia.org/P71507 and previous config saved to /var/cache/conftool/dbconfig/20241204-060808-root.json
  • 02:40 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1045.eqiad.wmnet with OS bookworm
  • 02:33 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1046.eqiad.wmnet with OS bookworm
  • 02:33 jclark@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 02:32 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1041.eqiad.wmnet with OS bookworm
  • 02:32 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 02:08 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1085.eqiad.wmnet with OS bullseye
  • 01:56 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 01:46 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1043.eqiad.wmnet with OS bookworm
  • 01:42 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1044.eqiad.wmnet with OS bookworm
  • 01:39 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 01:39 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1046.eqiad.wmnet with reason: host reimage
  • 01:36 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on es1046.eqiad.wmnet with reason: host reimage
  • 01:23 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:22 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:22 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage
  • 01:20 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1046.eqiad.wmnet with OS bookworm
  • 01:20 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1045.eqiad.wmnet with OS bookworm
  • 01:19 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on es1041.eqiad.wmnet with reason: host reimage
  • 01:15 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:15 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 01:03 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1041.eqiad.wmnet with OS bookworm
  • 01:02 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es1041.eqiad.wmnet with OS bookworm
  • 01:00 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:00 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:57 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1042.eqiad.wmnet with OS bookworm
  • 00:57 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 00:56 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 00:53 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:52 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:50 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:48 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:48 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:47 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1085.eqiad.wmnet with OS bullseye
  • 00:47 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:43 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:43 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:42 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:42 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:41 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1085.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:40 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage
  • 00:37 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:36 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on es1042.eqiad.wmnet with reason: host reimage
  • 00:31 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1085.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:30 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2020.codfw.wmnet w/ force delete existing files, repooling source-only afterwards
  • 00:26 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:18 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1085.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:18 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1085.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:16 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:13 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1084.eqiad.wmnet with OS bullseye
  • 00:13 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002"
  • 00:09 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 00:09 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply

2024-12-03

  • 23:58 amastilovic@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
  • 23:52 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002"
  • 23:50 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1085.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 23:48 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1085.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 23:48 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1044.eqiad.wmnet with OS bookworm
  • 23:42 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 23:41 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1043.eqiad.wmnet with OS bookworm
  • 23:41 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1042.eqiad.wmnet with OS bookworm
  • 23:40 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host es1041.eqiad.wmnet with OS bookworm
  • 23:39 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 23:39 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt ms-be1085 - vriley@cumin1002"
  • 23:39 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt ms-be1085 - vriley@cumin1002"
  • 23:37 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1086.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 23:36 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2020.codfw.wmnet w/ force delete existing files, repooling source-only afterwards
  • 23:36 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2019.codfw.wmnet w/ force delete existing files, repooling source-only afterwards
  • 23:35 vriley@cumin1002: START - Cookbook sre.dns.netbox
  • 23:34 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1084.eqiad.wmnet with reason: host reimage
  • 23:30 vriley@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1084.eqiad.wmnet with reason: host reimage
  • 23:29 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1086
  • 23:28 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1086
  • 23:27 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1089.eqiad.wmnet with OS bullseye
  • 23:27 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 23:25 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 23:22 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 23:22 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for ms-be - jclark@cumin1002"
  • 23:22 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for ms-be - jclark@cumin1002"
  • 23:20 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1084.eqiad.wmnet with OS bullseye
  • 23:19 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1083.eqiad.wmnet with OS bullseye
  • 23:19 vriley@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002"
  • 23:19 jclark@cumin1002: START - Cookbook sre.dns.netbox
  • 23:12 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1087.eqiad.wmnet with OS bullseye
  • 23:12 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 23:11 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 23:11 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1090.eqiad.wmnet with OS bullseye
  • 23:11 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 23:11 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 23:08 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1088.eqiad.wmnet with OS bullseye
  • 23:08 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 23:08 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1089.eqiad.wmnet with reason: host reimage
  • 23:08 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 23:04 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1089.eqiad.wmnet with reason: host reimage
  • 23:04 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1084.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 23:02 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wdqs[2018-2020,2026-2027].codfw.wmnet with reason: T376150
  • 23:02 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wdqs[2018-2020,2026-2027].codfw.wmnet with reason: T376150
  • 23:02 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wdqs[1026-1027].eqiad.wmnet with reason: T376150
  • 23:01 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on wdqs[1026-1027].eqiad.wmnet with reason: T376150
  • 22:57 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1002"
  • 22:53 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1084.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:53 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1089.eqiad.wmnet with OS bullseye
  • 22:52 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1090.eqiad.wmnet with reason: host reimage
  • 22:52 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1089.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:52 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1084.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:52 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1084.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:50 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1088.eqiad.wmnet with reason: host reimage
  • 22:46 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1087.eqiad.wmnet with reason: host reimage
  • 22:43 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1090.eqiad.wmnet with reason: host reimage
  • 22:43 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1088.eqiad.wmnet with reason: host reimage
  • 22:43 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2019.codfw.wmnet w/ force delete existing files, repooling source-only afterwards
  • 22:43 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1087.eqiad.wmnet with reason: host reimage
  • 22:42 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1089.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:38 ryankemper@deploy2002: Finished deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host (duration: 00m 13s)
  • 22:38 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host
  • 22:37 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1083.eqiad.wmnet with reason: host reimage
  • 22:35 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on wdqs[1026-1027].eqiad.wmnet with reason: T376150
  • 22:35 bking@cumin2002: START - Cookbook sre.hosts.downtime for 4:00:00 on wdqs[1026-1027].eqiad.wmnet with reason: T376150
  • 22:34 vriley@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1083.eqiad.wmnet with reason: host reimage
  • 22:32 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1088.eqiad.wmnet with OS bullseye
  • 22:32 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1090.eqiad.wmnet with OS bullseye
  • 22:32 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1087.eqiad.wmnet with OS bullseye
  • 22:32 ryankemper@deploy2002: Finished deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-scholarly host (duration: 00m 13s)
  • 22:32 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-scholarly host
  • 22:32 ryankemper@deploy2002: deploy aborted: deploy to fresh wdqs-internal-scholarly host (duration: 03m 59s)
  • 22:32 dancy@deploy2002: Installation of scap version "4.132.0" completed for 1 hosts
  • 22:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1090.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1088.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1087.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:31 dancy@deploy2002: Installing scap version "4.132.0" for 1 host(s)
  • 22:28 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-scholarly host
  • 22:23 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1083.eqiad.wmnet with OS bullseye
  • 22:21 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1089.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:21 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1090.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:21 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1088.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:21 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1089.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:21 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1087.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:15 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1084.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:15 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1084.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:12 brett@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 22:10 brett@cumin2002: START - Cookbook sre.dns.netbox
  • 21:52 ebernhardson@deploy2002: Finished scap sync-world: Backport for cirrus: Configure MLR buckets (T377128) (duration: 17m 47s)
  • 21:45 ebernhardson@deploy2002: ebernhardson: Continuing with sync
  • 21:40 ebernhardson@deploy2002: ebernhardson: Backport for cirrus: Configure MLR buckets (T377128) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 21:34 ebernhardson@deploy2002: Started scap sync-world: Backport for cirrus: Configure MLR buckets (T377128)
  • 21:32 ebernhardson@deploy2002: Finished scap sync-world: Backport for Rerunning Web browser extension survey (T380778), Reenable non-UI experiment quick survey (T379241), Deploy Vector22 To Wikis (T381041) (duration: 22m 00s)
  • 21:28 swfrench@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:28 swfrench@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Backfill allocations for mw-parsoid LVS VIPs - swfrench@cumin2002"
  • 21:28 swfrench@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Backfill allocations for mw-parsoid LVS VIPs - swfrench@cumin2002"
  • 21:24 ebernhardson@deploy2002: bwang, ebernhardson, lmora, jdrewniak: Continuing with sync
  • 21:23 swfrench@cumin2002: START - Cookbook sre.dns.netbox
  • 21:16 ebernhardson@deploy2002: bwang, ebernhardson, lmora, jdrewniak: Backport for Rerunning Web browser extension survey (T380778), Reenable non-UI experiment quick survey (T379241), Deploy Vector22 To Wikis (T381041) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 21:10 ebernhardson@deploy2002: Started scap sync-world: Backport for Rerunning Web browser extension survey (T380778), Reenable non-UI experiment quick survey (T379241), Deploy Vector22 To Wikis (T381041)
  • 21:08 dancy@deploy2002: Installation of scap version "4.132.0" completed for 1 hosts
  • 21:07 dancy@deploy2002: Installing scap version "4.132.0" for 1 host(s)
  • 20:49 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1087
  • 20:49 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1087
  • 20:49 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1088
  • 20:48 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1088
  • 20:48 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1090
  • 20:48 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1090
  • 20:48 jclark@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be1089
  • 20:48 jclark@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be1089
  • 20:46 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:46 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for ms-be - jclark@cumin1002"
  • 20:46 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for ms-be - jclark@cumin1002"
  • 20:42 jclark@cumin1002: START - Cookbook sre.dns.netbox
  • 20:38 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1278-1279].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 20:38 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1279.eqiad.wmnet with OS bookworm
  • 20:19 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1279.eqiad.wmnet with reason: host reimage
  • 20:15 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1279.eqiad.wmnet with reason: host reimage
  • 20:01 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1026.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 20:00 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal scholarly tier) xfer scholarly_articles from wdqs1023.eqiad.wmnet -> wdqs1027.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 19:57 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2018.codfw.wmnet w/ force delete existing files, repooling source-only afterwards
  • 19:55 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1279.eqiad.wmnet with OS bookworm
  • 19:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1278.eqiad.wmnet with OS bookworm
  • 19:34 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1278.eqiad.wmnet with reason: host reimage
  • 19:31 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1278.eqiad.wmnet with reason: host reimage
  • 19:19 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet
  • 19:18 jhuneidi@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.6 refs T375665
  • 19:15 topranks: rebooting rpki2003 to clear out tmpfs filesystem which is full
  • 19:15 cmooney@cumin1002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet
  • 19:14 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal scholarly tier) xfer scholarly_articles from wdqs1023.eqiad.wmnet -> wdqs1027.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 19:13 ryankemper@deploy2002: Finished deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-scholarly host (duration: 00m 07s)
  • 19:13 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-scholarly host
  • 19:13 ryankemper@deploy2002: Finished deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host (duration: 01m 09s)
  • 19:11 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1026.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 19:11 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host
  • 19:11 ryankemper@deploy2002: Finished deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host (duration: 02m 45s)
  • 19:10 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1278.eqiad.wmnet with OS bookworm
  • 19:09 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host
  • 19:04 bking@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1026.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 19:04 kamila@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1278-1279].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
  • 19:02 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1026.eqiad.wmnet w/ force delete existing files, repooling source-only afterwards
  • 19:00 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2018.codfw.wmnet w/ force delete existing files, repooling source-only afterwards
  • 18:59 bking@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T376150, initialize wdqs internal main tier) xfer scholarly_articles from wdqs2021.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling source-only afterwards
  • 18:58 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on wdqs2027.codfw.wmnet with reason: T376150
  • 18:58 bking@cumin2002: START - Cookbook sre.hosts.downtime for 4:00:00 on wdqs2027.codfw.wmnet with reason: T376150
  • 18:56 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T376150, initialize wdqs internal main tier) xfer scholarly_articles from wdqs2021.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling source-only afterwards
  • 18:49 ryankemper@deploy2002: Finished deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host (duration: 00m 14s)
  • 18:49 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host
  • 18:47 ryankemper@deploy2002: Finished deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host (duration: 00m 14s)
  • 18:47 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host
  • 18:43 ryankemper@deploy2002: Finished deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host (duration: 03m 31s)
  • 18:40 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-main host
  • 18:39 ryankemper@deploy2002: Finished deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-scholarly host (duration: 00m 11s)
  • 18:39 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-scholarly host
  • 18:39 ryankemper@deploy2002: Finished deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-scholarly host (duration: 00m 11s)
  • 18:39 ryankemper@deploy2002: Started deploy [wdqs/wdqs@9927a5a]: deploy to fresh wdqs-internal-scholarly host
  • 18:35 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1034-1035].eqiad.wmnet
  • 18:35 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1034-1035].eqiad.wmnet
  • 18:23 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 18:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1035.eqiad.wmnet with OS bookworm
  • 18:00 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1091.eqiad.wmnet with OS bullseye
  • 18:00 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 17:57 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 17:57 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 17:57 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 17:56 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 17:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1035.eqiad.wmnet with reason: host reimage
  • 17:50 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on wdqs2026.codfw.wmnet with reason: T376150
  • 17:50 bking@cumin2002: START - Cookbook sre.hosts.downtime for 4:00:00 on wdqs2026.codfw.wmnet with reason: T376150
  • 17:48 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1035.eqiad.wmnet with reason: host reimage
  • 17:47 brett@puppetserver1001: conftool action : set/pooled=yes; selector: dc=magru,service=cdn,name=cp7001.magru.wmnet
  • 17:46 brett: Removing RSA certificate support from haproxy/cp (T370837)
  • 17:38 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 17:32 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1035.eqiad.wmnet with OS bookworm
  • 17:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1034.eqiad.wmnet with OS bookworm
  • 17:20 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1091.eqiad.wmnet with reason: host reimage
  • 17:17 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1091.eqiad.wmnet with reason: host reimage
  • 17:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1034.eqiad.wmnet with reason: host reimage
  • 17:08 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1034.eqiad.wmnet with reason: host reimage
  • 17:07 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1091.eqiad.wmnet with OS bullseye
  • 16:58 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1091.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:52 brett@puppetserver1001: conftool action : set/pooled=no; selector: dc=magru,service=cdn,name=cp7001.magru.wmnet
  • 16:51 sbisson@deploy2002: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
  • 16:51 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1034.eqiad.wmnet with OS bookworm
  • 16:50 urbanecm@deploy2002: Finished scap sync-world: Backport for Revert "Increase Nuke max age to 90 days" (T380846) (duration: 12m 29s)
  • 16:49 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1034.eqiad.wmnet with OS bookworm
  • 16:47 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1091.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:44 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1091.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:44 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1091.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:38 urbanecm@deploy2002: Started scap sync-world: Backport for Revert "Increase Nuke max age to 90 days" (T380846)
  • 16:30 brett: Disabling puppet on A:cp to prep for RSA removal - T370837
  • 16:27 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1091.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:27 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1091.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 16:19 moritzm: rebalance Ganeti eqiad/B following server refreshes
  • 16:07 moritzm: installing intel-microcode security updates
  • 15:51 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1034.eqiad.wmnet with OS bookworm
  • 15:48 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1034.eqiad.wmnet wikikube-worker1035.eqiad.wmnet on all recursors
  • 15:48 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1034.eqiad.wmnet wikikube-worker1035.eqiad.wmnet on all recursors
  • 15:48 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1022 to wikikube-worker1035
  • 15:47 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1035
  • 15:45 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1035
  • 15:45 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:45 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1022 to wikikube-worker1035 - jelto@cumin1002"
  • 15:43 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1022 to wikikube-worker1035 - jelto@cumin1002"
  • 15:39 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 15:39 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1022 to wikikube-worker1035
  • 15:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1021 to wikikube-worker1034
  • 15:37 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1034
  • 15:36 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1034
  • 15:36 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:36 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1021 to wikikube-worker1034 - jelto@cumin1002"
  • 15:35 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1021 to wikikube-worker1034 - jelto@cumin1002"
  • 15:31 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 15:31 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1021 to wikikube-worker1034
  • 15:14 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1021-1022].eqiad.wmnet
  • 15:13 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1021-1022].eqiad.wmnet
  • 15:10 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2242.codfw.wmnet with OS bookworm
  • 15:10 jhancock@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
  • 15:10 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2241.codfw.wmnet with OS bookworm
  • 15:09 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
  • 14:57 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
  • 14:52 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
  • 14:44 urbanecm@deploy2002: Finished scap sync-world: Backport for fix: show thumbnails in surfacing popups (T381364), fix: show thumbnails in surfacing popups (T381364) (duration: 19m 24s)
  • 14:38 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2241.codfw.wmnet with reason: host reimage
  • 14:37 urbanecm@deploy2002: migr, urbanecm: Continuing with sync
  • 14:35 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2242.codfw.wmnet with reason: host reimage
  • 14:32 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2241.codfw.wmnet with reason: host reimage
  • 14:32 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2242.codfw.wmnet with reason: host reimage
  • 14:30 urbanecm@deploy2002: migr, urbanecm: Backport for fix: show thumbnails in surfacing popups (T381364), fix: show thumbnails in surfacing popups (T381364) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:25 urbanecm@deploy2002: Started scap sync-world: Backport for fix: show thumbnails in surfacing popups (T381364), fix: show thumbnails in surfacing popups (T381364)
  • 14:22 urbanecm@deploy2002: Finished scap sync-world: Backport for Increase Nuke max age to 90 days (T380846), knwiki: remove module namespace names from core-Namespaces.php (T346583), Remove temporary fix for badly set CentralAuth cookies (duration: 17m 04s)
  • 14:17 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host db2241.codfw.wmnet with OS bookworm
  • 14:17 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host db2242.codfw.wmnet with OS bookworm
  • 14:13 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1015-1016].eqiad.wmnet
  • 14:13 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1015-1016].eqiad.wmnet
  • 14:13 urbanecm@deploy2002: matmarex, chlod, urbanecm, anzx: Continuing with sync
  • 14:11 urbanecm@deploy2002: matmarex, chlod, urbanecm, anzx: Backport for Increase Nuke max age to 90 days (T380846), knwiki: remove module namespace names from core-Namespaces.php (T346583), Remove temporary fix for badly set CentralAuth cookies synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:05 urbanecm@deploy2002: Started scap sync-world: Backport for Increase Nuke max age to 90 days (T380846), knwiki: remove module namespace names from core-Namespaces.php (T346583), Remove temporary fix for badly set CentralAuth cookies
  • 13:57 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 13:41 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.restart_sanitarium (exit_code=99) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:41 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.restart_sanitarium (exit_code=0) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:40 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:39 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.restart_sanitarium (exit_code=99) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:39 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:35 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.restart_sanitarium (exit_code=99) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:34 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:34 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.restart_sanitarium (exit_code=0) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:34 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.restart_sanitarium (exit_code=0) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:33 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.restart_sanitarium (exit_code=0) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:32 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:30 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.restart_sanitarium (exit_code=99) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:30 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:28 fabfur: upgrade haproxykafka to version 0.3.4 (https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/commits/main?ref_type=heads) (T380583)
  • 13:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1022.eqiad.wmnet
  • 13:25 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:25 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1022.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 13:24 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1022.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 13:23 jelto@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
  • 13:22 jelto@deploy2002: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
  • 13:22 jelto@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
  • 13:22 jelto@deploy2002: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
  • 13:21 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:20 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 13:20 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:19 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.restart_sanitarium (exit_code=99) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:19 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:19 jelto@deploy2002: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
  • 13:18 jelto@deploy2002: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
  • 13:18 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.restart_sanitarium (exit_code=99) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:18 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1016.eqiad.wmnet with OS bookworm
  • 13:14 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1022.eqiad.wmnet
  • 13:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti1012.eqiad.wmnet
  • 13:14 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:14 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 13:13 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti1012.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 13:13 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.restart_sanitarium (exit_code=99) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:13 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:10 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 13:10 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.restart_sanitarium (exit_code=99) Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:10 arnaudb@cumin1002: START - Cookbook sre.mysql.restart_sanitarium Restart a pool of Sanitarium MariaDB instances and/or hosts.
  • 13:06 jnuche@deploy2002: Installation of scap version "4.132.0" completed for 1 hosts
  • 13:06 jnuche@deploy2002: Installing scap version "4.132.0" for 1 host(s)
  • 13:04 jmm@cumin2002: START - Cookbook sre.hosts.decommission for hosts ganeti1012.eqiad.wmnet
  • 12:57 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1016.eqiad.wmnet with reason: host reimage
  • 12:57 jnuche@deploy2002: Installing scap version "4.132.0" for 207 host(s)
  • 12:56 jnuche@deploy2002: Installation of scap version "4.132.0" completed for 1 hosts
  • 12:55 jnuche@deploy2002: Installing scap version "4.132.0" for 1 host(s)
  • 12:54 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1016.eqiad.wmnet with reason: host reimage
  • 12:54 jnuche@deploy2002: Installation of scap version "4.132.0" completed for 1 hosts
  • 12:53 jnuche@deploy2002: Installing scap version "4.132.0" for 1 host(s)
  • 12:47 klausman@cumin1002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ml-lab1001.eqiad.wmnet
  • 12:37 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1016.eqiad.wmnet with OS bookworm
  • 12:36 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1015.eqiad.wmnet with OS bookworm
  • 12:35 klausman@cumin1002: START - Cookbook sre.hosts.reboot-single for host ml-lab1001.eqiad.wmnet
  • 12:18 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1015.eqiad.wmnet with reason: host reimage
  • 12:15 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1015.eqiad.wmnet with reason: host reimage
  • 11:58 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1015.eqiad.wmnet with OS bookworm
  • 11:53 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kubernetes1019.eqiad.wmnet wikikube-worker1015.eqiad.wmnet kubernetes1020.eqiad.wmnet wikikube-worker1016.eqiad.wmnet on all recursors
  • 11:53 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache kubernetes1019.eqiad.wmnet wikikube-worker1015.eqiad.wmnet kubernetes1020.eqiad.wmnet wikikube-worker1016.eqiad.wmnet on all recursors
  • 11:50 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1020 to wikikube-worker1016
  • 11:50 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1016
  • 11:49 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1016
  • 11:49 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:49 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1020 to wikikube-worker1016 - jelto@cumin1002"
  • 11:49 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1020 to wikikube-worker1016 - jelto@cumin1002"
  • 11:45 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 11:44 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1020 to wikikube-worker1016
  • 11:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1019 to wikikube-worker1015
  • 11:43 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1015
  • 11:42 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1015
  • 11:42 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:42 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1019 to wikikube-worker1015 - jelto@cumin1002"
  • 11:41 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1019 to wikikube-worker1015 - jelto@cumin1002"
  • 11:37 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 11:37 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1019 to wikikube-worker1015
  • 11:33 volans@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1061.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:32 volans@cumin1002: START - Cookbook sre.hosts.provision for host cloudvirt1061.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 11:31 topranks: pushing new nftables rules to cloudgw1001 to block abuse from paws T381078
  • 11:20 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es2025.codfw.wmnet with reason: cloning
  • 11:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es2025.codfw.wmnet with reason: cloning
  • 11:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2025 to clone es2046', diff saved to https://phabricator.wikimedia.org/P71497 and previous config saved to /var/cache/conftool/dbconfig/20241203-112015-marostegui.json
  • 10:49 volans: installed spicerack v9.0.0 on cumin[12]002
  • 10:42 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[1019-1020].eqiad.wmnet
  • 10:41 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[1019-1020].eqiad.wmnet
  • 10:30 volans@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1061.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:27 volans@cumin1002: START - Cookbook sre.hosts.provision for host cloudvirt1061.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 100%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71496 and previous config saved to /var/cache/conftool/dbconfig/20241203-102143-root.json
  • 10:19 volans@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Update hieradata from Netbox - volans@cumin2002"
  • 10:19 volans@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Update hieradata from Netbox - volans@cumin2002"
  • 10:16 robh@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti7004.magru.wmnet with OS bookworm
  • 10:16 robh@cumin2002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - robh@cumin2002"
  • 10:16 bking@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1027.eqiad.wmnet with OS bullseye
  • 10:16 bking@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - bking@cumin1002"
  • 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 75%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71495 and previous config saved to /var/cache/conftool/dbconfig/20241203-100638-root.json
  • 09:53 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 09:53 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 09:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1006.eqiad.wmnet
  • 09:52 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1006.eqiad.wmnet
  • 09:51 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 50%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71494 and previous config saved to /var/cache/conftool/dbconfig/20241203-095133-root.json
  • 09:40 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 09:38 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 09:36 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 09:36 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 25%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71493 and previous config saved to /var/cache/conftool/dbconfig/20241203-093627-root.json
  • 09:31 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 09:31 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
  • 09:27 moritzm: rebalance Ganeti eqiad/A following server refreshes
  • 09:24 moritzm: removing ganeti1009 from active Ganeti nodes T378921
  • 09:22 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1009.eqiad.wmnet
  • 09:21 marostegui@cumin1002: dbctl commit (dc=all): 'es2041 (re)pooling @ 10%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71492 and previous config saved to /var/cache/conftool/dbconfig/20241203-092122-root.json
  • 08:45 jmm@cumin2002: END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: parse2017.codfw.wmnet
  • 08:45 jmm@cumin2002: START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: parse2017.codfw.wmnet
  • 08:37 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be1091.eqiad.wmnet with OS bullseye
  • 08:35 urbanecm@deploy2002: Finished scap sync-world: Backport for Growth: enable temporary Surfacing Alpha on pilot wikis (T379976) (duration: 21m 30s)
  • 08:34 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1091.eqiad.wmnet with OS bullseye
  • 08:32 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:31 elukey@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 08:27 moritzm: installing unbound security updates
  • 08:26 urbanecm@deploy2002: urbanecm, migr: Continuing with sync
  • 08:21 urbanecm@deploy2002: urbanecm, migr: Backport for Growth: enable temporary Surfacing Alpha on pilot wikis (T379976) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 08:14 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db1213 from dbctl T375593', diff saved to https://phabricator.wikimedia.org/P71489 and previous config saved to /var/cache/conftool/dbconfig/20241203-081434-marostegui.json
  • 08:13 urbanecm@deploy2002: Started scap sync-world: Backport for Growth: enable temporary Surfacing Alpha on pilot wikis (T379976)
  • 08:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1217.eqiad.wmnet with reason: Moving to m3
  • 08:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1217.eqiad.wmnet with reason: Moving to m3
  • 08:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1213.eqiad.wmnet with reason: Moving to m3
  • 08:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1213.eqiad.wmnet with reason: Moving to m3
  • 08:07 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1213', diff saved to https://phabricator.wikimedia.org/P71487 and previous config saved to /var/cache/conftool/dbconfig/20241203-080726-marostegui.json
  • 07:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es2021.codfw.wmnet with reason: cloning
  • 07:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es2021.codfw.wmnet with reason: cloning
  • 07:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2021', diff saved to https://phabricator.wikimedia.org/P71486 and previous config saved to /var/cache/conftool/dbconfig/20241203-075751-marostegui.json
  • 07:57 marostegui: Switchover es4 codfw master to es2022 dbmaint (this happened an hour ago) T381259
  • 07:28 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1009.eqiad.wmnet
  • 07:27 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1009.eqiad.wmnet
  • 07:21 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1009.eqiad.wmnet
  • 06:41 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 06:41 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Change VIPs for wdqs-internal-main and wdqs-internal-scholarly to avoid mw-parsoid collision - ryankemper@cumin2002"
  • 06:41 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Change VIPs for wdqs-internal-main and wdqs-internal-scholarly to avoid mw-parsoid collision - ryankemper@cumin2002"
  • 06:37 ryankemper@cumin2002: START - Cookbook sre.dns.netbox
  • 06:34 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es2022 to es4 master T381259', diff saved to https://phabricator.wikimedia.org/P71485 and previous config saved to /var/cache/conftool/dbconfig/20241203-063408-marostegui.json
  • 06:32 marostegui@cumin1002: dbctl commit (dc=all): 'es2020 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P71484 and previous config saved to /var/cache/conftool/dbconfig/20241203-063234-root.json
  • 06:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es2021.codfw.wmnet with reason: cloning
  • 06:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es2021.codfw.wmnet with reason: cloning
  • 06:17 marostegui@cumin1002: dbctl commit (dc=all): 'es2020 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P71483 and previous config saved to /var/cache/conftool/dbconfig/20241203-061729-root.json
  • 06:10 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 06:10 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add VIPs for wdqs-internal-main and wdqs-internal-scholarly - ryankemper@cumin2002"
  • 06:10 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add VIPs for wdqs-internal-main and wdqs-internal-scholarly - ryankemper@cumin2002"
  • 06:08 marostegui@cumin1002: dbctl commit (dc=all): 'Add es2041 to es4 with just minimal weight T381259', diff saved to https://phabricator.wikimedia.org/P71482 and previous config saved to /var/cache/conftool/dbconfig/20241203-060847-marostegui.json
  • 06:06 ryankemper@cumin2002: START - Cookbook sre.dns.netbox
  • 06:06 ryankemper: [Netbox] T379334 Aborted netbox sync cookbook due to wrong IPs for wdqs-internal-scholarly. Fixed in UI, re-running cookbook now
  • 06:06 marostegui@cumin1002: dbctl commit (dc=all): 'Add es2041 depooled T381259', diff saved to https://phabricator.wikimedia.org/P71481 and previous config saved to /var/cache/conftool/dbconfig/20241203-060614-marostegui.json
  • 06:06 ryankemper@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 06:02 marostegui@cumin1002: dbctl commit (dc=all): 'es2020 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P71480 and previous config saved to /var/cache/conftool/dbconfig/20241203-060224-root.json
  • 06:00 ryankemper@cumin2002: START - Cookbook sre.dns.netbox
  • 06:00 ryankemper: [Netbox] T379334 Added VIPs via UI for wdqs-internal-[main,scholarly].svc.[eqiad,codfw].wmnet
  • 05:47 marostegui@cumin1002: dbctl commit (dc=all): 'es2020 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P71479 and previous config saved to /var/cache/conftool/dbconfig/20241203-054718-root.json
  • 05:44 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on wdqs[2018-2020,2026-2027].codfw.wmnet with reason: T376150 non-prod hosts
  • 05:44 ryankemper@cumin2002: START - Cookbook sre.hosts.downtime for 12:00:00 on wdqs[2018-2020,2026-2027].codfw.wmnet with reason: T376150 non-prod hosts
  • 05:17 eileen: config revision changed from b3741848 to 694158ae
  • 05:17 eileen: civicrm upgraded from be7e5d33 to 6361a578
  • 05:01 mwpresync@deploy2002: Pruned MediaWiki: 1.44.0-wmf.3 (duration: 01m 27s)
  • 04:51 mwpresync@deploy2002: Finished scap sync-world: testwikis to 1.44.0-wmf.6 refs T375665 (duration: 48m 24s)
  • 04:02 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.44.0-wmf.6 refs T375665
  • 02:56 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1091.eqiad.wmnet with OS bullseye
  • 02:37 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1084.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 02:36 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1084.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 02:20 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 02:20 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt ms-be1084 - vriley@cumin1002"
  • 02:20 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt ms-be1084 - vriley@cumin1002"
  • 02:16 vriley@cumin1002: START - Cookbook sre.dns.netbox
  • 01:53 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:47 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:38 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:38 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:36 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1091.eqiad.wmnet with OS bullseye
  • 01:35 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:35 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:26 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1091.eqiad.wmnet with OS bullseye
  • 01:26 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:26 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:22 pt1979@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 01:22 pt1979@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:34 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:34 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be1091.eqiad.wmnet with OS bullseye
  • 00:34 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:32 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:32 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:31 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:30 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:25 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1091.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 00:15 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1091.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART

2024-12-02

  • 23:58 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1091.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 23:50 jclark@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1091.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 23:50 jclark@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 23:50 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for ms-be - jclark@cumin1002"
  • 23:50 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added mgmt for ms-be - jclark@cumin1002"
  • 23:46 jclark@cumin1002: START - Cookbook sre.dns.netbox
  • 22:27 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:27 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:27 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:26 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:25 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:24 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:21 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:21 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:20 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:20 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:18 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:18 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:16 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:16 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:05 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 22:05 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 21:45 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 21:40 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 21:36 urbanecm@deploy2002: Finished scap sync-world: Backport for testwiki: no growth experiment anymore (T380659), fix(surfacing): don't redirect to desktop (duration: 13m 22s)
  • 21:35 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 21:29 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 21:29 urbanecm@deploy2002: migr, urbanecm: Continuing with sync
  • 21:27 urbanecm@deploy2002: migr, urbanecm: Backport for testwiki: no growth experiment anymore (T380659), fix(surfacing): don't redirect to desktop synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 21:22 urbanecm@deploy2002: Started scap sync-world: Backport for testwiki: no growth experiment anymore (T380659), fix(surfacing): don't redirect to desktop
  • 21:21 urbanecm@deploy2002: Finished scap sync-world: Backport for Enable VisualEditor by default on Indonesian Wikiquote (T381214), votewiki, testwiki: add securepoll-edit-poll to electionadmin (T377531), cawiki: stop Flow being the default for some talk namespaces (T381295) (duration: 13m 40s)
  • 21:15 urbanecm@deploy2002: kemayo, urbanecm, nmw03, sd: Continuing with sync
  • 21:14 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 21:12 urbanecm@deploy2002: kemayo, urbanecm, nmw03, sd: Backport for Enable VisualEditor by default on Indonesian Wikiquote (T381214), votewiki, testwiki: add securepoll-edit-poll to electionadmin (T377531), cawiki: stop Flow being the default for some talk namespaces (T381295) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 21:12 vriley@cumin1002: START - Cookbook sre.hosts.provision for host ms-be1083.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
  • 21:09 vriley@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:09 vriley@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt ms-be1083 - vriley@cumin1002"
  • 21:09 vriley@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt ms-be1083 - vriley@cumin1002"
  • 21:08 urbanecm@deploy2002: Started scap sync-world: Backport for Enable VisualEditor by default on Indonesian Wikiquote (T381214), votewiki, testwiki: add securepoll-edit-poll to electionadmin (T377531), cawiki: stop Flow being the default for some talk namespaces (T381295)
  • 21:04 vriley@cumin1002: START - Cookbook sre.dns.netbox
  • 20:10 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp3069.esams.wmnet [reason: done: checking icinga alerts]
  • 19:55 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp3069.esams.wmnet [reason: checking icinga alerts]
  • 19:22 volans@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1001.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:22 volans@cumin1002: START - Cookbook sre.hosts.provision for host sretest1001.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:21 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs3010.esams.wmnet
  • 19:20 volans@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest1001.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 19:15 dancy@deploy2002: Installation of scap version "4.131.0" completed for 207 hosts
  • 19:14 sukhe@cumin1002: START - Cookbook sre.hosts.reboot-single for host lvs3010.esams.wmnet
  • 19:13 sukhe: rebooting lvs3010 to test CR 1093958
  • 19:11 dancy@deploy2002: Installing scap version "4.131.0" for 207 hosts
  • 19:07 sukhe: disable puppet on A:lvs to finish rolling out CR 1093958: T358260
  • 19:01 volans@cumin1002: START - Cookbook sre.hosts.provision for host sretest1001.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 18:39 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2242']
  • 18:39 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['db2241']
  • 18:39 urbanecm@deploy2002: Finished scap sync-world: Backport for ApiQueryLinkRecommendations: Do not use relative protocol URIs (T381277) (duration: 10m 35s)
  • 18:38 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2242']
  • 18:38 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['db2241']
  • 18:37 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
  • 18:36 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply
  • 18:35 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
  • 18:35 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
  • 18:35 jiji@deploy2002: helmfile [staging] START helmfile.d/services/eventstreams: apply
  • 18:35 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-page-content-change-enrich: apply
  • 18:34 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-page-content-change-enrich: apply
  • 18:34 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply
  • 18:34 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
  • 18:34 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/eventgate-main: apply
  • 18:34 jiji@deploy2002: helmfile [staging] START helmfile.d/services/eventgate-main: apply
  • 18:33 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/changeprop: apply
  • 18:33 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/changeprop: apply
  • 18:33 jiji@deploy2002: helmfile [staging] START helmfile.d/services/changeprop: apply
  • 18:33 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 18:32 urbanecm@deploy2002: urbanecm: Continuing with sync
  • 18:32 urbanecm@deploy2002: urbanecm: Backport for ApiQueryLinkRecommendations: Do not use relative protocol URIs (T381277) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 18:32 jiji@deploy2002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 18:32 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 18:31 jiji@deploy2002: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
  • 18:31 jiji@deploy2002: helmfile [staging] DONE helmfile.d/services/benthos-cache-invalidator: apply
  • 18:31 jiji@deploy2002: helmfile [staging] START helmfile.d/services/benthos-cache-invalidator: apply
  • 18:28 urbanecm@deploy2002: Started scap sync-world: Backport for ApiQueryLinkRecommendations: Do not use relative protocol URIs (T381277)
  • 18:19 jiji@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 18:18 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2242.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 18:18 jiji@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 18:17 jiji@deploy2002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 18:17 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2241.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 18:17 jiji@deploy2002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 18:17 jiji@deploy2002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 18:17 jiji@deploy2002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 18:17 jiji@deploy2002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 18:16 jiji@deploy2002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 18:16 jiji@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 18:16 jiji@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 18:16 jiji@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 18:15 jiji@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 18:15 jiji@deploy2002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 18:15 jiji@deploy2002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 18:15 jiji@deploy2002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 18:15 jiji@deploy2002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 18:00 urbanecm@deploy2002: urbanecm: Continuing with sync
  • 18:00 urbanecm@deploy2002: urbanecm: Backport for ApiQueryLinkRecommendations: Do not use relative protocol URIs (T381277) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 17:58 jiji@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad
  • 17:57 urbanecm@deploy2002: Started scap sync-world: Backport for ApiQueryLinkRecommendations: Do not use relative protocol URIs (T381277)
  • 17:54 urbanecm@deploy2002: Sync cancelled.
  • 17:54 urbanecm@deploy2002: urbanecm: Backport for ApiQueryLinkRecommendations: Do not use relative protocol URIs (T381277) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 17:54 fabfur@cumin1002: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox
  • 17:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1006.eqiad.wmnet with OS bookworm
  • 17:50 urbanecm@deploy2002: Started scap sync-world: Backport for ApiQueryLinkRecommendations: Do not use relative protocol URIs (T381277)
  • 17:48 jiji@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad
  • 17:48 dancy@deploy2002: Installation of scap version "4.129.0" completed for 207 hosts
  • 17:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2242.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 17:46 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host db2241.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
  • 17:44 dancy@deploy2002: Installing scap version "4.129.0" for 207 hosts
  • 17:43 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:43 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2241-2 to codfw - jhancock@cumin2002"
  • 17:43 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2241-2 to codfw - jhancock@cumin2002"
  • 17:38 jhancock@cumin2002: START - Cookbook sre.dns.netbox
  • 17:33 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1006.eqiad.wmnet with reason: host reimage
  • 17:31 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1006.eqiad.wmnet with reason: host reimage
  • 17:16 fabfur@cumin1002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough and A:wikidough
  • 17:13 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1006.eqiad.wmnet with OS bookworm
  • 17:07 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "missing data for wikikube-worker1006 - jayme@cumin1002"
  • 17:07 topranks: resetting ulsfo->eqsin link to normal metric to put all codfw->eqsin traffic back on Aerlion cct
  • 17:07 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "missing data for wikikube-worker1006 - jayme@cumin1002"
  • 17:03 fabfur@cumin1002: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough and A:wikidough
  • 16:55 fabfur@cumin1002: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox
  • 16:54 jdrewniak@deploy2002: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 02m 28s)
  • 16:52 jdrewniak@deploy2002: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 10m 36s)
  • 16:38 dancy@deploy2002: Installation of scap version "4.130.0" completed for 207 hosts
  • 16:34 dancy@deploy2002: Installing scap version "4.130.0" for 207 hosts
  • 16:32 jan_drewniak: starting portals deploy
  • 16:25 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1006.eqiad.wmnet with OS bookworm
  • 16:00 bking@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1025.eqiad.wmnet']
  • 16:00 bking@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wdqs1025.eqiad.wmnet']
  • 16:00 bking@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1025.eqiad.wmnet']
  • 15:59 bking@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1025.eqiad.wmnet']
  • 15:58 bking@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host wdqs1025.eqiad.wmnet with OS bullseye
  • 15:50 bking@cumin2002: START - Cookbook sre.hosts.reimage for host wdqs1025.eqiad.wmnet with OS bullseye
  • 15:47 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 15:46 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/citoid: apply
  • 15:42 volans: uploaded spicerack_9.0.0 to apt.wikimedia.org bullseye-wikimedia
  • 15:42 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 15:42 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/citoid: apply
  • 15:32 taavi@deploy2002: Finished scap sync-world: Backport for wikitech: Drop contentadmin group (T375950) (duration: 09m 42s)
  • 15:29 sukhe: sudo cumin -b1 -s10 "A:cp" 'run-puppet-agent --enable "merging CR 1091748"'
  • 15:26 taavi@deploy2002: taavi: Continuing with sync
  • 15:26 taavi@deploy2002: taavi: Backport for wikitech: Drop contentadmin group (T375950) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 15:24 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet [reason: [done] testing CR 1091748]
  • 15:22 taavi@deploy2002: Started scap sync-world: Backport for wikitech: Drop contentadmin group (T375950)
  • 15:17 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet [reason: testing CR 1091748]
  • 15:14 sukhe: sudo cumin "A:cp" 'disable-puppet "merging CR 1091748"' [trafficserver: remove inbound TLS and related settings]
  • 15:08 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1006.eqiad.wmnet with OS bookworm
  • 15:03 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kubernetes1018.eqiad.wmnet wikikube-worker1006.eqiad.wmnet on all recursors
  • 15:03 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache kubernetes1018.eqiad.wmnet wikikube-worker1006.eqiad.wmnet on all recursors
  • 15:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1018 to wikikube-worker1006
  • 15:01 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1006
  • 14:59 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1006
  • 14:59 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:58 marostegui: Deploy schema change on db1167 dbmaint eqiad - s8 sanitarium master, there will be days of lag in wikireplicas in s8 T367856
  • 14:57 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 14:55 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:53 sukhe@cumin1002: START - Cookbook sre.dns.netbox
  • 14:50 sukhe: running authdns-update for CR 1099713
  • 14:44 jelto@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 14:43 urbanecm@deploy2002: Finished scap sync-world: Backport for [Growth] testwiki: Enable Surfacing structured tasks (T379976), Prepare for surfacing structured tasks (squashed) (T379976) (duration: 19m 08s)
  • 14:36 urbanecm@deploy2002: migr, urbanecm: Continuing with sync
  • 14:34 moritzm: installing curl security updates
  • 14:29 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 14:29 jiji@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts mc-gp[1001-1003].eqiad.wmnet
  • 14:29 jiji@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 14:28 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1018 to wikikube-worker1006
  • 14:27 urbanecm@deploy2002: migr, urbanecm: Backport for [Growth] testwiki: Enable Surfacing structured tasks (T379976), Prepare for surfacing structured tasks (squashed) (T379976) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:27 jiji@cumin1002: START - Cookbook sre.dns.netbox
  • 14:25 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:25 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:24 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:24 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:23 urbanecm@deploy2002: Started scap sync-world: Backport for [Growth] testwiki: Enable Surfacing structured tasks (T379976), Prepare for surfacing structured tasks (squashed) (T379976)
  • 14:17 urbanecm@deploy2002: Finished scap sync-world: Backport for Drop $wgWikimediaCampaignEventsEnableCommunityList (T380075) (duration: 14m 37s)
  • 14:11 urbanecm@deploy2002: urbanecm, daimona: Continuing with sync
  • 14:08 jiji@cumin1002: START - Cookbook sre.hosts.decommission for hosts mc-gp[1001-1003].eqiad.wmnet
  • 14:07 urbanecm@deploy2002: urbanecm, daimona: Backport for Drop $wgWikimediaCampaignEventsEnableCommunityList (T380075) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:03 urbanecm@deploy2002: Started scap sync-world: Backport for Drop $wgWikimediaCampaignEventsEnableCommunityList (T380075)
  • 14:00 moritzm: removing ganeti1020 from active Ganeti nodes T378921
  • 13:57 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1007.eqiad.wmnet
  • 13:57 jiji@cumin1002: START - Cookbook sre.hosts.remove-downtime for kafka-main1007.eqiad.wmnet
  • 13:51 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on kafka-main[1003,1008].eqiad.wmnet with reason: Hardware refresh
  • 13:51 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on kafka-main[1003,1008].eqiad.wmnet with reason: Hardware refresh
  • 13:46 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P71471 and previous config saved to /var/cache/conftool/dbconfig/20241202-134648-root.json
  • 13:46 isaranto@deploy2002: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
  • 13:42 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 13:42 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 13:41 isaranto@deploy2002: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
  • 13:37 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on kafka-main[1002,1007].eqiad.wmnet with reason: Hardware refresh
  • 13:37 jiji@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on kafka-main[1002,1007].eqiad.wmnet with reason: Hardware refresh
  • 13:31 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P71470 and previous config saved to /var/cache/conftool/dbconfig/20241202-133143-root.json
  • 13:31 effie: repacing kafka-main1003 in production with kafka-main1008 - T363214
  • 13:30 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes1018.eqiad.wmnet
  • 13:30 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 13:30 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 13:29 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes1018.eqiad.wmnet
  • 13:27 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 13:26 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 13:24 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc-gp[2002-2003].codfw.wmnet
  • 13:24 jiji@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:24 jiji@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc-gp[2002-2003].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1002"
  • 13:21 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 13:21 jiji@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc-gp[2002-2003].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1002"
  • 13:21 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 13:20 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 13:20 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 13:18 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1005.eqiad.wmnet
  • 13:18 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1005.eqiad.wmnet
  • 13:17 jiji@cumin1002: START - Cookbook sre.dns.netbox
  • 13:16 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P71469 and previous config saved to /var/cache/conftool/dbconfig/20241202-131638-root.json
  • 13:06 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 13:05 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 13:05 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 13:01 jiji@cumin1002: START - Cookbook sre.hosts.decommission for hosts mc-gp[2002-2003].codfw.wmnet
  • 13:01 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P71467 and previous config saved to /var/cache/conftool/dbconfig/20241202-130132-root.json
  • 12:57 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1020.eqiad.wmnet
  • 12:22 topranks: re-routing traffic from Drmrs towards TECHLIB-TCZ - AS2852 - National Library of Technology, Prague, to avoid path via GEANT
  • 12:18 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2005-2006].codfw.wmnet
  • 12:18 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2005-2006].codfw.wmnet
  • 12:13 jiji@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mc-gp2001.codfw.wmnet
  • 12:13 jiji@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:13 jiji@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc-gp2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1002"
  • 12:06 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2005.codfw.wmnet with OS bookworm
  • 12:05 jiji@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: mc-gp2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jiji@cumin1002"
  • 12:04 jelto: homer 'cr*eqiad*' commit 'T377876'
  • 12:02 jiji@cumin1002: START - Cookbook sre.dns.netbox
  • 12:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1005.eqiad.wmnet with OS bookworm
  • 11:57 moritzm: upload mapnik 4.0.3+ds-2~wmf12u2 (adding a forward ported mapnik-config script to be consumed by node-mapnik even with the switch of mapnik 4 towards pkg-config) T327396
  • 11:56 jiji@cumin1002: START - Cookbook sre.hosts.decommission for hosts mc-gp2001.codfw.wmnet
  • 11:56 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2006.codfw.wmnet with OS bookworm
  • 11:55 marostegui: Stop mariadb on es2020 to clone es2041 T381259
  • 11:52 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1070.eqiad.wmnet
  • 11:52 mvernon@cumin2002: START - Cookbook sre.hosts.remove-downtime for ms-be1070.eqiad.wmnet
  • 11:46 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2005.codfw.wmnet with reason: host reimage
  • 11:46 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1070.eqiad.wmnet with reason: vacuum two overlarge container dbs
  • 11:45 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on ms-be1070.eqiad.wmnet with reason: vacuum two overlarge container dbs
  • 11:42 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1005.eqiad.wmnet with reason: host reimage
  • 11:42 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2005.codfw.wmnet with reason: host reimage
  • 11:38 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1005.eqiad.wmnet with reason: host reimage
  • 11:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es2020.codfw.wmnet with reason: cloning
  • 11:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es2020.codfw.wmnet with reason: cloning
  • 11:36 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2006.codfw.wmnet with reason: host reimage
  • 11:33 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2006.codfw.wmnet with reason: host reimage
  • 11:26 ladsgroup@deploy2002: Finished scap sync-world: Backport for Translate: Disable message group subscription feature for legalteamwiki (T372386 T381250) (duration: 11m 21s)
  • 11:23 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2005
  • 11:23 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2005
  • 11:23 topranks: rollback OSPF metric change on cr4-ulsfo to place all codfw to eqsin traffic back on primary transport link
  • 11:22 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1005.eqiad.wmnet with OS bookworm
  • 11:21 marostegui@cumin2002: dbctl commit (dc=all): 'Depool es2020 T381259', diff saved to https://phabricator.wikimedia.org/P71463 and previous config saved to /var/cache/conftool/dbconfig/20241202-112105-marostegui.json
  • 11:19 ladsgroup@deploy2002: abi, ladsgroup: Continuing with sync
  • 11:19 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2005
  • 11:19 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2005.codfw.wmnet 40.32.192.10.in-addr.arpa 0.4.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:19 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2005.codfw.wmnet 40.32.192.10.in-addr.arpa 0.4.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:19 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:19 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2005 - jayme@cumin2002"
  • 11:19 ladsgroup@deploy2002: abi, ladsgroup: Backport for Translate: Disable message group subscription feature for legalteamwiki (T372386 T381250) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 11:19 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2005 - jayme@cumin2002"
  • 11:15 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 11:15 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 11:15 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2088.codfw.wmnet with OS bullseye
  • 11:15 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kubernetes1017.eqiad.wmnet wikikube-worker1005.eqiad.wmnet on all recursors
  • 11:15 ladsgroup@deploy2002: Started scap sync-world: Backport for Translate: Disable message group subscription feature for legalteamwiki (T372386 T381250)
  • 11:14 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache kubernetes1017.eqiad.wmnet wikikube-worker1005.eqiad.wmnet on all recursors
  • 11:14 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 11:14 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2005
  • 11:13 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2006
  • 11:13 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2006
  • 11:13 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2006
  • 11:13 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2006.codfw.wmnet 141.32.192.10.in-addr.arpa 1.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:13 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2006.codfw.wmnet 141.32.192.10.in-addr.arpa 1.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
  • 11:13 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:13 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2006 - jayme@cumin2002"
  • 11:13 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2006 - jayme@cumin2002"
  • 11:09 ladsgroup@deploy2002: Started scap sync-world: Backport for Translate: Disable message group subscription feature for legalteamwiki (T372386 T381250)
  • 11:09 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 11:08 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 11:08 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 11:07 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 11:07 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1017 to wikikube-worker1005
  • 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1005
  • 11:05 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 11:05 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1005
  • 11:05 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:05 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1017 to wikikube-worker1005 - jelto@cumin1002"
  • 11:04 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1017 to wikikube-worker1005 - jelto@cumin1002"
  • 11:02 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2005.codfw.wmnet with OS bookworm
  • 11:02 jayme@cumin2002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2006
  • 11:02 jayme@cumin2002: START - Cookbook sre.hosts.reimage for host wikikube-worker2006.codfw.wmnet with OS bookworm
  • 11:01 jayme@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2005.codfw.wmnet wikikube-worker2006.codfw.wmnet on all recursors
  • 11:01 jayme@cumin2002: START - Cookbook sre.dns.wipe-cache wikikube-worker2005.codfw.wmnet wikikube-worker2006.codfw.wmnet on all recursors
  • 11:00 jelto@cumin1002: START - Cookbook sre.dns.netbox
  • 11:00 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1017 to wikikube-worker1005
  • 10:55 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2437 to wikikube-worker2006
  • 10:55 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2006
  • 10:54 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2006
  • 10:54 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:54 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2437 to wikikube-worker2006 - jayme@cumin2002"
  • 10:54 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2437 to wikikube-worker2006 - jayme@cumin2002"
  • 10:52 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw2436 to wikikube-worker2005
  • 10:52 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2088.codfw.wmnet with reason: host reimage
  • 10:51 jayme@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2005
  • 10:51 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:51 jayme@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2005
  • 10:51 jayme@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:51 jayme@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2436 to wikikube-worker2005 - jayme@cumin2002"
  • 10:50 jayme@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw2436 to wikikube-worker2005 - jayme@cumin2002"
  • 10:48 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2088.codfw.wmnet with reason: host reimage
  • 10:46 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2437 to wikikube-worker2006
  • 10:46 jayme@cumin2002: START - Cookbook sre.dns.netbox
  • 10:46 jayme@cumin2002: START - Cookbook sre.hosts.rename from mw2436 to wikikube-worker2005
  • 10:45 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes1017.eqiad.wmnet
  • 10:45 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes1017.eqiad.wmnet
  • 10:44 ladsgroup@deploy2002: Finished scap sync-world: Backport for Enable new ParserCache key schema on every page (T373037) (duration: 17m 25s)
  • 10:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc1017.eqiad.wmnet with OS bookworm
  • 10:37 ladsgroup@deploy2002: ladsgroup: Continuing with sync
  • 10:35 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be2088.codfw.wmnet with OS bullseye
  • 10:33 ladsgroup@deploy2002: ladsgroup: Backport for Enable new ParserCache key schema on every page (T373037) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 10:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
  • 10:32 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
  • 10:31 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2087.codfw.wmnet with OS bullseye
  • 10:26 ladsgroup@deploy2002: Started scap sync-world: Backport for Enable new ParserCache key schema on every page (T373037)
  • 10:16 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2437.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:16 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2436.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc1017.eqiad.wmnet with reason: host reimage
  • 10:12 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on pc1017.eqiad.wmnet with reason: host reimage
  • 10:12 marostegui: Deploy schema change on db1167 - s8 sanitarium master, there will be days of lag in wikireplicas in s8 T367856
  • 10:12 marostegui@cumin2002: dbctl commit (dc=all): 'Depool db1167 for an alter table', diff saved to https://phabricator.wikimedia.org/P71461 and previous config saved to /var/cache/conftool/dbconfig/20241202-101225-marostegui.json
  • 10:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: alter
  • 10:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: alter
  • 10:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 8:00:00 on db1167.eqiad.wmnet with reason: alter
  • 10:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 8:00:00 on db1167.eqiad.wmnet with reason: alter
  • 10:09 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2087.codfw.wmnet with reason: host reimage
  • 10:05 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2087.codfw.wmnet with reason: host reimage
  • 10:04 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2437.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 10:03 jayme@cumin2002: START - Cookbook sre.hosts.provision for host mw2436.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
  • 09:56 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host pc1017.eqiad.wmnet with OS bookworm
  • 09:52 jayme@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on mw[2436-2437].codfw.wmnet with reason: rename/reimage
  • 09:52 jayme@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on mw[2436-2437].codfw.wmnet with reason: rename/reimage
  • 09:52 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be2087.codfw.wmnet with OS bullseye
  • 09:51 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 09:51 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 09:48 jayme@cumin2002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[2436-2437].codfw.wmnet
  • 09:47 jayme@cumin2002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[2436-2437].codfw.wmnet
  • 09:45 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2086.codfw.wmnet with OS bullseye
  • 09:45 elukey@deploy2002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync
  • 09:45 elukey@deploy2002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: sync
  • 09:43 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: optimizing
  • 09:43 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: optimizing
  • 09:42 elukey@deploy2002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync
  • 09:41 elukey@deploy2002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: sync
  • 09:39 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 09:39 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 09:35 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 09:35 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 09:35 marostegui: Installing mariadb 10.6.20 on db1198 T378940
  • 09:28 marostegui@cumin2002: dbctl commit (dc=all): 'Depoll db1198 to install 10.6.20', diff saved to https://phabricator.wikimedia.org/P71460 and previous config saved to /var/cache/conftool/dbconfig/20241202-092854-marostegui.json
  • 09:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1198.eqiad.wmnet with reason: testing
  • 09:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db1198.eqiad.wmnet with reason: testing
  • 09:24 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2086.codfw.wmnet with reason: host reimage
  • 09:20 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2086.codfw.wmnet with reason: host reimage
  • 09:13 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1020.eqiad.wmnet
  • 09:09 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be2086.codfw.wmnet with OS bullseye
  • 08:59 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2086.codfw.wmnet with OS bullseye
  • 08:52 dcausse: restarting blazegraph on wdqs1019 (BlazegraphFreeAllocatorsDecreasingRapidly)
  • 08:49 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:49 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:47 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:47 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:45 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:44 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:42 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:42 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:36 kartik@deploy2002: Finished scap sync-world: Backport for Translate: Enable message group subscription feature for some wikis (T372386) (duration: 23m 39s)
  • 08:35 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host ms-be2086.codfw.wmnet with OS bullseye
  • 08:29 kartik@deploy2002: abi, kartik: Continuing with sync
  • 08:25 kartik@deploy2002: abi, kartik: Backport for Translate: Enable message group subscription feature for some wikis (T372386) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 08:21 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:21 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:20 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:19 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:19 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:19 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
  • 08:18 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 08:18 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
  • 08:15 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1020.eqiad.wmnet
  • 08:12 kartik@deploy2002: Started scap sync-world: Backport for Translate: Enable message group subscription feature for some wikis (T372386)
  • 08:11 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1020.eqiad.wmnet
  • 08:09 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1009.eqiad.wmnet
  • 08:07 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1009.eqiad.wmnet
  • 08:00 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on pc1017.eqiad.wmnet with reason: T378068, host is not pooled
  • 08:00 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on pc1017.eqiad.wmnet with reason: T378068, host is not pooled
  • 08:00 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on pc1013.eqiad.wmnet with reason: T373037, host is not pooled
  • 08:00 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on pc1013.eqiad.wmnet with reason: T373037, host is not pooled
  • 05:20 TimStarling: foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol=https
  • 05:14 TimStarling: on mwmaint2002: mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=idwikivoyage --force-protocol=https
  • 04:41 TimStarling: installed id.wikivoyage.org
  • 04:39 TimStarling: on db2123: grant alter ON `%wik%`.* TO `wikiadmin2023`@`10.%`
  • 04:26 tstarling@deploy2002: Finished scap sync-world: Backport for Create id.wikivoyage.org (T380726 T352113), Add messages for Indonesian Wikivoyage (idwikivoyage) (T380726) (duration: 31m 05s)
  • 04:13 tstarling@deploy2002: tstarling: Continuing with sync
  • 04:12 tstarling@deploy2002: tstarling: Backport for Create id.wikivoyage.org (T380726 T352113), Add messages for Indonesian Wikivoyage (idwikivoyage) (T380726) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 03:55 tstarling@deploy2002: Started scap sync-world: Backport for Create id.wikivoyage.org (T380726 T352113), Add messages for Indonesian Wikivoyage (idwikivoyage) (T380726)

2024-12-01

  • 23:53 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 23:52 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 23:52 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 23:52 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 23:52 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 23:52 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 23:50 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 23:50 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 23:50 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 23:50 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 23:50 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 23:50 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 13:17 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1156 gradually with 4 steps - Maint over (T381213)
  • 13:02 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1233 gradually with 4 steps - Maint over (T381213)
  • 12:31 ladsgroup@cumin1002: START - Cookbook sre.mysql.pool db1156 gradually with 4 steps - Maint over (T381213)
  • 12:16 ladsgroup@cumin1002: START - Cookbook sre.mysql.pool db1233 gradually with 4 steps - Maint over (T381213)
  • 12:10 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1156.eqiad.wmnet onto db1233.eqiad.wmnet
  • 10:45 ladsgroup@cumin1002: START - Cookbook sre.mysql.clone of db1156.eqiad.wmnet onto db1233.eqiad.wmnet
  • 10:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depool to reclone (T381213)', diff saved to https://phabricator.wikimedia.org/P71451 and previous config saved to /var/cache/conftool/dbconfig/20241201-104441-ladsgroup.json
  • 06:18 marostegui@cumin2002: dbctl commit (dc=all): 'Depoll db1233', diff saved to https://phabricator.wikimedia.org/P71450 and previous config saved to /var/cache/conftool/dbconfig/20241201-061841-marostegui.json


Archives

See Server Admin Log/Archives.