16:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{aux-k8s-worker[2006-2009].codfw.wmnet} and (A:aux-master-codfw or A:aux-worker-codfw)
16:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2009.codfw.wmnet
16:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2009.codfw.wmnet
16:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2009.codfw.wmnet
16:23 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2009.codfw.wmnet
16:23 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2008.codfw.wmnet
16:23 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2008.codfw.wmnet
16:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2008.codfw.wmnet
16:13 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2008.codfw.wmnet
16:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet
16:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet
16:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet
16:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet
16:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet
16:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet
16:06 cdobbins@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie
16:01 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet
16:01 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet
16:01 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{aux-k8s-worker[2006-2009].codfw.wmnet} and (A:aux-master-codfw or A:aux-worker-codfw)
10:54 kamila@deploy1003: Started scap sync-world: rebuild for T436488
10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{aux-k8s-worker[1006-1009].eqiad.wmnet} and (A:aux-master-eqiad or A:aux-worker-eqiad)
10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet
10:30 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet
10:25 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet
10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet
10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet
10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet
10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet
10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet
10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet
10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet
10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet
10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{aux-k8s-worker[1006-1009].eqiad.wmnet} and (A:aux-master-eqiad or A:aux-worker-eqiad)
09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet
09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet
09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet
09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet
09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet
09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet
09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad
09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet
09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet
09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet
08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning
08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections T407942
08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2
08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
08:49 moritzm: installing emacs security updates
08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
2026-08-28
21:16 ryankemper: T415073 admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs
21:05 ryankemper: T415073 beginning admin_ng namespace-certificates rollout (gerrit 1278562, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging
18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm
18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage
18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage
17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm
17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm
17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm
16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie
16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage
12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet
12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie
12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage
12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage
11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie
11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org T416709
11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003"
11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003"
11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors
11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors
11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003"
11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003"
11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet
11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T436069
10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T436069
08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet
08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet
06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
04:34 ryankemper: T435862 Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done
04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons.
03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons.
03:58 ryankemper: T435862 Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined
03:42 ryankemper: T435862 Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded
03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons.
03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons.
23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet
23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet
23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet
23:07 jasmine_: homer lsw1-c6-eqiad* commit 'T421711'
23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie
22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage
22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage
22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261
22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261
22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261
22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003"
22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003"
17:31 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2003.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Homer - add function to wmf plugin to build rpm probe data - cmooney@cumin1003
17:30 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2003.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Homer - add function to wmf plugin to build rpm probe data - cmooney@cumin1003
16:18 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
16:17 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
16:15 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1005.eqiad.wmnet
16:15 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1005.eqiad.wmnet
16:15 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1004.eqiad.wmnet
16:15 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1004.eqiad.wmnet
16:10 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1004.eqiad.wmnet
16:10 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1004.eqiad.wmnet
16:10 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1003.eqiad.wmnet
16:10 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1003.eqiad.wmnet
16:06 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1003.eqiad.wmnet
16:05 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1003.eqiad.wmnet
16:05 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1002.eqiad.wmnet
16:05 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1002.eqiad.wmnet
16:01 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1002.eqiad.wmnet
16:01 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
16:00 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1002.eqiad.wmnet
16:00 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad
15:58 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2002.codfw.wmnet with OS bookworm
15:58 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
15:50 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2002.codfw.wmnet with OS bookworm
15:46 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
15:38 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
15:33 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw
15:33 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2009.codfw.wmnet
15:33 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2009.codfw.wmnet
15:30 dwisehaupt@dns1005: END - running authdns-update
15:28 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage
15:28 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2009.codfw.wmnet
15:24 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage
15:23 krinkle@deploy1003: krinkle: Continuing with deployment
15:23 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2009.codfw.wmnet
15:23 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2008.codfw.wmnet
15:23 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2008.codfw.wmnet
15:20 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2003.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Homer - add function to wmf plugin to build rpm probe data - cmooney@cumin1003
15:18 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2003.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Homer - add function to wmf plugin to build rpm probe data - cmooney@cumin1003
15:17 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2008.codfw.wmnet
15:17 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2003.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Homer - add function to wmf plugin to build rpm probe data - cmooney@cumin1003
15:17 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2008.codfw.wmnet
15:17 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet
15:17 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet
15:15 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2003.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Homer - add function to wmf plugin to build rpm probe data - cmooney@cumin1003
22:23 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1260.eqiad.wmnet with reason: host reimage
22:20 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1260.eqiad.wmnet with reason: host reimage
21:59 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1260
21:59 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1260
21:58 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1260
21:58 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1260.eqiad.wmnet 28.32.64.10.in-addr.arpa 8.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
21:58 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1260.eqiad.wmnet 28.32.64.10.in-addr.arpa 8.2.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
21:58 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:58 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1260 - jasmine@cumin1003"
21:58 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1260 - jasmine@cumin1003"
20:57 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2002.codfw.wmnet with OS bookworm
20:48 cdobbins@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp5022.eqsin.wmnet with reason: needs repair
20:29 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2003.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Homer - remove eqsin workaround from wmf-plugin for eqsin L3 switches - cmooney@cumin1003
20:28 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2003.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Homer - remove eqsin workaround from wmf-plugin for eqsin L3 switches - cmooney@cumin1003
20:27 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2002.codfw.wmnet with OS bookworm
20:24 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2002.codfw.wmnet with OS bookworm
20:10 brett@cumin2003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, T435406]
20:10 brett@cumin2003: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, T435406]
20:08 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough-eqsin
20:06 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough-eqsin
17:02 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:02 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix dns entries for cr2-eqiad transport interface IPs - cmooney@cumin1003"
17:02 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix dns entries for cr2-eqiad transport interface IPs - cmooney@cumin1003"
16:25 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 12 hosts with reason: site migration to new Nokia switches
16:25 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 13 hosts with reason: site migration to new Nokia switches
16:24 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 12 hosts with reason: site migration to new Nokia switches
16:23 cmooney@cumin1003: END (PASS) - Cookbook sre.metamonitoring.downtime (exit_code=0) Downtime for 4:00:00 of prometheus/deadmanswitchnotified, prometheus/deadmanswitchonamdb, prometheus/extmon on 2 host(s) with reason: eqsin site rebuild
16:23 cmooney@cumin1003: START - Cookbook sre.metamonitoring.downtime Downtime for 4:00:00 of prometheus/deadmanswitchnotified, prometheus/deadmanswitchonamdb, prometheus/extmon on 2 host(s) with reason: eqsin site rebuild
16:18 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns1004.wikimedia.org with OS trixie
15:44 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:44 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix dns entries for cr2-eqiad transport interface IPs - cmooney@cumin1003"
15:44 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns1004.wikimedia.org with reason: host reimage
15:41 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix dns entries for cr2-eqiad transport interface IPs - cmooney@cumin1003"
15:37 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns1004.wikimedia.org with reason: host reimage
15:33 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:33 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix dns entries for cr2-eqiad transport interface IPs - cmooney@cumin1003"
15:33 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: fix dns entries for cr2-eqiad transport interface IPs - cmooney@cumin1003"
14:45 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on krb1004.eqiad.wmnet with reason: host reimage
14:42 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db-test2001.codfw.wmnet
14:42 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:42 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
14:42 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
14:23 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1004.eqiad.wmnet - bking@cumin2003"
14:23 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1004.eqiad.wmnet - bking@cumin2003"
14:23 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) krb1004.eqiad.wmnet on all recursors
14:23 bking@cumin2003: START - Cookbook sre.dns.wipe-cache krb1004.eqiad.wmnet on all recursors
14:23 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:23 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1004.eqiad.wmnet - bking@cumin2003"
14:21 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1004.eqiad.wmnet - bking@cumin2003"
13:16 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1010.eqiad.wmnet with reason: host reimage
13:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti3005.esams.wmnet
13:12 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db-test2002.codfw.wmnet
13:12 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:12 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
13:07 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Security Release - T436069
13:06 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 12 hosts with reason: site migration to new Nokia switches
13:06 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 13 hosts with reason: site migration to new Nokia switches
13:05 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 12 hosts with reason: site migration to new Nokia switches
13:04 cmooney@cumin1003: END (PASS) - Cookbook sre.metamonitoring.downtime (exit_code=0) Downtime for 3:00:00 of prometheus/deadmanswitchnotified, prometheus/deadmanswitchonamdb, prometheus/extmon on 2 host(s) with reason: eqsin site rebuild
13:04 cmooney@cumin1003: START - Cookbook sre.metamonitoring.downtime Downtime for 3:00:00 of prometheus/deadmanswitchnotified, prometheus/deadmanswitchonamdb, prometheus/extmon on 2 host(s) with reason: eqsin site rebuild
13:01 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1010.eqiad.wmnet with OS bookworm
12:49 Daimona: Running query from T436088#12256503 in x1.wikishared
12:49 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T436069
12:38 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Release - T436069
12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti3005.esams.wmnet with OS bookworm
11:54 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti3005.esams.wmnet with reason: host reimage
11:50 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti3005.esams.wmnet with reason: host reimage
11:49 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
11:26 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host ganeti3005.esams.wmnet with OS bookworm
09:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1022: Pool back pc2
09:55 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool pc1022: Pool back pc2
09:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1022: Pool back pc2
09:55 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool pc1022: Pool back
09:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool pc1022: Pool back
09:47 hashar@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.17 refs T430836
09:42 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2022.codfw.wmnet,pc1022.eqiad.wmnet with reason: Working on pc2 truncation
09:41 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99)
09:41 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts db-test2002.codfw.wmnet
09:39 fceratto@cumin1003: START - Cookbook sre.hosts.decommission for hosts db-test2002.codfw.wmnet
09:22 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' .
09:09 gkyziridis@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' .
09:07 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' .
09:07 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' .
09:01 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1010.eqiad.wmnet with OS bookworm
08:49 moritzm: installing openssl security updates on trixie
08:42 cmooney@cumin1003: END (PASS) - Cookbook sre.metamonitoring.downtime (exit_code=0) Downtime for 4:00:00 of prometheus/deadmanswitchnotified, prometheus/deadmanswitchonamdb, prometheus/extmon on 2 host(s) with reason: eqsin site rebuild
08:42 cmooney@cumin1003: START - Cookbook sre.metamonitoring.downtime Downtime for 4:00:00 of prometheus/deadmanswitchnotified, prometheus/deadmanswitchonamdb, prometheus/extmon on 2 host(s) with reason: eqsin site rebuild
08:28 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
08:28 fceratto@cumin1003: Removing db-test1003 from zarcillo T435912
08:21 hashar: Rolling back group 1 from 1.47.0-wmf.17 to 1.47.0-wmf.16 due to Special:EntityUsage being broken on Wikibase client wikis # T436060T430836
08:19 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2011.codfw.wmnet
08:17 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 12 hosts with reason: site migration to new Nokia switches
08:16 cmooney@cumin1003: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 5:00:00 on 12 hosts with reason: site migration to new Nokia switches
08:14 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 13 hosts with reason: site migration to new Nokia switches
08:14 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 12 hosts with reason: site migration to new Nokia switches
08:12 hashar@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.17 refs T430836
08:12 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host maps2011.codfw.wmnet
08:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps2012.codfw.wmnet
07:54 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host maps2012.codfw.wmnet
19:44 bking@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons.
19:14 bking@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons.
18:57 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns2004.wikimedia.org
18:57 cdobbins@cumin1003: START - Cookbook sre.hosts.remove-downtime for dns2004.wikimedia.org
18:04 aokoth@cumin1003: END (PASS) - Cookbook sre.vrts.upgrade (exit_code=0) on VRTS host vrts1003.eqiad.wmnet
18:02 aokoth@cumin1003: START - Cookbook sre.vrts.upgrade on VRTS host vrts1003.eqiad.wmnet
17:41 bking@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons.
17:28 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns2004.wikimedia.org with reason: host reimage
17:22 cdobbins@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns2004.wikimedia.org with reason: host reimage
17:11 bking@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons.
17:10 bking@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-canary cluster: Roll restart of all Presto's jvm daemons.
17:07 bking@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-canary cluster: Roll restart of all Presto's jvm daemons.
17:07 bking@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons.
17:05 bking@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons.
17:01 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host dns2004.wikimedia.org with OS trixie
13:25 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
13:25 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
13:24 jelto@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
13:23 godog: failover dumps-nfs from clouddumps1002 to clouddumps1001 - T411248
13:20 jelto@cumin1003: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
13:14 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
13:14 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
13:06 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db-test1003.eqiad.wmnet
13:06 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:05 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
13:01 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
12:58 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
12:34 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db-test1002.eqiad.wmnet
12:34 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:34 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
12:34 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling reboot on A:thanos-fe
12:33 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
12:32 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
11:38 gkyziridis@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
11:38 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db2903.codfw.wmnet with OS trixie
11:34 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' .
11:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
11:32 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db2903.codfw.wmnet - fceratto@cumin1003"
11:32 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db2903.codfw.wmnet - fceratto@cumin1003"
11:31 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db2903.codfw.wmnet on all recursors
11:31 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db2903.codfw.wmnet on all recursors
11:31 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:31 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db2903.codfw.wmnet - fceratto@cumin1003"
11:31 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db2903.codfw.wmnet - fceratto@cumin1003"
11:28 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:29 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1056.eqiad.wmnet
10:29 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1057.eqiad.wmnet
10:29 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1055.eqiad.wmnet
10:23 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudvirt1057.eqiad.wmnet
10:23 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudvirt1056.eqiad.wmnet
10:22 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudvirt1055.eqiad.wmnet
10:19 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1054.eqiad.wmnet
10:12 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudvirt1054.eqiad.wmnet
10:12 Emperor: upgrading apus eqiad cluster to Reef 18.2.8
10:09 filippo@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1051
10:08 filippo@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1051
10:02 filippo@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1051
10:02 filippo@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1051
09:58 filippo@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1051
09:58 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1049.eqiad.wmnet with OS trixie
09:58 filippo@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1051
09:54 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1051.eqiad.wmnet with OS trixie
09:46 marostegui@cumin1003: dbctl commit (dc=all): 'Add test-s4 to dbctl T427059', diff saved to https://phabricator.wikimedia.org/P96249 and previous config saved to /var/cache/conftool/dbconfig/20260825-094607-marostegui.json
09:43 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1176 and db2230 to dbctl T427059', diff saved to https://phabricator.wikimedia.org/P96248 and previous config saved to /var/cache/conftool/dbconfig/20260825-094345-marostegui.json
09:42 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
09:42 jelto@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
09:41 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2901.codfw.wmnet with reason: Cloning
09:40 jelto@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
09:39 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:36 cmooney@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
09:35 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db-test1001.eqiad.wmnet
09:35 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:35 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
09:34 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db-test1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
08:29 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1276: Pool back
08:29 moritzm: installing Expat security updates
08:29 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1276 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96242 and previous config saved to /var/cache/conftool/dbconfig/20260825-082922-marostegui.json
08:24 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1051.eqiad.wmnet with OS trixie
08:22 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1049.eqiad.wmnet with OS trixie
08:17 godog: put cloudvirts back in service T431682
08:17 filippo@cumin1003: END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host cloudvirt1049.eqiad.wmnet
08:14 hashar@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.17 refs T430836
03:29 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db1243 from dbctl T435892', diff saved to https://phabricator.wikimedia.org/P96240 and previous config saved to /var/cache/conftool/dbconfig/20260825-032952-marostegui.json
03:29 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[1217,1243].eqiad.wmnet with reason: Cloning
03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.17 refs T430836
22:28 RScout-WMF: Deployed security fix for T435822
22:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons.
17:38 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:38 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for telxius circuit IP in magru - cmooney@cumin1003"
17:38 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for telxius circuit IP in magru - cmooney@cumin1003"
17:32 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1003.eqiad.wmnet with OS bookworm
17:32 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - bking@cumin2003"
17:12 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1003.eqiad.wmnet with reason: host reimage
17:08 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2901.codfw.wmnet with reason: host reimage
17:07 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on krb1003.eqiad.wmnet with reason: host reimage
16:54 bking@cumin2003: START - Cookbook sre.hosts.reimage for host krb1003.eqiad.wmnet with OS bookworm
16:50 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db2901.codfw.wmnet with OS trixie
16:50 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db2901.codfw.wmnet - fceratto@cumin1003"
16:50 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db2901.codfw.wmnet - fceratto@cumin1003"
16:50 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db2901.codfw.wmnet on all recursors
16:50 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db2901.codfw.wmnet on all recursors
16:50 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:50 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db2901.codfw.wmnet - fceratto@cumin1003"
16:49 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db2901.codfw.wmnet - fceratto@cumin1003"
16:39 kevinbazira@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
16:39 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
16:26 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
09:46 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts db2901.codfw.wmnet
09:46 fceratto@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
09:46 fceratto@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2901.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
09:30 taavi: [09:26] kevinbazira@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
09:29 taavi: [09:25] kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
09:28 taavi: test
09:18 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
06:55 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db1182 from dbctl T434869', diff saved to https://phabricator.wikimedia.org/P96236 and previous config saved to /var/cache/conftool/dbconfig/20260824-065547-marostegui.json
06:46 marostegui: wikifunctionsclient_usage from s7 T434541
06:33 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1282: Pool back
06:33 marostegui: wikifunctionsclient_usage from s5 T434541
05:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 21 hosts with reason: Cloning
05:50 marostegui: Stop mariadb on sanitarium s1,s3,s8,s5,x3 there will be lag on wikireplicas for those sections T407942
05:48 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1282: Pool back
05:48 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1282 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96225 and previous config saved to /var/cache/conftool/dbconfig/20260824-054826-marostegui.json
05:23 marostegui: wikifunctionsclient_usage from s3 T434541
05:20 marostegui: wikifunctionsclient_usage from s2 T434541
17:32 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:32 sukhe@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: force HEAD to be26e30ae101 - sukhe@cumin1003"
17:32 sukhe@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: force HEAD to be26e30ae101 - sukhe@cumin1003"
17:28 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:28 sukhe@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: force HEAD to be26e30ae101 - sukhe@cumin1003"
17:28 sukhe@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: force HEAD to be26e30ae101 - sukhe@cumin1003"
16:41 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:41 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on eqsin<->codfw arelion - cmooney@cumin1003"
16:37 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on eqsin<->codfw arelion - cmooney@cumin1003"
16:00 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:00 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on eqord<->codfw arelion - cmooney@cumin1003"
15:56 cdobbins@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host dns5004.wikimedia.org with OS trixie
15:55 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on eqord<->codfw arelion - cmooney@cumin1003"
12:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2006.wikimedia.org with OS trixie
12:00 kevinbazira@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
11:58 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
11:43 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage
11:41 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
11:38 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2006.wikimedia.org with reason: host reimage
11:20 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host urldownloader2006.wikimedia.org with OS trixie
11:11 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader2005.wikimedia.org with OS trixie
10:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage
10:53 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader2005.wikimedia.org with reason: host reimage
10:43 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-worker-codfw
10:43 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2011.codfw.wmnet
10:43 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2011.codfw.wmnet
10:40 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2011.codfw.wmnet
10:34 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2011.codfw.wmnet
10:34 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2010.codfw.wmnet
10:34 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2010.codfw.wmnet
10:33 fnegri@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database bolwiki (T429954)
10:33 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host urldownloader2005.wikimedia.org with OS trixie
10:30 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2010.codfw.wmnet
10:25 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2010.codfw.wmnet
10:24 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2009.codfw.wmnet
10:24 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2009.codfw.wmnet
10:20 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1006.wikimedia.org with OS trixie
10:18 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2009.codfw.wmnet
10:13 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2009.codfw.wmnet
10:13 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2008.codfw.wmnet
10:13 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2008.codfw.wmnet
10:06 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2008.codfw.wmnet
10:04 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage
10:01 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2008.codfw.wmnet
10:01 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2007.codfw.wmnet
10:01 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2007.codfw.wmnet
09:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1006.wikimedia.org with reason: host reimage
09:56 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2007.codfw.wmnet
09:51 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2007.codfw.wmnet
09:51 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2006.codfw.wmnet
09:51 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2006.codfw.wmnet
09:46 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2006.codfw.wmnet
09:46 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host urldownloader1006.wikimedia.org with OS trixie
09:45 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host urldownloader1005.wikimedia.org with OS trixie
09:41 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2006.codfw.wmnet
09:41 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2005.codfw.wmnet
09:41 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2005.codfw.wmnet
09:38 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2013.codfw.wmnet
09:36 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2005.codfw.wmnet
09:35 fnegri@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database bolwiki (T429954)
09:35 fnegri@cumin1003: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database minwikiquote (T429946)
09:35 fnegri@cumin1003: START - Cookbook sre.wikireplicas.add-wiki for database minwikiquote (T429946)
09:32 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2013.codfw.wmnet
09:30 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb2011.codfw.wmnet
09:29 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage
09:26 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2005.codfw.wmnet
09:26 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2004.codfw.wmnet
09:26 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2004.codfw.wmnet
09:24 blake@cumin1003: START - Cookbook sre.hosts.reboot-single for host rdb2011.codfw.wmnet
09:22 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on urldownloader1005.wikimedia.org with reason: host reimage
09:21 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve2004.codfw.wmnet
09:16 blake@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rdb1015.eqiad.wmnet
09:16 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve2004.codfw.wmnet
09:16 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve2003.codfw.wmnet
09:16 klausman@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve2003.codfw.wmnet
09:13 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cr[1-2]-eqiad,pfw1-eqiad with reason: upgrade pfw1a-eqiad and pfw1b-eqiad pair
06:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1283: Pool back
06:17 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1283 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96213 and previous config saved to /var/cache/conftool/dbconfig/20260821-061743-marostegui.json
21:58 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1054.eqiad.wmnet with reason: host reimage
21:51 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1054.eqiad.wmnet with reason: host reimage
21:36 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1054.eqiad.wmnet with OS trixie
21:36 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:31 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:31 krinkle@deploy1003: krinkle: Continuing with deployment
21:31 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1054
21:31 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1054
16:51 swfrench-wmf: disable-puppet on A:cp for ATS Lua change - T427666
16:51 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2901.codfw.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
16:10 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host stat1011
16:10 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host stat1011
16:09 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host stat1011
16:09 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) stat1011.eqiad.wmnet 14.36.64.10.in-addr.arpa 4.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
16:09 bking@cumin2003: START - Cookbook sre.dns.wipe-cache stat1011.eqiad.wmnet 14.36.64.10.in-addr.arpa 4.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
16:09 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:09 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host stat1011 - bking@cumin2003"
16:09 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host stat1011 - bking@cumin2003"
15:14 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1903.eqiad.wmnet with reason: host reimage
15:07 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1903.eqiad.wmnet with reason: host reimage
14:54 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1903.eqiad.wmnet with OS trixie
14:53 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1903.eqiad.wmnet - fceratto@cumin1003"
14:53 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1903.eqiad.wmnet - fceratto@cumin1003"
14:53 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1903.eqiad.wmnet on all recursors
14:53 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db1903.eqiad.wmnet on all recursors
14:53 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:53 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1903.eqiad.wmnet - fceratto@cumin1003"
14:53 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1903.eqiad.wmnet - fceratto@cumin1003"
14:22 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:22 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries used on new transport backup eqiad codfw - cmooney@cumin1003"
14:19 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entries used on new transport backup eqiad codfw - cmooney@cumin1003"
13:38 fnegri@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1025.eqiad.wmnet with reason: Removing s6 from clouddb1025
13:34 klausman@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet
12:57 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2001.codfw.wmnet
12:56 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2002.codfw.wmnet
12:55 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:55 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad GTT vpls - cmooney@cumin1003"
12:54 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad GTT vpls - cmooney@cumin1003"
12:54 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2002.codfw.wmnet
12:54 klausman@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2003.codfw.wmnet
12:50 klausman@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2003.codfw.wmnet
12:31 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:31 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad cct - cmooney@cumin1003"
12:28 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad cct - cmooney@cumin1003"
12:28 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1902.eqiad.wmnet with reason: host reimage
12:25 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on stat1009.eqiad.wmnet with reason: host reimage
12:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on stat1010.eqiad.wmnet with reason: host reimage
12:22 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1902.eqiad.wmnet with reason: host reimage
12:21 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on stat1010.eqiad.wmnet with reason: host reimage
12:14 elukey: move the Docker Registry's /v2/dev/.* prefix to its dedicated S3 backend - T432829
12:12 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1902.eqiad.wmnet with OS trixie
12:09 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1902.eqiad.wmnet - fceratto@cumin1003"
12:09 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1902.eqiad.wmnet - fceratto@cumin1003"
12:09 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1902.eqiad.wmnet on all recursors
12:09 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db1902.eqiad.wmnet on all recursors
12:09 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:08 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1902.eqiad.wmnet - fceratto@cumin1003"
12:08 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1902.eqiad.wmnet - fceratto@cumin1003"
12:01 fceratto@cumin1003: START - Cookbook sre.ganeti.makevm for new host db1902.eqiad.wmnet
12:00 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host stat1010.eqiad.wmnet with OS bookworm
11:57 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1902.eqiad.wmnet
11:57 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:57 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1902.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
11:57 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1902.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
11:24 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host stat1010.eqiad.wmnet with OS bookworm
11:07 fceratto@cumin1003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host db1901.eqiad.wmnet
11:07 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1901.eqiad.wmnet with OS trixie
10:53 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1901.eqiad.wmnet with reason: host reimage
10:47 tappof: bump space for prometheus k8s-aux in codfw
10:47 tappof: bump space for prometheus k8s-dse in eqiad
10:47 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db1901.eqiad.wmnet with reason: host reimage
10:35 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1901.eqiad.wmnet with OS trixie
10:32 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1901.eqiad.wmnet - fceratto@cumin1003"
10:32 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1901.eqiad.wmnet - fceratto@cumin1003"
10:32 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1901.eqiad.wmnet on all recursors
10:32 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db1901.eqiad.wmnet on all recursors
10:31 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:31 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003"
10:31 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003"
06:21 brennen@deploy1003: Started deploy [phabricator/deployment@6b9b6ff]: deploy phab1005 for T435087
06:20 brennen@deploy1003: Finished deploy [phabricator/deployment@6b9b6ff]: deploy phab1004 for to pick up config values for T435087 (duration: 01m 46s)
06:18 brennen@deploy1003: Started deploy [phabricator/deployment@6b9b6ff]: deploy phab1004 for to pick up config values for T435087
18:03 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.15 upgrade (T434478)
17:56 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.15 upgrade (T434478)
17:50 cmooney@dns3003: END - running authdns-update
17:42 dancy@deploy1003: Installation of scap version "4.283.0" completed for 3 hosts
17:41 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum and A:durum
17:40 dancy@deploy1003: Installing scap version "4.283.0" for 3 host(s)
17:38 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:37 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on GTT VPLS - cmooney@cumin1003"
17:32 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on GTT VPLS - cmooney@cumin1003"
17:29 sbassett: Deployed security fix for T435210 (wmf.16)
16:15 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:15 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003"
16:13 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore[2005-2006].codfw.wmnet,sessionstore[1005-1006].eqiad.wmnet: Upgrade Cassandra to 5.0.8 & Java to 17 — T435154 - eevans@cumin1003
16:11 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003"
16:10 urbanecm@deploy1003: urbanecm: Continuing with deployment
15:48 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:48 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003"
15:45 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->codfw - cmooney@cumin1003"
15:44 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
15:36 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:36 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqsin - cmooney@cumin1003"
15:33 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqsin - cmooney@cumin1003"
13:10 cdobbins@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=97) Rolling upgrade of ATS on A:cp-codfw and A:cp - 9.2.15 upgrade (T434478)
13:10 cdobbins@cumin1003: END (ERROR) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=97) Rolling upgrade of ATS on A:cp-eqiad and A:cp - 9.2.15 upgrade (T434478)
12:45 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:45 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqord cct - cmooney@cumin1003"
12:45 elukey: move the /v2/releng.* prefix on the Docker Registry to its new s3 backend - T432829
12:43 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
12:42 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on ulsfo<->eqord cct - cmooney@cumin1003"
09:55 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors
09:52 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-rw2001.wikimedia.org with OS trixie
09:51 filippo@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
09:51 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) netbox.discovery.wmnet. on all recursors
09:51 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache netbox.discovery.wmnet. on all recursors
09:51 jmm@cumin2003: START - Cookbook sre.netbox.restart-reboot rolling reboot on A:netbox
09:46 filippo@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
09:44 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1056.eqiad.wmnet with OS trixie
09:44 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003"
09:36 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1065.eqiad.wmnet with reason: host reimage
09:36 topranks: make HE transport circuits from magru live
09:36 filippo@cumin1003: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
09:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
09:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-rw2001.wikimedia.org with reason: host reimage
09:31 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
09:30 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1065.eqiad.wmnet with reason: host reimage
09:28 arnaudb@dns1006: END - running authdns-update
09:27 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-rw2001.wikimedia.org with reason: host reimage
09:27 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
09:24 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1056.eqiad.wmnet with reason: host reimage
09:23 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
09:20 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.convert-disks (exit_code=0) for host ms-be1068
09:20 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1068.eqiad.wmnet with OS trixie
09:20 filippo@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
09:07 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1173.eqiad.wmnet with OS bookworm
09:06 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host ldap-rw2001.wikimedia.org with OS trixie
09:03 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.convert-disks (exit_code=0) for host ms-be1064
09:03 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1064.eqiad.wmnet with OS trixie
09:03 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1056.eqiad.wmnet with OS trixie
09:02 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1057.eqiad.wmnet with reason: host reimage
09:01 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
08:58 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1068.eqiad.wmnet with reason: host reimage
08:56 filippo@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1057.eqiad.wmnet with reason: host reimage
08:54 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1208.eqiad.wmnet with OS bookworm
08:53 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2902.codfw.wmnet with OS trixie
08:52 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1068.eqiad.wmnet with reason: host reimage
08:50 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1174.eqiad.wmnet with OS bookworm
08:46 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1172.eqiad.wmnet with OS bookworm
08:45 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1173.eqiad.wmnet with reason: host reimage
08:41 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1064.eqiad.wmnet with reason: host reimage
08:40 filippo@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1057.eqiad.wmnet with OS trixie
08:39 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
08:39 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1207.eqiad.wmnet with OS bookworm
08:38 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2902.codfw.wmnet with reason: host reimage
08:37 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1064.eqiad.wmnet with reason: host reimage
08:34 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be1068.eqiad.wmnet with OS trixie
08:34 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1208.eqiad.wmnet with reason: host reimage
08:31 filippo@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
08:29 filippo@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
08:28 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1222.eqiad.wmnet onto db1276.eqiad.wmnet
08:28 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Pool db1222.eqiad.wmnet in after cloning
08:28 fceratto@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2902.codfw.wmnet with reason: host reimage
08:27 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1055.eqiad.wmnet with OS trixie
08:27 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1003"
08:26 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1174.eqiad.wmnet with reason: host reimage
08:24 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet
08:23 topranks: reboot pfw1-codfw firewall pair to upgrade JunOS T434865
08:22 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1172.eqiad.wmnet with reason: host reimage
08:20 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be1064.eqiad.wmnet with OS trixie
08:20 mvernon@cumin2003: START - Cookbook sre.swift.convert-disks for host ms-be1065
08:18 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1207.eqiad.wmnet with reason: host reimage
08:17 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1174.eqiad.wmnet with reason: host reimage
08:17 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1173.eqiad.wmnet with reason: host reimage
08:17 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1172.eqiad.wmnet with reason: host reimage
08:15 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet
08:15 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1208.eqiad.wmnet with reason: host reimage
08:15 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1207.eqiad.wmnet with reason: host reimage
08:14 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db2902.codfw.wmnet with OS trixie
08:14 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.16 refs T430835
08:13 fceratto@cumin1003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host db2902.codfw.wmnet
08:13 fceratto@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2902.codfw.wmnet with OS trixie
08:10 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr[1-2]-codfw with reason: upgrade pfw1a-codfw and pfw1b-codfw pair
08:09 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1055.eqiad.wmnet with reason: host reimage
08:07 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pfw1-codfw with reason: upgrade pfw1a-codfw and pfw1b-codfw pair
08:03 filippo@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1055.eqiad.wmnet with reason: host reimage
08:02 arnaudb@dns1006: END - running authdns-update
08:02 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1208.eqiad.wmnet with OS bookworm
08:02 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1207.eqiad.wmnet with OS bookworm
08:01 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1174.eqiad.wmnet with OS bookworm
08:01 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1173.eqiad.wmnet with OS bookworm
08:01 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1172.eqiad.wmnet with OS bookworm
07:14 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1277: Pool back
07:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
07:07 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1280: Pool back
07:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
06:55 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
06:54 moritzm: installing php8.2 security updates
06:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1284: Pool back
06:49 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1222: Depool db1222.eqiad.wmnet to then clone it to db1276.eqiad.wmnet - marostegui@cumin1003
06:49 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1222: Depool db1222.eqiad.wmnet to then clone it to db1276.eqiad.wmnet - marostegui@cumin1003
06:49 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1222.eqiad.wmnet onto db1276.eqiad.wmnet
06:45 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet
06:41 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet
06:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet
06:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: db2209 repool
06:33 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet
06:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
06:29 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1277: Pool back
06:28 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1277 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96190 and previous config saved to /var/cache/conftool/dbconfig/20260819-062815-marostegui.json
06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
06:22 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1001.eqiad.wmnet
06:18 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host zuul1001.eqiad.wmnet
06:18 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1003.eqiad.wmnet
06:18 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1280: Pool back
06:17 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1284 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96186 and previous config saved to /var/cache/conftool/dbconfig/20260819-061743-marostegui.json
06:14 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host zuul1003.eqiad.wmnet
06:14 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1002.eqiad.wmnet
06:10 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host zuul1002.eqiad.wmnet
06:10 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2048.codfw.wmnet
06:09 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2048.codfw.wmnet
06:06 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1284: Pool back
06:06 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1284 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96184 and previous config saved to /var/cache/conftool/dbconfig/20260819-060621-marostegui.json
06:03 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2048.codfw.wmnet
20:49 ryankemper: `an-launcher1003` terminated process group `666809` (`rest_backfill_phase1.sh`) ~20 mins ago with `sudo kill -TERM -- -666809` after its local spark driver (`--driver-memory 64g`) repeatedly exhausted memory on the 32 GB VM and caused SSH to intermittently flap; host recovered to 27 GB available memory
17:48 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1156.eqiad.wmnet with reason: host reimage
17:46 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1146.eqiad.wmnet with reason: host reimage
17:45 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-esams and A:cp - 9.2.15 upgrade (T434478)
17:42 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-ctrl2006.codfw.wmnet with OS trixie
17:41 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1177.eqiad.wmnet with reason: host reimage
17:38 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1202.eqiad.wmnet with reason: host reimage
17:34 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1201.eqiad.wmnet with reason: host reimage
17:30 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1156.eqiad.wmnet with reason: host reimage
17:29 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1177.eqiad.wmnet with reason: host reimage
17:28 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1146.eqiad.wmnet with reason: host reimage
17:28 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1202.eqiad.wmnet with reason: host reimage
17:27 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1201.eqiad.wmnet with reason: host reimage
17:25 jasmine@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage
17:21 jasmine@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage
17:14 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1202.eqiad.wmnet with OS bookworm
17:14 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1201.eqiad.wmnet with OS bookworm
17:14 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1177.eqiad.wmnet with OS bookworm
17:14 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1156.eqiad.wmnet with OS bookworm
17:14 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1146.eqiad.wmnet with OS bookworm
14:57 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on phab2003.codfw.wmnet,phab[1004-1006].eqiad.wmnet with reason: maintenance
13:51 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db2902.codfw.wmnet - fceratto@cumin1003"
13:51 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db2902.codfw.wmnet - fceratto@cumin1003"
13:51 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2026.codfw.wmnet
13:51 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host puppetboard2003.codfw.wmnet
12:53 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1288-1289,1291-1301].eqiad.wmnet
12:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1273,1275-1287].eqiad.wmnet
12:53 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1273,1275-1287].eqiad.wmnet
12:52 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org
12:51 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db2902.codfw.wmnet on all recursors
12:51 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db2902.codfw.wmnet on all recursors
12:51 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:51 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db2902.codfw.wmnet - fceratto@cumin1003"
12:51 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db2902.codfw.wmnet - fceratto@cumin1003"
12:46 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org
12:45 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org
12:44 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1273,1275-1287].eqiad.wmnet
12:42 dhinus: repooled clouddb1032 that was currently {"weight": 0, "pooled": "inactive"} for both s4 and s6
12:41 dhinus: also depooled clouddb1017 (forgot it in the previous list)
12:21 jmm@cumin2003: START - Cookbook sre.ganeti.addnode for new host ganeti2046.codfw.wmnet to cluster codfw and group A
12:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1169.eqiad.wmnet onto db1283.eqiad.wmnet
12:21 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1169: Pool db1169.eqiad.wmnet in after cloning
12:19 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1248-1261].eqiad.wmnet
12:19 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1149-1153,1158,1240-1247].eqiad.wmnet
12:19 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1149-1153,1158,1240-1247].eqiad.wmnet
12:11 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1149-1153,1158,1240-1247].eqiad.wmnet
12:10 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1168.eqiad.wmnet onto db1282.eqiad.wmnet
12:10 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1168: Pool db1168.eqiad.wmnet in after cloning
12:06 jmm@cumin2003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
11:30 jmm@cumin2003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
11:27 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1114-1127].eqiad.wmnet
11:26 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1076-1081,1084-1087,1093-1095,1113].eqiad.wmnet
11:26 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1076-1081,1084-1087,1093-1095,1113].eqiad.wmnet
11:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1168: Pool db1168.eqiad.wmnet in after cloning
11:24 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1194.eqiad.wmnet with OS bookworm
11:23 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1181.eqiad.wmnet with OS bookworm
11:21 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2050.codfw.wmnet
11:21 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2050.codfw.wmnet
11:18 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1076-1081,1084-1087,1093-1095,1113].eqiad.wmnet
11:15 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2050.codfw.wmnet
11:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2004.codfw.wmnet
11:09 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1076-1081,1084-1087,1093-1095,1113].eqiad.wmnet
11:09 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1286: Pool back
11:09 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1045-1050,1056-1057,1064-1066,1073-1075].eqiad.wmnet
11:09 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1045-1050,1056-1057,1064-1066,1073-1075].eqiad.wmnet
11:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host build2004.codfw.wmnet
11:06 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on an-worker1194.eqiad.wmnet with reason: host reimage
11:06 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1194.eqiad.wmnet with reason: host reimage
11:05 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2049.codfw.wmnet
11:04 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2049.codfw.wmnet
11:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid1003.eqiad.wmnet
11:01 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1045-1050,1056-1057,1064-1066,1073-1075].eqiad.wmnet
10:59 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1181.eqiad.wmnet with reason: host reimage
10:59 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2049.codfw.wmnet
10:59 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid1003.eqiad.wmnet
10:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host failoid2003.codfw.wmnet
10:54 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1181.eqiad.wmnet with reason: host reimage
10:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host failoid2003.codfw.wmnet
10:51 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1901.eqiad.wmnet on all recursors
10:51 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db1901.eqiad.wmnet on all recursors
10:51 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:51 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003"
10:51 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003"
10:48 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1045-1050,1056-1057,1064-1066,1073-1075].eqiad.wmnet
10:48 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1035,1038-1044].eqiad.wmnet
10:48 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1035,1038-1044].eqiad.wmnet
10:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1168: Depool db1168.eqiad.wmnet to then clone it to db1282.eqiad.wmnet - marostegui@cumin1003
10:44 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1168: Depool db1168.eqiad.wmnet to then clone it to db1282.eqiad.wmnet - marostegui@cumin1003
10:44 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1168.eqiad.wmnet onto db1282.eqiad.wmnet
10:41 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2047.codfw.wmnet
10:40 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1035,1038-1044].eqiad.wmnet
10:33 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2032.codfw.wmnet
10:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2032.codfw.wmnet
10:32 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1035,1038-1044].eqiad.wmnet
10:32 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-eqiad
10:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2032.codfw.wmnet
10:24 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1286: Pool back
10:24 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1286 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96163 and previous config saved to /var/cache/conftool/dbconfig/20260818-102431-marostegui.json
10:22 moritzm: installing Django security updates
10:20 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul2002.codfw.wmnet
10:20 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul2001.codfw.wmnet
10:14 blake@deploy1003: Started scap sync-world: no-build deployment for T417800
10:12 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul2003.codfw.wmnet
10:12 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy2001.codfw.wmnet
10:12 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy3001.esams.wmnet
10:12 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy1001.eqiad.wmnet
10:08 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2031.codfw.wmnet
10:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2031.codfw.wmnet
10:08 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host zuul2003.codfw.wmnet
10:08 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy2001.codfw.wmnet
10:08 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy3001.esams.wmnet
10:08 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy1001.eqiad.wmnet
10:07 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy1002.eqiad.wmnet
10:05 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy2002.codfw.wmnet
10:05 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy3002.esams.wmnet
10:04 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy1002.eqiad.wmnet
10:03 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy4003.ulsfo.wmnet
10:03 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy5003.eqsin.wmnet
10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2031.codfw.wmnet
10:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1169: Depool db1169.eqiad.wmnet to then clone it to db1283.eqiad.wmnet - marostegui@cumin1003
10:01 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy2002.codfw.wmnet
10:01 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy4004.ulsfo.wmnet
10:01 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy3002.esams.wmnet
10:01 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1169: Depool db1169.eqiad.wmnet to then clone it to db1283.eqiad.wmnet - marostegui@cumin1003
10:01 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1169.eqiad.wmnet onto db1283.eqiad.wmnet
10:01 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy5004.eqsin.wmnet
09:59 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy4003.ulsfo.wmnet
09:59 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy4004.ulsfo.wmnet
09:59 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy5003.eqsin.wmnet
09:59 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy7001.magru.wmnet
09:59 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy5004.eqsin.wmnet
09:58 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy7002.magru.wmnet
09:57 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy6001.drmrs.wmnet
09:57 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy6002.drmrs.wmnet
09:54 filippo@cumin1003: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for 10 hosts
09:54 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast4006.wikimedia.org
09:53 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy6001.drmrs.wmnet
09:53 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy6002.drmrs.wmnet
09:53 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host releases1003.eqiad.wmnet
09:53 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy7001.magru.wmnet
09:53 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people2004.codfw.wmnet
09:52 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host people1005.eqiad.wmnet
09:52 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host tcp-proxy7002.magru.wmnet
09:50 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host releases2003.codfw.wmnet
09:49 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host releases2003.codfw.wmnet
09:49 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host releases1003.eqiad.wmnet
09:49 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host people2004.codfw.wmnet
09:48 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host people1005.eqiad.wmnet
09:46 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2040.codfw.wmnet
09:46 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2040.codfw.wmnet
09:43 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1901.eqiad.wmnet on all recursors
09:42 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db1901.eqiad.wmnet on all recursors
09:42 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:42 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003"
09:42 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003"
09:41 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2040.codfw.wmnet
09:40 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp2005.wikimedia.org
09:36 slyngshede@cumin1003: START - Cookbook sre.hosts.reboot-single for host idp2005.wikimedia.org
09:31 fceratto@cumin1003: START - Cookbook sre.ganeti.makevm for new host db1901.eqiad.wmnet
09:31 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1901.eqiad.wmnet
09:31 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:31 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1901.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
09:31 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1901.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
08:57 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2037.codfw.wmnet
08:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast5005.wikimedia.org
08:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2037.codfw.wmnet
08:56 filippo@cumin1003: START - Cookbook sre.mysql.multiinstance_reboot for 10 hosts
08:52 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2037.codfw.wmnet
08:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1289: Pool back
08:35 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudidp2001-dev.codfw.wmnet
08:34 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet
08:33 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1287: Pool back
08:33 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1287 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96150 and previous config saved to /var/cache/conftool/dbconfig/20260818-083311-marostegui.json
08:32 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet
08:31 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudidp2001-dev.codfw.wmnet
08:30 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudbackup1002-dev.eqiad.wmnet
08:29 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudbackup1001-dev.eqiad.wmnet
08:28 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2036.codfw.wmnet
08:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2036.codfw.wmnet
08:25 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1288: Pool back
08:23 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T435157
08:23 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1181.eqiad.wmnet with OS bookworm
08:23 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti2036.codfw.wmnet
08:22 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1288 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96148 and previous config saved to /var/cache/conftool/dbconfig/20260818-082234-marostegui.json
08:05 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1289: Pool back
08:05 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1289 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96145 and previous config saved to /var/cache/conftool/dbconfig/20260818-080531-marostegui.json
07:54 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T435157
19:44 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1160.eqiad.wmnet with reason: host reimage
19:41 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-codfw: Upgrade to Java 17 — T433026 - eevans@cumin1003
19:39 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1162.eqiad.wmnet with reason: host reimage
19:36 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-eqsin and A:cp - 9.2.15 upgrade (T434478)
17:39 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-eqsin and A:cp - 9.2.15 upgrade (T434478)
17:38 swfrench@deploy1003: Started scap sync-world: Deploy "Point Test Wiki to new docroot" - T432412
17:36 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1159.eqiad.wmnet with OS bookworm
17:34 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1158.eqiad.wmnet with OS bookworm
17:30 eevans@cumin1003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Upgrade to Java 17 — T433026 - eevans@cumin1003
17:30 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-ulsfo or A:cp-drmrs and A:cp - 9.2.15 upgrade (T434620)
17:28 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1157.eqiad.wmnet with OS bookworm
17:23 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1180.eqiad.wmnet with reason: host reimage
17:09 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1158.eqiad.wmnet with reason: host reimage
17:04 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1157.eqiad.wmnet with reason: host reimage
17:02 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1180
17:02 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1180
17:01 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-eqsin and A:liberica
17:01 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1193.eqiad.wmnet with reason: host reimage
16:57 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1183.eqiad.wmnet with reason: host reimage
16:56 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1054.eqiad.wmnet with OS trixie
16:55 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1056.eqiad.wmnet with OS trixie
16:54 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1193.eqiad.wmnet with reason: host reimage
16:54 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1192.eqiad.wmnet with reason: host reimage
16:51 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Upgrade to Java 17 — T433026 - eevans@cumin1003
16:50 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1180
16:50 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1180.eqiad.wmnet 17.36.64.10.in-addr.arpa 7.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
16:50 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1180.eqiad.wmnet 17.36.64.10.in-addr.arpa 7.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
16:50 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:50 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1180 - btullis@cumin1003"
16:50 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1180 - btullis@cumin1003"
16:49 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1158.eqiad.wmnet with reason: host reimage
16:49 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1157.eqiad.wmnet with reason: host reimage
16:49 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting A:liberica-eqsin and A:liberica
16:49 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1183.eqiad.wmnet with reason: host reimage
16:48 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1159.eqiad.wmnet with reason: host reimage
16:47 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1192.eqiad.wmnet with reason: host reimage
16:39 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1180
16:39 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1193.eqiad.wmnet with OS bookworm
16:39 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1180.eqiad.wmnet with OS bookworm
16:33 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1192.eqiad.wmnet with OS bookworm
16:33 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1183.eqiad.wmnet with OS bookworm
16:33 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1159.eqiad.wmnet with OS bookworm
16:33 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1158.eqiad.wmnet with OS bookworm
16:33 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1157.eqiad.wmnet with OS bookworm
16:31 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1057.eqiad.wmnet with OS trixie
16:30 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
16:29 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-drmrs and A:liberica
16:28 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1154.eqiad.wmnet with OS bookworm
16:23 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
16:23 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1055.eqiad.wmnet with OS trixie
16:22 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1057
16:22 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1057
16:21 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
16:21 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1057] - vriley@cumin1003"
15:42 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1191.eqiad.wmnet with reason: host reimage
15:39 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1190.eqiad.wmnet with reason: host reimage
15:36 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1054.eqiad.wmnet with OS trixie
15:35 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
15:35 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1056.eqiad.wmnet with OS trixie
15:35 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
15:34 moritzm: failover Ganeti master in eqiad to ganeti1046
15:34 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1176.eqiad.wmnet with reason: host reimage
15:33 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1191.eqiad.wmnet with reason: host reimage
15:33 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1190.eqiad.wmnet with reason: host reimage
15:26 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
15:25 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns2006.wikimedia.org with OS trixie
15:25 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1056
15:03 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1055.eqiad.wmnet with OS trixie
15:02 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
14:59 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1047.eqiad.wmnet
14:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1047.eqiad.wmnet
14:58 sukhe@cumin1003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy
14:57 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-codfw
14:57 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker-exp2001.codfw.wmnet
14:57 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker-exp2001.codfw.wmnet
14:57 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting A:liberica-magru and not P{lvs7003*} and A:liberica
14:57 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
14:56 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1055
14:56 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1055
14:55 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns2006.wikimedia.org with reason: host reimage
14:55 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:55 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1055] - vriley@cumin1003"
14:29 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1054.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
14:28 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2318-2331].codfw.wmnet
14:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2304-2317].codfw.wmnet
14:28 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2304-2317].codfw.wmnet
14:28 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1054
14:27 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1054
14:27 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:27 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1054] - vriley@cumin1003"
14:22 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1182.eqiad.wmnet with reason: host reimage
14:22 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
14:20 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2304-2317].codfw.wmnet
14:19 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host dns4004.wikimedia.org with OS trixie
14:19 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host dns2006.wikimedia.org with OS trixie
14:19 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1143.eqiad.wmnet with reason: host reimage
14:19 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host dns1006.wikimedia.org with OS trixie
14:17 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1189.eqiad.wmnet with reason: host reimage
14:15 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1182.eqiad.wmnet with reason: host reimage
14:14 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1143.eqiad.wmnet with reason: host reimage
14:13 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1051.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
14:12 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2304-2317].codfw.wmnet
14:12 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2290-2303].codfw.wmnet
14:12 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1049.eqiad.wmnet with OS trixie
14:12 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2290-2303].codfw.wmnet
14:12 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
14:11 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1051
14:11 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1051
14:11 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1170.eqiad.wmnet onto db1284.eqiad.wmnet
14:11 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1170: Pool db1170.eqiad.wmnet in after cloning
13:56 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-ulsfo or A:cp-drmrs and A:cp - 9.2.15 upgrade (T434620)
13:36 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1045.eqiad.wmnet
13:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1045.eqiad.wmnet
13:34 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2276-2289].codfw.wmnet
13:34 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1049.eqiad.wmnet with OS trixie
13:34 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2262-2275].codfw.wmnet
13:34 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2262-2275].codfw.wmnet
13:32 Lucas_WMDE: UTC afternoon backport+config window done
13:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1045.eqiad.wmnet
13:27 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2262-2275].codfw.wmnet
13:26 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1049.eqiad.wmnet with OS trixie
13:26 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host cloudvirt1049.eqiad.wmnet with OS trixie
13:26 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1170: Pool db1170.eqiad.wmnet in after cloning
13:23 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1049.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
13:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2262-2275].codfw.wmnet
13:14 lucaswerkmeister-wmde@deploy1003: ndkdd, lucaswerkmeister-wmde: Continuing with deployment
13:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2248-2261].codfw.wmnet
13:14 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2248-2261].codfw.wmnet
13:13 vriley@cumin1003: START - Cookbook sre.hosts.provision for host cloudvirt1049.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
13:12 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1049
13:11 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1049
13:10 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:10 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [cloudvirt1049] - vriley@cumin1003"
12:35 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2204-2215,2242-2243].codfw.wmnet
12:34 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2190-2203].codfw.wmnet
12:34 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2190-2203].codfw.wmnet
12:32 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1028.eqiad.wmnet
12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1028.eqiad.wmnet
12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint1001.eqiad.wmnet
12:30 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1170: Depool db1170.eqiad.wmnet to then clone it to db1284.eqiad.wmnet - marostegui@cumin1003
12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-maint1001.eqiad.wmnet
12:28 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1170: Depool db1170.eqiad.wmnet to then clone it to db1284.eqiad.wmnet - marostegui@cumin1003
12:27 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1170.eqiad.wmnet onto db1284.eqiad.wmnet
12:27 fceratto@cumin1003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host db2901.codfw.wmnet
12:27 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2190-2203].codfw.wmnet
12:27 fceratto@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
12:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1028.eqiad.wmnet
12:19 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2190-2203].codfw.wmnet
12:18 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2172-2179,2184-2189].codfw.wmnet
12:18 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2172-2179,2184-2189].codfw.wmnet
11:43 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
11:43 marostegui@cumin1003: Removing db1153 from zarcillo T434638
11:42 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1153.eqiad.wmnet
11:42 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:42 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1153.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003"
11:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1172.eqiad.wmnet onto db1286.eqiad.wmnet
11:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1172: Pool db1172.eqiad.wmnet in after cloning
11:42 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1153.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003"
11:41 fceratto@cumin1003: START - Cookbook sre.ganeti.makevm for new host db1902.eqiad.wmnet
11:41 fceratto@cumin1003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host db2901.codfw.wmnet
11:41 fceratto@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db2901.codfw.wmnet with OS trixie
11:13 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db2901.codfw.wmnet with OS trixie
11:12 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db2901.codfw.wmnet - fceratto@cumin1003"
11:12 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db2901.codfw.wmnet - fceratto@cumin1003"
11:12 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db2901.codfw.wmnet on all recursors
11:12 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db2901.codfw.wmnet on all recursors
11:12 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:12 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db2901.codfw.wmnet - fceratto@cumin1003"
11:12 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2130-2143].codfw.wmnet
11:11 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2107-2115,2124-2129].codfw.wmnet
11:11 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2107-2115,2124-2129].codfw.wmnet
11:06 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db2901.codfw.wmnet - fceratto@cumin1003"
11:03 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2107-2115,2124-2129].codfw.wmnet
10:57 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1172: Pool db1172.eqiad.wmnet in after cloning
10:54 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2107-2115,2124-2129].codfw.wmnet
10:54 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2078,2087-2095,2102-2106].codfw.wmnet
10:54 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2078,2087-2095,2102-2106].codfw.wmnet
10:38 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1056.eqiad.wmnet
10:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1056.eqiad.wmnet
10:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2078,2087-2095,2102-2106].codfw.wmnet
10:36 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2061-2062,2064-2065,2067-2077].codfw.wmnet
10:36 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2061-2062,2064-2065,2067-2077].codfw.wmnet
10:32 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1056.eqiad.wmnet
10:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2061-2062,2064-2065,2067-2077].codfw.wmnet
10:19 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2061-2062,2064-2065,2067-2077].codfw.wmnet
10:18 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2038-2039,2041-2042,2044,2046,2049-2051,2055-2060].codfw.wmnet
10:18 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2038-2039,2041-2042,2044,2046,2049-2051,2055-2060].codfw.wmnet
10:15 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1055.eqiad.wmnet
10:15 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1055.eqiad.wmnet
10:08 fceratto@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
10:08 fceratto@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1903.eqiad.wmnet - fceratto@cumin1003"
10:07 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1902.eqiad.wmnet with OS trixie
10:07 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1902.eqiad.wmnet - fceratto@cumin1003"
10:07 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1902.eqiad.wmnet - fceratto@cumin1003"
10:01 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1903.eqiad.wmnet - fceratto@cumin1003"
10:01 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2038-2039,2041-2042,2044,2046,2049-2051,2055-2060].codfw.wmnet
10:01 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2002,2005-2006,2011-2015,2017-2018,2033-2037].codfw.wmnet
10:01 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2002,2005-2006,2011-2015,2017-2018,2033-2037].codfw.wmnet
09:44 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2002,2005-2006,2011-2015,2017-2018,2033-2037].codfw.wmnet
09:43 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-codfw
09:43 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1174.eqiad.wmnet onto db1288.eqiad.wmnet
09:42 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1174: Pool db1174.eqiad.wmnet in after cloning
09:40 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1172: Depool db1172.eqiad.wmnet to then clone it to db1286.eqiad.wmnet - marostegui@cumin1003
09:39 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1172: Depool db1172.eqiad.wmnet to then clone it to db1286.eqiad.wmnet - marostegui@cumin1003
09:39 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1172.eqiad.wmnet onto db1286.eqiad.wmnet
09:33 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1175.eqiad.wmnet onto db1289.eqiad.wmnet
09:32 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1175: Pool db1175.eqiad.wmnet in after cloning
09:31 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1201.eqiad.wmnet onto db1287.eqiad.wmnet
09:31 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1201: Pool db1201.eqiad.wmnet in after cloning
09:28 fceratto@cumin1003: START - Cookbook sre.hosts.reimage for host db1901.eqiad.wmnet with OS trixie
09:27 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1901.eqiad.wmnet - fceratto@cumin1003"
09:27 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM db1901.eqiad.wmnet - fceratto@cumin1003"
09:27 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) db1901.eqiad.wmnet on all recursors
09:27 fceratto@cumin1003: START - Cookbook sre.dns.wipe-cache db1901.eqiad.wmnet on all recursors
09:26 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:26 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003"
09:26 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM db1901.eqiad.wmnet - fceratto@cumin1003"
09:15 fceratto@cumin1003: START - Cookbook sre.ganeti.makevm for new host db1901.eqiad.wmnet
09:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ldap-rw1001.wikimedia.org with OS trixie
08:57 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1174: Pool db1174.eqiad.wmnet in after cloning
08:54 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1054.eqiad.wmnet
08:54 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1054.eqiad.wmnet
08:48 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1054.eqiad.wmnet
08:47 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1175: Pool db1175.eqiad.wmnet in after cloning
08:46 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
08:46 marostegui@cumin1003: Removing db1152 from zarcillo T434480
08:46 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1152.eqiad.wmnet
08:46 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
08:46 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1152.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003"
08:46 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1201: Pool db1201.eqiad.wmnet in after cloning
08:46 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1152.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003"
08:08 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet
08:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet
08:03 godog: switch production to use dumps-nfs.w.o - T432212
08:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet
08:02 phuedx@deploy1003: nvdtn19, phuedx: Continuing with deployment
08:00 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1201: Depool db1201.eqiad.wmnet to then clone it to db1287.eqiad.wmnet - marostegui@cumin1003
08:00 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1201: Depool db1201.eqiad.wmnet to then clone it to db1287.eqiad.wmnet - marostegui@cumin1003
08:00 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1201.eqiad.wmnet onto db1287.eqiad.wmnet
08:00 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1057.eqiad.wmnet
08:00 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
08:00 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1057.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
07:59 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1057.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
07:59 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1275: Pool back
07:51 tappof: bump space for prometheus k8s-aux in eqiad
07:50 filippo@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudvirt1057.eqiad.wmnet
07:48 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1281: Pool back
07:47 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1281 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96108 and previous config saved to /var/cache/conftool/dbconfig/20260817-074749-marostegui.json
07:46 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet
07:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1174: Depool db1174.eqiad.wmnet to then clone it to db1288.eqiad.wmnet - marostegui@cumin1003
07:41 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1174: Depool db1174.eqiad.wmnet to then clone it to db1288.eqiad.wmnet - marostegui@cumin1003
07:41 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1174.eqiad.wmnet onto db1288.eqiad.wmnet
07:40 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet
07:40 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm2001.wikimedia.org
07:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet
07:39 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1056.eqiad.wmnet
07:39 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
07:39 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1056.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
07:38 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1056.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
07:36 slyngshede@cumin1003: START - Cookbook sre.hosts.reboot-single for host idm2001.wikimedia.org
07:35 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1279 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96104 and previous config saved to /var/cache/conftool/dbconfig/20260817-073542-marostegui.json
07:31 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1279: Pool back
07:30 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1279 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96102 and previous config saved to /var/cache/conftool/dbconfig/20260817-073038-marostegui.json
07:29 slyngshede@cumin1003: START - Cookbook sre.hosts.reboot-single for host idm-test1001.wikimedia.org
07:29 slyngshede@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm1001.wikimedia.org
07:28 filippo@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudvirt1056.eqiad.wmnet
07:28 moritzm: extend the disk of ldap-rw1001 by 80G T331699
07:28 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1055.eqiad.wmnet
07:28 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
07:28 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1055.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
07:27 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1055.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
07:26 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1044.eqiad.wmnet
07:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1044.eqiad.wmnet
07:25 slyngshede@cumin1003: START - Cookbook sre.hosts.reboot-single for host idm1001.wikimedia.org
07:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1175: Depool db1175.eqiad.wmnet to then clone it to db1289.eqiad.wmnet - marostegui@cumin1003
07:24 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1175: Depool db1175.eqiad.wmnet to then clone it to db1289.eqiad.wmnet - marostegui@cumin1003
07:24 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1175.eqiad.wmnet onto db1289.eqiad.wmnet
07:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1044.eqiad.wmnet
07:16 filippo@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudvirt1055.eqiad.wmnet
07:15 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1054.eqiad.wmnet
07:15 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
07:15 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
07:15 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1054.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
07:13 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1275: Pool back
07:12 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1275 to dbctl T407942', diff saved to https://phabricator.wikimedia.org/P96098 and previous config saved to /var/cache/conftool/dbconfig/20260817-071225-marostegui.json
07:05 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1043.eqiad.wmnet
07:05 filippo@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudvirt1054.eqiad.wmnet
07:05 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1051.eqiad.wmnet
07:05 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
07:05 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
07:05 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1043.eqiad.wmnet
07:04 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1051.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
06:59 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin1001.eqiad.wmnet
06:59 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1043.eqiad.wmnet
06:59 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin2001.codfw.wmnet
06:55 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcumin2001.codfw.wmnet
06:55 filippo@cumin1003: START - Cookbook sre.hosts.decommission for hosts cloudvirt1051.eqiad.wmnet
06:54 filippo@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1049.eqiad.wmnet
06:54 filippo@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
06:54 filippo@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
06:54 filippo@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1049.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - filippo@cumin1003"
06:54 filippo@cumin1003: START - Cookbook sre.hosts.reboot-single for host cloudcumin1001.eqiad.wmnet
12:33 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
12:33 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
12:33 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
12:27 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
12:26 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
12:26 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
12:26 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
12:24 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ldap-rw1001.wikimedia.org with reason: host reimage
12:20 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
12:17 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ldap-rw1001.wikimedia.org with reason: host reimage
12:14 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
12:14 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
12:03 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host ldap-rw1001.wikimedia.org with OS trixie
12:00 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cuminunpriv1001.eqiad.wmnet
11:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host cuminunpriv1001.eqiad.wmnet
11:27 aokoth@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org
11:24 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db1153 from dbctl T434638', diff saved to https://phabricator.wikimedia.org/P96097 and previous config saved to /var/cache/conftool/dbconfig/20260814-112449-marostegui.json
11:21 aokoth@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org
11:20 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T434668
11:17 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
11:17 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
11:17 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
11:10 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
11:10 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
11:10 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
11:10 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
11:03 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1181.eqiad.wmnet with OS bookworm
11:03 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
11:03 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
11:02 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
11:02 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
10:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1235.eqiad.wmnet with OS bookworm
10:57 cgoubert@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rdb-lock2003.codfw.wmnet
10:57 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb-lock2003.codfw.wmnet with OS trixie
10:56 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
10:54 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1153.eqiad.wmnet with OS bookworm
10:51 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
10:51 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
10:51 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
10:50 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T434668
10:49 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1187.eqiad.wmnet with OS bookworm
10:45 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
10:44 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
10:44 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
10:44 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb-lock2003.codfw.wmnet with reason: host reimage
10:38 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb-lock2003.codfw.wmnet with reason: host reimage
10:35 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1153.eqiad.wmnet with reason: host reimage
10:32 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1235.eqiad.wmnet with reason: host reimage
10:29 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1187.eqiad.wmnet with reason: host reimage
10:24 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1153.eqiad.wmnet with reason: host reimage
10:23 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1235.eqiad.wmnet with reason: host reimage
10:21 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1187.eqiad.wmnet with reason: host reimage
10:16 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host rdb-lock2003.codfw.wmnet with OS trixie
10:16 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1005.wikimedia.org
10:12 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
10:12 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
10:12 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors
10:12 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors
10:12 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:12 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
10:12 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
10:09 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install1005.wikimedia.org
10:07 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1235.eqiad.wmnet with OS bookworm
10:07 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1187.eqiad.wmnet with OS bookworm
10:07 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1181.eqiad.wmnet with OS bookworm
10:07 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1153.eqiad.wmnet with OS bookworm
10:07 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install2005.wikimedia.org
09:33 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1231.eqiad.wmnet with reason: host reimage
09:33 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install5004.wikimedia.org
09:32 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host centrallog1002.eqiad.wmnet
09:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install6003.wikimedia.org
09:30 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1231.eqiad.wmnet with reason: host reimage
09:28 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1152.eqiad.wmnet with reason: host reimage
09:25 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host centrallog1002.eqiad.wmnet
09:25 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1152.eqiad.wmnet with reason: host reimage
09:24 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install6003.wikimedia.org
09:22 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1232
09:22 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1232
09:22 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1232
09:22 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1232.eqiad.wmnet 25.53.64.10.in-addr.arpa 5.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
09:22 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan1001.eqiad.wmnet
09:22 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1232.eqiad.wmnet 25.53.64.10.in-addr.arpa 5.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
09:22 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:22 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1232 - btullis@cumin1003"
09:22 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1232 - btullis@cumin1003"
09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org
09:17 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1232
09:16 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1231
09:16 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1231
09:14 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1231
09:14 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1231.eqiad.wmnet 24.53.64.10.in-addr.arpa 4.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
09:14 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan1001.eqiad.wmnet
09:14 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1231.eqiad.wmnet 24.53.64.10.in-addr.arpa 4.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
09:14 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:14 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1231 - btullis@cumin1003"
09:14 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1231 - btullis@cumin1003"
09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org
09:08 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1231
09:08 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1152
09:08 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1152
09:06 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1152
09:06 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1152.eqiad.wmnet 16.53.64.10.in-addr.arpa 6.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
09:06 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1152.eqiad.wmnet 16.53.64.10.in-addr.arpa 6.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
09:06 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:06 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1152 - btullis@cumin1003"
09:06 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1152 - btullis@cumin1003"
09:03 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1151.eqiad.wmnet
09:03 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:03 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1151.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
08:55 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan2001.codfw.wmnet
08:54 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1151.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
08:54 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1152
08:53 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1232.eqiad.wmnet with OS bookworm
08:53 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1231.eqiad.wmnet with OS bookworm
08:53 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1152.eqiad.wmnet with OS bookworm
08:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pool back
08:51 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for an-worker1201.eqiad.wmnet
08:50 btullis@cumin1003: START - Cookbook sre.hosts.remove-downtime for an-worker1201.eqiad.wmnet
08:48 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1229.eqiad.wmnet with OS bookworm
08:47 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan2001.codfw.wmnet
08:46 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping2004.codfw.wmnet
08:43 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ping2004.codfw.wmnet
08:40 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-worker1230.eqiad.wmnet with OS bookworm
08:31 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan1002.eqiad.wmnet
08:28 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1229.eqiad.wmnet with reason: host reimage
08:25 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan1002.eqiad.wmnet
08:25 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1229.eqiad.wmnet with reason: host reimage
08:24 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ping1004.eqiad.wmnet
08:21 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ping1004.eqiad.wmnet
08:19 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1230.eqiad.wmnet with reason: host reimage
08:12 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1230.eqiad.wmnet with reason: host reimage
08:11 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1229
08:11 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1229
08:11 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1229
08:11 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1229.eqiad.wmnet 22.53.64.10.in-addr.arpa 2.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
08:11 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1229.eqiad.wmnet 22.53.64.10.in-addr.arpa 2.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
08:11 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
08:11 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1229 - btullis@cumin1003"
08:11 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1229 - btullis@cumin1003"
08:10 btullis@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-worker1201.eqiad.wmnet with reason: Fixing a disk
08:07 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host titan2002.codfw.wmnet
08:05 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db2209: Pool back
08:00 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host titan2002.codfw.wmnet
07:59 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1229
07:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1230
07:58 btullis@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1230
07:55 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1230
07:55 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1230.eqiad.wmnet 23.53.64.10.in-addr.arpa 3.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
07:55 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1230.eqiad.wmnet 23.53.64.10.in-addr.arpa 3.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
07:55 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
07:55 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1230 - btullis@cumin1003"
07:55 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1230 - btullis@cumin1003"
07:51 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestagemaster2005.codfw.wmnet with OS trixie
07:41 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1230
07:41 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1229.eqiad.wmnet with OS bookworm
07:41 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1230.eqiad.wmnet with OS bookworm
07:39 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db1152 from dbctl T434480', diff saved to https://phabricator.wikimedia.org/P96090 and previous config saved to /var/cache/conftool/dbconfig/20260814-073941-marostegui.json
07:29 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2005.codfw.wmnet with reason: host reimage
07:23 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2005.codfw.wmnet with reason: host reimage
07:04 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host kubestagemaster2005.codfw.wmnet with OS trixie
06:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1228.eqiad.wmnet with OS bookworm
06:44 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1227.eqiad.wmnet with OS bookworm
06:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1209.eqiad.wmnet with OS bookworm
06:36 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1175.eqiad.wmnet with OS bookworm
06:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1228.eqiad.wmnet with reason: host reimage
06:27 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1228.eqiad.wmnet with reason: host reimage
06:25 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1227.eqiad.wmnet with reason: host reimage
06:21 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1227.eqiad.wmnet with reason: host reimage
06:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1209.eqiad.wmnet with reason: host reimage
06:14 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1209.eqiad.wmnet with reason: host reimage
06:14 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1228
06:14 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1228
06:13 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1175.eqiad.wmnet with reason: host reimage
06:12 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1228
06:12 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1228.eqiad.wmnet 20.53.64.10.in-addr.arpa 0.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
06:12 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1228.eqiad.wmnet 20.53.64.10.in-addr.arpa 0.2.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
06:12 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
06:12 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1228 - ryankemper@cumin2003"
06:12 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1228 - ryankemper@cumin2003"
06:09 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1175.eqiad.wmnet with reason: host reimage
06:07 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1228
06:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1227
06:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1227
06:06 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1227
06:06 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1227.eqiad.wmnet 19.53.64.10.in-addr.arpa 9.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
06:06 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1227.eqiad.wmnet 19.53.64.10.in-addr.arpa 9.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
06:06 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
06:06 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1227 - ryankemper@cumin2003"
06:06 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1227 - ryankemper@cumin2003"
06:00 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1227
06:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1209
06:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1209
06:00 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1209
06:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1209.eqiad.wmnet 15.53.64.10.in-addr.arpa 5.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
06:00 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1209.eqiad.wmnet 15.53.64.10.in-addr.arpa 5.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
06:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
06:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1209 - ryankemper@cumin2003"
06:00 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1209 - ryankemper@cumin2003"
05:54 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1209
05:54 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1175
05:54 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1175
05:52 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1175
05:52 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1175.eqiad.wmnet 17.53.64.10.in-addr.arpa 7.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
05:52 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1175.eqiad.wmnet 17.53.64.10.in-addr.arpa 7.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
05:52 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
05:52 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1175 - ryankemper@cumin2003"
05:52 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1175 - ryankemper@cumin2003"
05:49 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1228.eqiad.wmnet with OS bookworm
05:49 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1227.eqiad.wmnet with OS bookworm
05:48 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1209.eqiad.wmnet with OS bookworm
00:59 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1226.eqiad.wmnet with OS bookworm
00:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1225.eqiad.wmnet with OS bookworm
00:41 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1224.eqiad.wmnet with OS bookworm
00:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1226.eqiad.wmnet with reason: host reimage
00:31 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1226.eqiad.wmnet with reason: host reimage
00:28 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1225.eqiad.wmnet with reason: host reimage
00:25 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1225.eqiad.wmnet with reason: host reimage
00:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1224.eqiad.wmnet with reason: host reimage
00:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1226
00:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1226
00:17 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1226
00:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1226.eqiad.wmnet 23.36.64.10.in-addr.arpa 3.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
00:17 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1226.eqiad.wmnet 23.36.64.10.in-addr.arpa 3.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
00:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
00:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1226 - ryankemper@cumin2003"
00:17 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1226 - ryankemper@cumin2003"
00:16 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1224.eqiad.wmnet with reason: host reimage
00:12 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1226
00:11 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1225
00:11 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1225
00:10 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1225
00:10 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1225.eqiad.wmnet 22.36.64.10.in-addr.arpa 2.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
00:10 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1225.eqiad.wmnet 22.36.64.10.in-addr.arpa 2.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
00:10 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
00:10 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1225 - ryankemper@cumin2003"
00:10 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1225 - ryankemper@cumin2003"
00:02 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1225
00:02 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1224
00:02 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1224
00:00 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1224
00:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1224.eqiad.wmnet 21.36.64.10.in-addr.arpa 1.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
00:00 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1224.eqiad.wmnet 21.36.64.10.in-addr.arpa 1.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
00:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
00:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1224 - ryankemper@cumin2003"
00:00 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1224 - ryankemper@cumin2003"
2026-08-13
23:54 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1226.eqiad.wmnet with OS bookworm
23:53 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1225.eqiad.wmnet with OS bookworm
23:07 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1223
23:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1223.eqiad.wmnet 20.36.64.10.in-addr.arpa 0.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
23:07 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1223.eqiad.wmnet 20.36.64.10.in-addr.arpa 0.2.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
23:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
23:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1223 - ryankemper@cumin2003"
23:03 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1223 - ryankemper@cumin2003"
23:01 sbassett: Deployed security updates for T430596, T120386
22:55 bking@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host kubestagemaster2005.codfw.wmnet
22:55 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubestagemaster2005.codfw.wmnet with OS bookworm
22:53 ryankemper@cumin2003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
22:49 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2005.codfw.wmnet with reason: host reimage
22:49 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1222.eqiad.wmnet with OS bookworm
22:42 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2005.codfw.wmnet with reason: host reimage
22:29 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1222.eqiad.wmnet with reason: host reimage
22:26 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1222.eqiad.wmnet with reason: host reimage
22:23 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host aux-k8s-etcd2003.codfw.wmnet
22:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd2003.codfw.wmnet with OS bookworm
22:23 bking@cumin2003: START - Cookbook sre.hosts.reimage for host kubestagemaster2005.codfw.wmnet with OS bookworm
22:23 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM kubestagemaster2005.codfw.wmnet - bking@cumin2003"
22:23 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM kubestagemaster2005.codfw.wmnet - bking@cumin2003"
22:22 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kubestagemaster2005.codfw.wmnet on all recursors
22:22 bking@cumin2003: START - Cookbook sre.dns.wipe-cache kubestagemaster2005.codfw.wmnet on all recursors
22:22 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
22:22 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM kubestagemaster2005.codfw.wmnet - bking@cumin2003"
22:22 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM kubestagemaster2005.codfw.wmnet - bking@cumin2003"
22:12 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host kubestagemaster2005.codfw.wmnet
22:12 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1222
22:12 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1222
22:11 sbassett: Deployed security updates for T429244, T434039
22:11 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1222
22:11 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1222.eqiad.wmnet 19.36.64.10.in-addr.arpa 9.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
22:11 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1222.eqiad.wmnet 19.36.64.10.in-addr.arpa 9.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
22:11 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
22:11 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1222 - ryankemper@cumin2003"
22:07 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1222 - ryankemper@cumin2003"
22:05 robh@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-wdqs2001.codfw.wmnet with reason: updating firmware
21:39 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1212.eqiad.wmnet with reason: host reimage
21:34 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1212.eqiad.wmnet with reason: host reimage
21:33 bking@cumin2003: START - Cookbook sre.hosts.reimage for host aux-k8s-etcd2003.codfw.wmnet with OS bookworm
21:32 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-etcd2003.codfw.wmnet - bking@cumin2003"
21:32 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-etcd2003.codfw.wmnet - bking@cumin2003"
21:32 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-etcd2003.codfw.wmnet on all recursors
21:32 bking@cumin2003: START - Cookbook sre.dns.wipe-cache aux-k8s-etcd2003.codfw.wmnet on all recursors
21:32 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:32 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-etcd2003.codfw.wmnet - bking@cumin2003"
21:31 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-etcd2003.codfw.wmnet - bking@cumin2003"
20:58 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-etcd2001.codfw.wmnet with OS bookworm
20:57 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003"
20:57 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003"
20:57 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2001.codfw.wmnet on all recursors
20:57 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2001.codfw.wmnet on all recursors
20:57 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
20:57 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003"
20:57 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003"
19:04 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1166.eqiad.wmnet onto db1280.eqiad.wmnet
19:03 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage
19:02 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1166: Pool db1166.eqiad.wmnet in after cloning
18:59 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage
18:58 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1234.eqiad.wmnet with OS bookworm
18:52 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie
18:51 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
18:49 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
18:46 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie
18:46 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS trixie
18:35 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1006.eqiad.wmnet with OS trixie
18:29 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1220.eqiad.wmnet with OS bookworm
18:29 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1234.eqiad.wmnet with reason: host reimage
18:27 bking@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dse-k8s-etcd2001.codfw.wmnet
18:27 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2001.codfw.wmnet on all recursors
18:27 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2001.codfw.wmnet on all recursors
18:27 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
18:27 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003"
18:27 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003"
18:25 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1234.eqiad.wmnet with reason: host reimage
18:25 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie
18:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1221.eqiad.wmnet with reason: host reimage
18:17 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1166: Pool db1166.eqiad.wmnet in after cloning
18:15 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.15 refs T430834
18:15 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1221.eqiad.wmnet with reason: host reimage
18:12 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
18:12 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2001.codfw.wmnet on all recursors
18:12 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2001.codfw.wmnet on all recursors
18:12 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
18:12 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003"
18:12 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2001.codfw.wmnet - bking@cumin2003"
18:10 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: quash java safepoint logspam - bking@cumin2003 - T434685
18:09 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1234
18:09 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1234
18:09 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1220.eqiad.wmnet with reason: host reimage
18:07 brennen: 1.47.0-wmf.15 train status (T430834) - no current blockers, rolling to all wikis
18:07 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1234
18:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1234.eqiad.wmnet 10.36.64.10.in-addr.arpa 0.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
18:01 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1234
18:01 dancy@deploy1003: Installing scap version "4.280.1" for 3 host(s)
18:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-worker1221
18:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-worker1221
18:00 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1221
18:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1221.eqiad.wmnet 18.36.64.10.in-addr.arpa 8.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
18:00 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1221.eqiad.wmnet 18.36.64.10.in-addr.arpa 8.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
18:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
18:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1221 - ryankemper@cumin2003"
17:51 ryankemper@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1220
17:51 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1220.eqiad.wmnet 11.36.64.10.in-addr.arpa 1.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
17:51 ryankemper@cumin2003: START - Cookbook sre.dns.wipe-cache an-worker1220.eqiad.wmnet 11.36.64.10.in-addr.arpa 1.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
17:51 ryankemper@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1220 - ryankemper@cumin2003"
17:50 ryankemper@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1220 - ryankemper@cumin2003"
17:47 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
17:47 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7003*} and A:liberica
17:46 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1234.eqiad.wmnet with OS bookworm
17:45 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1221.eqiad.wmnet with OS bookworm
17:45 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host an-worker1220
17:45 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host an-worker1220.eqiad.wmnet with OS bookworm
17:43 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin rebooting P{lvs7003*} and A:liberica
17:41 bking@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dse-k8s-etcd2001.codfw.wmnet
17:41 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host dse-k8s-etcd2001.codfw.wmnet
17:40 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
17:40 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
17:39 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
17:38 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
17:36 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
17:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1179.eqiad.wmnet with OS bookworm
17:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1150.eqiad.wmnet with OS bookworm
17:18 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns3004.wikimedia.org with OS trixie
17:15 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
17:09 swfrench@deploy1003: Started scap sync-world: Helmfile-only deployment for mediawiki chart bump - T427666
17:02 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
17:01 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
17:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1218.eqiad.wmnet with reason: host reimage
17:01 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
17:01 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
16:51 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1166: Depool db1166.eqiad.wmnet to then clone it to db1280.eqiad.wmnet - cwilliams@cumin1003
16:50 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1166: Depool db1166.eqiad.wmnet to then clone it to db1280.eqiad.wmnet - cwilliams@cumin1003
16:50 cwilliams@cumin1003: START - Cookbook sre.mysql.clone of db1166.eqiad.wmnet onto db1280.eqiad.wmnet
16:49 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1218.eqiad.wmnet with reason: host reimage
16:48 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1179.eqiad.wmnet with reason: host reimage
16:47 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1150.eqiad.wmnet with reason: host reimage
16:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for 1 hosts
16:41 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2220: Upgrade of db2220.codfw.wmnet completed
14:59 bking@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host dse-k8s-etcd2004.codfw.wmnet
14:58 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2004.codfw.wmnet on all recursors
14:58 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2004.codfw.wmnet on all recursors
14:58 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:58 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003"
14:58 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003"
14:55 zabe@deploy1003: zabe: Continuing with deployment
14:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-etcd2004.codfw.wmnet on all recursors
14:53 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-etcd2004.codfw.wmnet on all recursors
14:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:53 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003"
14:53 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM dse-k8s-etcd2004.codfw.wmnet - bking@cumin2003"
14:52 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl2001.codfw.wmnet
14:52 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl2001.codfw.wmnet
14:52 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-codfw
14:52 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl2001.codfw.wmnet
14:52 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl2001.codfw.wmnet
13:51 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus2005.codfw.wmnet
13:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host apifeatureusage1001.eqiad.wmnet with OS bookworm
13:50 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus7002.magru.wmnet
13:50 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1015.eqiad.wmnet
13:50 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:50 sukhe@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003"
13:49 sukhe@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003"
13:41 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow1003.eqiad.wmnet
13:40 sukhe@cumin1003: START - Cookbook sre.hosts.decommission for hosts lvs1015.eqiad.wmnet
13:40 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1014.eqiad.wmnet
13:40 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:40 sukhe@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003"
13:40 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1040.eqiad.wmnet
13:40 sukhe@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003"
13:39 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1214.eqiad.wmnet with reason: host reimage
13:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2004.codfw.wmnet
13:38 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus6002.drmrs.wmnet
13:38 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus5003.eqsin.wmnet
13:31 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus5003.eqsin.wmnet
13:31 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin2003
13:31 tappof@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus4003.ulsfo.wmnet
13:30 sukhe@cumin1003: START - Cookbook sre.hosts.decommission for hosts lvs1014.eqiad.wmnet
13:30 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lvs1013.eqiad.wmnet
13:30 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:30 sukhe@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003"
13:30 sukhe@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lvs1013.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - sukhe@cumin1003"
13:27 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1039.eqiad.wmnet
13:27 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1039.eqiad.wmnet
13:26 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1167.eqiad.wmnet onto db1281.eqiad.wmnet
13:26 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1167: Pool db1167.eqiad.wmnet in after cloning
13:25 tappof@cumin1003: START - Cookbook sre.hosts.reboot-single for host prometheus4003.ulsfo.wmnet
12:49 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1169: Depool db1169.eqiad.wmnet to then clone it to db1277.eqiad.wmnet - cwilliams@cumin1003
12:46 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1169: Depool db1169.eqiad.wmnet to then clone it to db1277.eqiad.wmnet - cwilliams@cumin1003
12:46 cwilliams@cumin1003: START - Cookbook sre.mysql.clone of db1169.eqiad.wmnet onto db1277.eqiad.wmnet
12:45 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:45 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns record for deleted IP reservations eqsin lvs vlan ints - cmooney@cumin1003"
12:44 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns record for deleted IP reservations eqsin lvs vlan ints - cmooney@cumin1003"
12:43 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1051.eqiad.wmnet
12:43 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1051.eqiad.wmnet
12:42 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1210.eqiad.wmnet with reason: host reimage
12:41 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1167: Pool db1167.eqiad.wmnet in after cloning
12:20 cgoubert@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rdb-lock2003.codfw.wmnet
12:20 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:20 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox dns entries for expanded public1-603-eqsin subnet - cmooney@cumin1003"
12:20 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox dns entries for expanded public1-603-eqsin subnet - cmooney@cumin1003"
12:20 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors
12:20 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors
12:19 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:19 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
12:19 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1049.eqiad.wmnet
12:19 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1049.eqiad.wmnet
12:19 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
12:18 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be1003.eqiad.wmnet
12:17 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet
12:10 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be1006.eqiad.wmnet
12:10 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors
12:10 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors
12:10 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:10 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
12:10 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
12:09 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netflow7002.magru.wmnet
11:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1213.eqiad.wmnet with OS bookworm
11:56 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host apus-be1005.eqiad.wmnet
11:55 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apus-be1004.eqiad.wmnet
11:54 cgoubert@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rdb-lock2003.codfw.wmnet
11:54 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors
11:53 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors
11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
11:53 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
11:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1038.eqiad.wmnet
11:49 mvernon@cumin2003: START - Cookbook sre.hosts.reboot-single for host apus-be1004.eqiad.wmnet
11:44 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
11:44 moritzm: installing Linux 5.10.262 on Bullseye hosts
11:40 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors
11:40 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors
11:40 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:40 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
11:40 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
11:38 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1213.eqiad.wmnet with reason: host reimage
11:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1167: Depool db1167.eqiad.wmnet to then clone it to db1281.eqiad.wmnet - marostegui@cumin1003
11:35 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2003.codfw.wmnet
11:35 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1167: Depool db1167.eqiad.wmnet to then clone it to db1281.eqiad.wmnet - marostegui@cumin1003
11:35 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1167.eqiad.wmnet onto db1281.eqiad.wmnet
11:34 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 22 hosts with reason: Cloning
11:34 moritzm: remove ganeti3005 from esams03 cluster, hardware issues T434646
11:32 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1213.eqiad.wmnet with reason: host reimage
11:28 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet
11:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1034.eqiad.wmnet
11:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1034.eqiad.wmnet
10:50 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Enabling notifications and pooling
10:45 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1033.eqiad.wmnet
10:45 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1033.eqiad.wmnet
10:40 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[1216,1225,1239-1240].eqiad.wmnet with reason: reboot
10:39 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti1033.eqiad.wmnet
10:38 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db1151 from dbctl T434538', diff saved to https://phabricator.wikimedia.org/P96055 and previous config saved to /var/cache/conftool/dbconfig/20260813-103828-marostegui.json
09:45 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 8 hosts with reason: reboot
09:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 22 hosts with reason: Cloning
09:43 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1165: Depool db1165.eqiad.wmnet to then clone it to db1279.eqiad.wmnet - marostegui@cumin1003
09:42 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1165: Depool db1165.eqiad.wmnet to then clone it to db1279.eqiad.wmnet - marostegui@cumin1003
09:42 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1165.eqiad.wmnet onto db1279.eqiad.wmnet
09:36 btullis@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host an-worker1151
09:36 btullis@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-worker1151.eqiad.wmnet 13.36.64.10.in-addr.arpa 3.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
09:36 btullis@cumin1003: START - Cookbook sre.dns.wipe-cache an-worker1151.eqiad.wmnet 13.36.64.10.in-addr.arpa 3.1.0.0.6.3.0.0.4.6.0.0.0.1.0.0.6.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
09:36 btullis@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:36 btullis@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1151 - btullis@cumin1003"
09:36 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on backupmon1001.eqiad.wmnet with reason: reboot
09:36 btullis@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-worker1151 - btullis@cumin1003"
09:35 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2003.codfw.wmnet
09:33 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 7 hosts
09:33 jynus@cumin1003: START - Cookbook sre.hosts.remove-downtime for 7 hosts
09:33 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
08:44 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts2002.codfw.wmnet
08:44 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc2003.codfw.wmnet
08:43 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host vrts1004.eqiad.wmnet
08:43 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab2003.codfw.wmnet
08:42 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host stewards2001.codfw.wmnet
08:42 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad2002.codfw.wmnet
08:41 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host stewards1001.eqiad.wmnet
08:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1273: Pool in s7
08:41 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org
08:40 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host doc2003.codfw.wmnet
08:40 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gerrit1003.wikimedia.org
08:39 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host etherpad1004.eqiad.wmnet
08:39 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host doc1004.eqiad.wmnet
08:39 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2002.wikimedia.org
08:38 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host etherpad2002.codfw.wmnet
08:37 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host phab2003.codfw.wmnet
08:36 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org
08:35 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host etherpad1004.eqiad.wmnet
08:35 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host doc1004.eqiad.wmnet
08:34 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0)
08:34 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org
08:34 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet2003.codfw.wmnet
08:34 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2003.wikimedia.org
08:33 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host planet1003.eqiad.wmnet
08:33 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org
08:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 22 hosts with reason: Cloning
08:32 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org
08:31 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict1002.eqiad.wmnet
08:30 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host planet2003.codfw.wmnet
08:29 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host planet1003.eqiad.wmnet
08:29 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org
08:28 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists2001.wikimedia.org
08:28 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab2003.wikimedia.org
08:27 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host aphlict1002.eqiad.wmnet
08:27 arnaudb@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict2001.codfw.wmnet
08:26 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org
08:23 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host aphlict2001.codfw.wmnet
08:23 btullis@cumin1003: START - Cookbook sre.hosts.move-vlan for host an-worker1151
08:22 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-worker1151.eqiad.wmnet with OS bookworm
08:22 arnaudb@cumin1003: START - Cookbook sre.hosts.reboot-single for host lists2001.wikimedia.org
08:20 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1161: Depool db1161.eqiad.wmnet to then clone it to db1275.eqiad.wmnet - marostegui@cumin1003
08:20 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1161: Depool db1161.eqiad.wmnet to then clone it to db1275.eqiad.wmnet - marostegui@cumin1003
08:20 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1161.eqiad.wmnet onto db1275.eqiad.wmnet
08:15 Emperor: roll-restart apus frontends in eqiad
07:56 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1273: Pool in s7
07:56 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1273 to dbctl T434288', diff saved to https://phabricator.wikimedia.org/P96035 and previous config saved to /var/cache/conftool/dbconfig/20260813-075611-marostegui.json
07:38 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1273.eqiad.wmnet with reason: Reboot
07:31 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.sanitize-wiki (exit_code=97) Managing sanitization for wikis testwiki in section s3
07:24 marostegui@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis testwiki in section s3
07:19 jayme@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on kubestagemaster2005.codfw.wmnet with reason: downtime because of hardware failure and no DRBD
05:42 arnaudb@dns1006: END - running authdns-update
21:32 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:31 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:30 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:30 vriley@cumin1003: START - Cookbook sre.hosts.provision for host db1245.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:17 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:14 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:14 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
20:39 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie
20:38 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
20:38 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5022.eqsin.wmnet with OS trixie
20:38 cdobbins@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cdobbins@cumin1003"
20:37 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
19:55 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.15 refs T430834
19:47 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie
19:35 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie
19:35 cdobbins@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie
19:32 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
19:30 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
19:29 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:35 jhancock@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp5022']
17:31 cdobbins@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on A:cp-magru and not (P{cp7001*} or P{cp7009*}) and A:cp - 9.2.15 upgrade (T434620)
16:02 cdobbins@cumin1003: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on A:cp-magru and not (P{cp7001*} or P{cp7009*}) and A:cp - 9.2.15 upgrade (T434620)
15:57 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp5022.mgmt.eqsin.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
15:56 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1272: New host
15:52 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2046.codfw.wmnet
15:52 jmm@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host ganeti2046.codfw.wmnet
15:42 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp5022.mgmt.eqsin.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
15:35 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp5022.mgmt.eqsin.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
15:23 eevans@cumin1003: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2024,1031]*.wmnet: Set storage compatability to UPGRADING — T433028 - eevans@cumin1003
15:23 brennen@deploy1003: brennen, daimona: Continuing with deployment
15:23 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host cp5022.mgmt.eqsin.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
15:22 cgoubert@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rdb-lock2003.codfw.wmnet
15:21 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors
15:21 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors
15:21 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:21 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
15:21 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
15:16 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors
15:16 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors
15:16 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:16 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
15:16 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
15:12 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2003.codfw.wmnet
15:11 cgoubert@cumin2003: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host rdb-lock2003.codfw.wmnet
15:11 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors
15:11 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors
15:11 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:11 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
15:11 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
15:04 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2003.codfw.wmnet on all recursors
15:04 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2003.codfw.wmnet on all recursors
15:04 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
15:04 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
15:03 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb-lock1003.eqiad.wmnet with reason: host reimage
15:03 reedy@deploy1003: reedy: Continuing with deployment
15:02 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2003.codfw.wmnet - cgoubert@cumin2003"
14:47 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host rdb-lock1003.eqiad.wmnet with OS trixie
14:46 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1003.eqiad.wmnet - cgoubert@cumin2003"
14:46 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1003.eqiad.wmnet - cgoubert@cumin2003"
14:45 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock1003.eqiad.wmnet on all recursors
14:45 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock1003.eqiad.wmnet on all recursors
14:45 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:45 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1003.eqiad.wmnet - cgoubert@cumin2003"
14:45 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Enabling notifications
14:44 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1003.eqiad.wmnet - cgoubert@cumin2003"
14:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2002.codfw.wmnet - cgoubert@cumin2003"
14:14 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2002.codfw.wmnet - cgoubert@cumin2003"
14:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2002.codfw.wmnet on all recursors
14:14 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2002.codfw.wmnet on all recursors
14:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2002.codfw.wmnet - cgoubert@cumin2003"
14:14 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2002.codfw.wmnet - cgoubert@cumin2003"
14:12 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet
14:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet
14:05 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2002.codfw.wmnet
14:05 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1002.eqiad.wmnet - cgoubert@cumin2003"
14:05 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1002.eqiad.wmnet - cgoubert@cumin2003"
14:05 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock1002.eqiad.wmnet on all recursors
14:05 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock1002.eqiad.wmnet on all recursors
14:05 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:05 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1002.eqiad.wmnet - cgoubert@cumin2003"
14:03 cgoubert@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host rdb-lock2001.codfw.wmnet
14:03 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rdb-lock2001.codfw.wmnet with OS trixie
14:02 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1002.eqiad.wmnet - cgoubert@cumin2003"
14:02 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on apifeatureusage2001.codfw.wmnet with reason: host reimage
13:47 cwilliams@cumin1003: dbctl commit (dc=all): 'Configuring db1272 for s3 pooling', diff saved to https://phabricator.wikimedia.org/P96021 and previous config saved to /var/cache/conftool/dbconfig/20260812-134732-cwilliams.json
13:44 bking@cumin2003: START - Cookbook sre.hosts.reimage for host apifeatureusage2001.codfw.wmnet with OS bookworm
13:43 cgoubert@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rdb-lock2001.codfw.wmnet with reason: host reimage
13:41 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2028.codfw.wmnet
13:41 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2028.codfw.wmnet
13:41 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1018.eqiad.wmnet with reason: host reimage
13:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rdb-lock1001.eqiad.wmnet with reason: host reimage
13:36 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1018.eqiad.wmnet with reason: host reimage
13:25 fceratto@cumin1003: END (FAIL) - Cookbook sre.mysql.sanitize-wiki (exit_code=99) Managing sanitization for wikis testwiki in section s3
13:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reimage for host rdb-lock2001.codfw.wmnet with OS trixie
13:24 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2001.codfw.wmnet - cgoubert@cumin2003"
13:24 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock2001.codfw.wmnet - cgoubert@cumin2003"
13:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock2001.codfw.wmnet on all recursors
13:23 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock2001.codfw.wmnet on all recursors
13:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2001.codfw.wmnet - cgoubert@cumin2003"
13:23 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock2001.codfw.wmnet - cgoubert@cumin2003"
13:17 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1001.eqiad.wmnet - cgoubert@cumin2003"
13:17 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM rdb-lock1001.eqiad.wmnet - cgoubert@cumin2003"
13:17 cgoubert@cumin2003: START - Cookbook sre.ganeti.makevm for new host rdb-lock2001.codfw.wmnet
13:17 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) rdb-lock1001.eqiad.wmnet on all recursors
13:17 cgoubert@cumin2003: START - Cookbook sre.dns.wipe-cache rdb-lock1001.eqiad.wmnet on all recursors
13:17 cgoubert@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:17 cgoubert@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1001.eqiad.wmnet - cgoubert@cumin2003"
13:17 cgoubert@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM rdb-lock1001.eqiad.wmnet - cgoubert@cumin2003"
11:14 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1016.eqiad.wmnet with reason: host reimage
11:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet
11:11 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1009.eqiad.wmnet with reason: host reimage
11:10 jmm@cumin2003: END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti3005.esams.wmnet
11:10 jmm@cumin2003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host ganeti3005.esams.wmnet
11:09 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1009.eqiad.wmnet with OS bookworm
11:08 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1158: Depool db1158.eqiad.wmnet to then clone it to db1273.eqiad.wmnet - marostegui@cumin1003
11:07 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1016.eqiad.wmnet with reason: host reimage
11:07 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1158: Depool db1158.eqiad.wmnet to then clone it to db1273.eqiad.wmnet - marostegui@cumin1003
11:07 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1158.eqiad.wmnet onto db1273.eqiad.wmnet
09:14 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Turnilo import: fix regexp escaping bug - oblivian@cumin1003"
09:14 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Turnilo import: fix regexp escaping bug - oblivian@cumin1003
09:13 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Turnilo import: fix regexp escaping bug - oblivian@cumin1003
09:13 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Turnilo import: fix regexp escaping bug - oblivian@cumin1003"
09:03 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-launcher1003.eqiad.wmnet with reason: host reimage
08:58 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-launcher1003.eqiad.wmnet with reason: host reimage
08:55 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1278: Pool in x1
08:55 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1278 to dbctl T434288', diff saved to https://phabricator.wikimedia.org/P95996 and previous config saved to /var/cache/conftool/dbconfig/20260812-085521-marostegui.json
08:51 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1012.eqiad.wmnet with reason: host reimage
08:45 fceratto@cumin1003: START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis testwiki in section s3
08:43 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1009.eqiad.wmnet with OS bookworm
08:42 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1012.eqiad.wmnet with reason: host reimage
08:41 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-launcher1003.eqiad.wmnet with OS bookworm
08:40 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1013.eqiad.wmnet with OS bookworm
08:38 marostegui@cumin1003: dbctl commit (dc=all): 'Pool back ms3', diff saved to https://phabricator.wikimedia.org/P95995 and previous config saved to /var/cache/conftool/dbconfig/20260812-083816-marostegui.json
08:38 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-master1003.eqiad.wmnet with OS bookworm
08:37 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1268 to dbctl T434288', diff saved to https://phabricator.wikimedia.org/P95994 and previous config saved to /var/cache/conftool/dbconfig/20260812-083722-marostegui.json
08:35 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-web1001.eqiad.wmnet with OS bookworm
08:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2252.codfw.wmnet,db[1153,1268].eqiad.wmnet with reason: Switching over ms3
08:28 marostegui@cumin1003: dbctl commit (dc=all): 'Depool ms3 T434288', diff saved to https://phabricator.wikimedia.org/P95993 and previous config saved to /var/cache/conftool/dbconfig/20260812-082852-marostegui.json
08:25 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1012.eqiad.wmnet with OS bookworm
08:25 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1011.eqiad.wmnet with OS bookworm
08:13 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-master1003.eqiad.wmnet with reason: host reimage
08:07 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-master1003.eqiad.wmnet with reason: host reimage
08:04 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-web1001.eqiad.wmnet with reason: host reimage
07:58 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-web1001.eqiad.wmnet with reason: host reimage
07:50 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-master1003.eqiad.wmnet with OS bookworm
07:38 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1011.eqiad.wmnet with reason: host reimage
07:38 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-master1003.eqiad.wmnet with OS bookworm
07:35 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3007.esams.wmnet
07:35 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3007.esams.wmnet
07:33 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1011.eqiad.wmnet with reason: host reimage
07:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti3007.esams.wmnet
20:42 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage
19:59 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage
19:54 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-ctrl2006.codfw.wmnet with reason: host reimage
19:04 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns6001.wikimedia.org
19:04 sukhe@cumin1003: START - Cookbook sre.hosts.remove-downtime for dns6001.wikimedia.org
19:04 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns5003.wikimedia.org
19:04 sukhe@cumin1003: START - Cookbook sre.hosts.remove-downtime for dns5003.wikimedia.org
19:03 brennen@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.15 refs T430834
18:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns5003.wikimedia.org with OS trixie
18:55 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns6001.wikimedia.org with OS trixie
18:19 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp7009.magru.wmnet} and A:cp - 9.2.15 Upgrade ()
18:14 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp7009.magru.wmnet} and A:cp - 9.2.15 Upgrade ()
18:13 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.15 refs T430834
18:12 brett@cumin2002: END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp7001.magru.wmnet} and A:cp - 9.2.15 Upgrade ()
18:09 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns5003.wikimedia.org with reason: host reimage
18:06 brett@cumin2002: START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp7001.magru.wmnet} and A:cp - 9.2.15 Upgrade ()
18:06 brennen: 1.47.0-wmf.15 train status (T430834) - no current blockers, rolling to group0
18:05 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dns5003.wikimedia.org with reason: host reimage
18:05 brett: import trafficserver-9.2.15~deb13+wmf1 into trixie-wikimedia (T434478)
18:01 ladsgroup@cumin1003: END (PASS) - Cookbook sre.mysql.sanitarium_restart (exit_code=0)
17:58 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns6001.wikimedia.org with reason: host reimage
16:01 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:55 cwilliams@cumin1003: START - Cookbook sre.mysql.pool pool db1157: Pool db1157.eqiad.wmnet in after cloning
15:54 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
15:54 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-ctrl2006.codfw.wmnet with OS trixie
15:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti7004.magru.wmnet
15:09 cwilliams@cumin1003: dbctl commit (dc=all): 'Promote db2207 to s2 primary T434565', diff saved to https://phabricator.wikimedia.org/P95983 and previous config saved to /var/cache/conftool/dbconfig/20260811-150905-cwilliams.json
15:08 cezmunsta: Starting s2 codfw failover from db2204 to db2207 - T434565
15:04 cwilliams@cumin1003: dbctl commit (dc=all): 'Set db2207 with weight 0 T434565', diff saved to https://phabricator.wikimedia.org/P95982 and previous config saved to /var/cache/conftool/dbconfig/20260811-150402-cwilliams.json
15:02 cwilliams@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s2 T434565
14:55 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1010.eqiad.wmnet with OS bookworm
14:49 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns4003.wikimedia.org with OS trixie
14:47 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1010.eqiad.wmnet with OS bookworm
14:47 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7001.wikimedia.org with OS trixie
14:44 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-presto1010.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
14:41 btullis@cumin1003: START - Cookbook sre.hosts.provision for host an-presto1010.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
14:40 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-presto1010.eqiad.wmnet with OS bookworm
14:39 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1009.eqiad.wmnet with OS bookworm
14:39 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-presto1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
14:36 btullis@cumin1003: START - Cookbook sre.hosts.provision for host an-presto1009.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
14:32 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1009.eqiad.wmnet with OS bookworm
14:32 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-master1004.eqiad.wmnet with OS bookworm
14:31 cwilliams@cumin1003: START - Cookbook sre.mysql.clone of db1157.eqiad.wmnet onto db1272.eqiad.wmnet
14:28 moritzm: failover ganeti master in ulsfo to ganeti4005
14:23 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7003.magru.wmnet
14:23 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7003.magru.wmnet
14:23 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-master1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
14:22 btullis@cumin1003: START - Cookbook sre.hosts.provision for host an-master1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
14:21 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-master1004.eqiad.wmnet with OS bookworm
14:19 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti4007.ulsfo.wmnet
14:19 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti4007.ulsfo.wmnet
14:18 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1010.eqiad.wmnet with OS bookworm
14:17 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1008.eqiad.wmnet with OS bookworm
14:15 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ganeti7003.magru.wmnet
13:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repool ms2 T434288', diff saved to https://phabricator.wikimedia.org/P95980 and previous config saved to /var/cache/conftool/dbconfig/20260811-130725-marostegui.json
13:06 urbanecm@deploy1003: urbanecm, superpes: Continuing with deployment
13:06 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1266 to dbctl T434288', diff saved to https://phabricator.wikimedia.org/P95979 and previous config saved to /var/cache/conftool/dbconfig/20260811-130627-marostegui.json
12:59 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2253.codfw.wmnet,db[1151,1266].eqiad.wmnet with reason: Switching over ms2
12:54 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1157: Using as clone source
12:53 cwilliams@cumin1003: START - Cookbook sre.mysql.depool depool db1157: Using as clone source
12:51 marostegui@cumin1003: dbctl commit (dc=all): 'Depool ms2 T434288', diff saved to https://phabricator.wikimedia.org/P95977 and previous config saved to /var/cache/conftool/dbconfig/20260811-125129-marostegui.json
12:47 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
12:46 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
12:45 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
12:44 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
12:44 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
12:43 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
12:43 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-coord1003.eqiad.wmnet with reason: host reimage
12:43 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' .
12:42 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'ores-legacy' for release 'main' .
12:42 cwilliams@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2165: Security update
12:40 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-coord1003.eqiad.wmnet with reason: host reimage
12:39 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'ores-legacy' for release 'main' .
12:38 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' .
12:38 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' .
12:37 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' .
11:28 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-presto1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
11:26 btullis@cumin1003: START - Cookbook sre.hosts.provision for host an-presto1008.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
11:23 btullis@dns1004: END - running authdns-update
11:20 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-presto1008.eqiad.wmnet with OS bookworm
11:20 moritzm: installing curl security updates
11:11 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1007.eqiad.wmnet with OS bookworm
10:45 tappof: bump space for prometheus k8s-dse in codfw
10:43 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1007.eqiad.wmnet with reason: host reimage
10:38 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1007.eqiad.wmnet with reason: host reimage
10:37 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-coord1004.eqiad.wmnet with OS bookworm
10:35 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1008.eqiad.wmnet with OS bookworm
10:34 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1179: Depool db1179.eqiad.wmnet to then clone it to db1278.eqiad.wmnet - marostegui@cumin1003
10:34 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1008.eqiad.wmnet with OS bookworm
10:25 fceratto@cumin1003: dbctl commit (dc=all): 'Remove db1177 T433474', diff saved to https://phabricator.wikimedia.org/P95964 and previous config saved to /var/cache/conftool/dbconfig/20260811-102527-fceratto.json
10:22 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1008.eqiad.wmnet with OS bookworm
10:22 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-presto1007.eqiad.wmnet with OS bookworm
10:21 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-presto1006.eqiad.wmnet with OS bookworm
10:18 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1179: Depool db1179.eqiad.wmnet to then clone it to db1278.eqiad.wmnet - marostegui@cumin1003
10:18 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1179.eqiad.wmnet onto db1278.eqiad.wmnet
10:09 blake@deploy1003: Stopping before sync operations
10:09 blake@deploy1003: Started scap sync-world: Non-deployment run to populate release values for T427668
10:04 fceratto@cumin1003: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
10:04 fceratto@cumin1003: Removing db1177 from zarcillo T433474
10:04 fceratto@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1177.eqiad.wmnet
10:04 fceratto@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:04 fceratto@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1177.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
10:03 fceratto@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1177.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - fceratto@cumin1003"
10:00 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1006.eqiad.wmnet with reason: host reimage
09:58 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-coord1004.eqiad.wmnet with reason: host reimage
09:57 marostegui: Failover m1 from db1164 to db1213 - T434493
09:57 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1006.eqiad.wmnet with reason: host reimage
08:48 marostegui: Switchover ms1 master in eqiad T434288
08:48 marostegui@cumin1003: dbctl commit (dc=all): 'Repool ms1 T434288', diff saved to https://phabricator.wikimedia.org/P95962 and previous config saved to /var/cache/conftool/dbconfig/20260811-084804-marostegui.json
08:40 marostegui@cumin1003: dbctl commit (dc=all): 'Add db1267 to dbctl T434288', diff saved to https://phabricator.wikimedia.org/P95961 and previous config saved to /var/cache/conftool/dbconfig/20260811-084054-marostegui.json
08:29 marostegui: Failover m1 from db1213 to db1164 - T434043
08:25 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db[2160,2232].codfw.wmnet,db[1164,1213,1217].eqiad.wmnet with reason: Primary switchover m1 T434043
08:22 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2251.codfw.wmnet,db[1152,1267].eqiad.wmnet with reason: Switching over ms1
08:22 marostegui@cumin1003: dbctl commit (dc=all): 'Depool ms1 T434288', diff saved to https://phabricator.wikimedia.org/P95960 and previous config saved to /var/cache/conftool/dbconfig/20260811-082201-marostegui.json
08:20 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1152: Switching over ms1
08:20 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99)
20:54 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie
20:32 andrewbogott: restarting puppetserver service on puppetserver* for T434339
19:52 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:45 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:32 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
19:25 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
18:41 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
18:35 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
12:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1156: Pool db1156.eqiad.wmnet in after cloning
11:38 bjensen: sudo -i reprepro -C main include trixie-wikimedia ${HOME}/httpbb/trixie/httpbb_${VERSION?}-1+deb13u1_amd64.changes #T434052
11:35 bjensen: sudo -i reprepro -C main include bookworm-wikimedia ${HOME}/httpbb/bookworm/httpbb_${VERSION?}-1_amd64.changes #T434052
11:30 marostegui@cumin1003: dbctl commit (dc=all): 'Adding db1271 to dbctl', diff saved to https://phabricator.wikimedia.org/P95945 and previous config saved to /var/cache/conftool/dbconfig/20260807-113006-marostegui.json
11:12 marostegui@cumin1003: START - Cookbook sre.mysql.pool pool db1156: Pool db1156.eqiad.wmnet in after cloning
10:06 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 21 hosts with reason: cloning
10:01 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1156: Depool db1156.eqiad.wmnet to then clone it to db1271.eqiad.wmnet - marostegui@cumin1003
09:59 marostegui@cumin1003: START - Cookbook sre.mysql.depool depool db1156: Depool db1156.eqiad.wmnet to then clone it to db1271.eqiad.wmnet - marostegui@cumin1003
09:59 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db1156.eqiad.wmnet onto db1271.eqiad.wmnet
09:15 jynus: started stress testing db1245 dbs T431115
08:19 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
08:18 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
08:16 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
08:14 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
08:13 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
08:10 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
08:06 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
08:05 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
08:00 klausman@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime to get full picture of current BIOS settings beyond what Redfish shows
08:00 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
07:54 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
07:54 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
07:53 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
07:52 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
07:51 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
07:50 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
07:49 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
07:48 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
07:47 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
07:45 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
07:45 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
07:41 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
07:38 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
07:37 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
06:35 jayme: updated istio to 1.29.4 on wikikube eqiad - T427401
06:08 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1178.eqiad.wmnet
06:08 marostegui@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
06:08 marostegui@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1178.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003"
06:06 marostegui@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1178.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003"
18:19 ladsgroup@deploy1003: Started scap sync-world: Deploying gerrit:1321603 (T107188)
18:18 ladsgroup@deploy1003: Stopping before sync operations
18:17 ladsgroup@deploy1003: Started scap sync-world: Deploying gerrit:1321603 (T107188)
17:55 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie
16:50 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-worker1002.eqiad.wmnet with OS bookworm
16:35 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-test-worker1001.eqiad.wmnet with OS bookworm
16:19 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-worker1002.eqiad.wmnet with reason: host reimage
16:16 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-worker1002.eqiad.wmnet with reason: host reimage
16:05 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-worker1001.eqiad.wmnet with reason: host reimage
16:00 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-worker1001.eqiad.wmnet with reason: host reimage
15:57 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1002.eqiad.wmnet with OS bookworm
15:43 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1001.eqiad.wmnet with OS bookworm
15:29 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-test-worker1001.eqiad.wmnet with OS bookworm
15:29 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-test-worker1002.eqiad.wmnet with OS bookworm
14:58 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1001.eqiad.wmnet with OS bookworm
14:40 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-test-worker1002.eqiad.wmnet with OS bookworm
14:38 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-ui1001.eqiad.wmnet with reason: host reimage
14:34 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-presto1001.eqiad.wmnet with reason: host reimage
14:28 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-ui1001.eqiad.wmnet with reason: host reimage
14:27 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-presto1001.eqiad.wmnet with reason: host reimage
13:49 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
13:44 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
13:31 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
13:29 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
13:26 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
13:23 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-client1002.eqiad.wmnet with reason: host reimage
13:22 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
13:19 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
13:18 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-client1002.eqiad.wmnet with reason: host reimage
13:18 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
13:17 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
13:16 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
13:13 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
13:11 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
13:09 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
13:06 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
13:06 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-client1002.eqiad.wmnet with OS bookworm
13:05 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
13:05 aikochou@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
13:05 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' .
13:04 brouberol@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-test-client1002.eqiad.wmnet with OS bookworm
13:04 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
13:03 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
13:02 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
13:00 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'readability' for release 'main' .
12:59 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'readability' for release 'main' .
12:58 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-client1002.eqiad.wmnet with reason: host reimage
12:57 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'logo-detection' for release 'main' .
12:57 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'logo-detection' for release 'main' .
12:57 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
12:55 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'llm' for release 'main' .
12:54 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
12:53 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-client1002.eqiad.wmnet with reason: host reimage
12:53 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' .
12:51 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' .
12:50 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
12:48 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
12:46 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
12:45 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-models' for release 'main' .
12:41 bwojtowicz@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
12:39 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
12:38 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-client1002.eqiad.wmnet with OS bookworm
09:39 marostegui@cumin1003: dbctl commit (dc=all): 'Pool back ms2', diff saved to https://phabricator.wikimedia.org/P95929 and previous config saved to /var/cache/conftool/dbconfig/20260806-093908-marostegui.json
09:36 marostegui@cumin1003: dbctl commit (dc=all): 'Remove db1178 from dbctl T433471', diff saved to https://phabricator.wikimedia.org/P95928 and previous config saved to /var/cache/conftool/dbconfig/20260806-093632-marostegui.json
09:10 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
09:09 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
09:07 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup2003.codfw.wmnet
09:07 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:07 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: backup2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
09:06 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: backup2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
08:57 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts backup2003.codfw.wmnet
08:55 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts backup1003.eqiad.wmnet
08:55 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
08:55 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: backup1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
08:54 marostegui@cumin1003: dbctl commit (dc=all): 'Pool back ms3', diff saved to https://phabricator.wikimedia.org/P95925 and previous config saved to /var/cache/conftool/dbconfig/20260806-085422-marostegui.json
08:53 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: backup1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
2026-08-05
23:24 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf1009.eqiad.wmnet with OS bookworm
23:03 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf1009.eqiad.wmnet with reason: host reimage
22:59 swfrench@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on conf1009.eqiad.wmnet with reason: host reimage
22:43 swfrench@cumin2002: START - Cookbook sre.hosts.reimage for host conf1009.eqiad.wmnet with OS bookworm
22:38 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf1009.eqiad.wmnet
22:34 swfrench@cumin2002: START - Cookbook sre.hosts.reboot-single for host conf1009.eqiad.wmnet
22:25 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host conf1008.eqiad.wmnet with OS bookworm
22:04 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on conf1008.eqiad.wmnet with reason: host reimage
22:00 swfrench@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on conf1008.eqiad.wmnet with reason: host reimage
21:48 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:47 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:46 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:44 swfrench@cumin2002: START - Cookbook sre.hosts.reimage for host conf1008.eqiad.wmnet with OS bookworm
21:41 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf1008.eqiad.wmnet
21:36 swfrench@cumin2002: START - Cookbook sre.hosts.reboot-single for host conf1008.eqiad.wmnet
21:14 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:08 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
21:07 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
21:03 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host zuul1006
21:02 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host zuul1006
21:00 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
20:56 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
20:56 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
20:55 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1006] - vriley@cumin1003"
19:46 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1006.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
19:30 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1007.eqiad.wmnet with OS trixie
19:30 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
19:13 swfrench@cumin2002: START - Cookbook sre.hosts.reimage for host conf1007.eqiad.wmnet with OS bookworm
19:12 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage
19:09 swfrench@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host conf1007.eqiad.wmnet
19:07 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage
19:03 swfrench@cumin2002: START - Cookbook sre.hosts.reboot-single for host conf1007.eqiad.wmnet
18:52 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host zuul1007.eqiad.wmnet with OS trixie
18:52 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host zuul1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
18:35 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
18:34 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
18:34 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
18:30 vriley@cumin1003: START - Cookbook sre.hosts.provision for host zuul1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
18:28 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
18:28 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt [zuul1007] - vriley@cumin1003"
18:22 vriley@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
18:22 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp5022.mgmt.eqsin.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
18:13 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp5022.mgmt.eqsin.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
17:31 jasmine@cumin2002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
17:12 mutante: LDAP - added vwalters to group ciadmin - T433615
15:54 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1159.eqiad.wmnet with OS trixie
15:38 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply logging and security config updates - bking@cumin2003 - T324335
15:33 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1159.eqiad.wmnet with reason: host reimage
15:32 bking@cumin2003: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: apply logging and security config updates - bking@cumin2003 - T324335
15:27 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1159.eqiad.wmnet with reason: host reimage
15:10 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1159
15:10 jayme@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1159
15:00 swfrench-wmf: begin rolling restart of confd in codfw, eqsin, ulsfo for hosts in the wikimedia.org domain - T428495
14:55 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2046.codfw.wmnet with OS trixie
14:54 swfrench-wmf: restarted navtiming on webperf1003 - T428495
14:52 jayme@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1159
14:52 jayme@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1159.eqiad.wmnet 129.48.64.10.in-addr.arpa 9.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
14:52 jayme@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1159.eqiad.wmnet 129.48.64.10.in-addr.arpa 9.2.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
14:52 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:52 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1159 - jayme@cumin1003"
14:52 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1159 - jayme@cumin1003"
14:15 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-worker1003.eqiad.wmnet with reason: host reimage
13:58 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1157.eqiad.wmnet with reason: host reimage
13:57 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on search-loader1002.eqiad.wmnet with reason: host reimage
13:54 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-worker1003.eqiad.wmnet with OS bookworm
13:46 bking@cumin2003: START - Cookbook sre.hosts.reimage for host search-loader1002.eqiad.wmnet with OS trixie
13:43 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1157
13:42 jayme@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1157
13:40 jayme@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1157
13:40 jayme@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1157.eqiad.wmnet 183.32.64.10.in-addr.arpa 3.8.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
13:40 jayme@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1157.eqiad.wmnet 183.32.64.10.in-addr.arpa 3.8.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
13:40 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
13:40 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1157 - jayme@cumin1003"
13:39 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1157 - jayme@cumin1003"
13:23 jayme@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1157
13:22 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1157.eqiad.wmnet with OS trixie
13:22 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1157.eqiad.wmnet
13:22 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1157.eqiad.wmnet
13:21 jayme@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1157.eqiad.wmnet
13:15 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1156.eqiad.wmnet
13:15 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1156.eqiad.wmnet
13:15 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1156.eqiad.wmnet
13:01 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1156.eqiad.wmnet with OS trixie
12:42 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1156.eqiad.wmnet with reason: host reimage
12:38 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1156.eqiad.wmnet with reason: host reimage
12:32 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
12:31 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
12:30 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
12:28 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
12:26 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
12:24 topranks: update bgp confed settings in eqsin
12:22 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1156
12:22 jayme@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1156
12:22 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
12:19 jayme@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1156
12:19 jayme@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1156.eqiad.wmnet 110.32.64.10.in-addr.arpa 0.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
12:19 jayme@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1156.eqiad.wmnet 110.32.64.10.in-addr.arpa 0.1.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
12:19 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
12:19 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1156 - jayme@cumin1003"
12:19 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1156 - jayme@cumin1003"
12:17 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
12:14 jiji@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2046.codfw.wmnet with OS trixie
12:09 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
12:06 bwojtowicz@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' .
10:04 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1155.eqiad.wmnet with OS trixie
09:52 marostegui@cumin1003: dbctl commit (dc=all): 'Pool back ms1', diff saved to https://phabricator.wikimedia.org/P95918 and previous config saved to /var/cache/conftool/dbconfig/20260805-095212-marostegui.json
09:44 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1152: after cloning
09:44 marostegui@cumin1003: END (FAIL) - Cookbook sre.mysql.parsercache (exit_code=99)
09:14 jayme@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1155
09:14 jayme@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1155.eqiad.wmnet 109.32.64.10.in-addr.arpa 9.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
09:14 jayme@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1155.eqiad.wmnet 109.32.64.10.in-addr.arpa 9.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
09:14 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
09:14 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1155 - jayme@cumin1003"
09:14 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1155 - jayme@cumin1003"
09:10 fceratto@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=1)
08:38 jayme@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1155
08:37 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1155.eqiad.wmnet with OS trixie
08:29 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1171.eqiad.wmnet
08:29 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
08:29 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1171.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
08:29 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 (T433990)', diff saved to https://phabricator.wikimedia.org/P95913 and previous config saved to /var/cache/conftool/dbconfig/20260805-082908-ladsgroup.json
08:27 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1171.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
08:18 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P95912 and previous config saved to /var/cache/conftool/dbconfig/20260805-081823-ladsgroup.json
08:17 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts db1171.eqiad.wmnet
08:17 jynus@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1150.eqiad.wmnet
08:17 jynus@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
08:17 jynus@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1150.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
08:15 jynus@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1150.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
08:15 jnuche@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.14 refs T430833
08:14 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1155.eqiad.wmnet
08:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1155.eqiad.wmnet
08:14 jayme@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1155.eqiad.wmnet
08:07 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249', diff saved to https://phabricator.wikimedia.org/P95911 and previous config saved to /var/cache/conftool/dbconfig/20260805-080737-ladsgroup.json
08:05 jynus@cumin1003: START - Cookbook sre.hosts.decommission for hosts db1150.eqiad.wmnet
07:56 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2249 (T433990)', diff saved to https://phabricator.wikimedia.org/P95910 and previous config saved to /var/cache/conftool/dbconfig/20260805-075650-ladsgroup.json
07:25 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2249 (T433990)', diff saved to https://phabricator.wikimedia.org/P95909 and previous config saved to /var/cache/conftool/dbconfig/20260805-072529-ladsgroup.json
07:24 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2249.codfw.wmnet with reason: Maintenance
07:24 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 (T433990)', diff saved to https://phabricator.wikimedia.org/P95908 and previous config saved to /var/cache/conftool/dbconfig/20260805-072426-ladsgroup.json
07:21 slyngshede@dns1004: END - running authdns-update
07:13 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P95906 and previous config saved to /var/cache/conftool/dbconfig/20260805-071340-ladsgroup.json
07:02 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231', diff saved to https://phabricator.wikimedia.org/P95905 and previous config saved to /var/cache/conftool/dbconfig/20260805-070253-ladsgroup.json
06:52 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2231 (T433990)', diff saved to https://phabricator.wikimedia.org/P95904 and previous config saved to /var/cache/conftool/dbconfig/20260805-065206-ladsgroup.json
06:45 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
06:22 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2231 (T433990)', diff saved to https://phabricator.wikimedia.org/P95903 and previous config saved to /var/cache/conftool/dbconfig/20260805-062240-ladsgroup.json
06:21 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2231.codfw.wmnet with reason: Maintenance
06:21 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 (T433990)', diff saved to https://phabricator.wikimedia.org/P95902 and previous config saved to /var/cache/conftool/dbconfig/20260805-062137-ladsgroup.json
06:10 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P95901 and previous config saved to /var/cache/conftool/dbconfig/20260805-061051-ladsgroup.json
06:00 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215', diff saved to https://phabricator.wikimedia.org/P95900 and previous config saved to /var/cache/conftool/dbconfig/20260805-060004-ladsgroup.json
05:49 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2215 (T433990)', diff saved to https://phabricator.wikimedia.org/P95899 and previous config saved to /var/cache/conftool/dbconfig/20260805-054918-ladsgroup.json
05:19 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2215 (T433990)', diff saved to https://phabricator.wikimedia.org/P95898 and previous config saved to /var/cache/conftool/dbconfig/20260805-051939-ladsgroup.json
05:18 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2215.codfw.wmnet with reason: Maintenance
04:30 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2201.codfw.wmnet with reason: Maintenance
03:40 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance
03:40 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 (T433990)', diff saved to https://phabricator.wikimedia.org/P95897 and previous config saved to /var/cache/conftool/dbconfig/20260805-034036-ladsgroup.json
03:29 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P95896 and previous config saved to /var/cache/conftool/dbconfig/20260805-032948-ladsgroup.json
03:19 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196', diff saved to https://phabricator.wikimedia.org/P95895 and previous config saved to /var/cache/conftool/dbconfig/20260805-031902-ladsgroup.json
03:08 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2196 (T433990)', diff saved to https://phabricator.wikimedia.org/P95894 and previous config saved to /var/cache/conftool/dbconfig/20260805-030815-ladsgroup.json
02:34 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2196 (T433990)', diff saved to https://phabricator.wikimedia.org/P95893 and previous config saved to /var/cache/conftool/dbconfig/20260805-023413-ladsgroup.json
02:33 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2196.codfw.wmnet with reason: Maintenance
02:33 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 (T433990)', diff saved to https://phabricator.wikimedia.org/P95892 and previous config saved to /var/cache/conftool/dbconfig/20260805-023310-ladsgroup.json
02:22 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P95891 and previous config saved to /var/cache/conftool/dbconfig/20260805-022223-ladsgroup.json
02:11 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186', diff saved to https://phabricator.wikimedia.org/P95890 and previous config saved to /var/cache/conftool/dbconfig/20260805-021137-ladsgroup.json
02:01 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
02:00 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2186 (T433990)', diff saved to https://phabricator.wikimedia.org/P95889 and previous config saved to /var/cache/conftool/dbconfig/20260805-020051-ladsgroup.json
01:30 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2186 (T433990)', diff saved to https://phabricator.wikimedia.org/P95888 and previous config saved to /var/cache/conftool/dbconfig/20260805-013029-ladsgroup.json
01:29 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2186.codfw.wmnet with reason: Maintenance
00:34 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
00:34 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 (T433990)', diff saved to https://phabricator.wikimedia.org/P95887 and previous config saved to /var/cache/conftool/dbconfig/20260805-003408-ladsgroup.json
00:23 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P95886 and previous config saved to /var/cache/conftool/dbconfig/20260805-002322-ladsgroup.json
00:12 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264', diff saved to https://phabricator.wikimedia.org/P95885 and previous config saved to /var/cache/conftool/dbconfig/20260805-001235-ladsgroup.json
00:01 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1264 (T433990)', diff saved to https://phabricator.wikimedia.org/P95884 and previous config saved to /var/cache/conftool/dbconfig/20260805-000148-ladsgroup.json
2026-08-04
23:45 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db1264 (T433990)', diff saved to https://phabricator.wikimedia.org/P95883 and previous config saved to /var/cache/conftool/dbconfig/20260804-234508-ladsgroup.json
23:44 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1264.eqiad.wmnet with reason: Maintenance
23:44 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1237 (T433990)', diff saved to https://phabricator.wikimedia.org/P95882 and previous config saved to /var/cache/conftool/dbconfig/20260804-234405-ladsgroup.json
23:33 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1237', diff saved to https://phabricator.wikimedia.org/P95881 and previous config saved to /var/cache/conftool/dbconfig/20260804-233317-ladsgroup.json
23:22 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1237', diff saved to https://phabricator.wikimedia.org/P95880 and previous config saved to /var/cache/conftool/dbconfig/20260804-232230-ladsgroup.json
23:11 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1237 (T433990)', diff saved to https://phabricator.wikimedia.org/P95879 and previous config saved to /var/cache/conftool/dbconfig/20260804-231144-ladsgroup.json
22:23 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db1237 (T433990)', diff saved to https://phabricator.wikimedia.org/P95878 and previous config saved to /var/cache/conftool/dbconfig/20260804-222345-ladsgroup.json
22:23 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1237.eqiad.wmnet with reason: Maintenance
21:13 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance
20:13 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2046.codfw.wmnet with reason: host reimage
20:09 jhancock@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on mc2046.codfw.wmnet with reason: host reimage
20:00 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1216.eqiad.wmnet with reason: Maintenance
20:00 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T433990)', diff saved to https://phabricator.wikimedia.org/P95877 and previous config saved to /var/cache/conftool/dbconfig/20260804-195957-ladsgroup.json
19:51 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host mc2046
19:50 jhancock@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2046
19:50 jhancock@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host mc2046
19:50 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mc2046.codfw.wmnet 120.16.192.10.in-addr.arpa 0.2.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
19:50 jhancock@cumin2002: START - Cookbook sre.dns.wipe-cache mc2046.codfw.wmnet 120.16.192.10.in-addr.arpa 0.2.1.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
19:50 jhancock@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
19:50 jhancock@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host mc2046 - jhancock@cumin2002"
19:50 jhancock@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host mc2046 - jhancock@cumin2002"
19:49 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P95876 and previous config saved to /var/cache/conftool/dbconfig/20260804-194911-ladsgroup.json
19:45 jhancock@cumin2002: START - Cookbook sre.hosts.move-vlan for host mc2046
19:45 jhancock@cumin2002: START - Cookbook sre.hosts.reimage for host mc2046.codfw.wmnet with OS bookworm
19:38 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P95875 and previous config saved to /var/cache/conftool/dbconfig/20260804-193825-ladsgroup.json
19:27 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T433990)', diff saved to https://phabricator.wikimedia.org/P95874 and previous config saved to /var/cache/conftool/dbconfig/20260804-192738-ladsgroup.json
18:06 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1203.eqiad.wmnet with reason: Maintenance
18:06 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T433990)', diff saved to https://phabricator.wikimedia.org/P95871 and previous config saved to /var/cache/conftool/dbconfig/20260804-180618-ladsgroup.json
17:55 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P95870 and previous config saved to /var/cache/conftool/dbconfig/20260804-175531-ladsgroup.json
17:55 jayme@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=1) Renumbering for host wikikube-worker1154.eqiad.wmnet
17:55 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1154.eqiad.wmnet
17:55 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1154.eqiad.wmnet
17:48 swfrench@deploy1003: swfrench: Continuing with deployment
17:46 swfrench@deploy1003: swfrench: Deploy helmfile-only msmtp timeout override cleanup - T383047 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
17:44 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179', diff saved to https://phabricator.wikimedia.org/P95869 and previous config saved to /var/cache/conftool/dbconfig/20260804-174445-ladsgroup.json
17:34 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T433990)', diff saved to https://phabricator.wikimedia.org/P95868 and previous config saved to /var/cache/conftool/dbconfig/20260804-173359-ladsgroup.json
17:33 swfrench@deploy1003: Finished scap sync-world: Pick up new PHP production image (duration: 28m 32s)
17:28 aokoth@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on phab1005.eqiad.wmnet with reason: Puppet Failure
17:05 swfrench@deploy1003: Started scap sync-world: Pick up new PHP production image
17:00 cwilliams@cumin1003: END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97)
16:25 dzahn@cumin1003: START - Cookbook sre.gerrit.restart-gerrit Restarting Gerrit on gerrit2003
16:24 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1179 (T433990)', diff saved to https://phabricator.wikimedia.org/P95866 and previous config saved to /var/cache/conftool/dbconfig/20260804-162424-ladsgroup.json
14:34 jayme@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1154
14:34 jayme@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1154.eqiad.wmnet 108.32.64.10.in-addr.arpa 8.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
14:34 jayme@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1154.eqiad.wmnet 108.32.64.10.in-addr.arpa 8.0.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
14:34 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
14:34 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1154 - jayme@cumin1003"
14:34 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1154 - jayme@cumin1003"
14:30 otto@deploy1003: otto: Continuing with deployment
12:49 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1096.eqiad.wmnet with reason: host reimage
12:43 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 1:00:00 on db1171.eqiad.wmnet with reason: decom
12:42 jynus@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 1:00:00 on db1150.eqiad.wmnet with reason: decom
12:42 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1096.eqiad.wmnet with reason: host reimage
12:38 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[1164,1217].eqiad.wmnet with reason: cloning
12:33 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2096.codfw.wmnet with OS trixie
12:22 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1096.eqiad.wmnet with OS trixie
12:14 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2096.codfw.wmnet with reason: host reimage
12:14 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1141.eqiad.wmnet with OS trixie
12:10 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2096.codfw.wmnet with reason: host reimage
12:10 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1289.eqiad.wmnet
12:05 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1289.eqiad.wmnet
12:05 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1288.eqiad.wmnet
11:59 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1288.eqiad.wmnet
11:59 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1287.eqiad.wmnet
11:54 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1097.eqiad.wmnet with OS trixie
11:54 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1287.eqiad.wmnet
11:54 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1286.eqiad.wmnet
11:53 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1141.eqiad.wmnet with reason: host reimage
11:51 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2096.codfw.wmnet with OS trixie
11:49 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1141.eqiad.wmnet with reason: host reimage
11:48 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1286.eqiad.wmnet
11:48 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1284.eqiad.wmnet
11:43 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2095.codfw.wmnet with OS trixie
11:43 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1284.eqiad.wmnet
11:43 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1283.eqiad.wmnet
11:42 klausman@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime to get full picture of current BIOS settings beyond what Redfish shows
11:39 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lswtest-d8-eqiad
11:39 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lswtest-d8-eqiad
11:37 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1283.eqiad.wmnet
11:37 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1282.eqiad.wmnet
11:37 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lswtest-d8-eqiad
11:37 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lswtest-d8-eqiad
11:33 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1141
11:33 jayme@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1141
11:32 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2095.codfw.wmnet with reason: host reimage
11:32 jayme@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1141
11:32 jayme@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1141.eqiad.wmnet 156.48.64.10.in-addr.arpa 6.5.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
11:32 jayme@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1141.eqiad.wmnet 156.48.64.10.in-addr.arpa 6.5.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
11:32 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
11:32 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1141 - jayme@cumin1003"
11:32 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1141 - jayme@cumin1003"
11:32 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1282.eqiad.wmnet
11:32 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1281.eqiad.wmnet
11:32 cmooney@cumin1003: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lswtest-d8-eqiad
11:32 cmooney@cumin1003: START - Cookbook sre.network.tls for network device lswtest-d8-eqiad
11:29 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2095.codfw.wmnet with reason: host reimage
11:29 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1097.eqiad.wmnet with reason: host reimage
11:26 mvernon@cumin2003: START - Cookbook sre.hosts.reimage for host ms-be2095.codfw.wmnet with OS trixie
11:26 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1281.eqiad.wmnet
11:26 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1280.eqiad.wmnet
11:24 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1097.eqiad.wmnet with reason: host reimage
11:22 jayme@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1141
11:21 jayme@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1141.eqiad.wmnet with OS trixie
11:20 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1280.eqiad.wmnet
11:20 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1279.eqiad.wmnet
11:20 cmooney@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mr1-eqsin with reason: upgrade new Nokia swtiches in eqsin to SR Linux v26
11:17 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1141.eqiad.wmnet
11:16 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1141.eqiad.wmnet
11:16 jayme@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1141.eqiad.wmnet
11:16 mvernon@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host ms-be2095.codfw.wmnet with OS trixie
11:15 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1279.eqiad.wmnet
11:15 cwilliams@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1278.eqiad.wmnet
11:14 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1139.eqiad.wmnet
11:14 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1139.eqiad.wmnet
11:13 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1070.eqiad.wmnet with OS trixie
11:13 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1140.eqiad.wmnet
11:13 jayme@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1140.eqiad.wmnet
11:13 jayme@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1140.eqiad.wmnet
11:09 cwilliams@cumin1003: START - Cookbook sre.hosts.reboot-single for host db1278.eqiad.wmnet
11:06 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1071.eqiad.wmnet with OS trixie
11:05 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1097.eqiad.wmnet with OS trixie
11:04 mvernon@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be1097.eqiad.wmnet with OS trixie
11:02 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1140.eqiad.wmnet with OS trixie
11:02 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1097.eqiad.wmnet with OS trixie
11:01 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1096.eqiad.wmnet with OS trixie
11:00 jayme@cumin1003: END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host wikikube-worker1139.eqiad.wmnet
11:00 jayme@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host wikikube-worker1139.eqiad.wmnet with OS trixie
10:56 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1069.eqiad.wmnet with OS trixie
10:56 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2095.codfw.wmnet with reason: host reimage
10:52 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1070.eqiad.wmnet with reason: host reimage
10:48 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1096.eqiad.wmnet with reason: host reimage
10:45 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1071.eqiad.wmnet with reason: host reimage
10:44 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1096.eqiad.wmnet with reason: host reimage
10:41 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1096.eqiad.wmnet with OS trixie
10:41 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1140.eqiad.wmnet with reason: host reimage
10:39 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1071.eqiad.wmnet with reason: host reimage
10:38 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1070.eqiad.wmnet with reason: host reimage
10:38 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1139.eqiad.wmnet with reason: host reimage
10:37 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1140.eqiad.wmnet with reason: host reimage
10:35 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1069.eqiad.wmnet with reason: host reimage
10:33 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1095.eqiad.wmnet with OS trixie
10:33 mvernon@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2095.codfw.wmnet with reason: host reimage
10:33 jayme@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1139.eqiad.wmnet with reason: host reimage
10:30 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1069.eqiad.wmnet with reason: host reimage
10:23 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1140
10:23 jayme@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1140
10:23 mvernon@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1095.eqiad.wmnet with reason: host reimage
10:22 jayme@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1140
10:21 jayme@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1140.eqiad.wmnet 155.48.64.10.in-addr.arpa 5.5.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
10:21 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1071.eqiad.wmnet with OS trixie
10:21 jayme@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1140.eqiad.wmnet 155.48.64.10.in-addr.arpa 5.5.1.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
10:21 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:21 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1140 - jayme@cumin1003"
10:21 jayme@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1140 - jayme@cumin1003"
10:21 mvernon@cumin1003: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1071
10:21 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1070.eqiad.wmnet with OS trixie
10:21 mvernon@cumin1003: END (FAIL) - Cookbook sre.swift.convert-disks (exit_code=99) for host ms-be1070
10:20 mvernon@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1095.eqiad.wmnet with reason: host reimage
10:17 mvernon@cumin1003: START - Cookbook sre.hosts.reimage for host ms-be1095.eqiad.wmnet with OS trixie
10:17 jayme@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1139
10:17 jayme@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1139
10:17 mvernon@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host ms-be1095.eqiad.wmnet with OS trixie
10:15 jayme@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1139
10:15 jayme@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1139.eqiad.wmnet 194.32.64.10.in-addr.arpa 4.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
10:15 jayme@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1139.eqiad.wmnet 194.32.64.10.in-addr.arpa 4.9.1.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
10:15 jayme@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
10:15 jayme@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1139 - jayme@cumin1003"
17:55 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host codesearch2001.codfw.wmnet with OS trixie
17:54 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM codesearch2001.codfw.wmnet - dzahn@cumin2002"
17:54 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM codesearch2001.codfw.wmnet - dzahn@cumin2002"
17:53 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) codesearch2001.codfw.wmnet on all recursors
17:53 dzahn@cumin2002: START - Cookbook sre.dns.wipe-cache codesearch2001.codfw.wmnet on all recursors
17:53 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:53 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM codesearch2001.codfw.wmnet - dzahn@cumin2002"
17:48 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM codesearch2001.codfw.wmnet - dzahn@cumin2002"
17:41 dzahn@cumin2002: START - Cookbook sre.ganeti.makevm for new host codesearch2001.codfw.wmnet
17:37 dzahn@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host codesearch1001.eqiad.wmnet
17:37 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host codesearch1001.eqiad.wmnet with OS trixie
17:24 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on codesearch1001.eqiad.wmnet with reason: host reimage
17:17 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on codesearch1001.eqiad.wmnet with reason: host reimage
17:08 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host codesearch1001.eqiad.wmnet with OS trixie
17:06 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM codesearch1001.eqiad.wmnet - dzahn@cumin2002"
17:06 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM codesearch1001.eqiad.wmnet - dzahn@cumin2002"
17:06 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) codesearch1001.eqiad.wmnet on all recursors
17:06 dzahn@cumin2002: START - Cookbook sre.dns.wipe-cache codesearch1001.eqiad.wmnet on all recursors
17:06 dzahn@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
17:06 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM codesearch1001.eqiad.wmnet - dzahn@cumin2002"
16:58 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM codesearch1001.eqiad.wmnet - dzahn@cumin2002"
16:57 mvernon@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2091.codfw.wmnet with OS trixie
10:36 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2245, db2246 and db2247 T433610', diff saved to https://phabricator.wikimedia.org/P95855 and previous config saved to /var/cache/conftool/dbconfig/20260803-103652-marostegui.json
10:35 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2248 from s4 T433610', diff saved to https://phabricator.wikimedia.org/P95854 and previous config saved to /var/cache/conftool/dbconfig/20260803-103535-marostegui.json
