Server Admin Log

2025-01-13

• 10:41 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2203
• 10:41 marostegui@cumin1002: dbctl commit (dc=all): 'db1170 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71994 and previous config saved to /var/cache/conftool/dbconfig/20250113-104115-root.json
• 10:41 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2203.codfw.wmnet with OS bookworm
• 10:36 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2203.codfw.wmnet wikikube-worker2204.codfw.wmnet wikikube-worker2205.codfw.wmnet on all recursors
• 10:36 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2203.codfw.wmnet wikikube-worker2204.codfw.wmnet wikikube-worker2205.codfw.wmnet on all recursors
• 10:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc1015.eqiad.wmnet with reason: cloning
• 10:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on pc1015.eqiad.wmnet with reason: cloning
• 10:28 stran@deploy2002: helmfile [eqiad] DONE helmfile.d/services/ipoid: apply
• 10:28 stran@deploy2002: helmfile [eqiad] START helmfile.d/services/ipoid: apply
• 10:26 stran@deploy2002: helmfile [staging] DONE helmfile.d/services/ipoid: apply
• 10:26 stran@deploy2002: helmfile [staging] START helmfile.d/services/ipoid: apply
• 10:25 stran@deploy2002: helmfile [codfw] DONE helmfile.d/services/ipoid: apply
• 10:25 stran@deploy2002: helmfile [codfw] START helmfile.d/services/ipoid: apply
• 10:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2044 to wikikube-worker2205
• 10:24 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2205
• 10:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repool pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71993 and previous config saved to /var/cache/conftool/dbconfig/20250113-102343-marostegui.json
• 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'Make pc1013 master in pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71992 and previous config saved to /var/cache/conftool/dbconfig/20250113-102152-marostegui.json
• 10:20 marostegui@cumin1002: dbctl commit (dc=all): 'Remove pc1015 from pc3', diff saved to https://phabricator.wikimedia.org/P71991 and previous config saved to /var/cache/conftool/dbconfig/20250113-102047-marostegui.json
• 10:19 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2205
• 10:18 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:18 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2044 to wikikube-worker2205 - jelto@cumin1002"
• 10:18 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2044 to wikikube-worker2205 - jelto@cumin1002"
• 10:14 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 10:14 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2044 to wikikube-worker2205
• 10:14 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2043 to wikikube-worker2204
• 10:13 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2204
• 10:13 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for pc1015.eqiad.wmnet
• 10:13 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2204
• 10:13 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:13 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2043 to wikikube-worker2204 - jelto@cumin1002"
• 10:12 root@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for pc2013.codfw.wmnet
• 10:12 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2043 to wikikube-worker2204 - jelto@cumin1002"
• 10:11 marostegui@cumin1002: dbctl commit (dc=all): 'Fix weights in pc3', diff saved to https://phabricator.wikimedia.org/P71990 and previous config saved to /var/cache/conftool/dbconfig/20250113-101132-marostegui.json
• 10:10 ladsgroup@deploy2002: Finished scap sync-world: Backport for Add wikitech.wikimedia.org to list of local vhosts (T376305) (duration: 22m 28s)
• 10:08 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 10:07 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2043 to wikikube-worker2204
• 10:07 marostegui: Upgrade pc2013 pc1015 pc3 dbmaint eqiad codfw T383398
• 10:07 root@cumin1002: START - Cookbook sre.mysql.upgrade for pc1015.eqiad.wmnet
• 10:06 root@cumin1002: START - Cookbook sre.mysql.upgrade for pc2013.codfw.wmnet
• 10:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depool pc3 T383398', diff saved to https://phabricator.wikimedia.org/P71989 and previous config saved to /var/cache/conftool/dbconfig/20250113-100554-marostegui.json
• 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2042 to wikikube-worker2203
• 10:04 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2203
• 10:04 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2203
• 10:04 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:04 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2042 to wikikube-worker2203 - jelto@cumin1002"
• 10:03 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2042 to wikikube-worker2203 - jelto@cumin1002"
• 10:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc2013.codfw.wmnet with reason: cloning
• 10:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on pc2013.codfw.wmnet with reason: cloning
• 10:02 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc1013.eqiad.wmnet with reason: cloning
• 10:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on pc1013.eqiad.wmnet with reason: cloning
• 10:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on pc1015.eqiad.wmnet with reason: cloning
• 10:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on pc1015.eqiad.wmnet with reason: cloning
• 10:00 ladsgroup@deploy2002: ladsgroup: Continuing with sync
• 10:00 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 10:00 ladsgroup@deploy2002: ladsgroup: Backport for Add wikitech.wikimedia.org to list of local vhosts (T376305) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 09:59 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2042 to wikikube-worker2203
• 09:56 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and not P{cp4052.ulsfo.wmnet} and A:cp
• 09:55 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[2042-2044].codfw.wmnet
• 09:51 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[2042-2044].codfw.wmnet
• 09:49 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Reboot
• 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1023.eqiad.wmnet with reason: cloning
• 09:49 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Reboot
• 09:49 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1023.eqiad.wmnet with reason: cloning
• 09:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1023', diff saved to https://phabricator.wikimedia.org/P71988 and previous config saved to /var/cache/conftool/dbconfig/20250113-094846-marostegui.json
• 09:48 marostegui@cumin1002: dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71987 and previous config saved to /var/cache/conftool/dbconfig/20250113-094833-marostegui.json
• 09:48 ladsgroup@deploy2002: Started scap sync-world: Backport for Add wikitech.wikimedia.org to list of local vhosts (T376305)
• 09:47 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2212.codfw.wmnet
• 09:42 ladsgroup@cumin1002: START - Cookbook sre.mysql.upgrade for db2212.codfw.wmnet
• 09:41 Amir1: dbmaint on pc5@eqiad (T382948)
• 09:38 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and not P{cp4052.ulsfo.wmnet} and A:cp
• 09:36 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and not P{cp4044.ulsfo.wmnet} and A:cp
• 09:17 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and not P{cp4044.ulsfo.wmnet} and A:cp
• 08:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2135.codfw.wmnet
• 08:57 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:57 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2135.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:57 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2135.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:53 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 08:49 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2135.codfw.wmnet
• 08:34 hashar@deploy2002: Finished deploy [integration/docroot@a81d82c]: build: Updating mediawiki/mediawiki-phan-config to 0.15.1 (duration: 00m 09s)
• 08:34 hashar@deploy2002: Started deploy [integration/docroot@a81d82c]: build: Updating mediawiki/mediawiki-phan-config to 0.15.1
• 08:27 moritzm: updated netboot image for bookworm to 12.9 T383537

2025-01-12

• 09:13 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 09:12 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.

2025-01-11

• 23:02 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2212.codfw.wmnet with reason: Replication lag
• 23:02 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db2212.codfw.wmnet with reason: Replication lag
• 23:02 fabfur@cumin1002: dbctl commit (dc=all): 'Depool db2212', diff saved to https://phabricator.wikimedia.org/P71985 and previous config saved to /var/cache/conftool/dbconfig/20250111-230213-fabfur.json
• 01:39 eileen: config revision changed from fba21538 to b41ed54d
• 00:06 eileen: config revision changed from f86e46bb to b1f34373

2025-01-10

• 23:33 eileen: config revision changed from 4947f9bd to f86e46bb
• 23:09 eileen: config revision changed from 5d411dbb to 4947f9bd
• 22:59 eileen: config revision changed from 51a3e52e to 5d411dbb
• 22:49 eileen: config revision changed from cf756e5f to 51a3e52e
• 22:45 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cloudelastic1005*,cloudelastic1006* for ban hosts prior to decom - bking@cumin2002 - T380937
• 22:45 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic1005*,cloudelastic1006* for ban hosts prior to decom - bking@cumin2002 - T380937
• 22:45 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: cloudelastic1005,cloudelastic1006 for ban hosts prior to decom - bking@cumin2002 - T380937
• 22:45 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Banning hosts: cloudelastic1005,cloudelastic1006 for ban hosts prior to decom - bking@cumin2002 - T380937
• 22:21 eileen: config revision changed from 2a572b99 to cf756e5f
• 22:11 eileen: config revision changed from e0866d2f to 2a572b99
• 20:01 eevans@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002
• 19:43 eevans@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002
• 19:41 eevans@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002
• 19:26 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on eventlog1003.eqiad.wmnet with reason: Shutting down VM in preparation for decommissioning
• 19:26 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on eventlog1003.eqiad.wmnet with reason: Shutting down VM in preparation for decommissioning
• 19:23 eevans@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002
• 18:49 sukhe: sudo cumin 'P:Mediawiki::Maintenance' 'run-puppet-agent': CR 1109755
• 16:56 cmooney@dns2005: END - running authdns-update
• 16:55 cmooney@dns2005: START - running authdns-update
• 16:50 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:50 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names in newly assigned wmcs private ipv6 ranges - cmooney@cumin1002"
• 16:50 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names in newly assigned wmcs private ipv6 ranges - cmooney@cumin1002"
• 16:47 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 16:28 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:28 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names in newly assigned wmcs private ipv6 ranges - cmooney@cumin1002"
• 16:28 eevans@deploy2002: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
• 16:28 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns names in newly assigned wmcs private ipv6 ranges - cmooney@cumin1002"
• 16:27 eevans@deploy2002: helmfile [codfw] START helmfile.d/services/data-gateway: apply
• 16:26 eevans@deploy2002: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
• 16:25 eevans@deploy2002: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
• 16:23 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 16:23 cmooney@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 16:19 cmooney@cumin1002: START - Cookbook sre.dns.netbox
• 15:46 eevans@deploy2002: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
• 15:45 eevans@deploy2002: helmfile [staging] START helmfile.d/services/data-gateway: apply
• 15:36 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2199-2202].codfw.wmnet
• 15:36 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2199-2202].codfw.wmnet
• 15:34 jelto: homer 'cr*codfw*' commit 'T377877'
• 15:33 jelto: homer 'lsw1-d1-codfw*' commit 'T377877'
• 15:33 jelto: homer 'lsw1-d5-codfw*' commit 'T377877'
• 15:32 jelto: homer 'lsw1-d3-codfw*' commit 'T377877'
• 15:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2202.codfw.wmnet with OS bookworm
• 15:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2201.codfw.wmnet with OS bookworm
• 15:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2240.codfw.wmnet with reason: maintenance
• 15:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 10:00:00 on db2240.codfw.wmnet with reason: maintenance
• 15:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2240.codfw.wmnet with reason: maintenance
• 15:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db2240.codfw.wmnet with reason: maintenance
• 15:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2240 to make it candidate master', diff saved to https://phabricator.wikimedia.org/P71984 and previous config saved to /var/cache/conftool/dbconfig/20250110-152035-marostegui.json
• 15:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2202.codfw.wmnet with reason: host reimage
• 15:08 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2202.codfw.wmnet with reason: host reimage
• 15:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2201.codfw.wmnet with reason: host reimage
• 15:02 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2201.codfw.wmnet with reason: host reimage
• 14:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2202
• 14:49 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2202
• 14:48 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2202
• 14:48 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2202.codfw.wmnet 226.48.192.10.in-addr.arpa 6.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:48 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2202.codfw.wmnet 226.48.192.10.in-addr.arpa 6.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:48 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:46 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 14:46 jelto@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 14:46 jelto@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2202 - jelto@cumin1002"
• 14:42 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2201
• 14:42 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2201
• 14:42 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2201
• 14:42 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2201.codfw.wmnet 227.48.192.10.in-addr.arpa 7.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:42 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2201.codfw.wmnet 227.48.192.10.in-addr.arpa 7.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 14:42 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:42 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2201 - jelto@cumin1002"
• 14:33 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2202 - jelto@cumin1002"
• 14:29 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 14:27 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2201 - jelto@cumin1002"
• 14:26 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2202
• 14:26 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2202.codfw.wmnet with OS bookworm
• 14:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2200.codfw.wmnet with OS bookworm
• 14:24 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 14:24 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2201
• 14:23 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2201.codfw.wmnet with OS bookworm
• 14:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2199.codfw.wmnet with OS bookworm
• 14:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2200.codfw.wmnet with reason: host reimage
• 14:03 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2199.codfw.wmnet with reason: host reimage
• 14:02 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2200.codfw.wmnet with reason: host reimage
• 13:59 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2199.codfw.wmnet with reason: host reimage
• 13:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: maintenance
• 13:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: maintenance
• 13:56 aqu@deploy2002: Finished deploy [airflow-dags/analytics@7a1a552]: Backfill 2024 12: cassandra_load_pageview_per_article (duration: 01m 19s)
• 13:54 aqu@deploy2002: Started deploy [airflow-dags/analytics@7a1a552]: Backfill 2024 12: cassandra_load_pageview_per_article
• 13:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1170.eqiad.wmnet with reason: maintenance
• 13:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1170.eqiad.wmnet with reason: maintenance
• 13:49 jynus@cumin1002: dbctl commit (dc=all): 'depool db1170', diff saved to https://phabricator.wikimedia.org/P71983 and previous config saved to /var/cache/conftool/dbconfig/20250110-134954-jynus.json
• 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2200
• 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2200
• 13:43 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2200
• 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2200.codfw.wmnet 228.48.192.10.in-addr.arpa 8.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:43 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2200.codfw.wmnet 228.48.192.10.in-addr.arpa 8.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:40 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2199
• 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2199
• 13:40 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2199
• 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2199.codfw.wmnet 229.48.192.10.in-addr.arpa 9.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:40 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2199.codfw.wmnet 229.48.192.10.in-addr.arpa 9.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2199 - jelto@cumin1002"
• 13:40 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2199 - jelto@cumin1002"
• 13:38 marostegui: Move pc2013 to pc4 dbmaint codfw - T383398
• 13:36 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 13:35 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2200
• 13:35 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2199
• 13:35 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2200.codfw.wmnet with OS bookworm
• 13:34 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2199.codfw.wmnet with OS bookworm
• 13:32 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2199.codfw.wmnet wikikube-worker2200.codfw.wmnet wikikube-worker2201.codfw.wmnet wikikube-worker2202.codfw.wmnet on all recursors
• 13:32 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2199.codfw.wmnet wikikube-worker2200.codfw.wmnet wikikube-worker2201.codfw.wmnet wikikube-worker2202.codfw.wmnet on all recursors
• 13:30 marostegui: Move pc1013 to pc3 dbmaint eqiad - T383398
• 13:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2048 to wikikube-worker2202
• 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2202
• 13:29 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2202
• 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:29 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2048 to wikikube-worker2202 - jelto@cumin1002"
• 13:28 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2048 to wikikube-worker2202 - jelto@cumin1002"
• 13:25 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 13:25 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2048 to wikikube-worker2202
• 13:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2047 to wikikube-worker2201
• 13:23 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2201
• 13:22 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2201
• 13:22 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:22 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2047 to wikikube-worker2201 - jelto@cumin1002"
• 13:21 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2047 to wikikube-worker2201 - jelto@cumin1002"
• 13:18 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 13:17 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2047 to wikikube-worker2201
• 13:17 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2046 to wikikube-worker2200
• 13:16 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2200
• 13:15 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2200
• 13:15 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:14 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2046 to wikikube-worker2200 - jelto@cumin1002"
• 13:10 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 13:10 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2046 to wikikube-worker2200
• 13:09 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2045 to wikikube-worker2199
• 13:09 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2199
• 13:08 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2199
• 13:08 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:08 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2045 to wikikube-worker2199 - jelto@cumin1002"
• 13:08 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2045 to wikikube-worker2199 - jelto@cumin1002"
• 13:05 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 13:04 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2045 to wikikube-worker2199
• 13:00 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[2045-2048].codfw.wmnet
• 12:58 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[2045-2048].codfw.wmnet
• 12:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P71982 and previous config saved to /var/cache/conftool/dbconfig/20250110-124438-root.json
• 12:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2126.codfw.wmnet
• 12:30 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:30 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2126.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 12:30 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2126.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 12:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P71981 and previous config saved to /var/cache/conftool/dbconfig/20250110-122933-root.json
• 12:27 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 12:22 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2126.codfw.wmnet
• 12:22 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2126 from dbctl T383395', diff saved to https://phabricator.wikimedia.org/P71980 and previous config saved to /var/cache/conftool/dbconfig/20250110-122206-marostegui.json
• 12:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2126 T383395', diff saved to https://phabricator.wikimedia.org/P71979 and previous config saved to /var/cache/conftool/dbconfig/20250110-121657-marostegui.json
• 12:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P71978 and previous config saved to /var/cache/conftool/dbconfig/20250110-121427-root.json
• 12:07 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
• 11:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71977 and previous config saved to /var/cache/conftool/dbconfig/20250110-115922-root.json
• 11:54 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2195-2198].codfw.wmnet
• 11:54 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2195-2198].codfw.wmnet
• 11:51 jelto: homer 'cr*codw*' commit 'T377877'
• 11:51 jelto: homer 'cr*eqiad*' commit 'T377876'
• 11:50 jelto: homer 'lsw1-d6-codfw*' commit 'T377877'
• 11:50 jelto: homer 'lsw1-d8-codfw*' commit 'T377877'
• 11:49 jelto: homer 'lsw1-d5-codfw*' commit 'T377877'
• 11:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2197.codfw.wmnet with OS bookworm
• 11:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2192 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71976 and previous config saved to /var/cache/conftool/dbconfig/20250110-114417-root.json
• 11:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2198.codfw.wmnet with OS bookworm
• 11:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2197.codfw.wmnet with reason: host reimage
• 11:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2198.codfw.wmnet with reason: host reimage
• 11:21 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2197.codfw.wmnet with reason: host reimage
• 11:19 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2198.codfw.wmnet with reason: host reimage
• 11:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P71975 and previous config saved to /var/cache/conftool/dbconfig/20250110-110643-root.json
• 11:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P71974 and previous config saved to /var/cache/conftool/dbconfig/20250110-110633-root.json
• 11:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2197
• 11:02 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2197
• 11:01 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2197
• 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2197.codfw.wmnet 223.48.192.10.in-addr.arpa 3.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:01 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2197.codfw.wmnet 223.48.192.10.in-addr.arpa 3.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2198
• 10:59 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2198
• 10:59 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 10:58 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2198
• 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2198.codfw.wmnet 222.48.192.10.in-addr.arpa 2.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:58 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2198.codfw.wmnet 222.48.192.10.in-addr.arpa 2.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:58 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2198 - jelto@cumin1002"
• 10:58 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2198 - jelto@cumin1002"
• 10:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2192.codfw.wmnet with reason: maintenance
• 10:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db2192.codfw.wmnet with reason: maintenance
• 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2192 to change binlog format', diff saved to https://phabricator.wikimedia.org/P71973 and previous config saved to /var/cache/conftool/dbconfig/20250110-105514-marostegui.json
• 10:55 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 10:54 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2198
• 10:54 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2197
• 10:54 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2198.codfw.wmnet with OS bookworm
• 10:54 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2197.codfw.wmnet with OS bookworm
• 10:52 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 100%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P71972 and previous config saved to /var/cache/conftool/dbconfig/20250110-105202-root.json
• 10:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2195.codfw.wmnet with OS bookworm
• 10:51 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P71971 and previous config saved to /var/cache/conftool/dbconfig/20250110-105137-root.json
• 10:51 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P71970 and previous config saved to /var/cache/conftool/dbconfig/20250110-105127-root.json
• 10:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2196.codfw.wmnet with OS bookworm
• 10:47 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2123 T383388', diff saved to https://phabricator.wikimedia.org/P71969 and previous config saved to /var/cache/conftool/dbconfig/20250110-104739-marostegui.json
• 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 75%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P71968 and previous config saved to /var/cache/conftool/dbconfig/20250110-103657-root.json
• 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P71967 and previous config saved to /var/cache/conftool/dbconfig/20250110-103632-root.json
• 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P71966 and previous config saved to /var/cache/conftool/dbconfig/20250110-103622-root.json
• 10:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2195.codfw.wmnet with reason: host reimage
• 10:30 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1093-1095].eqiad.wmnet
• 10:30 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1093-1095].eqiad.wmnet
• 10:28 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2196.codfw.wmnet with reason: host reimage
• 10:28 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2195.codfw.wmnet with reason: host reimage
• 10:25 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2196.codfw.wmnet with reason: host reimage
• 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 50%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P71965 and previous config saved to /var/cache/conftool/dbconfig/20250110-102152-root.json
• 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71964 and previous config saved to /var/cache/conftool/dbconfig/20250110-102126-root.json
• 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71963 and previous config saved to /var/cache/conftool/dbconfig/20250110-102116-root.json
• 10:19 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1022.eqiad.wmnet with reason: maintenance
• 10:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1022.eqiad.wmnet with reason: maintenance
• 10:07 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2195
• 10:07 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2195
• 10:07 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2195
• 10:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2195.codfw.wmnet 225.48.192.10.in-addr.arpa 5.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:07 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2195.codfw.wmnet 225.48.192.10.in-addr.arpa 5.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 25%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P71962 and previous config saved to /var/cache/conftool/dbconfig/20250110-100646-root.json
• 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71961 and previous config saved to /var/cache/conftool/dbconfig/20250110-100621-root.json
• 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71960 and previous config saved to /var/cache/conftool/dbconfig/20250110-100611-root.json
• 10:06 elukey: restart dump_cloud_ip_ranges on puppetserver1001 - unit failed due to errors while fetching new data from upstream, trying to see if it was a temporary issue
• 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2196
• 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2196
• 10:05 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2196
• 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2196.codfw.wmnet 224.48.192.10.in-addr.arpa 4.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:05 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 10:05 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2196.codfw.wmnet 224.48.192.10.in-addr.arpa 4.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2196 - jelto@cumin1002"
• 10:05 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2196 - jelto@cumin1002"
• 10:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db[2128,2186,2228].codfw.wmnet with reason: maintenance
• 10:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db[2128,2186,2228].codfw.wmnet with reason: maintenance
• 10:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2128 db2228 T373579', diff saved to https://phabricator.wikimedia.org/P71959 and previous config saved to /var/cache/conftool/dbconfig/20250110-100248-marostegui.json
• 10:02 elukey: elukey@cumin1002:~$ sudo cumin -b 20 'an-worker*' 'apt-get clean' (safety to free space and avoid issues on hadoop) - T383320
• 10:01 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 10:00 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2196
• 10:00 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2195
• 10:00 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2196.codfw.wmnet with OS bookworm
• 10:00 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2195.codfw.wmnet with OS bookworm
• 09:57 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2195.codfw.wmnet wikikube-worker2196.codfw.wmnet wikikube-worker2197.codfw.wmnet wikikube-worker2198.codfw.wmnet on all recursors
• 09:57 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2195.codfw.wmnet wikikube-worker2196.codfw.wmnet wikikube-worker2197.codfw.wmnet wikikube-worker2198.codfw.wmnet on all recursors
• 09:57 elukey: kill hanging jupyterhub process on stat1009 to allow puppet to run an delete a user
• 09:56 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2052 to wikikube-worker2198
• 09:56 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2198
• 09:56 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2198
• 09:56 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:56 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2052 to wikikube-worker2198 - jelto@cumin1002"
• 09:55 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2052 to wikikube-worker2198 - jelto@cumin1002"
• 09:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2132,2160,2232].codfw.wmnet with reason: maintenance
• 09:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db[2132,2160,2232].codfw.wmnet with reason: maintenance
• 09:52 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 09:51 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2052 to wikikube-worker2198
• 09:51 marostegui@cumin1002: dbctl commit (dc=all): 'es1022 (re)pooling @ 10%: Repooling after maintenance', diff saved to https://phabricator.wikimedia.org/P71957 and previous config saved to /var/cache/conftool/dbconfig/20250110-095141-root.json
• 09:50 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2051 to wikikube-worker2197
• 09:50 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2197
• 09:50 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2197
• 09:50 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:50 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2051 to wikikube-worker2197 - jelto@cumin1002"
• 09:49 elukey: elukey@cumin1002:~$ sudo cumin 'an-worker11[39,15,54,90,75,57,89,18,06,24]*' 'apt-get clean'
• 09:49 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2051 to wikikube-worker2197 - jelto@cumin1002"
• 09:46 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 09:45 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2051 to wikikube-worker2197
• 09:45 elukey: elukey@cumin1002:~$ sudo cumin 'an-worker11[16,43,19,47,56,72,69]*' 'apt-get clean'
• 09:45 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2050 to wikikube-worker2196
• 09:44 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2196
• 09:44 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2196
• 09:44 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:44 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2050 to wikikube-worker2196 - jelto@cumin1002"
• 09:43 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2050 to wikikube-worker2196 - jelto@cumin1002"
• 09:40 elukey: `apt-get clean` on an-worker1147 to free space on the root partition
• 09:39 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 09:39 elukey: `apt-get clean` on an-worker1117 to free space on the root partition
• 09:39 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2050 to wikikube-worker2196
• 09:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2049 to wikikube-worker2195
• 09:38 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2195
• 09:37 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2195
• 09:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:37 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2049 to wikikube-worker2195 - jelto@cumin1002"
• 09:36 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2049 to wikikube-worker2195 - jelto@cumin1002"
• 09:33 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 09:32 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2049 to wikikube-worker2195
• 09:28 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[2049-2052].codfw.wmnet
• 09:25 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[2049-2052].codfw.wmnet
• 09:21 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2022.codfw.wmnet
• 09:21 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2022.codfw.wmnet
• 09:19 jelto: homer 'lsw1-c6-codfw*' commit 'T377877'
• 09:00 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
• 08:39 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1243.eqiad.wmnet
• 08:39 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1243.eqiad.wmnet
• 08:36 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1081.eqiad.wmnet
• 08:36 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1081.eqiad.wmnet
• 08:34 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet
• 08:34 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet
• 08:31 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet
• 08:31 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet
• 08:25 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1057.eqiad.wmnet
• 08:25 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet
• 08:21 jelto: homer 'lsw1-e3-eqiad*' commit 'T377876'
• 08:15 jelto: homer 'cr*eqiad*' commit 'T377876'
• 05:02 eileen: civicrm upgraded from b357a6fd to 1ea537d3
• 02:36 eileen: civicrm upgraded from e8326943 to b357a6fd
• 01:32 eileen: config revision changed from fc1c1a6b to e0866d2f
• 01:07 eileen: civicrm upgraded from 24f3b57f to e8326943

2025-01-09

• 23:00 inflatador: bking@pcc-db1002.puppet-diffs.eqiad1.wikimedia.cloud sudo -u jenkins-deploy /usr/local/sbin/pcc_facts_processor T378368
• 23:00 inflatador: bking@puppetserver1001:~$ sudo /usr/local/sbin/puppet-facts-upload --proxy http://webproxy.eqiad.wmnet:8080 T378368
• 21:44 bking@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Sync cloudelastic1011 status change after Netbox update - bking@cumin2002 - T378368"
• 21:43 bking@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Sync cloudelastic1011 status change after Netbox update - bking@cumin2002 - T378368"
• 20:30 aqu@deploy2002: Finished deploy [airflow-dags/analytics@0e4370e]: Canary event fix (duration: 01m 23s)
• 20:29 aqu@deploy2002: Started deploy [airflow-dags/analytics@0e4370e]: Canary event fix
• 20:06 dcausse@deploy2002: Finished deploy [airflow-dags/search@718e870]: search: switch query_clicks to SparkSqlOperator (duration: 00m 27s)
• 20:05 dcausse@deploy2002: Started deploy [airflow-dags/search@718e870]: search: switch query_clicks to SparkSqlOperator
• 19:54 cdanis@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 19:54 cdanis@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 19:49 cdanis@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
• 19:34 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.11 refs T382362
• 18:49 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply
• 18:46 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply
• 18:46 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-wikifunctions: apply
• 18:45 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply
• 18:45 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
• 18:44 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1095.eqiad.wmnet with OS bookworm
• 18:44 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
• 18:44 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
• 18:42 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
• 18:42 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
• 18:41 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
• 18:41 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
• 18:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1094.eqiad.wmnet with OS bookworm
• 18:39 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
• 18:37 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
• 18:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1093.eqiad.wmnet with OS bookworm
• 18:36 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
• 18:35 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
• 18:33 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-web: apply
• 18:27 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
• 18:26 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
• 18:25 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1095.eqiad.wmnet with reason: host reimage
• 18:24 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 18:23 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 18:20 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1094.eqiad.wmnet with reason: host reimage
• 18:17 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1093.eqiad.wmnet with reason: host reimage
• 18:14 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1094.eqiad.wmnet with reason: host reimage
• 18:13 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1095.eqiad.wmnet with reason: host reimage
• 18:13 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1093.eqiad.wmnet with reason: host reimage
• 17:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1095
• 17:58 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1095
• 17:58 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1095.eqiad.wmnet with OS bookworm
• 17:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1094
• 17:58 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1094
• 17:58 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1094.eqiad.wmnet with OS bookworm
• 17:58 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1093
• 17:58 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1093
• 17:58 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1093.eqiad.wmnet with OS bookworm
• 17:55 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1095.eqiad.wmnet with OS bookworm
• 17:55 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1094.eqiad.wmnet with OS bookworm
• 17:55 kamila@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1093.eqiad.wmnet with OS bookworm
• 17:50 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply
• 17:49 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply
• 17:49 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-wikifunctions: apply
• 17:48 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply
• 17:48 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 17:47 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 17:47 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
• 17:45 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-web: apply
• 17:45 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
• 17:44 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
• 17:44 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
• 17:42 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
• 17:42 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
• 17:41 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
• 17:41 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
• 17:41 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 17:39 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
• 17:39 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
• 17:38 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
• 17:38 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
• 17:37 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1095
• 17:37 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1095
• 17:37 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1095.eqiad.wmnet with OS bookworm
• 17:37 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
• 17:37 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1094
• 17:37 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1094
• 17:37 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1094.eqiad.wmnet with OS bookworm
• 17:26 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 17:25 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 17:25 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
• 17:22 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-web: apply
• 17:22 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
• 17:21 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
• 17:21 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
• 17:21 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1093
• 17:20 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1093
• 17:20 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1093.eqiad.wmnet with OS bookworm
• 17:20 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2022.codfw.wmnet with reason: host reimage
• 17:19 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
• 17:19 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
• 17:18 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
• 17:18 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
• 17:17 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
• 17:17 cdanis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply
• 17:16 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2022.codfw.wmnet with reason: host reimage
• 17:16 cdanis@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply
• 17:16 cdanis@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-wikifunctions: apply
• 17:15 cdanis@deploy2002: helmfile [codfw] START helmfile.d/services/mw-wikifunctions: apply
• 17:02 cdanis@deploy2002: Finished scap sync-world: Backport for group1: enable OpenTelemetry exports (T340552) (duration: 14m 22s)
• 16:55 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 16:55 elukey@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 16:55 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 16:54 elukey@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2022
• 16:54 elukey@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2022
• 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2022.codfw.wmnet 212.32.192.10.in-addr.arpa 2.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:54 elukey@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2022.codfw.wmnet 212.32.192.10.in-addr.arpa 2.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:54 elukey@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2022 - elukey@cumin1002"
• 16:54 elukey@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2022 - elukey@cumin1002"
• 16:53 cdanis@deploy2002: cdanis: Continuing with sync
• 16:53 cdanis@deploy2002: cdanis: Backport for group1: enable OpenTelemetry exports (T340552) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 16:50 elukey@cumin1002: START - Cookbook sre.dns.netbox
• 16:50 elukey@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 16:48 cdanis@deploy2002: Started scap sync-world: Backport for group1: enable OpenTelemetry exports (T340552)
• 16:40 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.11 refs T382362
• 16:33 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1243.eqiad.wmnet with OS bookworm
• 16:33 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 16:29 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
• 16:28 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
• 16:27 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 16:21 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1095.eqiad.wmnet with OS bookworm
• 16:21 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1094.eqiad.wmnet with OS bookworm
• 16:21 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1093.eqiad.wmnet with OS bookworm
• 16:21 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
• 16:21 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1069.eqiad.wmnet with OS bookworm
• 16:21 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 16:20 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
• 16:12 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 16:08 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1243.eqiad.wmnet with reason: host reimage
• 16:05 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1243.eqiad.wmnet with reason: host reimage
• 15:56 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1057.eqiad.wmnet with OS bookworm
• 15:56 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 15:55 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 15:54 sukhe@dns1004:: END - running authdns-update
• 15:53 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1069.eqiad.wmnet with reason: host reimage
• 15:52 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
• 15:52 sukhe@dns1004:: START - running authdns-update
• 15:52 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
• 15:49 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1069.eqiad.wmnet with reason: host reimage
• 15:46 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1243.eqiad.wmnet with OS bookworm
• 15:46 inflatador: bking@an-airflow1005 stopping airflow-search services as part of k8s migration T380615
• 15:46 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
• 15:45 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
• 15:39 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1073.eqiad.wmnet with OS bookworm
• 15:39 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 15:39 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 15:37 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1057.eqiad.wmnet with reason: host reimage
• 15:33 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
• 15:33 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
• 15:33 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1057.eqiad.wmnet with reason: host reimage
• 15:31 sukhe@dns1004:: END - running authdns-update
• 15:31 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1069.eqiad.wmnet with OS bookworm
• 15:30 sukhe@dns1004:: START - running authdns-update
• 15:28 logmsgbot: testing update from dns host
• 15:27 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2193-2194].codfw.wmnet
• 15:27 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2193-2194].codfw.wmnet
• 15:25 jelto: homer 'cr*codfw*' commit 'T377877'
• 15:24 jelto: homer 'lsw1-c5-codfw*' commit 'T377877'
• 15:23 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1095
• 15:23 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1095
• 15:23 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1095.eqiad.wmnet with OS bookworm
• 15:23 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1094
• 15:23 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1094
• 15:23 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1094.eqiad.wmnet with OS bookworm
• 15:23 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1093
• 15:23 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1093
• 15:23 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1093.eqiad.wmnet with OS bookworm
• 15:22 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
• 15:22 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
• 15:21 jelto: homer 'lsw1-d3-codfw*' commit 'T377877'
• 15:21 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1093.eqiad.wmnet wikikube-worker1094.eqiad.wmnet wikikube-worker1095.eqiad.wmnet on all recursors
• 15:21 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1093.eqiad.wmnet wikikube-worker1094.eqiad.wmnet wikikube-worker1095.eqiad.wmnet on all recursors
• 15:21 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1457 to wikikube-worker1093
• 15:20 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1093
• 15:20 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1073.eqiad.wmnet with reason: host reimage
• 15:19 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2192.codfw.wmnet with OS bookworm
• 15:19 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1093
• 15:18 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:18 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1459 to wikikube-worker1095
• 15:17 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1095
• 15:16 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1073.eqiad.wmnet with reason: host reimage
• 15:16 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1458 to wikikube-worker1094
• 15:16 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 15:16 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1095
• 15:16 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:15 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1094
• 15:15 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1057.eqiad.wmnet with OS bookworm
• 15:14 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1094
• 15:14 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:14 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1458 to wikikube-worker1094 - kamila@cumin1002"
• 15:14 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 15:12 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1458 to wikikube-worker1094 - kamila@cumin1002"
• 15:09 tchin@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
• 15:09 tchin@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
• 15:09 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 15:09 kamila@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 15:08 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1459 to wikikube-worker1095
• 15:07 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1458 to wikikube-worker1094
• 15:06 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 15:06 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1457 to wikikube-worker1093
• 15:06 Lucas_WMDE: UTC afternoon backport+config window done
• 15:04 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for Enable Translate message bundle Scribunto library on MetaWiki (T379892) (duration: 32m 53s)
• 14:58 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1073.eqiad.wmnet with OS bookworm
• 14:56 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2194.codfw.wmnet with OS bookworm
• 14:54 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1457-1459].eqiad.wmnet
• 14:54 lucaswerkmeister-wmde@deploy2002: abi, lucaswerkmeister-wmde: Continuing with sync
• 14:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2193.codfw.wmnet with OS bookworm
• 14:52 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1457-1459].eqiad.wmnet
• 14:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2192
• 14:41 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2192
• 14:41 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2192.codfw.wmnet with OS bookworm
• 14:38 lucaswerkmeister-wmde@deploy2002: abi, lucaswerkmeister-wmde: Backport for Enable Translate message bundle Scribunto library on MetaWiki (T379892) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:37 dcaro@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
• 14:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2194.codfw.wmnet with reason: host reimage
• 14:33 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2193.codfw.wmnet with reason: host reimage
• 14:32 dcaro@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
• 14:31 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for Enable Translate message bundle Scribunto library on MetaWiki (T379892)
• 14:31 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2194.codfw.wmnet with reason: host reimage
• 14:29 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2193.codfw.wmnet with reason: host reimage
• 14:24 dcaro@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1012.eqiad.wmnet with OS bullseye
• 14:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2194
• 14:12 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2194
• 14:12 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2194.codfw.wmnet with OS bookworm
• 14:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2193
• 14:11 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2193
• 14:11 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2193.codfw.wmnet with OS bookworm
• 14:05 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2194.codfw.wmnet with OS bookworm
• 14:00 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2193.codfw.wmnet with OS bookworm
• 13:58 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2192.codfw.wmnet with OS bookworm
• 13:39 moritzm: installing jinja2 security updates
• 13:21 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1088-1092].eqiad.wmnet
• 13:21 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1088-1092].eqiad.wmnet
• 13:21 aqu@deploy2002: Finished deploy [airflow-dags/analytics@9073e46]: Refine refactoring (duration: 02m 51s)
• 13:18 aqu@deploy2002: Started deploy [airflow-dags/analytics@9073e46]: Refine refactoring
• 13:11 moritzm: installing sqlparse security updates
• 13:08 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71948 and previous config saved to /var/cache/conftool/dbconfig/20250109-130818-root.json
• 12:54 aqu@deploy2002: deploy aborted: Refine refactoring (duration: 00m 20s)
• 12:54 aqu@deploy2002: Started deploy [airflow-dags/analytics_test@9073e46]: Refine refactoring
• 12:53 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71947 and previous config saved to /var/cache/conftool/dbconfig/20250109-125313-root.json
• 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2194
• 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2194
• 12:45 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2194
• 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2194.codfw.wmnet 224.32.192.10.in-addr.arpa 4.2.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:45 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2194.codfw.wmnet 224.32.192.10.in-addr.arpa 4.2.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:45 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2194 - jelto@cumin1002"
• 12:44 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2194 - jelto@cumin1002"
• 12:43 dcaro@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcephosd1012.eqiad.wmnet with OS bullseye
• 12:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2193
• 12:41 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2193
• 12:41 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 12:41 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2193
• 12:41 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2193.codfw.wmnet 62.48.192.10.in-addr.arpa 2.6.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:41 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2193.codfw.wmnet 62.48.192.10.in-addr.arpa 2.6.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:41 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:41 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2193 - jelto@cumin1002"
• 12:41 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2193 - jelto@cumin1002"
• 12:40 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2194
• 12:40 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2194.codfw.wmnet with OS bookworm
• 12:38 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71946 and previous config saved to /var/cache/conftool/dbconfig/20250109-123806-root.json
• 12:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2192
• 12:38 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2192
• 12:37 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2192
• 12:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2192.codfw.wmnet 221.48.192.10.in-addr.arpa 1.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:37 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 12:37 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2192.codfw.wmnet 221.48.192.10.in-addr.arpa 1.2.2.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:37 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:37 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2192 - jelto@cumin1002"
• 12:37 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2192 - jelto@cumin1002"
• 12:37 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2193
• 12:37 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2193.codfw.wmnet with OS bookworm
• 12:34 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 12:34 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2192
• 12:34 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2192.codfw.wmnet with OS bookworm
• 12:31 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2192.codfw.wmnet wikikube-worker2193.codfw.wmnet wikikube-worker2194.codfw.wmnet on all recursors
• 12:31 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2192.codfw.wmnet wikikube-worker2193.codfw.wmnet wikikube-worker2194.codfw.wmnet on all recursors
• 12:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2058 to wikikube-worker2194
• 12:30 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2194
• 12:30 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2194
• 12:30 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:30 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2058 to wikikube-worker2194 - jelto@cumin1002"
• 12:29 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2058 to wikikube-worker2194 - jelto@cumin1002"
• 12:26 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 12:25 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2058 to wikikube-worker2194
• 12:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2056 to wikikube-worker2193
• 12:25 root@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Piccardi out of all services on: 2310 hosts
• 12:24 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2193
• 12:24 root@cumin2002: START - Cookbook sre.idm.logout Logging Piccardi out of all services on: 2310 hosts
• 12:23 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71945 and previous config saved to /var/cache/conftool/dbconfig/20250109-122301-root.json
• 12:19 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2193
• 12:19 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:19 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2056 to wikikube-worker2193 - jelto@cumin1002"
• 12:18 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2056 to wikikube-worker2193 - jelto@cumin1002"
• 12:15 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 12:14 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2056 to wikikube-worker2193
• 12:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes2053 to wikikube-worker2192
• 12:13 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2192
• 12:13 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2192
• 12:12 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:12 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2053 to wikikube-worker2192 - jelto@cumin1002"
• 12:12 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes2053 to wikikube-worker2192 - jelto@cumin1002"
• 12:08 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
• 12:07 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71944 and previous config saved to /var/cache/conftool/dbconfig/20250109-120755-root.json
• 12:07 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
• 12:06 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
• 12:05 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
• 12:05 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 12:05 jelto@cumin1002: START - Cookbook sre.hosts.rename from kubernetes2053 to wikikube-worker2192
• 12:04 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
• 12:03 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
• 11:58 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
• 11:57 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/changeprop: apply
• 11:54 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
• 11:53 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
• 11:53 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
• 11:53 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/changeprop: apply
• 11:52 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 9%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71943 and previous config saved to /var/cache/conftool/dbconfig/20250109-115250-root.json
• 11:37 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 8%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71942 and previous config saved to /var/cache/conftool/dbconfig/20250109-113744-root.json
• 11:29 elukey@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 100%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71940 and previous config saved to /var/cache/conftool/dbconfig/20250109-112543-root.json
• 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71939 and previous config saved to /var/cache/conftool/dbconfig/20250109-112534-root.json
• 11:24 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 11:24 elukey@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 11:23 elukey@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 11:22 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 7%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71938 and previous config saved to /var/cache/conftool/dbconfig/20250109-112239-root.json
• 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 75%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71937 and previous config saved to /var/cache/conftool/dbconfig/20250109-111038-root.json
• 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 75%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71936 and previous config saved to /var/cache/conftool/dbconfig/20250109-111029-root.json
• 11:07 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 6%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71935 and previous config saved to /var/cache/conftool/dbconfig/20250109-110734-root.json
• 10:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc4', diff saved to https://phabricator.wikimedia.org/P71934 and previous config saved to /var/cache/conftool/dbconfig/20250109-105708-ladsgroup.json
• 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 50%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71933 and previous config saved to /var/cache/conftool/dbconfig/20250109-105533-root.json
• 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 50%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71932 and previous config saved to /var/cache/conftool/dbconfig/20250109-105523-root.json
• 10:52 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71931 and previous config saved to /var/cache/conftool/dbconfig/20250109-105228-root.json
• 10:45 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for pc2015.codfw.wmnet
• 10:45 root@cumin2002: END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for dbprov2004.codfw.wmnet: Renew puppet certificate - root@cumin2002
• 10:45 root@cumin2002: START - Cookbook sre.puppet.renew-cert for dbprov2004.codfw.wmnet: Renew puppet certificate - root@cumin2002
• 10:43 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc1016.eqiad.wmnet with reason: Reboot
• 10:43 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on pc1016.eqiad.wmnet with reason: Reboot
• 10:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P71930 and previous config saved to /var/cache/conftool/dbconfig/20250109-104043-root.json
• 10:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 25%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71929 and previous config saved to /var/cache/conftool/dbconfig/20250109-104027-root.json
• 10:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 25%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71928 and previous config saved to /var/cache/conftool/dbconfig/20250109-104017-root.json
• 10:40 ladsgroup@cumin1002: START - Cookbook sre.mysql.upgrade for pc2015.codfw.wmnet
• 10:38 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for pc1016.eqiad.wmnet
• 10:37 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71927 and previous config saved to /var/cache/conftool/dbconfig/20250109-103723-root.json
• 10:34 ladsgroup@cumin1002: START - Cookbook sre.mysql.upgrade for pc1016.eqiad.wmnet
• 10:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling pc4', diff saved to https://phabricator.wikimedia.org/P71926 and previous config saved to /var/cache/conftool/dbconfig/20250109-102700-ladsgroup.json
• 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P71925 and previous config saved to /var/cache/conftool/dbconfig/20250109-102538-root.json
• 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 10%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71924 and previous config saved to /var/cache/conftool/dbconfig/20250109-102522-root.json
• 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 10%: Repooling after moving sanitarium', diff saved to https://phabricator.wikimedia.org/P71923 and previous config saved to /var/cache/conftool/dbconfig/20250109-102512-root.json
• 10:22 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71922 and previous config saved to /var/cache/conftool/dbconfig/20250109-102218-root.json
• 10:21 marostegui: Move db2187:3312 under db2226 s2 codfw dbmaint T373579
• 10:20 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db[2126,2187,2226].codfw.wmnet with reason: maintenance
• 10:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db[2126,2187,2226].codfw.wmnet with reason: maintenance
• 10:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2126 db2226 T373579', diff saved to https://phabricator.wikimedia.org/P71921 and previous config saved to /var/cache/conftool/dbconfig/20250109-102010-marostegui.json
• 10:12 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubernetes[2053,2056,2058].codfw.wmnet
• 10:11 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host kubernetes[2053,2056,2058].codfw.wmnet
• 10:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P71920 and previous config saved to /var/cache/conftool/dbconfig/20250109-101033-root.json
• 10:09 klausman@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2001.codfw.wmnet
• 10:07 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71919 and previous config saved to /var/cache/conftool/dbconfig/20250109-100712-root.json
• 10:06 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbprov2004.codfw.wmnet with reason: os upgrade
• 10:05 jynus@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbprov2004.codfw.wmnet with reason: os upgrade
• 09:58 moritzm: installing glibc bugfix updates for Bookworm
• 09:55 klausman@cumin2002: START - Cookbook sre.hosts.reboot-single for host ml-serve2001.codfw.wmnet
• 09:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71918 and previous config saved to /var/cache/conftool/dbconfig/20250109-095527-root.json
• 09:55 klausman@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve2001.codfw.wmnet
• 09:52 marostegui@cumin1002: dbctl commit (dc=all): 'db2231 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71917 and previous config saved to /var/cache/conftool/dbconfig/20250109-095207-root.json
• 09:43 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71916 and previous config saved to /var/cache/conftool/dbconfig/20250109-094355-root.json
• 09:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2131 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71915 and previous config saved to /var/cache/conftool/dbconfig/20250109-094022-root.json
• 09:28 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71914 and previous config saved to /var/cache/conftool/dbconfig/20250109-092850-root.json
• 09:23 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp4044.ulsfo.wmnet,cp4052.ulsfo.wmnet} and A:cp
• 09:21 klausman@cumin2002: START - Cookbook sre.hosts.reboot-single for host ml-serve2001.codfw.wmnet
• 09:20 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet
• 09:19 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet
• 09:19 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet
• 09:19 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet
• 09:19 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet
• 09:19 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet
• 09:19 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2013.codfw.wmnet with OS bookworm
• 09:18 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp4044.ulsfo.wmnet,cp4052.ulsfo.wmnet} and A:cp
• 09:17 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp4044.ulsfo.wmnet,cp4052.ulsfo.wmnet} and A:cp
• 09:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2012.codfw.wmnet with OS bookworm
• 09:13 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71913 and previous config saved to /var/cache/conftool/dbconfig/20250109-091345-root.json
• 09:13 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp4044.ulsfo.wmnet,cp4052.ulsfo.wmnet} and A:cp
• 09:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2011.codfw.wmnet with OS bookworm
• 09:02 vgutierrez: update to haproxy 2.8.13 on component thirdparty/haproxy28 bullseye-wikimedia (apt.wm.o) - T383111
• 08:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2013.codfw.wmnet with reason: host reimage
• 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71912 and previous config saved to /var/cache/conftool/dbconfig/20250109-085840-root.json
• 08:57 root@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2131.codfw.wmnet onto db2231.codfw.wmnet
• 08:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2012.codfw.wmnet with reason: host reimage
• 08:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2011.codfw.wmnet with reason: host reimage
• 08:49 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2013.codfw.wmnet with reason: host reimage
• 08:49 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2012.codfw.wmnet with reason: host reimage
• 08:49 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2011.codfw.wmnet with reason: host reimage
• 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71911 and previous config saved to /var/cache/conftool/dbconfig/20250109-084335-root.json
• 08:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1043.eqiad.wmnet with reason: cloning
• 08:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1043.eqiad.wmnet with reason: cloning
• 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2012
• 08:32 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2012
• 08:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2013
• 08:32 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2013
• 08:31 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2012.codfw.wmnet with OS bookworm
• 08:31 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2013.codfw.wmnet with OS bookworm
• 08:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2011
• 08:31 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2011
• 08:31 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2011.codfw.wmnet with OS bookworm
• 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 9%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71910 and previous config saved to /var/cache/conftool/dbconfig/20250109-082829-root.json
• 08:26 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2011-2013].codfw.wmnet
• 08:24 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2011-2013].codfw.wmnet
• 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet
• 08:22 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet
• 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet
• 08:22 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet
• 08:21 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2017.codfw.wmnet with OS bookworm
• 08:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2014.codfw.wmnet with OS bookworm
• 08:13 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 8%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71909 and previous config saved to /var/cache/conftool/dbconfig/20250109-081324-root.json
• 08:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2017.codfw.wmnet with reason: host reimage
• 07:58 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 7%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71908 and previous config saved to /var/cache/conftool/dbconfig/20250109-075820-root.json
• 07:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2014.codfw.wmnet with reason: host reimage
• 07:52 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2017.codfw.wmnet with reason: host reimage
• 07:52 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2014.codfw.wmnet with reason: host reimage
• 07:43 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 6%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71907 and previous config saved to /var/cache/conftool/dbconfig/20250109-074314-root.json
• 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P71906 and previous config saved to /var/cache/conftool/dbconfig/20250109-074214-root.json
• 07:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2017
• 07:34 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2017
• 07:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2014
• 07:34 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2014
• 07:34 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2017.codfw.wmnet with OS bookworm
• 07:34 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2014.codfw.wmnet with OS bookworm
• 07:32 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2014,2017].codfw.wmnet
• 07:29 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2014,2017].codfw.wmnet
• 07:28 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71905 and previous config saved to /var/cache/conftool/dbconfig/20250109-072809-root.json
• 07:27 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P71904 and previous config saved to /var/cache/conftool/dbconfig/20250109-072709-root.json
• 07:13 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71903 and previous config saved to /var/cache/conftool/dbconfig/20250109-071305-root.json
• 07:12 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P71902 and previous config saved to /var/cache/conftool/dbconfig/20250109-071203-root.json
• 06:58 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71901 and previous config saved to /var/cache/conftool/dbconfig/20250109-065759-root.json
• 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P71900 and previous config saved to /var/cache/conftool/dbconfig/20250109-065658-root.json
• 06:53 kart_: Updated cxserver to 2025-01-07-045930-production (T377966, T377813, T381379)
• 06:52 root@cumin1002: START - Cookbook sre.mysql.clone of db2131.codfw.wmnet onto db2231.codfw.wmnet
• 06:51 marostegui@cumin1002: dbctl commit (dc=all): 'Add db2231 to dbctl depooled T373579', diff saved to https://phabricator.wikimedia.org/P71899 and previous config saved to /var/cache/conftool/dbconfig/20250109-065114-marostegui.json
• 06:47 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2131.codfw.wmnet with reason: cloning db2231
• 06:47 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2131.codfw.wmnet with reason: cloning db2231
• 06:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2131 T373579', diff saved to https://phabricator.wikimedia.org/P71898 and previous config saved to /var/cache/conftool/dbconfig/20250109-064556-marostegui.json
• 06:45 kartik@deploy2002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
• 06:44 kartik@deploy2002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
• 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71897 and previous config saved to /var/cache/conftool/dbconfig/20250109-064254-root.json
• 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P71896 and previous config saved to /var/cache/conftool/dbconfig/20250109-064153-root.json
• 06:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1022.eqiad.wmnet with reason: cloning es1042
• 06:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1022.eqiad.wmnet with reason: cloning es1042
• 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1022 T382569', diff saved to https://phabricator.wikimedia.org/P71895 and previous config saved to /var/cache/conftool/dbconfig/20250109-064117-marostegui.json
• 06:40 kartik@deploy2002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
• 06:40 kartik@deploy2002: helmfile [codfw] START helmfile.d/services/cxserver: apply
• 06:31 kartik@deploy2002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
• 06:30 kartik@deploy2002: helmfile [staging] START helmfile.d/services/cxserver: apply
• 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'es1042 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71894 and previous config saved to /var/cache/conftool/dbconfig/20250109-062749-root.json
• 06:26 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P71893 and previous config saved to /var/cache/conftool/dbconfig/20250109-062647-root.json
• 06:17 marostegui@cumin1002: dbctl commit (dc=all): 'Add es1042 depooled T382569', diff saved to https://phabricator.wikimedia.org/P71892 and previous config saved to /var/cache/conftool/dbconfig/20250109-061724-marostegui.json
• 06:11 marostegui@cumin1002: dbctl commit (dc=all): 'es1021 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P71891 and previous config saved to /var/cache/conftool/dbconfig/20250109-061142-root.json
• 00:43 ladsgroup@deploy2002: Finished scap sync-world: Backport for filerepo: Fix schema compatibility constant usage (T383269) (duration: 13m 34s)
• 00:36 ladsgroup@deploy2002: ladsgroup, cdanis: Continuing with sync
• 00:36 ladsgroup@deploy2002: ladsgroup, cdanis: Backport for filerepo: Fix schema compatibility constant usage (T383269) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 00:30 ladsgroup@deploy2002: Started scap sync-world: Backport for filerepo: Fix schema compatibility constant usage (T383269)

2025-01-08

• 23:03 eileen: civicrm upgraded from 1c93f7a1 to 24f3b57f
• 22:27 eileen: civicrm upgraded from 82b02ce5 to 1c93f7a1
• 21:28 dzahn@cumin2002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: security release 20250108
• 21:16 aokoth@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Security Update
• 21:14 dzahn@cumin2002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: security release 20250108
• 21:08 aokoth@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Security Update
• 20:05 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.11 refs T382362
• 19:59 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wikikube-worker1001.eqiad.wmnet
• 19:59 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 19:59 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1002"
• 19:55 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1002"
• 19:54 dduvall: rolling back wmf.11 to group0 due to `Table 'commonswiki.file' doesn't exist` errors
• 19:53 kamila_: homer 'cr*eqiad*' commit 'T365571'
• 19:52 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 19:46 kamila@cumin1002: START - Cookbook sre.hosts.decommission for hosts wikikube-worker1001.eqiad.wmnet
• 19:32 sfaci@deploy2002: Finished deploy [airflow-dags/analytics@b2b5707]: (no justification provided) (duration: 03m 06s)
• 19:29 sfaci@deploy2002: Started deploy [airflow-dags/analytics@b2b5707]: (no justification provided)
• 19:29 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.11 refs T382362
• 19:17 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1001.eqiad.wmnet
• 19:17 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1001.eqiad.wmnet
• 19:08 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1089.eqiad.wmnet with OS bookworm
• 19:03 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1091.eqiad.wmnet with OS bookworm
• 18:59 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1090.eqiad.wmnet with OS bookworm
• 18:55 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1092.eqiad.wmnet with OS bookworm
• 18:52 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1088.eqiad.wmnet with OS bookworm
• 18:48 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1089.eqiad.wmnet with reason: host reimage
• 18:43 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1091.eqiad.wmnet with reason: host reimage
• 18:40 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1090.eqiad.wmnet with reason: host reimage
• 18:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1092.eqiad.wmnet with reason: host reimage
• 18:33 swfrench@deploy2002: Finished scap sync-world: Deployment to switch migration release files to 8.1 - T377040 (duration: 13m 57s)
• 18:32 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1088.eqiad.wmnet with reason: host reimage
• 18:29 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1092.eqiad.wmnet with reason: host reimage
• 18:29 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1091.eqiad.wmnet with reason: host reimage
• 18:29 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1090.eqiad.wmnet with reason: host reimage
• 18:28 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1089.eqiad.wmnet with reason: host reimage
• 18:27 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1088.eqiad.wmnet with reason: host reimage
• 18:19 swfrench@deploy2002: Started scap sync-world: Deployment to switch migration release files to 8.1 - T377040
• 18:17 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol1011.eqiad.wmnet with OS bookworm
• 18:17 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 18:16 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 18:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1092
• 18:13 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1092
• 18:13 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1092.eqiad.wmnet with OS bookworm
• 18:13 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1091
• 18:13 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1091
• 18:13 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1091.eqiad.wmnet with OS bookworm
• 18:12 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1090
• 18:12 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1090
• 18:12 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1090.eqiad.wmnet with OS bookworm
• 18:12 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1089
• 18:12 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1089
• 18:12 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1089.eqiad.wmnet with OS bookworm
• 18:11 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1088
• 18:11 kamila@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1088
• 18:11 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1088.eqiad.wmnet with OS bookworm
• 18:09 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1088.eqiad.wmnet wikikube-worker1089.eqiad.wmnet wikikube-worker1090.eqiad.wmnet wikikube-worker1091.eqiad.wmnet wikikube-worker1092.eqiad.wmnet on all recursors
• 18:09 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1088.eqiad.wmnet wikikube-worker1089.eqiad.wmnet wikikube-worker1090.eqiad.wmnet wikikube-worker1091.eqiad.wmnet wikikube-worker1092.eqiad.wmnet on all recursors
• 18:09 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1455 to wikikube-worker1092
• 18:08 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1092
• 18:07 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1092
• 18:07 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:07 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1455 to wikikube-worker1092 - kamila@cumin1002"
• 18:07 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1455 to wikikube-worker1092 - kamila@cumin1002"
• 18:04 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1454 to wikikube-worker1091
• 18:04 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1091
• 18:03 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 18:02 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1091
• 18:02 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:02 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1454 to wikikube-worker1091 - kamila@cumin1002"
• 18:02 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1454 to wikikube-worker1091 - kamila@cumin1002"
• 18:01 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1455 to wikikube-worker1092
• 17:59 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 17:58 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1454 to wikikube-worker1091
• 17:58 kamila@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1088.eqiad.wmnet wikikube-worker1089.eqiad.wmnet wikikube-worker1090.eqiad.wmnet wikikub on all recursors
• 17:58 kamila@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1088.eqiad.wmnet wikikube-worker1089.eqiad.wmnet wikikube-worker1090.eqiad.wmnet wikikub on all recursors
• 17:57 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol1011.eqiad.wmnet with reason: host reimage
• 17:56 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1452 to wikikube-worker1089
• 17:56 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1089
• 17:55 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1089
• 17:55 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:54 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol1011.eqiad.wmnet with reason: host reimage
• 17:53 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1453 to wikikube-worker1090
• 17:52 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1090
• 17:52 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 17:51 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1090
• 17:51 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:51 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1453 to wikikube-worker1090 - kamila@cumin1002"
• 17:51 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1453 to wikikube-worker1090 - kamila@cumin1002"
• 17:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from mw1451 to wikikube-worker1088
• 17:48 kamila@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1088
• 17:47 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 17:47 kamila@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1088
• 17:47 kamila@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:47 kamila@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1451 to wikikube-worker1088 - kamila@cumin1002"
• 17:46 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1453 to wikikube-worker1090
• 17:46 kamila@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming mw1451 to wikikube-worker1088 - kamila@cumin1002"
• 17:44 kamila@cumin1002: END (FAIL) - Cookbook sre.hosts.rename (exit_code=99) from mw1453 to wikikube-worker1090
• 17:43 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1453 to wikikube-worker1090
• 17:43 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1452 to wikikube-worker1089
• 17:43 kamila@cumin1002: START - Cookbook sre.dns.netbox
• 17:42 kamila@cumin1002: START - Cookbook sre.hosts.rename from mw1451 to wikikube-worker1088
• 17:41 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host mw[1451-1455].eqiad.wmnet
• 17:36 kamila@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host mw[1451-1455].eqiad.wmnet
• 17:22 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
• 17:05 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
• 17:04 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2019.codfw.wmnet
• 17:04 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2019.codfw.wmnet
• 17:02 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
• 17:02 jelto: sudo homer 'cr*codfw*' commit 'T377877'
• 17:01 jelto: sudo homer 'lsw1-c3-codfw*' commit 'T377877'
• 17:00 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2019.codfw.wmnet
• 17:00 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host cloudcontrol1011.eqiad.wmnet with OS bookworm
• 16:59 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2019.codfw.wmnet
• 16:59 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet
• 16:59 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet
• 16:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2019.codfw.wmnet with OS bookworm
• 16:50 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1081.eqiad.wmnet with OS bookworm
• 16:50 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 16:50 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
• 16:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2018.codfw.wmnet with OS bookworm
• 16:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2019.codfw.wmnet with reason: host reimage
• 16:31 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1081.eqiad.wmnet with reason: host reimage
• 16:29 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 16:27 jhathaway@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS bookworm
• 16:26 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2019.codfw.wmnet with reason: host reimage
• 16:25 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1081.eqiad.wmnet with reason: host reimage
• 16:24 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2018.codfw.wmnet with reason: host reimage
• 16:20 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 16:20 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2018.codfw.wmnet with reason: host reimage
• 16:08 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 16:08 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1081.eqiad.wmnet with OS bookworm
• 16:07 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2019
• 16:07 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2019
• 16:07 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2019
• 16:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2019.codfw.wmnet 117.32.192.10.in-addr.arpa 7.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:07 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2019.codfw.wmnet 117.32.192.10.in-addr.arpa 7.1.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 16:07 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:07 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2019 - jelto@cumin1002"
• 16:07 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2019 - jelto@cumin1002"
• 16:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 16:04 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 16:03 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2019
• 16:03 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2018
• 16:03 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2018
• 16:03 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2019.codfw.wmnet with OS bookworm
• 16:03 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2018.codfw.wmnet with OS bookworm
• 16:02 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2018-2019].codfw.wmnet
• 16:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71886 and previous config saved to /var/cache/conftool/dbconfig/20250108-160158-root.json
• 16:00 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2018-2019].codfw.wmnet
• 15:57 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
• 15:57 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
• 15:57 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 15:54 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
• 15:54 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
• 15:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 15:51 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 15:51 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 15:51 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 15:49 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 15:48 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 15:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71885 and previous config saved to /var/cache/conftool/dbconfig/20250108-154653-root.json
• 15:39 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 15:36 moritzm: installing jinja2 security updates
• 15:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71884 and previous config saved to /var/cache/conftool/dbconfig/20250108-153147-root.json
• 15:29 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 15:27 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 15:27 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 15:22 gengh@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 15:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 15:22 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 15:22 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 15:21 gengh@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 15:21 gengh@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 15:20 gengh@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 15:18 gengh@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 15:17 gengh@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 15:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71883 and previous config saved to /var/cache/conftool/dbconfig/20250108-151642-root.json
• 15:12 gengh@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 15:11 gengh@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 15:11 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 15:10 gengh@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 15:10 gengh@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 15:08 gengh@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 15:07 gengh@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 15:07 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 15:07 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 15:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 15:06 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 15:06 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 15:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71882 and previous config saved to /var/cache/conftool/dbconfig/20250108-150136-root.json
• 15:00 joelyrookewmde: Finished populateSitesTable for tigwiki (T381382)
• 14:53 elukey@cumin1002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2022.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:48 jelto@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 14:48 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repool pc5 (T373037)', diff saved to https://phabricator.wikimedia.org/P71881 and previous config saved to /var/cache/conftool/dbconfig/20250108-144805-ladsgroup.json
• 14:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71880 and previous config saved to /var/cache/conftool/dbconfig/20250108-144631-root.json
• 14:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 100%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71879 and previous config saved to /var/cache/conftool/dbconfig/20250108-144429-root.json
• 14:42 ladsgroup@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for pc1014.eqiad.wmnet
• 14:40 elukey: elukey@puppetserver1001:~$ sudo puppetserver ca clean --certname kubernetes1061.eqiad.wmnet
• 14:39 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 14:38 joelyrookewmde: joelyrookewmde@mwmaint2002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https
• 14:37 ladsgroup@cumin1002: START - Cookbook sre.mysql.upgrade for pc1014.eqiad.wmnet
• 14:33 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 14:33 elukey@cumin1002: START - Cookbook sre.hosts.provision for host wikikube-worker2022.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71878 and previous config saved to /var/cache/conftool/dbconfig/20250108-143126-root.json
• 14:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 75%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71877 and previous config saved to /var/cache/conftool/dbconfig/20250108-142923-root.json
• 14:23 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 14:21 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 14:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71876 and previous config saved to /var/cache/conftool/dbconfig/20250108-141620-root.json
• 14:14 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on pc2017.codfw.wmnet,pc[1014,1017].eqiad.wmnet with reason: Reboot
• 14:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 50%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71875 and previous config saved to /var/cache/conftool/dbconfig/20250108-141418-root.json
• 14:14 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on pc2017.codfw.wmnet,pc[1014,1017].eqiad.wmnet with reason: Reboot
• 14:12 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 14:12 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 14:12 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 14:11 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2020.codfw.wmnet
• 14:11 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2020.codfw.wmnet
• 14:11 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2021.codfw.wmnet
• 14:11 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2021.codfw.wmnet
• 14:09 jelto: sudo homer 'cr*codfw*' commit 'T377877'
• 14:09 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 14:07 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71874 and previous config saved to /var/cache/conftool/dbconfig/20250108-140115-root.json
• 13:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 25%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71873 and previous config saved to /var/cache/conftool/dbconfig/20250108-135913-root.json
• 13:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2228 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71872 and previous config saved to /var/cache/conftool/dbconfig/20250108-134610-root.json
• 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71871 and previous config saved to /var/cache/conftool/dbconfig/20250108-134544-root.json
• 13:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 10%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71870 and previous config saved to /var/cache/conftool/dbconfig/20250108-134408-root.json
• 13:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2021.codfw.wmnet with OS bookworm
• 13:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2020.codfw.wmnet with OS bookworm
• 13:37 elukey: elukey@puppetserver1001:~$ sudo puppetserver ca clean --certname kubernetes1021.eqiad.wmnet
• 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71869 and previous config saved to /var/cache/conftool/dbconfig/20250108-133038-root.json
• 13:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling pc5 in codfw for test (T373037)', diff saved to https://phabricator.wikimedia.org/P71868 and previous config saved to /var/cache/conftool/dbconfig/20250108-132708-ladsgroup.json
• 13:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling pc5 in eqiad for test (T373037)', diff saved to https://phabricator.wikimedia.org/P71867 and previous config saved to /var/cache/conftool/dbconfig/20250108-132506-ladsgroup.json
• 13:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2021.codfw.wmnet with reason: host reimage
• 13:22 ladsgroup@deploy2002: Finished scap sync-world: Backport for Fully depool ParserCache section if load of the primary is zero (T373037 T383137) (duration: 19m 19s)
• 13:19 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71866 and previous config saved to /var/cache/conftool/dbconfig/20250108-131953-root.json
• 13:19 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2021.codfw.wmnet with reason: host reimage
• 13:17 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2020.codfw.wmnet with reason: host reimage
• 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71865 and previous config saved to /var/cache/conftool/dbconfig/20250108-131533-root.json
• 13:14 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2020.codfw.wmnet with reason: host reimage
• 13:13 ladsgroup@deploy2002: ladsgroup: Continuing with sync
• 13:12 ladsgroup@deploy2002: ladsgroup: Backport for Fully depool ParserCache section if load of the primary is zero (T373037 T383137) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 13:04 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 75%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71864 and previous config saved to /var/cache/conftool/dbconfig/20250108-130448-root.json
• 13:03 ladsgroup@deploy2002: Started scap sync-world: Backport for Fully depool ParserCache section if load of the primary is zero (T373037 T383137)
• 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2021
• 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2021
• 13:01 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2021
• 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2021.codfw.wmnet 210.32.192.10.in-addr.arpa 0.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:01 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2021.codfw.wmnet 210.32.192.10.in-addr.arpa 0.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:01 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2021 - jelto@cumin1002"
• 13:01 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2021 - jelto@cumin1002"
• 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71863 and previous config saved to /var/cache/conftool/dbconfig/20250108-130028-root.json
• 12:57 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 12:57 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2021
• 12:57 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2020
• 12:56 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2020
• 12:56 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2020
• 12:56 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2020.codfw.wmnet 208.32.192.10.in-addr.arpa 8.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:56 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2020.codfw.wmnet 208.32.192.10.in-addr.arpa 8.0.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 12:56 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:56 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2020 - jelto@cumin1002"
• 12:56 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2020 - jelto@cumin1002"
• 12:53 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1086.eqiad.wmnet
• 12:53 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1086.eqiad.wmnet
• 12:52 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 12:52 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2020
• 12:52 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2020.codfw.wmnet with OS bookworm
• 12:52 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2021.codfw.wmnet with OS bookworm
• 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2020-2021].codfw.wmnet
• 12:50 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2020-2021].codfw.wmnet
• 12:50 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 12:49 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 50%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71862 and previous config saved to /var/cache/conftool/dbconfig/20250108-124943-root.json
• 12:49 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2023.codfw.wmnet
• 12:49 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2023.codfw.wmnet
• 12:48 mvolz@deploy2002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
• 12:48 mvolz@deploy2002: helmfile [eqiad] START helmfile.d/services/citoid: apply
• 12:47 mvolz@deploy2002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
• 12:47 mvolz@deploy2002: helmfile [codfw] START helmfile.d/services/citoid: apply
• 12:46 root@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2128.codfw.wmnet onto db2228.codfw.wmnet
• 12:45 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 12:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71861 and previous config saved to /var/cache/conftool/dbconfig/20250108-124522-root.json
• 12:41 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/citoid: apply
• 12:40 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/citoid: apply
• 12:39 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 12:34 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 25%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71860 and previous config saved to /var/cache/conftool/dbconfig/20250108-123437-root.json
• 12:30 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 12:30 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71859 and previous config saved to /var/cache/conftool/dbconfig/20250108-123017-root.json
• 12:21 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1086.eqiad.wmnet with OS bookworm
• 12:20 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 12:20 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 12:19 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 10%: Repooling after cloning', diff saved to https://phabricator.wikimedia.org/P71858 and previous config saved to /var/cache/conftool/dbconfig/20250108-121931-root.json
• 12:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71857 and previous config saved to /var/cache/conftool/dbconfig/20250108-121512-root.json
• 12:09 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1084-1085,1087].eqiad.wmnet
• 12:09 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1084-1085,1087].eqiad.wmnet
• 12:08 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 12:03 root@cumin1002: START - Cookbook sre.mysql.clone of db2128.codfw.wmnet onto db2228.codfw.wmnet
• 12:02 root@cumin1002: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2128.codfw.wmnet onto db2228.codfw.wmnet
• 12:02 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1086.eqiad.wmnet with reason: host reimage
• 12:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71856 and previous config saved to /var/cache/conftool/dbconfig/20250108-120006-root.json
• 11:59 root@cumin1002: START - Cookbook sre.mysql.clone of db2128.codfw.wmnet onto db2228.codfw.wmnet
• 11:59 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1086.eqiad.wmnet with reason: host reimage
• 11:59 marostegui@cumin1002: dbctl commit (dc=all): 'Add db2228 to dbctl depooled T373579', diff saved to https://phabricator.wikimedia.org/P71855 and previous config saved to /var/cache/conftool/dbconfig/20250108-115908-marostegui.json
• 11:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2128,2186].codfw.wmnet with reason: cloning
• 11:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db[2128,2186].codfw.wmnet with reason: cloning
• 11:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2128 T373579', diff saved to https://phabricator.wikimedia.org/P71854 and previous config saved to /var/cache/conftool/dbconfig/20250108-115206-marostegui.json
• 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71853 and previous config saved to /var/cache/conftool/dbconfig/20250108-114501-root.json
• 11:41 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1086
• 11:41 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1086
• 11:40 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1086.eqiad.wmnet with OS bookworm
• 11:40 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1087.eqiad.wmnet with OS bookworm
• 11:40 jayme@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1086.eqiad.wmnet with OS bookworm
• 11:35 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1085.eqiad.wmnet with OS bookworm
• 11:34 moritzm: installing php7.4 security updates
• 11:31 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1084.eqiad.wmnet with OS bookworm
• 11:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2226 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71852 and previous config saved to /var/cache/conftool/dbconfig/20250108-112956-root.json
• 11:28 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on ganeti2027.codfw.wmnet with reason: reimage pending, blocked by T383207
• 11:28 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on ganeti2027.codfw.wmnet with reason: reimage pending, blocked by T383207
• 11:26 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ganeti2027.codfw.wmnet with reason: reimage pending, blocked by T383207
• 11:26 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 3:00:00 on ganeti2027.codfw.wmnet with reason: reimage pending, blocked by T383207
• 11:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 11:25 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 11:25 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 11:20 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1087.eqiad.wmnet with reason: host reimage
• 11:18 jelto: sudo homer 'lsw1-c6-codfw*' commit 'T377877'
• 11:17 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1085.eqiad.wmnet with reason: host reimage
• 11:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 11:13 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 11:12 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1084.eqiad.wmnet with reason: host reimage
• 11:10 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1085.eqiad.wmnet with reason: host reimage
• 11:09 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1087.eqiad.wmnet with reason: host reimage
• 11:08 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1084.eqiad.wmnet with reason: host reimage
• 11:08 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-airflow1002.eqiad.wmnet with reason: Migrating to kubernetes
• 11:07 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on an-airflow1002.eqiad.wmnet with reason: Migrating to kubernetes
• 11:06 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2023.codfw.wmnet with OS bookworm
• 11:00 moritzm: installing graphviz bugfix updates from bookworm point release
• 10:54 moritzm: installing numpy bugfix updates from bookworm point release
• 10:52 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1085
• 10:52 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1085
• 10:52 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1085.eqiad.wmnet with OS bookworm
• 10:52 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1087
• 10:52 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1087
• 10:52 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1087.eqiad.wmnet with OS bookworm
• 10:51 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1086
• 10:51 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1086
• 10:51 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1086.eqiad.wmnet with OS bookworm
• 10:51 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1084
• 10:51 jayme@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1084
• 10:51 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1084.eqiad.wmnet with OS bookworm
• 10:50 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1084.eqiad.wmnet wikikube-worker1085.eqiad.wmnet wikikube-worker1086.eqiad.wmnet wikikube-worker1087.eqiad.wmnet on all recursors
• 10:50 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker1084.eqiad.wmnet wikikube-worker1085.eqiad.wmnet wikikube-worker1086.eqiad.wmnet wikikube-worker1087.eqiad.wmnet on all recursors
• 10:48 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1060 to wikikube-worker1085
• 10:47 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1085
• 10:47 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 10:47 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 10:47 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 10:46 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 10:46 moritzm: installing libnvme bugfix updates from bookworm point release
• 10:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2023.codfw.wmnet with reason: host reimage
• 10:44 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1085
• 10:44 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:43 moritzm: uploaded php7.4 1:7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2+icu67u4 (backport of latest PHP security fixes to our PHP build) T378173
• 10:42 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 10:42 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1059 to wikikube-worker1084
• 10:42 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1061 to wikikube-worker1086
• 10:42 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2023.codfw.wmnet with reason: host reimage
• 10:41 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1084
• 10:41 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1086
• 10:41 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1086
• 10:41 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:38 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 10:38 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1084
• 10:38 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:37 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from kubernetes1062 to wikikube-worker1087
• 10:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 10:37 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 10:37 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 10:37 jayme@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1087
• 10:37 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ganeti2027.codfw.wmnet with reason: reimage to bookworm
• 10:37 jayme@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1087
• 10:37 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:37 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1062 to wikikube-worker1087 - jayme@cumin1002"
• 10:36 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 3:00:00 on ganeti2027.codfw.wmnet with reason: reimage to bookworm
• 10:36 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming kubernetes1062 to wikikube-worker1087 - jayme@cumin1002"
• 10:36 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 10:36 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 10:34 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1267-1269].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 10:34 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1269.eqiad.wmnet with OS bookworm
• 10:31 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 10:31 jayme@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 10:30 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1062 to wikikube-worker1087
• 10:29 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1061 to wikikube-worker1086
• 10:29 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1060 to wikikube-worker1085
• 10:29 jayme@cumin1002: START - Cookbook sre.dns.netbox
• 10:28 jayme@cumin1002: START - Cookbook sre.hosts.rename from kubernetes1059 to wikikube-worker1084
• 10:24 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2022
• 10:24 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 10:24 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 100%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71850 and previous config saved to /var/cache/conftool/dbconfig/20250108-102416-root.json
• 10:24 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2023
• 10:24 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2023
• 10:23 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2023
• 10:23 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2023.codfw.wmnet 213.32.192.10.in-addr.arpa 3.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:23 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2023.codfw.wmnet 213.32.192.10.in-addr.arpa 3.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 10:23 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:23 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2023 - jelto@cumin1002"
• 10:23 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2023 - jelto@cumin1002"
• 10:21 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 10:20 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 10:19 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2023
• 10:19 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 10:19 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host wikikube-worker2022
• 10:19 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2022
• 10:19 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2023.codfw.wmnet with OS bookworm
• 10:19 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2022.codfw.wmnet with OS bookworm
• 10:15 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2022-2023].codfw.wmnet
• 10:15 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1269.eqiad.wmnet with reason: host reimage
• 10:14 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2022-2023].codfw.wmnet
• 10:12 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1269.eqiad.wmnet with reason: host reimage
• 10:09 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 75%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71849 and previous config saved to /var/cache/conftool/dbconfig/20250108-100910-root.json
• 10:04 moritzm: imported osmborder 0.1.0+wmf12u1 to apt.wikimedia.org/bookworm T381565
• 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet
• 10:02 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2024.codfw.wmnet
• 10:02 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2024.codfw.wmnet
• 10:02 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2025.codfw.wmnet
• 10:02 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2025.codfw.wmnet
• 10:00 jelto: sudo homer 'cr*codfw*' commit 'T377877'
• 10:00 jelto: sudo homer 'lsw1-c6-codfw*' commit 'T377877'
• 09:59 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2025.codfw.wmnet with OS bookworm
• 09:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2024.codfw.wmnet with OS bookworm
• 09:54 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 50%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71848 and previous config saved to /var/cache/conftool/dbconfig/20250108-095405-root.json
• 09:52 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1269.eqiad.wmnet with OS bookworm
• 09:50 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1268.eqiad.wmnet with OS bookworm
• 09:42 root@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2126.codfw.wmnet onto db2226.codfw.wmnet
• 09:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2025.codfw.wmnet with reason: host reimage
• 09:39 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 25%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71847 and previous config saved to /var/cache/conftool/dbconfig/20250108-093900-root.json
• 09:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2024.codfw.wmnet with reason: host reimage
• 09:31 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1268.eqiad.wmnet with reason: host reimage
• 09:30 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2024.codfw.wmnet with reason: host reimage
• 09:27 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2025.codfw.wmnet with reason: host reimage
• 09:25 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1268.eqiad.wmnet with reason: host reimage
• 09:23 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 10%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71846 and previous config saved to /var/cache/conftool/dbconfig/20250108-092354-root.json
• 09:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2024
• 09:13 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2024
• 09:13 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2024.codfw.wmnet with OS bookworm
• 09:12 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2024.codfw.wmnet with OS bookworm
• 09:10 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2025
• 09:10 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2025
• 09:10 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2025.codfw.wmnet with OS bookworm
• 09:08 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 5%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71845 and previous config saved to /var/cache/conftool/dbconfig/20250108-090849-root.json
• 09:08 jelto@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2025.codfw.wmnet with OS bookworm
• 09:05 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1268.eqiad.wmnet with OS bookworm
• 09:04 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1267.eqiad.wmnet with OS bookworm
• 08:53 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 4%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71844 and previous config saved to /var/cache/conftool/dbconfig/20250108-085344-root.json
• 08:44 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1267.eqiad.wmnet with reason: host reimage
• 08:42 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1267.eqiad.wmnet with reason: host reimage
• 08:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2024.codfw.wmnet
• 08:41 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:41 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2024.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:41 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2024.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:38 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 3%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71843 and previous config saved to /var/cache/conftool/dbconfig/20250108-083838-root.json
• 08:38 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet
• 08:37 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet
• 08:37 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 08:36 jmm@cumin2002: START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet
• 08:36 elukey: kill hanging processes on stat1011 to allow puppet to properly clean up absented users
• 08:32 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2024.codfw.wmnet
• 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2024 from dbctl T383028', diff saved to https://phabricator.wikimedia.org/P71842 and previous config saved to /var/cache/conftool/dbconfig/20250108-082807-marostegui.json
• 08:27 jmm@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Aitolkyn out of all services on: 2309 hosts
• 08:26 jmm@cumin2002: START - Cookbook sre.idm.logout Logging Aitolkyn out of all services on: 2309 hosts
• 08:23 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 2%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71841 and previous config saved to /var/cache/conftool/dbconfig/20250108-082333-root.json
• 08:23 elukey: destroy puppet cert for cloudelastic1011.eqiad.wmnet on puppetmaster1001 (cruft from old/wrong reimages)
• 08:21 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1267.eqiad.wmnet with OS bookworm
• 08:19 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1267-1269].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 08:08 marostegui@cumin1002: dbctl commit (dc=all): 'es1041 (re)pooling @ 1%: Repooling for the first time', diff saved to https://phabricator.wikimedia.org/P71840 and previous config saved to /var/cache/conftool/dbconfig/20250108-080828-root.json
• 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2024
• 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2024
• 07:51 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2024
• 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2024.codfw.wmnet 214.32.192.10.in-addr.arpa 4.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 07:51 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2024.codfw.wmnet 214.32.192.10.in-addr.arpa 4.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:51 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2024 - jelto@cumin1002"
• 07:51 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2024 - jelto@cumin1002"
• 07:50 moritzm: truncate /var/log/debug on seaborgium to unblock some disk space
• 07:49 root@cumin1002: START - Cookbook sre.mysql.clone of db2126.codfw.wmnet onto db2226.codfw.wmnet
• 07:48 marostegui@cumin1002: dbctl commit (dc=all): 'Add db2226 depooled', diff saved to https://phabricator.wikimedia.org/P71839 and previous config saved to /var/cache/conftool/dbconfig/20250108-074856-marostegui.json
• 07:47 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 07:47 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2024
• 07:47 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2025
• 07:47 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2025
• 07:47 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2025.codfw.wmnet with OS bookworm
• 07:47 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2024.codfw.wmnet with OS bookworm
• 07:45 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2024-2025].codfw.wmnet
• 07:44 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2024-2025].codfw.wmnet
• 07:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: cloning
• 07:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: cloning
• 07:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2126.codfw.wmnet with reason: cloning
• 07:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2126.codfw.wmnet with reason: cloning
• 07:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2126 T373579', diff saved to https://phabricator.wikimedia.org/P71838 and previous config saved to /var/cache/conftool/dbconfig/20250108-073603-marostegui.json
• 07:27 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 100%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71837 and previous config saved to /var/cache/conftool/dbconfig/20250108-072733-root.json
• 07:12 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 75%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71836 and previous config saved to /var/cache/conftool/dbconfig/20250108-071228-root.json
• 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 50%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71834 and previous config saved to /var/cache/conftool/dbconfig/20250108-065723-root.json
• 06:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy1021.eqiad.wmnet
• 06:53 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 06:53 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1021.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 06:53 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1021.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 06:50 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 06:44 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy1021.eqiad.wmnet
• 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 25%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71833 and previous config saved to /var/cache/conftool/dbconfig/20250108-064217-root.json
• 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 10%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71832 and previous config saved to /var/cache/conftool/dbconfig/20250108-062712-root.json
• 06:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1021.eqiad.wmnet with reason: cloning es1042
• 06:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1021.eqiad.wmnet with reason: cloning es1042
• 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'Add es1041 to dbctl depooled T382569', diff saved to https://phabricator.wikimedia.org/P71831 and previous config saved to /var/cache/conftool/dbconfig/20250108-062447-marostegui.json
• 06:19 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1021 T382569', diff saved to https://phabricator.wikimedia.org/P71830 and previous config saved to /var/cache/conftool/dbconfig/20250108-061928-marostegui.json
• 06:19 marostegui@cumin1002: dbctl commit (dc=all): 'Switchover es4 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71829 and previous config saved to /var/cache/conftool/dbconfig/20250108-061914-marostegui.json
• 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'es1020 (re)pooling @ 5%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71828 and previous config saved to /var/cache/conftool/dbconfig/20250108-061207-root.json

2025-01-07

• 23:58 eileen: civicrm upgraded from d8ec1e91 to 82b02ce5
• 23:49 urbanecm@deploy2002: Finished scap sync-world: Backport for [Growth] enwiki: Deploy Add Link to 5% of users (T382382) (duration: 13m 34s)
• 23:42 urbanecm@deploy2002: urbanecm: Continuing with sync
• 23:42 urbanecm@deploy2002: urbanecm: Backport for [Growth] enwiki: Deploy Add Link to 5% of users (T382382) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 23:36 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS bookworm
• 23:36 urbanecm@deploy2002: Started scap sync-world: Backport for [Growth] enwiki: Deploy Add Link to 5% of users (T382382)
• 23:35 cdanis@deploy2002: Finished scap sync-world: Backport for tracing: Disable tracing in CLI mode (T340552) (duration: 14m 23s)
• 23:27 cdanis@deploy2002: cdanis: Continuing with sync
• 23:27 cdanis@deploy2002: cdanis: Backport for tracing: Disable tracing in CLI mode (T340552) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 23:20 cdanis@deploy2002: Started scap sync-world: Backport for tracing: Disable tracing in CLI mode (T340552)
• 23:19 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
• 23:16 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
• 23:05 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 22:47 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 22:35 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 22:34 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 22:22 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 22:22 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 22:15 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 22:09 jhathaway@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm
• 22:02 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm
• 20:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 20:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 20:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 20:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 20:46 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on doc1003.eqiad.wmnet with reason: maintenance
• 20:45 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on doc1003.eqiad.wmnet with reason: maintenance
• 20:19 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 20:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 20:16 cdanis@deploy2002: Finished scap sync-world: Backport for group0: enable OpenTelemetry exports (T340552) (duration: 16m 06s)
• 20:15 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
• 20:15 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
• 20:08 cdanis@deploy2002: cdanis: Continuing with sync
• 20:07 cdanis@deploy2002: cdanis: Backport for group0: enable OpenTelemetry exports (T340552) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 20:00 cdanis@deploy2002: Started scap sync-world: Backport for group0: enable OpenTelemetry exports (T340552)
• 20:00 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 20:00 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 19:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 19:55 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 19:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 19:52 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-dump-rev-content-reconcile-enrich: apply
• 19:36 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.44.0-wmf.11 refs T382362
• 18:41 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 18:41 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 18:27 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: sync
• 18:27 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: sync
• 18:17 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 18:17 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 18:12 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8 days, 0:00:00 on ms-be2075.codfw.wmnet with reason: host is awaiting attention from Dell
• 18:11 mvernon@cumin1002: START - Cookbook sre.hosts.downtime for 8 days, 0:00:00 on ms-be2075.codfw.wmnet with reason: host is awaiting attention from Dell
• 17:50 ottomata: Disable varnish handling of /beacon/event to decommission eventlogging backend [puppet] - T238230 T353817
• 17:49 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
• 17:49 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
• 17:43 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
• 17:43 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
• 17:29 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
• 17:29 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
• 17:26 dzahn@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muhammad Jaziraly out of all services on: 4 hosts
• 17:26 dzahn@cumin2002: START - Cookbook sre.idm.logout Logging Muhammad Jaziraly out of all services on: 4 hosts
• 17:24 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
• 17:24 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
• 17:20 dzahn@cumin2002: END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Muhammad Jaziraly out of all services on: 2313 hosts
• 17:18 dzahn@cumin2002: START - Cookbook sre.idm.logout Logging Muhammad Jaziraly out of all services on: 2313 hosts
• 16:53 ejegg: fundraising civicrm upgraded from 64cfe3a0 to d8ec1e91
• 16:44 jayme: puppet ca destroy citoid.discovery.wmnet - T381474
• 16:44 jayme: puppet ca destroy mathoid.discovery.wmnet - T381474
• 16:44 jayme: puppet ca destroy termbox.discovery.wmnet - T381474
• 16:43 mutante: krb1001 - sudo manage_principals.py delete muja@WIKIMEDIA (T383056)
• 16:32 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 16:32 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 16:31 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
• 16:31 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
• 16:31 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
• 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
• 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
• 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
• 16:30 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
• 16:29 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox: apply
• 16:29 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 16:28 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 16:28 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
• 16:27 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
• 16:27 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 16:27 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 16:27 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
• 16:27 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
• 16:26 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
• 16:26 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
• 16:26 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
• 16:26 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox: apply
• 16:25 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
• 16:24 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-video: apply
• 16:24 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
• 16:24 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
• 16:24 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 16:23 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 16:23 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
• 16:23 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-media: apply
• 16:22 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
• 16:22 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
• 16:22 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox: apply
• 16:22 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox: apply
• 16:06 elukey: reloaded postgres config on puppetdb1003 to pick up new wal size settings
• 15:59 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2026.codfw.wmnet
• 15:59 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2026.codfw.wmnet
• 15:58 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2027.codfw.wmnet
• 15:58 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2027.codfw.wmnet
• 15:58 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2026.codfw.wmnet with OS bookworm
• 15:58 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
• 15:57 lucaswerkmeister-wmde@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
• 15:57 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
• 15:57 lucaswerkmeister-wmde@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
• 15:55 lucaswerkmeister-wmde@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
• 15:55 lucaswerkmeister-wmde@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
• 15:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2027.codfw.wmnet with OS bookworm
• 15:51 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1264-1266].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 15:51 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1266.eqiad.wmnet with OS bookworm
• 15:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2026.codfw.wmnet with reason: host reimage
• 15:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2027.codfw.wmnet with reason: host reimage
• 15:31 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1266.eqiad.wmnet with reason: host reimage
• 15:31 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1253-1255].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 15:31 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1255.eqiad.wmnet with OS bookworm
• 15:29 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2027.codfw.wmnet with reason: host reimage
• 15:29 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2026.codfw.wmnet with reason: host reimage
• 15:26 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1266.eqiad.wmnet with reason: host reimage
• 15:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: sync
• 15:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: sync
• 15:13 Lucas_WMDE: UTC afternoon backport+config window done
• 15:13 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for Enable AutoModerator on zhwiki (T367306), cswikivoyage: Change the wordmark v2 (T382779) (duration: 30m 38s)
• 15:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2026
• 15:11 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2026
• 15:11 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2027
• 15:11 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2027
• 15:11 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1255.eqiad.wmnet with reason: host reimage
• 15:11 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2026.codfw.wmnet with OS bookworm
• 15:11 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2027.codfw.wmnet with OS bookworm
• 15:09 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2026-2027].codfw.wmnet
• 15:08 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2026-2027].codfw.wmnet
• 15:08 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1255.eqiad.wmnet with reason: host reimage
• 15:07 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2028.codfw.wmnet
• 15:07 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2028.codfw.wmnet
• 15:07 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2029.codfw.wmnet
• 15:07 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2029.codfw.wmnet
• 15:05 lucaswerkmeister-wmde@deploy2002: zhaofjx, lucaswerkmeister-wmde: Continuing with sync
• 14:58 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2029.codfw.wmnet with OS bookworm
• 14:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2028.codfw.wmnet with OS bookworm
• 14:51 moritzm: installing intel-microcode security updates
• 14:49 lucaswerkmeister-wmde@deploy2002: zhaofjx, lucaswerkmeister-wmde: Backport for Enable AutoModerator on zhwiki (T367306), cswikivoyage: Change the wordmark v2 (T382779) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:42 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for Enable AutoModerator on zhwiki (T367306), cswikivoyage: Change the wordmark v2 (T382779)
• 14:37 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2029.codfw.wmnet with reason: host reimage
• 14:36 ladsgroup@deploy2002: Finished scap sync-world: Backport for Reactivate Parsoid+Kartographer on hewiki and commonswiki (T373454 T373460) (duration: 24m 56s)
• 14:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2028.codfw.wmnet with reason: host reimage
• 14:34 elukey@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
• 14:33 elukey@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
• 14:31 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2029.codfw.wmnet with reason: host reimage
• 14:31 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2028.codfw.wmnet with reason: host reimage
• 14:26 ladsgroup@deploy2002: ladsgroup, ihurbain: Continuing with sync
• 14:20 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1266.eqiad.wmnet with OS bookworm
• 14:20 ladsgroup@deploy2002: ladsgroup, ihurbain: Backport for Reactivate Parsoid+Kartographer on hewiki and commonswiki (T373454 T373460) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:18 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1265.eqiad.wmnet with OS bookworm
• 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet
• 14:16 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1255.eqiad.wmnet with OS bookworm
• 14:14 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1254.eqiad.wmnet with OS bookworm
• 14:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2029
• 14:13 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2029
• 14:13 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2028
• 14:13 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2028
• 14:13 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2029.codfw.wmnet with OS bookworm
• 14:13 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2028.codfw.wmnet with OS bookworm
• 14:12 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2028-2029].codfw.wmnet
• 14:11 ladsgroup@deploy2002: Started scap sync-world: Backport for Reactivate Parsoid+Kartographer on hewiki and commonswiki (T373454 T373460)
• 14:11 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet
• 14:10 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2028-2029].codfw.wmnet
• 14:08 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2031.codfw.wmnet
• 14:08 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2031.codfw.wmnet
• 14:08 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2030.codfw.wmnet
• 14:08 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2030.codfw.wmnet
• 14:02 elukey: re-enable puppet fleetwide after maintenance
• 13:59 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1265.eqiad.wmnet with reason: host reimage
• 13:59 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1265.eqiad.wmnet with reason: host reimage
• 13:57 elukey: start postgres and puppetdb on puppetdb2003
• 13:54 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on wikikube-worker1254.eqiad.wmnet with reason: host reimage
• 13:53 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1254.eqiad.wmnet with reason: host reimage
• 13:52 elukey@cumin1002: END (FAIL) - Cookbook sre.postgresql.postgres-init (exit_code=99)
• 13:41 elukey@cumin1002: START - Cookbook sre.postgresql.postgres-init
• 13:39 elukey: stop puppetdb and postgres on puppetdb2003 - T383114
• 13:39 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1265.eqiad.wmnet with OS bookworm
• 13:35 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on puppetdb2003.codfw.wmnet with reason: Resync postgres
• 13:35 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on puppetdb2003.codfw.wmnet with reason: Resync postgres
• 13:35 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1264.eqiad.wmnet with OS bookworm
• 13:33 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1254.eqiad.wmnet with OS bookworm
• 13:32 elukey: disable puppet fleetwide to allow maintenance on puppetdb2003
• 13:31 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1253.eqiad.wmnet with OS bookworm
• 13:30 moritzm: install gst-plugins-base1.0 security updates
• 13:30 Amir1: running mwscript purgeParserCache.php --wiki=aawiki --tag pc5 --age=2592000 --msleep 200 in *eqiad* (T382948)
• 13:15 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1264.eqiad.wmnet with reason: host reimage
• 13:12 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1253.eqiad.wmnet with reason: host reimage
• 13:09 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1264.eqiad.wmnet with reason: host reimage
• 13:08 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1253.eqiad.wmnet with reason: host reimage
• 12:49 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1264.eqiad.wmnet with OS bookworm
• 12:48 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1253.eqiad.wmnet with OS bookworm
• 12:47 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1264-1266].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 12:47 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2030.codfw.wmnet with OS bookworm
• 12:46 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1253-1255].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 12:44 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2031.codfw.wmnet with OS bookworm
• 12:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2030.codfw.wmnet with reason: host reimage
• 12:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2031.codfw.wmnet with reason: host reimage
• 12:20 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2030.codfw.wmnet with reason: host reimage
• 12:19 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2031.codfw.wmnet with reason: host reimage
• 12:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy1020.eqiad.wmnet
• 12:03 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:03 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1020.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 12:02 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy1020.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 12:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2030
• 12:02 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2030
• 12:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2031
• 12:02 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2031
• 12:01 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2030.codfw.wmnet with OS bookworm
• 12:01 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2031.codfw.wmnet with OS bookworm
• 12:00 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2030-2031].codfw.wmnet
• 11:59 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2030-2031].codfw.wmnet
• 11:58 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 11:58 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2032.codfw.wmnet
• 11:58 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2032.codfw.wmnet
• 11:57 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet
• 11:57 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet
• 11:56 jelto: sudo homer 'cr*codfw*' commit 'T377877'
• 11:55 jelto: sudo homer 'lsw1-c6-codfw*' commit 'T377877'
• 11:52 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy1020.eqiad.wmnet
• 11:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2032.codfw.wmnet with OS bookworm
• 11:46 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
• 11:46 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
• 11:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2033.codfw.wmnet with OS bookworm
• 11:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2032.codfw.wmnet with reason: host reimage
• 11:26 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2032.codfw.wmnet with reason: host reimage
• 11:25 moritzm: refreshed Ganeti internal cert for ulsfo (after adding a manual temp cert to unblock itself) T382873
• 11:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2033.codfw.wmnet with reason: host reimage
• 11:18 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2033.codfw.wmnet with reason: host reimage
• 11:13 hashar@deploy2002: Finished deploy [integration/docroot@5d32766]: Remove Flow from being listed on doc.wikimedia.org frontpage - T379671 (duration: 00m 11s)
• 11:13 hashar@deploy2002: Started deploy [integration/docroot@5d32766]: Remove Flow from being listed on doc.wikimedia.org frontpage - T379671
• 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2032
• 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2032
• 11:08 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2032
• 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2032.codfw.wmnet 215.32.192.10.in-addr.arpa 5.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:08 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2032.codfw.wmnet 215.32.192.10.in-addr.arpa 5.1.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:07 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2032 - jelto@cumin1002"
• 11:07 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2032 - jelto@cumin1002"
• 11:03 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 11:02 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2032
• 11:01 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2032.codfw.wmnet with OS bookworm
• 11:01 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2033
• 11:01 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2033
• 11:01 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2033.codfw.wmnet with OS bookworm
• 10:56 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2032-2033].codfw.wmnet
• 10:53 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2032-2033].codfw.wmnet
• 10:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet
• 10:52 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet
• 10:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet
• 10:51 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet
• 10:51 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ganeti[4005-4008].ulsfo.wmnet with reason: renew certs
• 10:50 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2035.codfw.wmnet with OS bookworm
• 10:50 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 3:00:00 on ganeti[4005-4008].ulsfo.wmnet with reason: renew certs
• 10:50 cmooney@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 269180
• 10:49 cmooney@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 269180
• 10:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2034.codfw.wmnet with OS bookworm
• 10:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2035.codfw.wmnet with reason: host reimage
• 10:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2034.codfw.wmnet with reason: host reimage
• 10:25 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2035.codfw.wmnet with reason: host reimage
• 10:23 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2034.codfw.wmnet with reason: host reimage
• 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2035
• 10:05 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2035
• 10:05 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2034
• 10:05 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2034
• 10:05 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2034.codfw.wmnet with OS bookworm
• 10:05 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2035.codfw.wmnet with OS bookworm
• 10:05 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet
• 10:03 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2034-2035].codfw.wmnet
• 10:02 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2034-2035].codfw.wmnet
• 10:01 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet
• 10:01 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet
• 10:01 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet
• 10:01 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet
• 10:01 cmooney@cumin1002: START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet
• 09:51 jelto: sudo homer 'cr*codfw*' commit 'T377877'
• 09:50 jelto: sudo homer 'lsw1-c1-codfw*' commit 'T377877'
• 09:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2037.codfw.wmnet with OS bookworm
• 09:40 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2036.codfw.wmnet with OS bookworm
• 09:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2037.codfw.wmnet with reason: host reimage
• 09:22 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2037.codfw.wmnet with reason: host reimage
• 09:20 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2036.codfw.wmnet with reason: host reimage
• 09:17 moritzm: installing systemd bugfix updates from Bookworm point release
• 09:16 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2036.codfw.wmnet with reason: host reimage
• 09:08 moritzm: installing Java 21.0.5 security updates
• 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2037
• 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2037
• 09:02 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2037
• 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2037.codfw.wmnet 146.32.192.10.in-addr.arpa 6.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:02 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2037.codfw.wmnet 146.32.192.10.in-addr.arpa 6.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:02 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2037 - jelto@cumin1002"
• 09:02 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2037 - jelto@cumin1002"
• 08:59 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 08:58 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2037
• 08:57 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2037.codfw.wmnet with OS bookworm
• 08:57 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2036
• 08:57 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2036
• 08:56 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2036.codfw.wmnet with OS bookworm
• 08:54 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2036-2037].codfw.wmnet
• 08:53 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2036-2037].codfw.wmnet
• 08:48 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet
• 08:48 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet
• 08:48 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet
• 08:48 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet
• 08:47 jelto: sudo homer 'lsw1-c5-codfw*' commit 'T377877'
• 08:44 jelto: sudo homer 'cr*codfw*' commit 'T377877'
• 08:44 jelto: sudo homer 'lsw1-c1-codfw*' commit 'T377877'
• 08:43 jelto: sudo homer 'lsw1-c7-codfw*' commit 'T377877'
• 08:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on es1020.eqiad.wmnet with reason: cloning es1041
• 08:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on es1020.eqiad.wmnet with reason: cloning es1041
• 08:39 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1020 T382569', diff saved to https://phabricator.wikimedia.org/P71822 and previous config saved to /var/cache/conftool/dbconfig/20250107-083930-marostegui.json
• 08:38 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2038.codfw.wmnet with OS bookworm
• 08:38 marostegui@cumin1002: dbctl commit (dc=all): 'Switchover es5 eqiad master dbmaint T382569', diff saved to https://phabricator.wikimedia.org/P71821 and previous config saved to /var/cache/conftool/dbconfig/20250107-083811-marostegui.json
• 08:26 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2039.codfw.wmnet with OS bookworm
• 08:20 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2025.codfw.wmnet
• 08:20 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:20 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2025.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:19 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2025.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:17 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2038.codfw.wmnet with reason: host reimage
• 08:14 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 08:13 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2038.codfw.wmnet with reason: host reimage
• 08:09 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2025.codfw.wmnet
• 08:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2039.codfw.wmnet with reason: host reimage
• 08:03 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2039.codfw.wmnet with reason: host reimage
• 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2038
• 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2038
• 07:53 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2038
• 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2038.codfw.wmnet 147.32.192.10.in-addr.arpa 7.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 07:53 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2038.codfw.wmnet 147.32.192.10.in-addr.arpa 7.4.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:53 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2038 - jelto@cumin1002"
• 07:53 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2038 - jelto@cumin1002"
• 07:49 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 07:49 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2038
• 07:49 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2038.codfw.wmnet with OS bookworm
• 07:43 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2039
• 07:43 jelto@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2039
• 07:43 jelto@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2039
• 07:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker2039.codfw.wmnet 150.32.192.10.in-addr.arpa 0.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 07:43 jelto@cumin1002: START - Cookbook sre.dns.wipe-cache wikikube-worker2039.codfw.wmnet 150.32.192.10.in-addr.arpa 0.5.1.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 07:43 jelto@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:42 jelto@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2039 - jelto@cumin1002"
• 07:41 jelto@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker2039 - jelto@cumin1002"
• 07:37 jelto@cumin1002: START - Cookbook sre.dns.netbox
• 07:37 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2039
• 07:37 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2039.codfw.wmnet with OS bookworm
• 07:33 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2038-2039].codfw.wmnet
• 07:32 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2038-2039].codfw.wmnet
• 06:50 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2025 from dbctl T381848', diff saved to https://phabricator.wikimedia.org/P71820 and previous config saved to /var/cache/conftool/dbconfig/20250107-064958-marostegui.json
• 05:40 marostegui: Switchover m5 eqiad proxy dbmaint eqiad T368874
• 04:51 mwpresync@deploy2002: Finished scap sync-world: testwikis to 1.44.0-wmf.11 refs T382362 (duration: 48m 35s)
• 04:02 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.44.0-wmf.11 refs T382362
• 00:24 kamila@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1257-1263].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 00:24 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1263.eqiad.wmnet with OS bookworm
• 00:05 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1263.eqiad.wmnet with reason: host reimage
• 00:01 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1263.eqiad.wmnet with reason: host reimage

2025-01-06

• 23:41 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1263.eqiad.wmnet with OS bookworm
• 23:39 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1262.eqiad.wmnet with OS bookworm
• 23:19 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1262.eqiad.wmnet with reason: host reimage
• 23:16 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1262.eqiad.wmnet with reason: host reimage
• 22:56 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1262.eqiad.wmnet with OS bookworm
• 22:55 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS bookworm
• 22:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage
• 22:33 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage
• 22:13 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS bookworm
• 22:10 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1260.eqiad.wmnet with OS bookworm
• 21:49 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1260.eqiad.wmnet with reason: host reimage
• 21:45 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1260.eqiad.wmnet with reason: host reimage
• 21:39 cjming: end of UTC late backport window
• 21:31 cjming@deploy2002: Finished scap sync-world: Backport for Move logic for type infering to server (T382042) (duration: 19m 31s)
• 21:25 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1260.eqiad.wmnet with OS bookworm
• 21:25 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1259.eqiad.wmnet with OS bookworm
• 21:22 cjming@deploy2002: cjming, jdlrobson: Continuing with sync
• 21:16 cjming@deploy2002: cjming, jdlrobson: Backport for Move logic for type infering to server (T382042) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:12 cjming@deploy2002: Started scap sync-world: Backport for Move logic for type infering to server (T382042)
• 21:06 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1259.eqiad.wmnet with reason: host reimage
• 21:02 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1259.eqiad.wmnet with reason: host reimage
• 20:49 eileen: tools upgraded from c7b53ecd to 6ddfb22f
• 20:41 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1259.eqiad.wmnet with OS bookworm
• 20:39 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1258.eqiad.wmnet with OS bookworm
• 20:24 brett: running authdns-update for CR 1108142
• 20:19 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1258.eqiad.wmnet with reason: host reimage
• 20:15 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1258.eqiad.wmnet with reason: host reimage
• 19:55 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1258.eqiad.wmnet with OS bookworm
• 19:55 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1257.eqiad.wmnet with OS bookworm
• 19:36 kamila@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1257.eqiad.wmnet with reason: host reimage
• 19:32 kamila@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1257.eqiad.wmnet with reason: host reimage
• 19:11 kamila@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1257.eqiad.wmnet with OS bookworm
• 19:09 kamila@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1257-1263].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 18:47 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1244.eqiad.wmnet
• 18:47 jayme@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1244.eqiad.wmnet
• 18:32 ChrisDobbins901_: cdobbins@dns1004 running authdns-update for CR 1097521
• 18:31 ChrisDobbins901_: cdobbins@cumin1002 running authdns-update for CR 1097521
• 17:59 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1244.eqiad.wmnet with OS bookworm
• 17:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T370903)', diff saved to https://phabricator.wikimedia.org/P71818 and previous config saved to /var/cache/conftool/dbconfig/20250106-174024-ladsgroup.json
• 17:39 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1244.eqiad.wmnet with reason: host reimage
• 17:37 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1244.eqiad.wmnet with reason: host reimage
• 17:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P71817 and previous config saved to /var/cache/conftool/dbconfig/20250106-172517-ladsgroup.json
• 17:16 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1244.eqiad.wmnet with OS bookworm
• 17:15 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1250-1252].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 17:15 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1252.eqiad.wmnet with OS bookworm
• 17:15 jayme@cumin1002: END (FAIL) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=1) rolling reimage on P{wikikube-worker[1240-1244].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 17:15 jayme@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wikikube-worker1244.eqiad.wmnet with OS bookworm
• 17:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P71816 and previous config saved to /var/cache/conftool/dbconfig/20250106-171010-ladsgroup.json
• 17:00 jdrewniak@deploy2002: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 02m 43s)
• 16:58 jdrewniak@deploy2002: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 12m 29s)
• 16:56 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1252.eqiad.wmnet with reason: host reimage
• 16:55 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 16:55 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T370903)', diff saved to https://phabricator.wikimedia.org/P71815 and previous config saved to /var/cache/conftool/dbconfig/20250106-165503-ladsgroup.json
• 16:54 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 16:54 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 16:54 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 16:54 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 16:53 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
• 16:52 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1252.eqiad.wmnet with reason: host reimage
• 16:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T370903)', diff saved to https://phabricator.wikimedia.org/P71813 and previous config saved to /var/cache/conftool/dbconfig/20250106-164215-ladsgroup.json
• 16:42 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 16:42 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 16:39 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 16:38 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 16:37 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 15:51 moritzm: uploaded openjdk-21 21.0.5+11-1~deb12u1 to apt.wikimedia.org component/jdk21
• 15:39 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2143.codfw.wmnet with reason: onsite maintenance
• 15:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2143.codfw.wmnet with reason: onsite maintenance
• 15:31 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet
• 15:31 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet
• 15:31 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2040.codfw.wmnet
• 15:31 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2040.codfw.wmnet
• 15:29 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2040.codfw.wmnet with OS bookworm
• 15:27 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1252.eqiad.wmnet with OS bookworm
• 15:25 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1251.eqiad.wmnet with OS bookworm
• 15:23 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1244.eqiad.wmnet with OS bookworm
• 15:23 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS bookworm
• 15:13 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 15:12 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 15:12 Lucas_WMDE: UTC afternoon backport+config window done
• 15:11 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for Remove EntitySchema DataType feature flag - is always enabled (T333667) (duration: 13m 25s)
• 15:09 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2040.codfw.wmnet with reason: host reimage
• 15:06 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1251.eqiad.wmnet with reason: host reimage
• 15:04 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, arthurtaylor: Continuing with sync
• 15:03 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage
• 15:02 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, arthurtaylor: Backport for Remove EntitySchema DataType feature flag - is always enabled (T333667) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 15:00 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1251.eqiad.wmnet with reason: host reimage
• 15:00 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T371742)', diff saved to https://phabricator.wikimedia.org/P71811 and previous config saved to /var/cache/conftool/dbconfig/20250106-150040-ladsgroup.json
• 14:58 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2040.codfw.wmnet with reason: host reimage
• 14:58 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage
• 14:58 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for Remove EntitySchema DataType feature flag - is always enabled (T333667)
• 14:56 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for Add bfw, gju-arab, gju-deva, hoc and kgg to wmgExtraLanguageNames (T381934) (duration: 17m 11s)
• 14:49 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, jhsoby: Continuing with sync
• 14:46 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, jhsoby: Backport for Add bfw, gju-arab, gju-deva, hoc and kgg to wmgExtraLanguageNames (T381934) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:45 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P71810 and previous config saved to /var/cache/conftool/dbconfig/20250106-144534-ladsgroup.json
• 14:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2040
• 14:40 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2040
• 14:40 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2041
• 14:40 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2041
• 14:40 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2040.codfw.wmnet with OS bookworm
• 14:40 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS bookworm
• 14:40 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1251.eqiad.wmnet with OS bookworm
• 14:39 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for Add bfw, gju-arab, gju-deva, hoc and kgg to wmgExtraLanguageNames (T381934)
• 14:39 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2040-2041].codfw.wmnet
• 14:38 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1250.eqiad.wmnet with OS bookworm
• 14:38 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for ExtensionDistributor: Mark 1.43 as stable; remove 1.41 as EOL (T372331 T376550) (duration: 14m 14s)
• 14:35 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2040-2041].codfw.wmnet
• 14:34 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2042.codfw.wmnet
• 14:34 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet
• 14:34 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2043.codfw.wmnet
• 14:34 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2043.codfw.wmnet
• 14:33 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2042.codfw.wmnet with OS bookworm
• 14:33 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1243.eqiad.wmnet with OS bookworm
• 14:30 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2043.codfw.wmnet with OS bookworm
• 14:30 lucaswerkmeister-wmde@deploy2002: macfan4000, lucaswerkmeister-wmde: Continuing with sync
• 14:30 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P71809 and previous config saved to /var/cache/conftool/dbconfig/20250106-143027-ladsgroup.json
• 14:28 moritzm: installing libvirt bugfix updates
• 14:27 lucaswerkmeister-wmde@deploy2002: macfan4000, lucaswerkmeister-wmde: Backport for ExtensionDistributor: Mark 1.43 as stable; remove 1.41 as EOL (T372331 T376550) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:23 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for ExtensionDistributor: Mark 1.43 as stable; remove 1.41 as EOL (T372331 T376550)
• 14:22 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for Add mergehistory to import and transwiki on en.wikibooks (T382785), Add suppressredirect and delete-redirect to en.wikinews reviewers (T382887) (duration: 15m 02s)
• 14:18 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1250.eqiad.wmnet with reason: host reimage
• 14:15 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, dreamrimmer: Continuing with sync
• 14:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T371742)', diff saved to https://phabricator.wikimedia.org/P71808 and previous config saved to /var/cache/conftool/dbconfig/20250106-141520-ladsgroup.json
• 14:14 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2042.codfw.wmnet with reason: host reimage
• 14:12 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, dreamrimmer: Backport for Add mergehistory to import and transwiki on en.wikibooks (T382785), Add suppressredirect and delete-redirect to en.wikinews reviewers (T382887) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 14:12 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1250.eqiad.wmnet with reason: host reimage
• 14:10 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2043.codfw.wmnet with reason: host reimage
• 14:10 marostegui: Deploy schema change on x1 dbmaint codfw T383052
• 14:08 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2042.codfw.wmnet with reason: host reimage
• 14:07 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2043.codfw.wmnet with reason: host reimage
• 14:07 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for Add mergehistory to import and transwiki on en.wikibooks (T382785), Add suppressredirect and delete-redirect to en.wikinews reviewers (T382887)
• 14:04 marostegui: Deploy schema change on x1 dbmaint eqiad T383052
• 13:51 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1250.eqiad.wmnet with OS bookworm
• 13:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2042
• 13:49 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2042
• 13:49 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2043
• 13:49 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2043
• 13:49 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2042.codfw.wmnet with OS bookworm
• 13:49 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2043.codfw.wmnet with OS bookworm
• 13:47 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1250-1252].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 13:47 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2042-2043].codfw.wmnet
• 13:46 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2042-2043].codfw.wmnet
• 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet
• 13:40 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet
• 13:40 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2045.codfw.wmnet
• 13:40 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2045.codfw.wmnet
• 13:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2045.codfw.wmnet with OS bookworm
• 13:34 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2044.codfw.wmnet with OS bookworm
• 13:18 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2045.codfw.wmnet with reason: host reimage
• 13:15 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2044.codfw.wmnet with reason: host reimage
• 13:13 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1243.eqiad.wmnet with OS bookworm
• 13:12 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2045.codfw.wmnet with reason: host reimage
• 13:11 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2044.codfw.wmnet with reason: host reimage
• 12:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2044
• 12:55 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2044
• 12:54 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2045
• 12:54 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2045
• 12:54 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2044.codfw.wmnet with OS bookworm
• 12:54 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2045.codfw.wmnet with OS bookworm
• 12:54 ladsgroup@deploy2002: Finished scap sync-world: Backport for ParserCache: Set connect and recieve timeouts (T378076 T373037) (duration: 13m 39s)
• 12:53 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2044-2045].codfw.wmnet
• 12:52 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2044-2045].codfw.wmnet
• 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2046.codfw.wmnet
• 12:51 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet
• 12:51 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2048.codfw.wmnet
• 12:51 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2048.codfw.wmnet
• 12:48 ladsgroup@deploy2002: ladsgroup: Continuing with sync
• 12:46 ladsgroup@deploy2002: ladsgroup: Backport for ParserCache: Set connect and recieve timeouts (T378076 T373037) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 12:40 ladsgroup@deploy2002: Started scap sync-world: Backport for ParserCache: Set connect and recieve timeouts (T378076 T373037)
• 12:37 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1217.eqiad.wmnet with reason: upgrade kernel
• 12:37 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on db1217.eqiad.wmnet with reason: upgrade kernel
• 12:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T371742)', diff saved to https://phabricator.wikimedia.org/P71807 and previous config saved to /var/cache/conftool/dbconfig/20250106-123416-ladsgroup.json
• 12:34 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 12:34 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 12:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T371742)', diff saved to https://phabricator.wikimedia.org/P71806 and previous config saved to /var/cache/conftool/dbconfig/20250106-123405-ladsgroup.json
• 12:30 jayme@cumin1002: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker[1245-1249].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 12:30 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1249.eqiad.wmnet with OS bookworm
• 12:27 Emperor: swift post wikipedia-commons-local-thumb.f8 --read-acl 'mw:thumbor,mw:media,.r:*' --write-acl 'mw:thumbor,mw:media' ms-fe2009 per T383034
• 12:18 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P71805 and previous config saved to /var/cache/conftool/dbconfig/20250106-121858-ladsgroup.json
• 12:14 jayme@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1243.eqiad.wmnet with OS bookworm
• 12:11 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1249.eqiad.wmnet with reason: host reimage
• 12:07 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1249.eqiad.wmnet with reason: host reimage
• 12:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2023.codfw.wmnet
• 12:04 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:04 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2023.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 12:04 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2023.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 12:03 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P71804 and previous config saved to /var/cache/conftool/dbconfig/20250106-120351-ladsgroup.json
• 12:01 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 11:56 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2023.codfw.wmnet
• 11:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2046.codfw.wmnet with OS bookworm
• 11:52 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2048.codfw.wmnet with OS bookworm
• 11:48 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T371742)', diff saved to https://phabricator.wikimedia.org/P71803 and previous config saved to /var/cache/conftool/dbconfig/20250106-114844-ladsgroup.json
• 11:47 moritzm: fix /etc/network/interfaces on doc2002 T382610
• 11:47 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1249.eqiad.wmnet with OS bookworm
• 11:45 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1248.eqiad.wmnet with OS bookworm
• 11:35 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2046.codfw.wmnet with reason: host reimage
• 11:32 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2048.codfw.wmnet with reason: host reimage
• 11:28 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2046.codfw.wmnet with reason: host reimage
• 11:26 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2048.codfw.wmnet with reason: host reimage
• 11:24 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1248.eqiad.wmnet with reason: host reimage
• 11:20 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1248.eqiad.wmnet with reason: host reimage
• 11:16 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 11:16 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2048
• 11:08 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2046
• 11:08 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2046
• 11:08 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2048
• 11:08 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2046.codfw.wmnet with OS bookworm
• 11:08 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2048.codfw.wmnet with OS bookworm
• 11:07 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2046,2048].codfw.wmnet
• 11:06 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2046,2048].codfw.wmnet
• 11:00 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1248.eqiad.wmnet with OS bookworm
• 10:58 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1247.eqiad.wmnet with OS bookworm
• 10:52 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1243.eqiad.wmnet with OS bookworm
• 10:51 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1242.eqiad.wmnet with OS bookworm
• 10:49 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2050.codfw.wmnet
• 10:49 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet
• 10:49 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet
• 10:49 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet
• 10:39 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1247.eqiad.wmnet with reason: host reimage
• 10:39 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2049.codfw.wmnet with OS bookworm
• 10:36 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2050.codfw.wmnet with OS bookworm
• 10:35 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1247.eqiad.wmnet with reason: host reimage
• 10:32 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1242.eqiad.wmnet with reason: host reimage
• 10:28 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1242.eqiad.wmnet with reason: host reimage
• 10:19 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2049.codfw.wmnet with reason: host reimage
• 10:16 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2050.codfw.wmnet with reason: host reimage
• 10:15 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1247.eqiad.wmnet with OS bookworm
• 10:15 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2049.codfw.wmnet with reason: host reimage
• 10:13 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1246.eqiad.wmnet with OS bookworm
• 10:13 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 10:13 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 10:13 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 10:13 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2050.codfw.wmnet with reason: host reimage
• 10:12 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 10:12 cgoubert@deploy2002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 10:11 cgoubert@deploy2002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 10:11 cgoubert@deploy2002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 10:10 cgoubert@deploy2002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 10:10 cgoubert@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 10:09 cgoubert@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 10:09 cgoubert@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 10:09 cgoubert@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 10:08 cgoubert@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 10:08 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1242.eqiad.wmnet with OS bookworm
• 10:08 cgoubert@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 10:07 cgoubert@deploy2002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
• 10:07 cgoubert@deploy2002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
• 10:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T371742)', diff saved to https://phabricator.wikimedia.org/P71801 and previous config saved to /var/cache/conftool/dbconfig/20250106-100706-ladsgroup.json
• 10:07 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 10:06 claime: Deploying admin_ng external services changes on all kubernetes clusters
• 10:06 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 10:06 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 10:06 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 10:06 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1241.eqiad.wmnet with OS bookworm
• 09:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2050
• 09:55 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2050
• 09:55 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2049
• 09:55 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2049
• 09:55 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2050.codfw.wmnet with OS bookworm
• 09:55 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2049.codfw.wmnet with OS bookworm
• 09:54 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2049-2050].codfw.wmnet
• 09:54 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1246.eqiad.wmnet with reason: host reimage
• 09:53 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2049-2050].codfw.wmnet
• 09:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2052.codfw.wmnet
• 09:52 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2052.codfw.wmnet
• 09:52 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2051.codfw.wmnet
• 09:52 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet
• 09:51 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2052.codfw.wmnet with OS bookworm
• 09:50 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1246.eqiad.wmnet with reason: host reimage
• 09:47 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1241.eqiad.wmnet with reason: host reimage
• 09:46 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2051.codfw.wmnet with OS bookworm
• 09:43 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1241.eqiad.wmnet with reason: host reimage
• 09:41 dcausse: repooling wdqs1012
• 09:31 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2052.codfw.wmnet with reason: host reimage
• 09:29 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1246.eqiad.wmnet with OS bookworm
• 09:27 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2051.codfw.wmnet with reason: host reimage
• 09:26 dcausse: depooling wdqs1012 (high lag, forgot to keep it depooled after restarting blazegraph)
• 09:26 marostegui: Reboot db2160 for kernel upgrade T376905
• 09:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2160.codfw.wmnet with reason: upgrade kernel
• 09:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on db2160.codfw.wmnet with reason: upgrade kernel
• 09:25 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1245.eqiad.wmnet with OS bookworm
• 09:24 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2052.codfw.wmnet with reason: host reimage
• 09:23 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2051.codfw.wmnet with reason: host reimage
• 09:23 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1241.eqiad.wmnet with OS bookworm
• 09:21 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1240.eqiad.wmnet with OS bookworm
• 09:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2052
• 09:06 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2052
• 09:06 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2051
• 09:06 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2051
• 09:06 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2052.codfw.wmnet with OS bookworm
• 09:06 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2051.codfw.wmnet with OS bookworm
• 09:06 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1245.eqiad.wmnet with reason: host reimage
• 09:05 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2051-2052].codfw.wmnet
• 09:04 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2051-2052].codfw.wmnet
• 09:03 awight: UTC morning deployment finished
• 09:02 awight@deploy2002: Finished scap sync-world: Backport for bjnwikiquote: add wordmark (T382777) (duration: 11m 59s)
• 09:02 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1240.eqiad.wmnet with reason: host reimage
• 09:00 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1245.eqiad.wmnet with reason: host reimage
• 08:58 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1240.eqiad.wmnet with reason: host reimage
• 08:57 awight@deploy2002: awight, anzx: Continuing with sync
• 08:55 awight@deploy2002: awight, anzx: Backport for bjnwikiquote: add wordmark (T382777) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 08:50 awight@deploy2002: Started scap sync-world: Backport for bjnwikiquote: add wordmark (T382777)
• 08:46 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2053.codfw.wmnet
• 08:46 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2053.codfw.wmnet
• 08:46 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2054.codfw.wmnet
• 08:46 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2054.codfw.wmnet
• 08:46 awight@deploy2002: Finished scap sync-world: Backport for Change license on ptwikinews, nlwikinews and rowikinews to cc-by-4.0 (T382649) (duration: 09m 28s)
• 08:44 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker[2053-2054].codfw.wmnet
• 08:44 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2053-2054].codfw.wmnet
• 08:43 jelto@cumin1002: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker[2053-2054].codfw.wmnet
• 08:43 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2053-2054].codfw.wmnet
• 08:42 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2053.codfw.wmnet with OS bookworm
• 08:41 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2054.codfw.wmnet with OS bookworm
• 08:41 awight@deploy2002: dreamrimmer, awight: Continuing with sync
• 08:41 awight@deploy2002: dreamrimmer, awight: Backport for Change license on ptwikinews, nlwikinews and rowikinews to cc-by-4.0 (T382649) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 08:39 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1245.eqiad.wmnet with OS bookworm
• 08:38 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker1240.eqiad.wmnet with OS bookworm
• 08:37 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1245-1249].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 08:37 jayme@cumin1002: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker[1240-1244].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 08:36 awight@deploy2002: Started scap sync-world: Backport for Change license on ptwikinews, nlwikinews and rowikinews to cc-by-4.0 (T382649)
• 08:29 awight@deploy2002: awight, dreamrimmer: Backport for Change license on ptwikinews, nlwikinews and rowikinews to cc-by-4.0 (T382649) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 08:25 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2053.codfw.wmnet with reason: host reimage
• 08:24 awight@deploy2002: Started scap sync-world: Backport for Change license on ptwikinews, nlwikinews and rowikinews to cc-by-4.0 (T382649)
• 08:22 awight@deploy2002: Finished scap sync-world: Backport for [arwiki] Add templateeditor user group (T382784) (duration: 14m 48s)
• 08:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2054.codfw.wmnet with reason: host reimage
• 08:18 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2053.codfw.wmnet with reason: host reimage
• 08:18 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2054.codfw.wmnet with reason: host reimage
• 08:18 dcausse: restarting blazegraph on wdqs1012 (stuck with high thread count)
• 08:17 awight@deploy2002: awight, hubaishan: Continuing with sync
• 08:15 dcausse: restarting blazegraph on wdqs1014 (BlazegraphFreeAllocatorsDecreasingRapidly)
• 08:13 awight@deploy2002: awight, hubaishan: Backport for [arwiki] Add templateeditor user group (T382784) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 08:13 moritzm: installing fastnetmon security updates
• 08:08 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2024 and es2025 T381848', diff saved to https://phabricator.wikimedia.org/P71798 and previous config saved to /var/cache/conftool/dbconfig/20250106-080845-marostegui.json
• 08:07 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2023 from dbctl and promote es2046 to es5 master T381848 T383026', diff saved to https://phabricator.wikimedia.org/P71797 and previous config saved to /var/cache/conftool/dbconfig/20250106-080755-marostegui.json
• 08:07 awight@deploy2002: Started scap sync-world: Backport for [arwiki] Add templateeditor user group (T382784)
• 08:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2024 T381848', diff saved to https://phabricator.wikimedia.org/P71796 and previous config saved to /var/cache/conftool/dbconfig/20250106-080609-marostegui.json
• 08:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2023 T383026', diff saved to https://phabricator.wikimedia.org/P71795 and previous config saved to /var/cache/conftool/dbconfig/20250106-080405-marostegui.json
• 08:00 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2054
• 08:00 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2054
• 08:00 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker2053
• 08:00 jelto@cumin1002: START - Cookbook sre.hosts.move-vlan for host wikikube-worker2053
• 08:00 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2054.codfw.wmnet with OS bookworm
• 08:00 jelto@cumin1002: START - Cookbook sre.hosts.reimage for host wikikube-worker2053.codfw.wmnet with OS bookworm
• 07:59 jelto@cumin1002: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2053-2054].codfw.wmnet
• 07:55 jelto@cumin1002: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2053-2054].codfw.wmnet
• 07:29 moritzm: installing systemd bugfix updates from Bookworm point release
• 07:13 marostegui: dbmaint Switchover m3 (phabricator) eqiad master dbproxy1020 -> dbproxy1028 T368874
• 06:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2022.codfw.wmnet
• 06:54 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 06:54 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2022.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 06:54 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2022.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 06:51 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 06:50 marostegui@cumin1002: dbctl commit (dc=all): 'db1193 (re)pooling @ 100%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P71794 and previous config saved to /var/cache/conftool/dbconfig/20250106-065050-root.json
• 06:46 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2022.codfw.wmnet
• 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2022 from dbctl T382946', diff saved to https://phabricator.wikimedia.org/P71792 and previous config saved to /var/cache/conftool/dbconfig/20250106-063940-marostegui.json
• 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'db1193 (re)pooling @ 75%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P71791 and previous config saved to /var/cache/conftool/dbconfig/20250106-063545-root.json
• 06:27 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2020.codfw.wmnet
• 06:27 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 06:27 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 06:27 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2020.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 06:23 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'db1193 (re)pooling @ 50%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P71790 and previous config saved to /var/cache/conftool/dbconfig/20250106-062040-root.json
• 06:19 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2020.codfw.wmnet
• 06:18 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2020 from dbctl T382945', diff saved to https://phabricator.wikimedia.org/P71789 and previous config saved to /var/cache/conftool/dbconfig/20250106-061832-marostegui.json
• 06:08 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es2021.codfw.wmnet
• 06:08 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 06:08 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2021.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 06:08 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es2021.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'db1193 (re)pooling @ 25%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P71787 and previous config saved to /var/cache/conftool/dbconfig/20250106-060534-root.json
• 06:04 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 06:00 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts es2021.codfw.wmnet
• 05:57 marostegui@cumin1002: dbctl commit (dc=all): 'Remove es2021 from dbctl T382944', diff saved to https://phabricator.wikimedia.org/P71786 and previous config saved to /var/cache/conftool/dbconfig/20250106-055726-marostegui.json
• 05:50 marostegui@cumin1002: dbctl commit (dc=all): 'db1193 (re)pooling @ 10%: Repooling after schema change', diff saved to https://phabricator.wikimedia.org/P71785 and previous config saved to /var/cache/conftool/dbconfig/20250106-055029-root.json
• 03:39 tstarling@deploy2002: Finished scap sync-world: Backport for Enable canShellboxGetTempUrl everywhere (T292322) (duration: 21m 53s)
• 03:31 tstarling@deploy2002: tstarling: Continuing with sync
• 03:31 tstarling@deploy2002: tstarling: Backport for Enable canShellboxGetTempUrl everywhere (T292322) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 03:17 tstarling@deploy2002: Started scap sync-world: Backport for Enable canShellboxGetTempUrl everywhere (T292322)

2025-01-04

• 20:02 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on doc1003.eqiad.wmnet with reason: dz
• 20:01 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on doc1003.eqiad.wmnet with reason: dz

2025-01-03

• 19:09 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on doc2002.codfw.wmnet with reason: Disk Change
• 19:09 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on doc2002.codfw.wmnet with reason: Disk Change
• 18:47 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on doc2002.codfw.wmnet with reason: Disk Change
• 18:46 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on doc2002.codfw.wmnet with reason: Disk Change
• 17:15 thcipriani@deploy2002: Finished deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit1003.wikimedia.org only) (duration: 00m 10s)
• 17:15 thcipriani@deploy2002: Started deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit1003.wikimedia.org only)
• 17:06 thcipriani@deploy2002: Finished deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit2002.wikimedia.org only) (duration: 00m 08s)
• 17:06 thcipriani@deploy2002: Started deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit2002.wikimedia.org only)
• 17:00 isaranto@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 16:55 thcipriani@deploy2002: Finished deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit2003.wikimedia.org only) (duration: 00m 08s)
• 16:55 thcipriani@deploy2002: Started deploy [gerrit/gerrit@44854d4]: remove developer satisfaction survey banner (gerrit2003.wikimedia.org only)
• 12:10 moritzm: renewed internal Ganeti certs in eqsin (would have expired in two days) T382873
• 12:01 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2022 T381848', diff saved to https://phabricator.wikimedia.org/P71781 and previous config saved to /var/cache/conftool/dbconfig/20250103-120132-marostegui.json
• 11:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2020 T382945', diff saved to https://phabricator.wikimedia.org/P71780 and previous config saved to /var/cache/conftool/dbconfig/20250103-111255-marostegui.json
• 11:05 marostegui: Switchover es4 codfw master to es2043 dbmaint T381848
• 11:04 marostegui@cumin1002: dbctl commit (dc=all): 'Switchover es4 codfw master', diff saved to https://phabricator.wikimedia.org/P71779 and previous config saved to /var/cache/conftool/dbconfig/20250103-110440-marostegui.json
• 10:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es2021 T381848', diff saved to https://phabricator.wikimedia.org/P71778 and previous config saved to /var/cache/conftool/dbconfig/20250103-103513-marostegui.json
• 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2236 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71777 and previous config saved to /var/cache/conftool/dbconfig/20250103-100603-root.json
• 09:50 marostegui@cumin1002: dbctl commit (dc=all): 'db2236 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71776 and previous config saved to /var/cache/conftool/dbconfig/20250103-095057-root.json
• 09:35 marostegui@cumin1002: dbctl commit (dc=all): 'db2236 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71774 and previous config saved to /var/cache/conftool/dbconfig/20250103-093552-root.json
• 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'db2236 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71773 and previous config saved to /var/cache/conftool/dbconfig/20250103-092046-root.json
• 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'db2236 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71772 and previous config saved to /var/cache/conftool/dbconfig/20250103-090541-root.json
• 09:03 marostegui: Upgrade db2236 to 10.11.10 s4 codfw dbmaint T378940
• 09:02 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2236.codfw.wmnet with reason: upgrade
• 09:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db2236.codfw.wmnet with reason: upgrade
• 09:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2236 to upgrade to 10.11.10 T378940', diff saved to https://phabricator.wikimedia.org/P71771 and previous config saved to /var/cache/conftool/dbconfig/20250103-090215-marostegui.json
• 08:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2115.codfw.wmnet
• 08:35 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:35 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2115.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:33 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2115.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:29 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 08:24 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2115.codfw.wmnet
• 07:36 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on db1193.eqiad.wmnet with reason: maintenance
• 07:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on db1193.eqiad.wmnet with reason: maintenance
• 07:23 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2115 from dbctl T362949', diff saved to https://phabricator.wikimedia.org/P71770 and previous config saved to /var/cache/conftool/dbconfig/20250103-072349-marostegui.json
• 06:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2116.codfw.wmnet
• 06:32 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 06:32 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2116.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 06:32 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2116.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 06:28 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 06:23 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2116.codfw.wmnet

2025-01-02

• 22:23 wfan: SmashPig upgraded from 17ac74f2 to 1d060a11
• 21:26 urbanecm@deploy2002: Finished scap sync-world: Backport for [Growth] Remove Marketing campaign (T382499), gomwiki: Use wikitext talk pages by default (T382810) (duration: 21m 42s)
• 21:18 urbanecm@deploy2002: urbanecm: Continuing with sync
• 21:17 urbanecm@deploy2002: urbanecm: Backport for [Growth] Remove Marketing campaign (T382499), gomwiki: Use wikitext talk pages by default (T382810) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
• 21:04 urbanecm@deploy2002: Started scap sync-world: Backport for [Growth] Remove Marketing campaign (T382499), gomwiki: Use wikitext talk pages by default (T382810)
• 18:23 bd808@deploy2002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
• 18:23 bd808@deploy2002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
• 18:23 bd808@deploy2002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
• 18:22 bd808@deploy2002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
• 18:22 bd808@deploy2002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
• 18:22 bd808@deploy2002: helmfile [staging] START helmfile.d/services/developer-portal: apply
• 18:22 bd808@deploy2002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
• 18:21 bd808@deploy2002: helmfile [staging] START helmfile.d/services/developer-portal: apply
• 14:28 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2116 from dbctl T362950', diff saved to https://phabricator.wikimedia.org/P71768 and previous config saved to /var/cache/conftool/dbconfig/20250102-142806-marostegui.json
• 11:49 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be1066.eqiad.wmnet
• 11:49 mvernon@cumin1002: START - Cookbook sre.hosts.remove-downtime for ms-be1066.eqiad.wmnet
• 11:31 marostegui: dbmaint s8 db1193 eqiad rebuild pagelinks and recentchanges and deploy schema change on revision table T367856 T382842
• 11:29 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1066.eqiad.wmnet with reason: vacuum three container dbs
• 11:28 mvernon@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on ms-be1066.eqiad.wmnet with reason: vacuum three container dbs
• 11:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on db1193.eqiad.wmnet with reason: maintenance
• 11:22 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5 days, 0:00:00 on db1193.eqiad.wmnet with reason: maintenance
• 11:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1193 T381993', diff saved to https://phabricator.wikimedia.org/P71766 and previous config saved to /var/cache/conftool/dbconfig/20250102-111105-marostegui.json
• 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1209 to s8 primary T381993', diff saved to https://phabricator.wikimedia.org/P71765 and previous config saved to /var/cache/conftool/dbconfig/20250102-110923-marostegui.json
• 11:09 marostegui: Starting s8 eqiad failover from db1193 to db1209 - T381993
• 11:03 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db1209 from API/vslow/dump T381993', diff saved to https://phabricator.wikimedia.org/P71764 and previous config saved to /var/cache/conftool/dbconfig/20250102-110305-root.json
• 11:02 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1209 with weight 0 T381993', diff saved to https://phabricator.wikimedia.org/P71763 and previous config saved to /var/cache/conftool/dbconfig/20250102-110232-root.json
• 11:00 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s8 T381993
• 11:00 root@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s8 T381993
• 10:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy2004.codfw.wmnet
• 10:56 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:56 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2004.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 10:56 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2004.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 10:56 marostegui: dbmaint codfw Decommissioned dbproxy200[1-4] m1 m2 m3 m5 T381962
• 10:56 marostegui: dbmaint codfw Decommissioned dbproxy200[1-4] T381962
• 10:52 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 10:48 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy2004.codfw.wmnet
• 10:14 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy2003.codfw.wmnet
• 10:14 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:14 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 10:14 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 10:10 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 10:06 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy2003.codfw.wmnet
• 09:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 100%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71762 and previous config saved to /var/cache/conftool/dbconfig/20250102-095838-root.json
• 09:56 Emperor: restart swift-object on ms-be1075
• 09:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 75%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71761 and previous config saved to /var/cache/conftool/dbconfig/20250102-094332-root.json
• 09:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 50%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71760 and previous config saved to /var/cache/conftool/dbconfig/20250102-092827-root.json
• 09:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 25%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71759 and previous config saved to /var/cache/conftool/dbconfig/20250102-091322-root.json
• 09:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy2002.codfw.wmnet
• 09:01 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:01 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 09:01 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 10%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71758 and previous config saved to /var/cache/conftool/dbconfig/20250102-085816-root.json
• 08:57 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 08:53 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy2002.codfw.wmnet
• 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 5%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71757 and previous config saved to /var/cache/conftool/dbconfig/20250102-084312-root.json
• 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 1%: Repooling after recloning', diff saved to https://phabricator.wikimedia.org/P71756 and previous config saved to /var/cache/conftool/dbconfig/20250102-082806-root.json
• 08:17 marostegui@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 100%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71755 and previous config saved to /var/cache/conftool/dbconfig/20250102-081722-root.json
• 08:08 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy2001.codfw.wmnet
• 08:08 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 08:08 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:08 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
• 08:04 marostegui@cumin1002: START - Cookbook sre.dns.netbox
• 08:02 marostegui@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 75%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71754 and previous config saved to /var/cache/conftool/dbconfig/20250102-080216-root.json
• 08:00 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts dbproxy2001.codfw.wmnet
• 07:47 marostegui@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 50%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71753 and previous config saved to /var/cache/conftool/dbconfig/20250102-074711-root.json
• 07:35 marostegui: Stop haproxy on dbproxy200[14] T381962
• 07:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on dbproxy2004.codfw.wmnet with reason: maintenance
• 07:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on dbproxy2004.codfw.wmnet with reason: maintenance
• 07:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on dbproxy2003.codfw.wmnet with reason: maintenance
• 07:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on dbproxy2003.codfw.wmnet with reason: maintenance
• 07:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on dbproxy2002.codfw.wmnet with reason: maintenance
• 07:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on dbproxy2002.codfw.wmnet with reason: maintenance
• 07:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on dbproxy2001.codfw.wmnet with reason: maintenance
• 07:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on dbproxy2001.codfw.wmnet with reason: maintenance
• 07:32 marostegui@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 25%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71752 and previous config saved to /var/cache/conftool/dbconfig/20250102-073206-root.json
• 07:17 marostegui@cumin1002: dbctl commit (dc=all): 'db2171 (re)pooling @ 10%: Repooling after upgrade', diff saved to https://phabricator.wikimedia.org/P71751 and previous config saved to /var/cache/conftool/dbconfig/20250102-071700-root.json
• 07:13 root@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2171.codfw.wmnet onto db2123.codfw.wmnet
• 06:37 root@cumin1002: START - Cookbook sre.mysql.clone of db2171.codfw.wmnet onto db2123.codfw.wmnet
• 06:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: maintenance
• 06:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2171.codfw.wmnet with reason: maintenance
• 06:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2171 to clone db2123 T382744', diff saved to https://phabricator.wikimedia.org/P71750 and previous config saved to /var/cache/conftool/dbconfig/20250102-063218-marostegui.json
• 06:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2123.codfw.wmnet with reason: maintenance
• 06:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2123.codfw.wmnet with reason: maintenance
• 00:03 sukhe@cumin1002: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool site eqsin for service: ncredir-addrs [reason: no reason specified, no task ID specified]
• 00:03 sukhe@cumin1002: START - Cookbook sre.dns.admin DNS admin: depool site eqsin for service: ncredir-addrs [reason: no reason specified, no task ID specified]

Archives

See Server Admin Log/Archives.