Server Admin Log

2026-10-08

• 19:00 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
• 18:56 oblivian@cumin1004: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet
• 17:47 andrew@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
• 17:46 andrew@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
• 17:46 andrew@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
• 17:45 andrew@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
• 17:36 andrew@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
• 17:36 andrew@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
• 17:31 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2141-2179].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 17:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2179.codfw.wmnet
• 17:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2179.codfw.wmnet
• 17:10 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 17:09 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 17:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2179.codfw.wmnet
• 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2179.codfw.wmnet
• 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2178.codfw.wmnet
• 17:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2178.codfw.wmnet
• 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2178.codfw.wmnet
• 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2178.codfw.wmnet
• 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2177.codfw.wmnet
• 16:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2177.codfw.wmnet
• 16:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1057.eqiad.wmnet
• 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet
• 16:35 btullis@cumin1004: END (FAIL) - Cookbook sre.druid.reboot-workers (exit_code=99) for Druid public cluster: Reboot Druid nodes
• 16:33 oblivian@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet
• 16:28 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 16:26 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 16:25 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 16:25 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 16:25 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 16:21 dancy@deploy1003: Installation of scap version "4.296.0" completed for 3 hosts
• 16:19 dancy@deploy1003: Installing scap version "4.296.0" for 3 host(s)
• 16:16 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-c1-eqiad
• 16:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-c1-eqiad
• 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:14 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
• 16:13 swfrench-wmf: begin rolling restarts of confd in ulsfo, eqsin, codfw to pick up etcd client SRV change
• 16:10 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sretest2009.codfw.wmnet with OS trixie
• 16:07 swfrench@dns1004: END - running authdns-update
• 16:04 swfrench@dns1004: START - running authdns-update
• 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2009.codfw.wmnet (live debug)
• 15:55 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2009.codfw.wmnet (live debug)
• 15:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage
• 15:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:47 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage
• 15:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie
• 15:29 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
• 15:25 cmooney@cumin1004: START - Cookbook sre.dns.netbox
• 15:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1081.eqiad.wmnet
• 15:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1081.eqiad.wmnet
• 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1080.eqiad.wmnet
• 15:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1080.eqiad.wmnet
• 15:17 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:17 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2902: Pool db2902.codfw.wmnet in after cloning
• 15:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1080.eqiad.wmnet
• 15:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1080.eqiad.wmnet
• 15:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1079.eqiad.wmnet
• 15:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1079.eqiad.wmnet
• 15:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:59 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes
• 14:58 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 14:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1079.eqiad.wmnet
• 14:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1079.eqiad.wmnet
• 14:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1078.eqiad.wmnet
• 14:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1078.eqiad.wmnet
• 14:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:54 cmooney@dns3003: END - running authdns-update
• 14:51 cmooney@dns3003: START - running authdns-update
• 14:50 cgoubert@dns2004: END - running authdns-update
• 14:48 cgoubert@dns2004: START - running authdns-update
• 14:48 moritzm: installing pyasn1 security updates
• 14:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1078.eqiad.wmnet
• 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1078.eqiad.wmnet
• 14:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1077.eqiad.wmnet
• 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:43 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
• 14:43 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
• 14:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1077.eqiad.wmnet
• 14:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:39 cmooney@cumin1004: START - Cookbook sre.dns.netbox
• 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:39 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
• 14:35 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004"
• 14:35 jdrewniak@deploy1003: Finished scap sync-world: Backport for Bumping portals to master (T128546) (duration: 36m 09s)
• 14:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1077.eqiad.wmnet
• 14:32 cmooney@cumin1004: START - Cookbook sre.dns.netbox
• 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1077.eqiad.wmnet
• 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1076.eqiad.wmnet
• 14:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1076.eqiad.wmnet
• 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:29 jdrewniak@deploy1003: jdrewniak: Continuing with deployment
• 14:23 moritzm: installing perl security updates
• 14:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1076.eqiad.wmnet
• 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1076.eqiad.wmnet
• 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1075.eqiad.wmnet
• 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1075.eqiad.wmnet
• 14:20 brouberol@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
• 14:19 brouberol@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
• 14:18 brouberol@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
• 14:17 brouberol@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
• 14:17 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Pool db2901.codfw.wmnet in after cloning
• 14:09 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db2901: Pool db2901.codfw.wmnet in after cloning
• 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004
• 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004
• 14:03 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004
• 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004
• 14:03 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet
• 14:03 jdrewniak@deploy1003: jdrewniak: Backport for Bumping portals to master (T128546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:58 jdrewniak@deploy1003: Started scap sync-world: Backport for Bumping portals to master (T128546)
• 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1075.eqiad.wmnet
• 13:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1075.eqiad.wmnet
• 13:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1074.eqiad.wmnet
• 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1074.eqiad.wmnet
• 13:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:47 samtar@deploy1003: Finished scap sync-world: Backport for Home Experience: Feature flag to enable Home (T439738) (duration: 38m 16s)
• 13:47 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 13:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1074.eqiad.wmnet
• 13:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1074.eqiad.wmnet
• 13:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet
• 13:42 samtar@deploy1003: emc-wmf, samtar: Continuing with deployment
• 13:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet
• 13:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:38 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet
• 13:38 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet
• 13:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet
• 13:33 fceratto@cumin1004: START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet
• 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1073.eqiad.wmnet
• 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1073.eqiad.wmnet
• 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1072.eqiad.wmnet
• 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1072.eqiad.wmnet
• 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:28 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug)
• 13:27 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug)
• 13:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet
• 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet
• 13:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1071.eqiad.wmnet
• 13:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1071.eqiad.wmnet
• 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:13 samtar@deploy1003: emc-wmf, samtar: Backport for Home Experience: Feature flag to enable Home (T439738) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet
• 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet
• 13:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet
• 13:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet
• 13:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:09 samtar@deploy1003: Started scap sync-world: Backport for Home Experience: Feature flag to enable Home (T439738)
• 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:04 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test
• 13:04 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 13:04 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
• 13:03 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1070.eqiad.wmnet
• 13:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1070.eqiad.wmnet
• 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet
• 13:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet
• 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2177.codfw.wmnet
• 12:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2177.codfw.wmnet
• 12:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2176.codfw.wmnet
• 12:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2176.codfw.wmnet
• 12:58 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:55 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
• 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
• 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 12:54 fceratto@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1901: test
• 12:54 fceratto@cumin1004: START - Cookbook sre.mysql.depool depool db1901: test
• 12:54 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
• 12:54 dbrant@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
• 12:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1069.eqiad.wmnet
• 12:53 dbrant@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1069.eqiad.wmnet
• 12:53 dbrant@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
• 12:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1068.eqiad.wmnet
• 12:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1068.eqiad.wmnet
• 12:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:52 dbrant@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
• 12:51 dbrant@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 12:50 dbrant@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 12:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2176.codfw.wmnet
• 12:49 zabe: zabe@deploy1003:~$ mwscript recountCategories.php commonswiki --mode all # T440023 T439902
• 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2176.codfw.wmnet
• 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2175.codfw.wmnet
• 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2175.codfw.wmnet
• 12:47 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 12:44 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1068.eqiad.wmnet
• 12:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1068.eqiad.wmnet
• 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1067.eqiad.wmnet
• 12:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1067.eqiad.wmnet
• 12:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:39 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 12:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1067.eqiad.wmnet
• 12:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1067.eqiad.wmnet
• 12:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1066.eqiad.wmnet
• 12:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1066.eqiad.wmnet
• 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:31 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2175.codfw.wmnet
• 12:30 zabe@deploy1003: Finished scap sync-world: Backport for Use Hadoop for UnusedTemplates on commonswiki (T422443) (duration: 09m 47s)
• 12:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2175.codfw.wmnet
• 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2174.codfw.wmnet
• 12:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2174.codfw.wmnet
• 12:25 zabe@deploy1003: zabe: Continuing with deployment
• 12:24 zabe@deploy1003: zabe: Backport for Use Hadoop for UnusedTemplates on commonswiki (T422443) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1066.eqiad.wmnet
• 12:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1066.eqiad.wmnet
• 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1065.eqiad.wmnet
• 12:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1065.eqiad.wmnet
• 12:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:20 zabe@deploy1003: Started scap sync-world: Backport for Use Hadoop for UnusedTemplates on commonswiki (T422443)
• 12:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2174.codfw.wmnet
• 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2174.codfw.wmnet
• 12:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2173.codfw.wmnet
• 12:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2173.codfw.wmnet
• 12:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1065.eqiad.wmnet
• 12:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
• 12:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1065.eqiad.wmnet
• 12:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1064.eqiad.wmnet
• 12:10 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 12:10 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
• 12:09 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
• 12:09 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 12:08 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1064.eqiad.wmnet
• 12:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:00 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
• 12:00 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
• 11:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1064.eqiad.wmnet
• 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1064.eqiad.wmnet
• 11:58 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 11:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2173.codfw.wmnet
• 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2173.codfw.wmnet
• 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2172.codfw.wmnet
• 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2172.codfw.wmnet
• 11:44 oblivian@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1057.eqiad.wmnet with reason: HW failure
• 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:42 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
• 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
• 11:41 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
• 11:40 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
• 11:39 blake@deploy1003: Finished scap sync-world: update mcrouter exporter template for T417800 (duration: 04m 15s)
• 11:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2172.codfw.wmnet
• 11:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2172.codfw.wmnet
• 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2171.codfw.wmnet
• 11:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2171.codfw.wmnet
• 11:36 blake@deploy1003: Started scap sync-world: update mcrouter exporter template for T417800
• 11:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2171.codfw.wmnet
• 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2171.codfw.wmnet
• 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2170.codfw.wmnet
• 11:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2170.codfw.wmnet
• 11:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:11 aikochou@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 11:10 topranks: cr1-codfw switch back to master routing engine T426197
• 11:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2170.codfw.wmnet
• 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2170.codfw.wmnet
• 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2169.codfw.wmnet
• 11:06 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2169.codfw.wmnet
• 11:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:57 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 10:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2169.codfw.wmnet
• 10:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2169.codfw.wmnet
• 10:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2168.codfw.wmnet
• 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2168.codfw.wmnet
• 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2168.codfw.wmnet
• 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2168.codfw.wmnet
• 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2167.codfw.wmnet
• 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2167.codfw.wmnet
• 10:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:36 topranks: cr1-codfw switch to backup routing engine T426197
• 10:31 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
• 10:30 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
• 10:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2167.codfw.wmnet
• 10:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2167.codfw.wmnet
• 10:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2166.codfw.wmnet
• 10:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2166.codfw.wmnet
• 10:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:19 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Pooling back after cloning a host
• 10:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2166.codfw.wmnet
• 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2166.codfw.wmnet
• 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2165.codfw.wmnet
• 10:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2165.codfw.wmnet
• 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:08 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug)
• 10:06 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug)
• 10:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2165.codfw.wmnet
• 10:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2165.codfw.wmnet
• 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2164.codfw.wmnet
• 10:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2164.codfw.wmnet
• 10:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:59 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 09:52 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade
• 09:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:52 cmooney@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade
• 09:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2164.codfw.wmnet
• 09:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2164.codfw.wmnet
• 09:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2163.codfw.wmnet
• 09:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2163.codfw.wmnet
• 09:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Repooling after switchover
• 09:37 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2163.codfw.wmnet
• 09:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2163.codfw.wmnet
• 09:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2162.codfw.wmnet
• 09:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2162.codfw.wmnet
• 09:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:34 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Pooling back after cloning a host
• 09:29 topranks: begin draining traffic from cr1-codfw in advance of JunOS upgrade T417871
• 09:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2162.codfw.wmnet
• 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2162.codfw.wmnet
• 09:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2161.codfw.wmnet
• 09:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2161.codfw.wmnet
• 09:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2161.codfw.wmnet
• 09:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2161.codfw.wmnet
• 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2160.codfw.wmnet
• 09:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2160.codfw.wmnet
• 09:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2160.codfw.wmnet
• 08:59 moritzm: installing Wireshark security updates
• 08:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2160.codfw.wmnet
• 08:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2159.codfw.wmnet
• 08:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2159.codfw.wmnet
• 08:57 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover
• 08:53 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2191: Repooling after switchover
• 08:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover
• 08:52 marostegui@cumin1004: dbctl commit (dc=all): 'Depool db2191 T436922', diff saved to https://phabricator.wikimedia.org/P96627 and previous config saved to /var/cache/conftool/dbconfig/20261008-085238-marostegui.json
• 08:50 marostegui@cumin1004: dbctl commit (dc=all): 'Promote db2249 to x1 primary T436922', diff saved to https://phabricator.wikimedia.org/P96626 and previous config saved to /var/cache/conftool/dbconfig/20261008-085050-marostegui.json
• 08:50 marostegui: Starting x1 codfw failover from db2191 to db2249 - T436922
• 08:46 marostegui@cumin1004: dbctl commit (dc=all): 'Set db2249 with weight 0 T436922', diff saved to https://phabricator.wikimedia.org/P96625 and previous config saved to /var/cache/conftool/dbconfig/20261008-084653-marostegui.json
• 08:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 17 hosts with reason: Primary switchover x1 T436922
• 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
• 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
• 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:46 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
• 08:46 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
• 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1057.eqiad.wmnet
• 08:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2159.codfw.wmnet
• 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2159.codfw.wmnet
• 08:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2158.codfw.wmnet
• 08:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1057.eqiad.wmnet
• 08:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1056.eqiad.wmnet
• 08:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2158.codfw.wmnet
• 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:42 moritzm: installing python3.13 security updates
• 08:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1056.eqiad.wmnet
• 08:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:33 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1056.eqiad.wmnet
• 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1056.eqiad.wmnet
• 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet
• 08:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet
• 08:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1055.eqiad.wmnet
• 08:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1055.eqiad.wmnet
• 08:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet
• 08:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet
• 08:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.23 refs T438219
• 08:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet
• 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet
• 08:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet
• 08:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet
• 08:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet
• 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet
• 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet
• 08:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet
• 08:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:01 moritzm: installing mbedtls security updates
• 07:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet
• 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet
• 07:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet
• 07:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet
• 07:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2158.codfw.wmnet
• 07:46 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2158.codfw.wmnet
• 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet
• 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2157.codfw.wmnet
• 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2157.codfw.wmnet
• 07:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet
• 07:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1050.eqiad.wmnet
• 07:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1050.eqiad.wmnet
• 07:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:34 ladsgroup@deploy1003: Finished scap sync-world: Backport for nrwiki, xhwiki: update logo and tagline (T440138 T440139) (duration: 13m 44s)
• 07:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2157.codfw.wmnet
• 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2157.codfw.wmnet
• 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2156.codfw.wmnet
• 07:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2156.codfw.wmnet
• 07:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1050.eqiad.wmnet
• 07:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1050.eqiad.wmnet
• 07:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1049.eqiad.wmnet
• 07:29 ladsgroup@deploy1003: anzx, ladsgroup: Continuing with deployment
• 07:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1049.eqiad.wmnet
• 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:25 ladsgroup@deploy1003: anzx, ladsgroup: Backport for nrwiki, xhwiki: update logo and tagline (T440138 T440139) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2156.codfw.wmnet
• 07:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2156.codfw.wmnet
• 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2155.codfw.wmnet
• 07:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2155.codfw.wmnet
• 07:20 ladsgroup@deploy1003: Started scap sync-world: Backport for nrwiki, xhwiki: update logo and tagline (T440138 T440139)
• 07:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1049.eqiad.wmnet
• 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1049.eqiad.wmnet
• 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1048.eqiad.wmnet
• 07:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1048.eqiad.wmnet
• 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:14 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 07:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 07:12 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 07:12 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1265.eqiad.wmnet with reason: preparing backups
• 07:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2155.codfw.wmnet
• 07:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2155.codfw.wmnet
• 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2154.codfw.wmnet
• 07:07 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1048.eqiad.wmnet
• 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2154.codfw.wmnet
• 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1048.eqiad.wmnet
• 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1047.eqiad.wmnet
• 07:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1047.eqiad.wmnet
• 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2154.codfw.wmnet
• 06:55 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1047.eqiad.wmnet
• 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2154.codfw.wmnet
• 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2153.codfw.wmnet
• 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1047.eqiad.wmnet
• 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1046.eqiad.wmnet
• 06:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1046.eqiad.wmnet
• 06:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2153.codfw.wmnet
• 06:52 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1046.eqiad.wmnet
• 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1046.eqiad.wmnet
• 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1045.eqiad.wmnet
• 06:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1045.eqiad.wmnet
• 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:41 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2153.codfw.wmnet
• 06:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2153.codfw.wmnet
• 06:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2152.codfw.wmnet
• 06:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2152.codfw.wmnet
• 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:37 Amir1: mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model articlequality --old (T440358)
• 06:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1045.eqiad.wmnet
• 06:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1045.eqiad.wmnet
• 06:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1044.eqiad.wmnet
• 06:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1044.eqiad.wmnet
• 06:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2152.codfw.wmnet
• 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2152.codfw.wmnet
• 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2151.codfw.wmnet
• 06:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2151.codfw.wmnet
• 06:26 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1044.eqiad.wmnet
• 06:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1044.eqiad.wmnet
• 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1043.eqiad.wmnet
• 06:21 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
• 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1043.eqiad.wmnet
• 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:19 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
• 06:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2151.codfw.wmnet
• 06:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2151.codfw.wmnet
• 06:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2150.codfw.wmnet
• 06:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2150.codfw.wmnet
• 06:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:13 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1043.eqiad.wmnet
• 06:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1043.eqiad.wmnet
• 06:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1042.eqiad.wmnet
• 06:12 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
• 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1042.eqiad.wmnet
• 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:09 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
• 06:02 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1262.eqiad.wmnet with reason: preparing backups
• 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:02 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: preparing backups
• 06:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2150.codfw.wmnet
• 06:02 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1262: preparing backups
• 06:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1042.eqiad.wmnet
• 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2150.codfw.wmnet
• 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2149.codfw.wmnet
• 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1042.eqiad.wmnet
• 06:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2149.codfw.wmnet
• 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1041.eqiad.wmnet
• 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1041.eqiad.wmnet
• 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
• 05:58 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
• 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1041.eqiad.wmnet
• 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1041.eqiad.wmnet
• 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1040.eqiad.wmnet
• 05:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2149.codfw.wmnet
• 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1040.eqiad.wmnet
• 05:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2149.codfw.wmnet
• 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2148.codfw.wmnet
• 05:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2148.codfw.wmnet
• 05:46 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1040.eqiad.wmnet
• 05:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1040.eqiad.wmnet
• 05:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1039.eqiad.wmnet
• 05:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1039.eqiad.wmnet
• 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2148.codfw.wmnet
• 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2148.codfw.wmnet
• 05:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2147.codfw.wmnet
• 05:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2147.codfw.wmnet
• 05:32 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet
• 05:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet
• 05:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1038.eqiad.wmnet
• 05:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1038.eqiad.wmnet
• 05:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2147.codfw.wmnet
• 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2147.codfw.wmnet
• 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2146.codfw.wmnet
• 05:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2146.codfw.wmnet
• 05:19 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1038.eqiad.wmnet
• 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1038.eqiad.wmnet
• 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet
• 05:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet
• 05:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2146.codfw.wmnet
• 05:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet
• 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2146.codfw.wmnet
• 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2145.codfw.wmnet
• 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet
• 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet
• 05:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet
• 05:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2145.codfw.wmnet
• 05:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:59 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:59 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet
• 04:58 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet
• 04:58 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1035.eqiad.wmnet
• 04:57 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1035.eqiad.wmnet
• 04:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2145.codfw.wmnet
• 04:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2145.codfw.wmnet
• 04:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2144.codfw.wmnet
• 04:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2144.codfw.wmnet
• 04:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1035.eqiad.wmnet
• 04:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1035.eqiad.wmnet
• 04:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1034.eqiad.wmnet
• 04:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1034.eqiad.wmnet
• 04:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2144.codfw.wmnet
• 04:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2144.codfw.wmnet
• 04:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2143.codfw.wmnet
• 04:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2143.codfw.wmnet
• 04:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1034.eqiad.wmnet
• 04:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1034.eqiad.wmnet
• 04:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1021.eqiad.wmnet
• 04:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1021.eqiad.wmnet
• 04:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2143.codfw.wmnet
• 04:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2143.codfw.wmnet
• 04:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2142.codfw.wmnet
• 04:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1021.eqiad.wmnet
• 04:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1021.eqiad.wmnet
• 04:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1016.eqiad.wmnet
• 04:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2142.codfw.wmnet
• 04:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1016.eqiad.wmnet
• 04:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2142.codfw.wmnet
• 04:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2142.codfw.wmnet
• 04:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2141.codfw.wmnet
• 04:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1016.eqiad.wmnet
• 04:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1016.eqiad.wmnet
• 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1015.eqiad.wmnet
• 04:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2141.codfw.wmnet
• 04:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 04:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1015.eqiad.wmnet
• 04:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2141.codfw.wmnet
• 03:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 03:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1015.eqiad.wmnet
• 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2141.codfw.wmnet
• 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2141-2179].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1015.eqiad.wmnet
• 03:56 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 23s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
• 01:38 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 01:38 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 01:38 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 01:38 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 01:38 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 01:37 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
• 00:18 mutante: gerrit - added content-security-policy headers on <Location /r/Documentation/> - style-src 'self' 'unsafe-inline'; font-src 'self' - blocking Google font (T393763)

2026-10-07

• 22:07 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 22:05 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 22:04 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm
• 21:52 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host krb1005.eqiad.wmnet
• 21:52 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1005.eqiad.wmnet with OS bookworm
• 21:36 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage
• 21:32 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage
• 21:22 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
• 21:22 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
• 21:22 eevans@deploy1003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
• 21:21 eevans@deploy1003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
• 21:17 bking@cumin2003: START - Cookbook sre.hosts.reimage for host krb1005.eqiad.wmnet with OS bookworm
• 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003"
• 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003"
• 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) krb1005.eqiad.wmnet on all recursors
• 21:16 bking@cumin2003: START - Cookbook sre.dns.wipe-cache krb1005.eqiad.wmnet on all recursors
• 21:16 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 21:16 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003"
• 21:16 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003"
• 21:08 bking@cumin2003: START - Cookbook sre.dns.netbox
• 21:08 bking@cumin2003: START - Cookbook sre.ganeti.makevm for new host krb1005.eqiad.wmnet
• 20:28 kemayo@deploy1003: Finished scap sync-world: Backport for EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733) (duration: 11m 11s)
• 20:23 kemayo@deploy1003: kemayo: Continuing with deployment
• 20:21 kemayo@deploy1003: kemayo: Backport for EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:17 kemayo@deploy1003: Started scap sync-world: Backport for EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)
• 19:55 kemayo@deploy1003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
• 19:55 kemayo@deploy1003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
• 19:54 kemayo@deploy1003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
• 19:54 kemayo@deploy1003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
• 19:19 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
• 19:18 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
• 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2124-2140].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2140.codfw.wmnet
• 18:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2140.codfw.wmnet
• 18:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 18:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 18:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2140.codfw.wmnet
• 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2140.codfw.wmnet
• 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2139.codfw.wmnet
• 18:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2139.codfw.wmnet
• 18:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 18:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2139.codfw.wmnet
• 18:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2139.codfw.wmnet
• 18:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2138.codfw.wmnet
• 18:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2138.codfw.wmnet
• 18:07 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2138.codfw.wmnet
• 17:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2138.codfw.wmnet
• 17:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2137.codfw.wmnet
• 17:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2137.codfw.wmnet
• 17:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:42 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2137.codfw.wmnet
• 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2137.codfw.wmnet
• 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2136.codfw.wmnet
• 17:41 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2136.codfw.wmnet
• 17:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:30 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2136.codfw.wmnet
• 17:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2136.codfw.wmnet
• 17:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2135.codfw.wmnet
• 17:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2135.codfw.wmnet
• 17:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:26 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
• 17:26 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
• 17:25 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 17:25 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 17:24 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 17:24 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 17:21 dancy@deploy1003: Installation of scap version "4.295.0" completed for 3 hosts
• 17:19 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 17:19 dancy@deploy1003: Installing scap version "4.295.0" for 3 host(s)
• 17:19 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 17:16 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2135.codfw.wmnet
• 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2135.codfw.wmnet
• 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2134.codfw.wmnet
• 17:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2134.codfw.wmnet
• 17:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
• 17:12 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
• 17:08 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 17:08 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 17:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2134.codfw.wmnet
• 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2134.codfw.wmnet
• 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2133.codfw.wmnet
• 17:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2133.codfw.wmnet
• 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt
• 16:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt
• 16:57 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt
• 16:57 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt
• 16:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2133.codfw.wmnet
• 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2133.codfw.wmnet
• 16:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2132.codfw.wmnet
• 16:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2132.codfw.wmnet
• 16:49 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:47 cgoubert@deploy1003: Finished scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki | 1351442 | T432733 (duration: 10m 40s)
• 16:38 cgoubert@deploy1003: Started scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki | 1351442 | T432733
• 16:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2132.codfw.wmnet
• 16:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2132.codfw.wmnet
• 16:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2131.codfw.wmnet
• 16:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2131.codfw.wmnet
• 16:36 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:32 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie
• 16:32 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 16:31 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 16:31 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 16:29 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 16:28 cgoubert@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 16:26 cgoubert@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 16:26 cgoubert@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 16:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2131.codfw.wmnet
• 16:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2131.codfw.wmnet
• 16:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2130.codfw.wmnet
• 16:23 cgoubert@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2130.codfw.wmnet
• 16:22 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
• 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:21 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
• 16:21 blake@deploy1003: Finished scap sync-world: add restart policies for T417800 (duration: 05m 23s)
• 16:18 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
• 16:17 blake@deploy1003: Started scap sync-world: add restart policies for T417800
• 16:12 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
• 16:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2130.codfw.wmnet
• 16:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2130.codfw.wmnet
• 16:10 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (remove config only)
• 16:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2129.codfw.wmnet
• 16:10 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (remove config only)
• 16:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2129.codfw.wmnet
• 16:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:03 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
• 15:59 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
• 15:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2129.codfw.wmnet
• 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2129.codfw.wmnet
• 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2128.codfw.wmnet
• 15:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2128.codfw.wmnet
• 15:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:56 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (push config only)
• 15:56 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (push config only)
• 15:55 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie
• 15:51 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr[1-2]-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: cr2-codfw upgrade
• 15:48 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt with reason: Junos upgrade
• 15:47 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade
• 15:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2128.codfw.wmnet
• 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2128.codfw.wmnet
• 15:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2127.codfw.wmnet
• 15:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2127.codfw.wmnet
• 15:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: troubleshooting
• 15:18 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
• 15:18 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
• 15:11 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: Junos upgrade
• 15:10 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: Junos upgrade
• 15:09 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: Junos upgrade
• 15:08 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: Junos upgrade
• 15:07 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ulsfo,cr4-ulsfo IPv6,cr2-ulsfo.mgmt with reason: Junos upgrade
• 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: Junos upgrade
• 15:07 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqiad,cr2-eqiad IPv6,re0.cr2-eqiad.mgmt with reason: Junos upgrade
• 15:06 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ceqiad IPv6,cr2-eqiad,re0.cr2-eqiad.mgmt with reason: Junos upgrade
• 15:06 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pfw1-codfw with reason: Junos upgrade
• 15:05 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: Junos upgrade
• 15:04 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-f1-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade
• 15:04 pt1979@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ssw1-f8-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade
• 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-d8-codfw,ssw1-d8-codfw IPv6,ssw1-d8-codfw.mgmt with reason: Junos upgrade
• 15:03 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a8-codfw,ssw1-a8-codfw IPv6,ssw1-a8-codfw.mgmt with reason: Junos upgrade
• 14:53 jforrester@deploy1003: Finished scap sync-world: Backport for WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432) (duration: 10m 09s)
• 14:48 jforrester@deploy1003: jforrester: Continuing with deployment
• 14:47 jforrester@deploy1003: jforrester: Backport for WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams
• 14:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams
• 14:43 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs
• 14:43 jforrester@deploy1003: Started scap sync-world: Backport for WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)
• 14:41 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs
• 14:41 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin
• 14:40 jforrester@deploy1003: Finished scap sync-world: Backport for Factor out Abstract Wikipedia client mode into a dblist, Wikifunctions: Switch the editor from Ace to CodeMirror (T400015) (duration: 18m 26s)
• 14:37 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin
• 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo
• 14:34 jforrester@deploy1003: jforrester: Continuing with deployment
• 14:33 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo
• 14:31 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:27 moritzm: installing PHP 8.2 security updates
• 14:27 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-magru
• 14:26 jforrester@deploy1003: jforrester: Backport for Factor out Abstract Wikipedia client mode into a dblist, Wikifunctions: Switch the editor from Ace to CodeMirror (T400015) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2127.codfw.wmnet
• 14:24 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-magru
• 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2127.codfw.wmnet
• 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2126.codfw.wmnet
• 14:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2126.codfw.wmnet
• 14:22 jforrester@deploy1003: Started scap sync-world: Backport for Factor out Abstract Wikipedia client mode into a dblist, Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)
• 14:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
• 14:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
• 14:20 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs T438219
• 14:19 moritzm: installing postgresql-15 security updates
• 14:18 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 14:18 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 14:18 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:17 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:16 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:16 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:14 vgutierrez: upload liberica 0.26 to bookworm-wikimedia (apt.wm.o) - T439840
• 14:12 apine@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 14:12 apine@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:11 apine@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:11 apine@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:10 apine@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:10 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 14:09 apine@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2126.codfw.wmnet
• 14:05 jforrester@deploy1003: Finished scap sync-world: Backport for PublishDialog: Use one slot for the summary field warning (T372149 T440446) (duration: 10m 59s)
• 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2126.codfw.wmnet
• 14:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2125.codfw.wmnet
• 14:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2125.codfw.wmnet
• 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica
• 14:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7003.magru.wmnet} and A:liberica
• 14:04 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica
• 14:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica
• 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica
• 14:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica
• 14:00 jforrester@deploy1003: daphnesmit, jforrester: Continuing with deployment
• 13:59 jforrester@deploy1003: daphnesmit, jforrester: Backport for PublishDialog: Use one slot for the summary field warning (T372149 T440446) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:54 jforrester@deploy1003: Started scap sync-world: Backport for PublishDialog: Use one slot for the summary field warning (T372149 T440446)
• 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica
• 13:54 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P{lvs7003.magru.wmnet} and A:liberica
• 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:54 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica
• 13:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica
• 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica
• 13:53 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica
• 13:49 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2125.codfw.wmnet
• 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2125.codfw.wmnet
• 13:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2124.codfw.wmnet
• 13:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet
• 13:45 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade
• 13:44 papaul: start maintenance on cr2-codfw
• 13:42 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for Group fix for incorrect group mapping. (T439067) (duration: 15m 31s)
• 13:37 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment
• 13:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2124.codfw.wmnet
• 13:31 seanleong-wmde@deploy1003: seanleong-wmde: Backport for Group fix for incorrect group mapping. (T439067) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2124.codfw.wmnet
• 13:31 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2124-2140].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:27 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad
• 13:27 seanleong-wmde@deploy1003: Started scap sync-world: Backport for Group fix for incorrect group mapping. (T439067)
• 13:27 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad
• 13:22 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad
• 13:21 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad
• 13:19 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429) (duration: 11m 07s)
• 13:15 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad
• 13:15 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad
• 13:14 seanleong-wmde@deploy1003: seanleong-wmde: Continuing with deployment
• 13:13 moritzm: installing mbedtls security updates
• 13:12 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad
• 13:12 seanleong-wmde@deploy1003: seanleong-wmde: Backport for UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:12 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad
• 13:08 seanleong-wmde@deploy1003: Started scap sync-world: Backport for UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)
• 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2115.codfw.wmnet
• 13:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2115.codfw.wmnet
• 13:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
• 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
• 12:54 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
• 12:54 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
• 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2115.codfw.wmnet
• 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2115.codfw.wmnet
• 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2114.codfw.wmnet
• 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2114.codfw.wmnet
• 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad
• 12:47 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-a8-eqiad
• 12:47 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad
• 12:46 cmooney@cumin1004: START - Cookbook sre.network.tls for network device ssw1-a8-eqiad
• 12:45 moritzm: uploaded librsvg 2.60.0+dfsg-1+wmf13u2 to component/thumbor (backport of RUSTSEC-2026-0305 to our internal build)
• 12:38 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2114.codfw.wmnet
• 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2114.codfw.wmnet
• 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2113.codfw.wmnet
• 12:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2113.codfw.wmnet
• 12:37 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:29 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:29 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2113.codfw.wmnet
• 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2113.codfw.wmnet
• 12:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2112.codfw.wmnet
• 12:28 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2112.codfw.wmnet
• 12:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: apply
• 12:26 oblivian@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: apply
• 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: apply
• 12:25 oblivian@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: apply
• 12:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2112.codfw.wmnet
• 12:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2112.codfw.wmnet
• 12:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2111.codfw.wmnet
• 12:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet
• 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2111.codfw.wmnet
• 12:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2111.codfw.wmnet
• 12:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2110.codfw.wmnet
• 12:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2110.codfw.wmnet
• 12:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:57 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2110.codfw.wmnet
• 11:56 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2110.codfw.wmnet
• 11:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2109.codfw.wmnet
• 11:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet
• 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2109.codfw.wmnet
• 11:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2109.codfw.wmnet
• 11:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2108.codfw.wmnet
• 11:45 hnowlan: remove graphite config from puppet, moved graphite hosts to insetup
• 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2108.codfw.wmnet
• 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2108.codfw.wmnet
• 11:31 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2108.codfw.wmnet
• 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet
• 11:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet
• 11:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2107.codfw.wmnet
• 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2107.codfw.wmnet
• 11:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet
• 11:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet
• 11:21 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs T438219
• 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica
• 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7003.magru.wmnet} and A:liberica
• 11:09 andre: Rolling back again wmf.23 from group1 to group0 - T438219
• 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica
• 11:09 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica
• 11:09 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica
• 11:08 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica
• 11:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2106.codfw.wmnet
• 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2106.codfw.wmnet
• 11:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet
• 11:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet
• 11:02 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs7003.magru.wmnet} and A:liberica
• 11:02 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs7003.magru.wmnet} and A:liberica
• 10:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica
• 10:55 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P{lvs7003.magru.wmnet} and A:liberica
• 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica
• 10:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica
• 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica
• 10:55 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica
• 10:50 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:45 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2105.codfw.wmnet
• 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2105.codfw.wmnet
• 10:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2104.codfw.wmnet
• 10:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2104.codfw.wmnet
• 10:39 blake@deploy1003: Finished scap sync-world: update vendored modules for T417800 (duration: 07m 32s)
• 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:33 blake@deploy1003: Started scap sync-world: update vendored modules for T417800
• 10:32 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 10:30 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 10:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2104.codfw.wmnet
• 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2104.codfw.wmnet
• 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet
• 10:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet
• 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 10:26 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 10:25 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 10:19 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 10:17 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
• 10:16 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
• 10:16 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 10:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2103.codfw.wmnet
• 10:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2103.codfw.wmnet
• 10:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2102.codfw.wmnet
• 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 10:07 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 10:05 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2102.codfw.wmnet
• 10:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2102.codfw.wmnet
• 09:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2102.codfw.wmnet
• 09:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet
• 09:54 duesen: just a test for SAL
• 09:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet
• 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:50 daniel@deploy1003: Finished scap sync-world: Backport for REST: Accept message objects as parameter help (T433534 T440377) (duration: 37m 19s)
• 09:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2095.codfw.wmnet
• 09:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2095.codfw.wmnet
• 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2094.codfw.wmnet
• 09:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2094.codfw.wmnet
• 09:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 09:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 09:36 daniel@deploy1003: daniel: Continuing with deployment
• 09:36 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
• 09:35 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
• 09:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2094.codfw.wmnet
• 09:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2094.codfw.wmnet
• 09:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2093.codfw.wmnet
• 09:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2093.codfw.wmnet
• 09:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:26 moritzm: restarting pybal on lvs1018 T439730
• 09:20 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2093.codfw.wmnet
• 09:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2093.codfw.wmnet
• 09:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2092.codfw.wmnet
• 09:19 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2092.codfw.wmnet
• 09:17 daniel@deploy1003: daniel: Backport for REST: Accept message objects as parameter help (T433534 T440377) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 09:13 daniel@deploy1003: Started scap sync-world: Backport for REST: Accept message objects as parameter help (T433534 T440377)
• 09:02 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] (duration: 04m 42s)
• 08:57 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33]
• 08:56 aqu@deploy1003: Finished deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] (duration: 00m 44s)
• 08:55 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:55 aqu@deploy1003: Started deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33]
• 08:50 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet
• 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet
• 08:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet
• 08:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet
• 08:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:29 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs T438219
• 08:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2091.codfw.wmnet
• 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2091.codfw.wmnet
• 08:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2090.codfw.wmnet
• 08:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2090.codfw.wmnet
• 08:20 aklapper@deploy1003: Rolling back deployment
• 08:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2090.codfw.wmnet
• 08:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2090.codfw.wmnet
• 08:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2089.codfw.wmnet
• 08:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2089.codfw.wmnet
• 08:02 awight@deploy1003: Finished scap sync-world: Backport for Adds metrics for entity usage limit baseline (T439125) (duration: 37m 41s)
• 07:57 awight@deploy1003: nicholusmuwonge, awight: Continuing with deployment
• 07:56 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2089.codfw.wmnet
• 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2089.codfw.wmnet
• 07:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet
• 07:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet
• 07:42 awight@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
• 07:41 awight@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
• 07:40 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
• 07:40 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
• 07:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet
• 07:39 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
• 07:39 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
• 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet
• 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2087.codfw.wmnet
• 07:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2087.codfw.wmnet
• 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
• 07:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:39 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
• 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
• 07:38 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
• 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
• 07:37 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
• 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'.
• 07:36 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'.
• 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
• 07:35 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
• 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'.
• 07:34 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'.
• 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
• 07:33 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
• 07:30 moritzm: uploaded openjdk-21 21.0.12.1+1-1~deb12u1 to component/jdk21 for bookworm-wikimedia T392465
• 07:29 awight@deploy1003: nicholusmuwonge, awight: Backport for Adds metrics for entity usage limit baseline (T439125) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:28 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2087.codfw.wmnet
• 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2087.codfw.wmnet
• 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet
• 07:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet
• 07:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:26 elukey@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
• 07:26 elukey@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
• 07:25 elukey@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
• 07:25 awight@deploy1003: Started scap sync-world: Backport for Adds metrics for entity usage limit baseline (T439125)
• 07:24 elukey@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
• 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
• 07:22 elukey@deploy1003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
• 07:20 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2245: Pooling back after cloning db2250
• 07:19 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:19 elukey@deploy1003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
• 07:19 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2078.codfw.wmnet
• 07:18 elukey@deploy1003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
• 07:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2078.codfw.wmnet
• 07:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet
• 07:17 elukey@deploy1003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
• 07:16 elukey@deploy1003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
• 07:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet
• 07:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'.
• 07:14 elukey@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'.
• 07:12 elukey@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'.
• 07:11 elukey@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'.
• 07:08 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
• 07:08 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2077.codfw.wmnet
• 07:08 elukey@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
• 07:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2077.codfw.wmnet
• 07:06 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet
• 07:06 elukey@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'.
• 07:05 elukey@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'.
• 07:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet
• 07:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:56 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:56 moritzm: installing bird security updates
• 06:56 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2076.codfw.wmnet
• 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2076.codfw.wmnet
• 06:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet
• 06:55 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet
• 06:44 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2075.codfw.wmnet
• 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2075.codfw.wmnet
• 06:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet
• 06:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet
• 06:35 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2245: Pooling back after cloning db2250
• 06:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:22 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2074.codfw.wmnet
• 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet
• 06:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2073.codfw.wmnet
• 06:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2073.codfw.wmnet
• 06:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:11 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2073.codfw.wmnet
• 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2073.codfw.wmnet
• 06:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2072.codfw.wmnet
• 06:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2072.codfw.wmnet
• 06:09 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 06:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2072.codfw.wmnet
• 06:00 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2072.codfw.wmnet
• 06:00 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2071.codfw.wmnet
• 05:59 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2071.codfw.wmnet
• 05:59 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:54 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning
• 05:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2071.codfw.wmnet
• 05:50 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2071.codfw.wmnet
• 05:50 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2070.codfw.wmnet
• 05:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2070.codfw.wmnet
• 05:48 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:41 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning
• 05:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2070.codfw.wmnet
• 05:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2070.codfw.wmnet
• 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2069.codfw.wmnet
• 05:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2069.codfw.wmnet
• 05:39 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2069.codfw.wmnet
• 05:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2069.codfw.wmnet
• 05:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2068.codfw.wmnet
• 05:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2068.codfw.wmnet
• 05:29 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:21 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2068.codfw.wmnet
• 05:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2068.codfw.wmnet
• 05:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2067.codfw.wmnet
• 05:18 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2067.codfw.wmnet
• 05:18 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:10 kart_: Updated Apertium to 2026-09-30-051826-production (T437213)
• 05:09 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 05:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2067.codfw.wmnet
• 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2067.codfw.wmnet
• 05:08 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 05:04 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
• 05:04 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/apertium: apply
• 05:01 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
• 05:00 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/apertium: apply
• 04:54 marostegui: Stop mariadb on x4 db2245 to clone the backup host T437279
• 04:52 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: cloning
• 04:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2245: to clone the backup host T437279
• 04:51 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2245: to clone the backup host T437279
• 04:50 marostegui: Stop mariadb on x4 db2245 to clone the backup host
• 04:50 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/apertium: apply
• 04:50 kartik@deploy1003: helmfile [staging] START helmfile.d/services/apertium: apply
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-10-06

• 22:07 jdlrobson@deploy1003: Finished scap sync-world: Backport for Revert "Scroll the TOC along with the page scrolling" (T440296) (duration: 11m 24s)
• 22:01 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
• 22:00 jdlrobson@deploy1003: jdlrobson: Backport for Revert "Scroll the TOC along with the page scrolling" (T440296) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:55 jdlrobson@deploy1003: Started scap sync-world: Backport for Revert "Scroll the TOC along with the page scrolling" (T440296)
• 20:15 catrope@deploy1003: Finished scap sync-world: Backport for Email confirmation: Temporarily disable banner modification, Email confirmation: Temporarily disable banner modification, EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate(), EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate() (duration: 12m 17
• 20:10 catrope@deploy1003: catrope: Continuing with deployment
• 20:07 catrope@deploy1003: catrope: Backport for Email confirmation: Temporarily disable banner modification, Email confirmation: Temporarily disable banner modification, EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate(), EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate() synced to the testservers (see htt
• 20:03 catrope@deploy1003: Started scap sync-world: Backport for Email confirmation: Temporarily disable banner modification, Email confirmation: Temporarily disable banner modification, EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate(), EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()
• 18:48 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
• 18:48 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
• 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Pooling back db1218 after stress test
• 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pooling back db2209 after stress test
• 18:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: Pooling back db2207 after stress test
• 18:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet
• 18:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet
• 17:47 marostegui: Repool db1218, db2207 and db2209 after stress testing
• 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1218: Pooling back db1218 after stress test
• 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2209: Pooling back db2209 after stress test
• 17:47 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test
• 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2055-2062,2064-2065].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 17:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2065.codfw.wmnet
• 17:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2065.codfw.wmnet
• 17:46 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2207: Pooling back db2207 after stress test
• 17:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:44 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test
• 17:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2065.codfw.wmnet
• 17:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2065.codfw.wmnet
• 17:35 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2064.codfw.wmnet
• 17:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2064.codfw.wmnet
• 17:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:33 fceratto@cumin1004: END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1901: test
• 17:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2064.codfw.wmnet
• 17:24 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2064.codfw.wmnet
• 17:24 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2062.codfw.wmnet
• 17:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2062.codfw.wmnet
• 17:22 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2062.codfw.wmnet
• 17:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2062.codfw.wmnet
• 17:13 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2061.codfw.wmnet
• 17:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2061.codfw.wmnet
• 17:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:12 bking@cumin2003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM matomo1003.eqiad.wmnet
• 17:08 bking@cumin2003: START - Cookbook sre.ganeti.reboot-vm for VM matomo1003.eqiad.wmnet
• 17:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:05 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2061.codfw.wmnet
• 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2061.codfw.wmnet
• 17:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2060.codfw.wmnet
• 17:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2060.codfw.wmnet
• 17:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:53 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2060.codfw.wmnet
• 16:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2060.codfw.wmnet
• 16:52 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2059.codfw.wmnet
• 16:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2059.codfw.wmnet
• 16:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2059.codfw.wmnet
• 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2059.codfw.wmnet
• 16:42 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2058.codfw.wmnet
• 16:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2058.codfw.wmnet
• 16:41 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:39 bking@dns2004: END - running authdns-update
• 16:37 bking@dns2004: START - running authdns-update
• 16:34 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2058.codfw.wmnet
• 16:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2058.codfw.wmnet
• 16:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2057.codfw.wmnet
• 16:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2057.codfw.wmnet
• 16:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2057.codfw.wmnet
• 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2057.codfw.wmnet
• 16:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2056.codfw.wmnet
• 16:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2056.codfw.wmnet
• 16:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2056.codfw.wmnet
• 16:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2056.codfw.wmnet
• 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2055.codfw.wmnet
• 16:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2055.codfw.wmnet
• 16:12 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2055.codfw.wmnet
• 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2055.codfw.wmnet
• 16:04 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2055-2062,2064-2065].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 15:58 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
• 15:56 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
• 15:56 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 15:56 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
• 15:55 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 15:55 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 15:54 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 15:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
• 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 15:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 15:36 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
• 15:35 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
• 15:26 daniel@deploy1003: Finished scap sync-world: Backport for REST: enable the content/v2-beta module on testwiki (T433534) (duration: 15m 44s)
• 15:21 daniel@deploy1003: daniel: Continuing with deployment
• 15:17 daniel@deploy1003: daniel: Backport for REST: enable the content/v2-beta module on testwiki (T433534) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:11 moritzm: installing python3.13 security updates
• 15:11 jayme@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 15:11 jayme@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 15:11 daniel@deploy1003: Started scap sync-world: Backport for REST: enable the content/v2-beta module on testwiki (T433534)
• 15:11 jayme@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 15:10 jayme@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 15:08 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 15:07 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw
• 15:06 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 15:04 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab1005 for T440340 (duration: 00m 42s)
• 15:03 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab1005 for T440340
• 15:03 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1005.eqiad.wmnet with reason: Deployment
• 15:03 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
• 15:03 jayme@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 15:03 jayme@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 15:03 brennen@deploy1003: Finished deploy [phabricator/deployment@2b620eb]: deploy phab2003 for T440340 (duration: 00m 51s)
• 15:02 jayme@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 15:02 jayme@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 15:02 brennen@deploy1003: Started deploy [phabricator/deployment@2b620eb]: deploy phab2003 for T440340
• 15:02 aokoth@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet with reason: Deployment
• 15:02 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
• 14:59 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
• 14:59 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
• 14:57 moritzm: restarting pybal on lvs1018 T439730
• 14:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet
• 14:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet
• 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 14:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2051.codfw.wmnet
• 14:48 moritzm: restarting pybal on lvs1020 T439730
• 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet
• 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:41 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw
• 14:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet
• 14:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet
• 14:39 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2050.codfw.wmnet
• 14:38 ladsgroup@deploy1003: Finished scap sync-world: Backport for Enable $wgCiteDefaultRefDirAuto on fawiki (T432544) (duration: 11m 37s)
• 14:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet
• 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:33 ladsgroup@deploy1003: ladsgroup, mamouri: Continuing with deployment
• 14:33 moritzm: restarting pybal on lvs2012 T439730
• 14:32 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2050.codfw.wmnet
• 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2050.codfw.wmnet
• 14:31 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqsin
• 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet
• 14:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet
• 14:31 ladsgroup@deploy1003: ladsgroup, mamouri: Backport for Enable $wgCiteDefaultRefDirAuto on fawiki (T432544) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:28 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 14:27 ladsgroup@deploy1003: Started scap sync-world: Backport for Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)
• 14:27 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 14:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:23 jelto@deploy1003: Finished scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig (duration: 01m 54s)
• 14:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2049.codfw.wmnet
• 14:23 moritzm: restarting pybal on lvs2014 T439730
• 14:23 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb
• 14:22 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb
• 14:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2049.codfw.wmnet
• 14:22 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2046.codfw.wmnet
• 14:21 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb
• 14:21 jelto@deploy1003: Started scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig
• 14:21 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet
• 14:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:13 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2046.codfw.wmnet
• 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2046.codfw.wmnet
• 14:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet
• 14:12 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet
• 14:11 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• {{safesubst:SAL entry|1=14:08 ladsgroup@deploy1003: Finished scap sync-world: Backport for fix(AccountSetup): Send onboarding exposure from the post-signup redirect, refactor(AccountSetup): Move post-signup onboarding checks to a shared service, fix(AccountSetup): Send onboarding exposure from the post-signup redirect, [[gerrit:1352031|refactor(AccountSetup): Move post-signup onboarding chec}}
• 14:06 ladsgroup@deploy1003: migr, ladsgroup: Continuing with deployment
• 14:06 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqsin
• 14:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet
• 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet
• 14:03 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2042.codfw.wmnet
• {{safesubst:SAL entry|1=14:03 ladsgroup@deploy1003: migr, ladsgroup: Backport for fix(AccountSetup): Send onboarding exposure from the post-signup redirect, refactor(AccountSetup): Move post-signup onboarding checks to a shared service, fix(AccountSetup): Send onboarding exposure from the post-signup redirect, [[gerrit:1352031|refactor(AccountSetup): Move post-signup onboarding checks to a s}}
• 14:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet
• 14:02 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• {{safesubst:SAL entry|1=14:01 ladsgroup@deploy1003: Started scap sync-world: Backport for fix(AccountSetup): Send onboarding exposure from the post-signup redirect, refactor(AccountSetup): Move post-signup onboarding checks to a shared service, fix(AccountSetup): Send onboarding exposure from the post-signup redirect, [[gerrit:1352031|refactor(AccountSetup): Move post-signup onboarding check}}
• 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 13:55 ladsgroup@deploy1003: Finished scap sync-world: Backport for ltwiki: Enable extended confirmed user group (T440216) (duration: 13m 47s)
• 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
• 13:55 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
• 13:54 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2042.codfw.wmnet
• 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2042.codfw.wmnet
• 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:50 ladsgroup@deploy1003: tryvix1509, ladsgroup: Continuing with deployment
• 13:46 ladsgroup@deploy1003: tryvix1509, ladsgroup: Backport for ltwiki: Enable extended confirmed user group (T440216) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:42 ladsgroup@deploy1003: Started scap sync-world: Backport for ltwiki: Enable extended confirmed user group (T440216)
• 13:38 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet
• 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet
• 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet
• 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet
• 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2033-2039].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:35 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2033-2039].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:34 moritzm: installing php8.2 security updates
• 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:32 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 (duration: 214m 10s)
• 13:31 slyngshede@dns1004: END - running authdns-update
• 13:29 slyngshede@dns1004: START - running authdns-update
• 13:25 moritzm: installing libssh2 security updates
• 13:20 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
• 13:18 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
• 13:14 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
• 13:14 moritzm: installing glibc security updates
• 13:11 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
• 13:04 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
• 13:04 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
• 13:03 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
• 13:02 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply
• 12:59 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply
• 12:58 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply
• 12:45 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
• 12:44 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
• 12:41 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
• 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
• 12:40 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
• 12:39 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 12:39 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db2*-bin.* -maxdepth 1 -type f -mtime +365 -delete'
• 12:38 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 12:38 Amir1: sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db1*-bin.* -maxdepth 1 -type f -mtime +365 -delete'
• 12:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 12:02 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 11:56 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 11:52 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 11:41 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
• 11:41 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
• 11:33 moritzm: installing chromium security updates
• 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 11:21 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004"
• 11:21 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004"
• 11:09 blake@deploy1003: helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply
• 11:08 blake@deploy1003: helmfile [staging] START helmfile.d/services/mw-mcrouter: apply
• 11:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:02 jmm@cumin2003: START - Cookbook sre.dns.netbox
• 11:01 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage
• 10:56 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage
• 10:40 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 10:38 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1055.eqiad.wmnet with OS bookworm
• 10:38 filippo@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004"
• 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: pooling in s7
• 10:37 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling back
• 10:37 filippo@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004"
• 10:19 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 10:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 10:18 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage
• 10:14 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 visualeditor-editor (T406724)
• 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Pooling back pc1
• 10:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache
• 10:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool pc1021: Pooling back pc1
• 10:12 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 10:12 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage
• 10:10 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
• 10:10 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
• 10:10 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 10:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrading
• 10:10 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 10:09 gkyziridis@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' .
• 10:09 marostegui: Upgrade pc1 to mariadb 10.11.19 T436082
• 10:09 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' .
• 10:09 moritzm: installing curl security updates
• 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: testing
• 10:08 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.parsercache
• 10:08 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool pc1021: testing
• 10:02 elukey: `elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m | tee gc_run_06102026` - T437297
• 09:58 gkyziridis@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
• 09:58 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' .
• 09:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling in s7
• 09:58 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297
• 09:57 elukey@deploy1003: Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 (duration: 00m 19s)
• 09:57 elukey@deploy1003: Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297
• 09:55 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm
• 09:53 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2228: pooling in s7
• 09:52 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling back
• 09:45 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply
• 09:45 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply
• 09:45 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
• 09:45 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
• 09:32 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 09:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet
• 09:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet
• 09:15 oblivian@cumin1004: END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet
• 09:14 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet
• 09:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet
• 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet
• 09:13 oblivian@cumin1004: START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 09:13 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling in s7
• 09:13 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: testing
• 09:12 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: testing
• 09:12 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 09:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 09:02 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 09:00 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 08:56 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 08:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 08:30 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2169,2228].codfw.wmnet with reason: cloning
• 08:30 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: moving cloning
• 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2228: moving cloning
• 08:29 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving cloning
• 08:29 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving cloning
• 08:28 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
• 08:28 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 08:28 aklapper@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs T438219
• 08:17 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 08:16 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
• 08:16 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
• 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 08:14 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
• 08:14 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
• 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
• 08:14 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
• 08:14 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
• 08:13 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 08:10 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling in s7
• 08:04 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling back
• 08:00 matthiasmullie: UTC morning backport window done
• 08:00 mlitn@deploy1003: Finished scap sync-world: Backport for Properly render carousel preview dialog sizing in Webkit (T439007), Change lead section infobox exclusion (T439943), [Image Browsing] Prevent layout shift when loading carousel experiment (T439948) (duration: 10m 18s)
• 07:55 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment
• 07:54 mlitn@deploy1003: mlitn, ksarabia: Backport for Properly render carousel preview dialog sizing in Webkit (T439007), Change lead section infobox exclusion (T439943), [Image Browsing] Prevent layout shift when loading carousel experiment (T439948) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:50 mlitn@deploy1003: Started scap sync-world: Backport for Properly render carousel preview dialog sizing in Webkit (T439007), Change lead section infobox exclusion (T439943), [Image Browsing] Prevent layout shift when loading carousel experiment (T439948)
• 07:45 dcausse@deploy1003: Finished scap sync-world: Backport for Do not suggest duplicates when completing special pages (T440147), Find canonical pages on special page completion search (T440151) (duration: 13m 53s)
• 07:38 dcausse@deploy1003: dcausse: Continuing with deployment
• 07:38 dcausse@deploy1003: dcausse: Backport for Do not suggest duplicates when completing special pages (T440147), Find canonical pages on special page completion search (T440151) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:34 fceratto@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test
• 07:33 fceratto@cumin1004: START - Cookbook sre.mysql.pool pool db1901: test
• 07:31 dcausse@deploy1003: Started scap sync-world: Backport for Do not suggest duplicates when completing special pages (T440147), Find canonical pages on special page completion search (T440151)
• 07:30 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 07:25 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2169: Pooling in s7
• 07:21 awight@deploy1003: Finished scap sync-world: Backport for Config change for launch of stopping sending LL notifications into group1. (T437577) (duration: 17m 31s)
• 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2168: Pooling back
• 07:08 awight@deploy1003: awight, seanleong-wmde: Continuing with deployment
• 07:08 awight@deploy1003: awight, seanleong-wmde: Backport for Config change for launch of stopping sending LL notifications into group1. (T437577) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:03 awight@deploy1003: Started scap sync-world: Backport for Config change for launch of stopping sending LL notifications into group1. (T437577)
• 05:36 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet
• 05:36 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet
• 05:31 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: moving to s7
• 05:31 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2168: moving to s7
• 05:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2168-2169].codfw.wmnet with reason: cloning
• 05:26 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet
• 05:26 marostegui@cumin1004: START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet
• 05:21 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving to s7
• 05:19 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2169: moving to s7
• 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.20 (duration: 01m 00s)
• 03:30 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.23 refs T438219 (duration: 27m 17s)
• 03:28 mwpresync@deploy1003: Rolling back deployment
• 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.23 refs T438219
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 37s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-10-05

• 23:24 derenrich@deploy1003: Finished scap sync-world: Backport for Enable the discord previews from WikimediaCustomizations on enwiki (T439914) (duration: 11m 59s)
• 23:18 derenrich@deploy1003: derenrich: Continuing with deployment
• 23:16 derenrich@deploy1003: derenrich: Backport for Enable the discord previews from WikimediaCustomizations on enwiki (T439914) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 23:12 derenrich@deploy1003: Started scap sync-world: Backport for Enable the discord previews from WikimediaCustomizations on enwiki (T439914)
• 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1003.eqiad.wmnet
• 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 22:00 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
• 22:00 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
• 21:53 dzahn@cumin1004: START - Cookbook sre.dns.netbox
• 21:48 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1003.eqiad.wmnet
• 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1002.eqiad.wmnet
• 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 21:45 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
• 21:45 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
• 21:39 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet
• 21:39 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet
• 21:38 dzahn@cumin1004: START - Cookbook sre.dns.netbox
• 21:35 jasmine@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts testreduce1002.eqiad.wmnet
• 21:34 jasmine@cumin2003: START - Cookbook sre.hosts.decommission for hosts testreduce1002.eqiad.wmnet
• 21:33 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1002.eqiad.wmnet
• 21:10 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 21:05 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 21:02 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:58 logmsgbot: dreamyjazz Deployed security patch for T439719
• 20:50 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:45 logmsgbot: dreamyjazz Deployed security patch for T439974
• 20:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-ntp (exit_code=0) rolling restart_daemons on A:dnsbox
• 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 20:01 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 19:51 logmsgbot: dreamyjazz Deployed security patch for T439932
• 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 19:33 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 19:09 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-esams
• 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 18:58 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 18:46 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 18:45 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 18:44 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-esams
• 18:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-drmrs
• 18:23 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 18:14 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-drmrs
• 18:08 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo
• 17:58 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 17:58 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 17:58 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
• 17:57 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
• 17:53 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 17:52 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 17:46 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
• 17:46 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
• 17:43 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo
• 17:42 sbassett: deployed csv file to security.wikimedia.org (miscweb)
• 17:41 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-ulsfo
• 17:40 swfrench@deploy1003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 17:39 swfrench@deploy1003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 17:33 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Maliner --performer=Maliner --as-job # T439577
• 17:29 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deployment after logging and configuration changes (duration: 01m 31s)
• 17:29 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
• 17:28 swfrench@deploy1003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
• 17:27 swfrench@deploy1003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
• 17:27 swfrench@deploy1003: Started scap deploy-service shellbox: Test deployment after logging and configuration changes
• 17:16 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
• 17:16 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-ulsfo
• 17:15 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-magru
• 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
• 17:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply
• 17:15 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply
• 17:14 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
• 17:13 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
• 17:12 swfrench@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply
• 17:11 swfrench@deploy1003: helmfile [codfw] START helmfile.d/services/mw-pretrain: apply
• 16:50 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-magru
• 16:32 dcausse: closing the "Distributed search queries experiment" window
• 16:30 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: test perf of moving completion out of eqiad" (T438984) (duration: 10m 48s)
• 16:25 dcausse@deploy1003: dcausse: Continuing with deployment
• 16:23 dcausse@deploy1003: dcausse: Backport for Revert "search: test perf of moving completion out of eqiad" (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 16:19 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-ntp rolling restart_daemons on A:dnsbox
• 16:19 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: test perf of moving completion out of eqiad" (T438984)
• 16:16 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqiad
• 16:05 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: measure impact of enwiki" (T438984), search: test perf of moving completion out of eqiad (T438984) (duration: 10m 44s)
• 15:59 dcausse@deploy1003: dcausse: Continuing with deployment
• 15:58 dcausse@deploy1003: dcausse: Backport for Revert "search: measure impact of enwiki" (T438984), search: test perf of moving completion out of eqiad (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:54 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: measure impact of enwiki" (T438984), search: test perf of moving completion out of eqiad (T438984)
• 15:51 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqiad
• 15:47 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: test impact of cross project searches" (T438984), search: measure impact of enwiki (T438984) (duration: 25m 12s)
• 15:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm
• 15:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2039.codfw.wmnet
• 15:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet
• 15:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox
• 15:33 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet
• 15:32 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
• 15:32 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
• 15:32 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet
• 15:32 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
• 15:32 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
• 15:32 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2039.codfw.wmnet
• 15:32 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage
• 15:30 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
• 15:30 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
• 15:29 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
• 15:29 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2039.codfw.wmnet
• 15:29 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
• 15:29 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage
• 15:28 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:27 dcausse@deploy1003: dcausse: Continuing with deployment
• 15:26 dcausse@deploy1003: dcausse: Backport for Revert "search: test impact of cross project searches" (T438984), search: measure impact of enwiki (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:23 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2039.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2039.codfw.wmnet
• 15:22 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: test impact of cross project searches" (T438984), search: measure impact of enwiki (T438984)
• 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2039.codfw.wmnet
• 15:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2038.codfw.wmnet
• 15:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet
• 15:21 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2038.codfw.wmnet
• 15:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2038.codfw.wmnet
• 15:20 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 15:17 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 15:16 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm
• 15:16 elukey@dns1004: END - running authdns-update
• 15:15 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: test impact of the function score window size" (T438984), search: test impact of cross project searches (T438984) (duration: 10m 11s)
• 15:14 elukey@dns1004: START - running authdns-update
• 15:12 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2038.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 15:12 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2038.codfw.wmnet
• 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 15:12 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 15:12 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 15:11 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2038.codfw.wmnet
• 15:11 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2037.codfw.wmnet
• 15:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2037.codfw.wmnet
• 15:10 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:10 dcausse@deploy1003: dcausse: Continuing with deployment
• 15:09 dcausse@deploy1003: dcausse: Backport for Revert "search: test impact of the function score window size" (T438984), search: test impact of cross project searches (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:05 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: test impact of the function score window size" (T438984), search: test impact of cross project searches (T438984)
• 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 15:02 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2037.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:02 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2037.codfw.wmnet
• 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2037.codfw.wmnet
• 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2036.codfw.wmnet
• 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2036.codfw.wmnet
• 15:00 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm
• 14:58 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: test impact of MLR" (T438984), search: test impact of the function score window size (T438984) (duration: 10m 12s)
• 14:53 dcausse@deploy1003: dcausse: Continuing with deployment
• 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 14:53 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 14:52 dcausse@deploy1003: dcausse: Backport for Revert "search: test impact of MLR" (T438984), search: test impact of the function score window size (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2036.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:49 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 14:49 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 14:49 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 14:49 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 14:48 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: test impact of MLR" (T438984), search: test impact of the function score window size (T438984)
• 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 14:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2036.codfw.wmnet
• 14:48 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply
• 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2036.codfw.wmnet
• 14:47 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2035.codfw.wmnet
• 14:47 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet
• 14:47 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
• 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 14:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2035.codfw.wmnet
• 14:46 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
• 14:46 dancy@deploy1003: Installation of scap version "4.294.0" completed for 3 hosts
• 14:46 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 14:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2035.codfw.wmnet
• 14:45 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 14:45 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 14:45 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 14:45 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:44 dancy@deploy1003: Installing scap version "4.294.0" for 3 host(s)
• 14:44 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 14:43 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 14:43 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 14:43 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 14:42 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 14:38 sbassett@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 14:38 dcausse@deploy1003: Finished scap sync-world: Backport for Revert "search: test impact of the MLR window size" (T438984), search: test impact of MLR (T438984) (duration: 10m 04s)
• 14:38 sbassett@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 14:38 sbassett@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 14:37 sbassett@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage
• 14:36 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2035.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:36 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2035.codfw.wmnet
• 14:36 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox
• 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2035.codfw.wmnet
• 14:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2034.codfw.wmnet
• 14:35 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet
• 14:34 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2034.codfw.wmnet
• 14:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2034.codfw.wmnet
• 14:33 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:33 dcausse@deploy1003: dcausse: Continuing with deployment
• 14:32 dcausse@deploy1003: dcausse: Backport for Revert "search: test impact of the MLR window size" (T438984), search: test impact of MLR (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:32 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage
• 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 14:30 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 14:28 dcausse@deploy1003: Started scap sync-world: Backport for Revert "search: test impact of the MLR window size" (T438984), search: test impact of MLR (T438984)
• 14:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2034.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:26 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2034.codfw.wmnet
• 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2034.codfw.wmnet
• 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2033.codfw.wmnet
• 14:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2033.codfw.wmnet
• 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:20 dcausse@deploy1003: Finished scap sync-world: Backport for search: test impact of the MLR window size (T438984) (duration: 11m 12s)
• 14:20 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm
• 14:20 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1006.eqiad.wmnet with OS bookworm
• 14:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2033.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:18 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2033.codfw.wmnet
• 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2033.codfw.wmnet
• 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2018.codfw.wmnet
• 14:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2018.codfw.wmnet
• 14:16 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:15 dcausse@deploy1003: dcausse: Continuing with deployment
• 14:13 dcausse@deploy1003: dcausse: Backport for search: test impact of the MLR window size (T438984) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 14:12 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 14:10 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2018.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:09 dcausse@deploy1003: Started scap sync-world: Backport for search: test impact of the MLR window size (T438984)
• 14:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm
• 14:07 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1006.eqiad.wmnet with OS bookworm
• 14:04 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2018.codfw.wmnet
• 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2018.codfw.wmnet
• 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2017.codfw.wmnet
• 14:03 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2017.codfw.wmnet
• 14:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 14:03 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:58 logmsgbot: dreamyjazz Deployed security patch for T439968
• 13:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 13:57 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2017.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:56 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 13:54 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 13:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2017.codfw.wmnet
• 13:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2017.codfw.wmnet
• 13:51 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2016.codfw.wmnet
• 13:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet
• 13:50 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet
• 13:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet
• 13:48 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2016.codfw.wmnet
• 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2016.codfw.wmnet
• 13:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:42 zabe@deploy1003: Finished scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753) (duration: 09m 59s)
• 13:42 elukey: restart pybal on lvs10[20,19] to pick up a new LVS service
• 13:42 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 13:40 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2016.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2016.codfw.wmnet
• 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2003.codfw.wmnet
• 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:39 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 13:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2016.codfw.wmnet
• 13:39 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2015.codfw.wmnet
• 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet
• 13:38 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet
• 13:37 zabe@deploy1003: zabe: Continuing with deployment
• 13:37 elukey: restart pybal on lvs201[34] to pick up a new LVS service
• 13:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet
• 13:36 zabe@deploy1003: zabe: Backport for Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2015.codfw.wmnet
• 13:35 tappof@cumin1004: START - Cookbook sre.dns.netbox
• 13:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2015.codfw.wmnet
• 13:34 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:32 zabe@deploy1003: Started scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for (fr|ja)wiki (T439753)
• 13:32 zabe@deploy1003: Finished scap sync-world: Backport for Enable ReadingList CTA on Minerva for all reading list wikis (T438779), Update pulsating blue dot behavior for anonymous bookmark (T438779) (duration: 11m 23s)
• 13:28 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2015.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:27 zabe@deploy1003: jdlrobson, zabe, annet: Continuing with deployment
• 13:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2015.codfw.wmnet
• 13:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2015.codfw.wmnet
• 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet
• 13:25 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 13:25 zabe@deploy1003: jdlrobson, zabe, annet: Backport for Enable ReadingList CTA on Minerva for all reading list wikis (T438779), Update pulsating blue dot behavior for anonymous bookmark (T438779) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:24 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 13:24 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 13:22 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 13:21 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 13:21 zabe@deploy1003: Started scap sync-world: Backport for Enable ReadingList CTA on Minerva for all reading list wikis (T438779), Update pulsating blue dot behavior for anonymous bookmark (T438779)
• 13:20 zabe@deploy1003: Finished scap sync-world: Backport for throttle.php: Lift IP cap for edit-a-thon (T439991) (duration: 14m 48s)
• 13:20 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2014.codfw.wmnet
• 13:20 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2014.codfw.wmnet
• 13:20 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:20 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 13:19 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2011.codfw.wmnet
• 13:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet
• 13:19 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 13:14 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 13:12 zabe@deploy1003: zabe, tryvix1509: Continuing with deployment
• 13:12 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2010.codfw.wmnet
• 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet
• 13:11 zabe@deploy1003: zabe, tryvix1509: Backport for throttle.php: Lift IP cap for edit-a-thon (T439991) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:10 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2003.codfw.wmnet
• 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2002.codfw.wmnet
• 13:05 zabe@deploy1003: Started scap sync-world: Backport for throttle.php: Lift IP cap for edit-a-thon (T439991)
• 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:05 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 13:05 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 13:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2009.codfw.wmnet
• 13:01 tappof@cumin1004: START - Cookbook sre.dns.netbox
• 13:01 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 12:59 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 12:58 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:57 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm
• 12:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:56 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2002.codfw.wmnet
• 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging2001.codfw.wmnet
• 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:55 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 12:55 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 12:51 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2014.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:50 tappof@cumin1004: START - Cookbook sre.dns.netbox
• 12:49 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2014.codfw.wmnet
• 12:49 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host an-druid1005.eqiad.wmnet with OS bookworm
• 12:49 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2014.codfw.wmnet
• 12:48 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2013.codfw.wmnet
• 12:48 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet
• 12:48 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 12:47 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet
• 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet
• 12:46 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 12:46 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2013.codfw.wmnet
• 12:46 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging2001.codfw.wmnet
• 12:45 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 12:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2013.codfw.wmnet
• 12:45 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:39 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2001.codfw.wmnet with OS trixie
• 12:38 elukey@cumin1004: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004"
• 12:38 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for instrument(AccountSetup): expose motivation data for control (T439887) (duration: 55m 26s)
• 12:35 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 12:31 moritzm: installing sqlite3 security updates
• 12:26 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Continuing with deployment
• 12:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage
• 12:16 moritzm: installing mesa security updates
• 12:12 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage
• 12:01 lucaswerkmeister-wmde@deploy1003: volker-e, lucaswerkmeister-wmde: Backport for instrument(AccountSetup): expose motivation data for control (T439887) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1005.wikimedia.org
• 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica1006.wikimedia.org
• 12:00 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2007.wikimedia.org
• 11:58 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host an-druid1005
• 11:55 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host an-druid1005
• 11:55 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2013.codfw.wmnet
• 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2013.codfw.wmnet
• 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet
• 11:54 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet
• 11:54 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:53 jmm@puppetserver1001: conftool action : set/pooled=yes; selector: name=ldap-replica2008.wikimedia.org
• 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1006.wikimedia.org
• 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica1005.wikimedia.org
• 11:53 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host an-druid1005
• 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:53 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache an-druid1005.eqiad.wmnet 13.53.64.10.in-addr.arpa 3.1.0.0.3.5.0.0.4.6.0.0.0.1.0.0.8.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:53 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004"
• 11:53 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2007.wikimedia.org
• 11:52 jmm@puppetserver1001: conftool action : set/weight=10; selector: name=ldap-replica2008.wikimedia.org
• 11:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host an-druid1005 - btullis@cumin1004"
• 11:48 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2012.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:45 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 11:45 btullis@cumin1004: START - Cookbook sre.hosts.move-vlan for host an-druid1005
• 11:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1005.eqiad.wmnet with OS bookworm
• 11:44 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2012.codfw.wmnet
• 11:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2012.codfw.wmnet
• 11:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet
• 11:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet
• 11:43 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2011.codfw.wmnet
• 11:43 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for instrument(AccountSetup): expose motivation data for control (T439887)
• 11:42 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet
• 11:42 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 11:35 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2011.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:35 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet
• 11:34 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet
• 11:34 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2006.codfw.wmnet
• 11:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2006.codfw.wmnet
• 11:31 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:29 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 11:28 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 11:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:27 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2006.codfw.wmnet
• 11:26 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2006.codfw.wmnet
• 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2005.codfw.wmnet
• 11:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2005.codfw.wmnet
• 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1004.eqiad.wmnet with OS bookworm
• 11:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:18 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:14 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2005.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:14 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2005.codfw.wmnet
• 11:13 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2005.codfw.wmnet
• 11:10 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2002.codfw.wmnet
• 11:10 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet
• 11:09 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2002.codfw.wmnet
• 11:08 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2002.codfw.wmnet
• 11:08 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 11:00 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2002.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:58 elukey@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1004"
• 10:57 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 10:55 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 10:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage
• 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2002.codfw.wmnet
• 10:52 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2002.codfw.wmnet
• 10:52 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2001.codfw.wmnet
• 10:51 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2001.codfw.wmnet
• 10:51 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:50 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage
• 10:46 tappof@deploy1003: Finished scap sync-world: T432444 - Remove kafka-logging200[1-3] (duration: 04m 44s)
• 10:43 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
• 10:43 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 10:43 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet
• 10:43 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet
• 10:42 tappof@deploy1003: Started scap sync-world: T432444 - Remove kafka-logging200[1-3]
• 10:41 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
• 10:40 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
• 10:40 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2001.codfw.wmnet with reason: host reimage
• 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2001.codfw.wmnet
• 10:40 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
• 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2001.codfw.wmnet
• 10:39 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
• 10:38 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only)
• 10:38 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only)
• 10:37 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 10:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1218.eqiad.wmnet with reason: stress testing
• 10:33 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1218: stress tests
• 10:33 marostegui: Stop MariaDB on db1218, db2207 and db2209 for stress testing
• 10:33 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db1218: stress tests
• 10:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1004.eqiad.wmnet with OS bookworm
• 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2209: stress tests
• 10:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2209: stress tests
• 10:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2209.codfw.wmnet with reason: stress testing
• 10:31 marostegui: Stop MariaDB on db2207 and db2209 for stress testing
• 10:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1007.eqiad.wmnet
• 10:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1007.eqiad.wmnet
• 10:30 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
• 10:29 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only)
• 10:29 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only)
• 10:29 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie
• 10:29 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2207.codfw.wmnet with reason: stress testing
• 10:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1003.eqiad.wmnet with OS bookworm
• 10:28 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2207: stress tests
• 10:27 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2207: stress tests
• 10:25 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1007.eqiad.wmnet
• 10:25 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1007.eqiad.wmnet
• 10:20 elukey@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2001.codfw.wmnet with OS trixie
• 10:19 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS trixie
• 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 10:16 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only)
• 10:15 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only)
• 10:14 blake@deploy1003: Finished scap sync-world: helm-only deploy for T417800 (duration: 09m 40s)
• 10:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage
• 10:06 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage
• 10:05 blake@deploy1003: Started scap sync-world: helm-only deploy for T417800
• 10:05 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only)
• 10:05 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only)
• 10:05 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 10:01 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1007.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
• 09:50 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm
• 09:50 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm
• 09:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet
• 09:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet
• 09:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet
• 09:20 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet
• 09:17 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 08:58 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS bookworm
• 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
• 08:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 08:52 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 08:46 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 08:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
• 08:37 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
• 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
• 08:35 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 08:34 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 08:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
• 08:22 volans: upgrading cloudcumin2001 in-place to trixie T436667
• 08:20 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS bookworm
• 07:49 dpogorzelski@dns1004: END - running authdns-update
• 07:46 dpogorzelski@dns1004: START - running authdns-update
• 07:42 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 07:41 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 07:40 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
• 07:39 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
• 07:18 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 07:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 07:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 07:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 46s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-10-04

• 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
• 16:13 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
• 14:17 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 14:17 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 14:17 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 14:17 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 13:27 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 13:27 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 13:27 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 13:27 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 13:25 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 13:25 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 13:25 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 13:25 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 27s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-10-03

• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-10-02

• 21:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 21:01 jclark@cumin1004: START - Cookbook sre.hosts.provision for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 20:54 inflatador: bking@an-druid100[4-7] `tune2fs -m0 /dev/mapper/vg0-srv` T440033
• 20:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet with OS bookworm
• 20:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet with OS bookworm
• 20:40 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm
• 20:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage
• 20:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage
• 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage
• 20:26 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage
• 20:23 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 20:23 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 20:23 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 20:23 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 20:22 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 20:21 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 20:21 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 20:20 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 20:17 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie
• 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2010.codfw.wmnet with OS bookworm
• 20:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2011.codfw.wmnet with OS bookworm
• 19:58 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage
• 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage
• 19:47 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003']
• 19:46 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003']
• 19:39 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie
• 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003']
• 19:37 bking@cumin2003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003']
• 19:37 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003']
• 19:33 mutante: zuul1004 - reimage stuck in d-i on leftover software RAID/LVM from its previous life as a ganeti host; wiping old md/LVM metadata and partition tables from the installer shell same fix as on zuul1005 (as also mentioned by Valerie/dcops) - T427353
• 19:08 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie
• 18:40 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet
• 18:36 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet
• 18:03 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie
• 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1001.eqiad.wmnet
• 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:01 dzahn@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
• 18:01 dzahn@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004"
• 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet
• 18:00 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet
• 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet
• 17:56 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet
• 17:51 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet
• 17:50 dzahn@cumin1004: START - Cookbook sre.dns.netbox
• 17:47 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet
• 17:46 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet
• 17:45 dzahn@cumin1004: START - Cookbook sre.hosts.decommission for hosts zuul1001.eqiad.wmnet
• 17:44 dzahn@dns1004: END - running authdns-update
• 17:42 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet
• 17:41 dzahn@dns1004: START - running authdns-update
• 17:39 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
• 17:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet
• 17:35 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet
• 17:25 bking@cumin2003: START - Cookbook sre.wdqs.reboot
• 17:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
• 16:44 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet with OS bookworm
• 16:33 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 16:31 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 16:28 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 16:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage
• 16:24 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage
• 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2003 to dse-k8s-worker2011
• 16:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2002 to dse-k8s-worker2010
• 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2011
• 16:21 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2011
• 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2011 on all recursors
• 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2010
• 16:21 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2011 on all recursors
• 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004"
• 16:13 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2010
• 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2010 on all recursors
• 16:13 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2010 on all recursors
• 16:13 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:10 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 16:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004"
• 16:07 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2009.codfw.wmnet with OS bookworm
• 16:05 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 16:05 btullis@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 16:04 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2003 to dse-k8s-worker2011
• 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2001 to dse-k8s-worker2009
• 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2009
• 16:03 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 16:03 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2009
• 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2009 on all recursors
• 16:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2009 on all recursors
• 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:02 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004"
• 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 16:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 16:02 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2002 to dse-k8s-worker2010
• 16:01 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004"
• 15:57 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 15:56 btullis@cumin1004: START - Cookbook sre.hosts.rename from ml-cache2001 to dse-k8s-worker2009
• 15:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1002.eqiad.wmnet
• 15:51 bking@cumin2003: START - Cookbook sre.wdqs.reboot
• 15:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-test-client1002.eqiad.wmnet
• 15:47 btullis@cumin1004: END (ERROR) - Cookbook sre.druid.reboot-workers (exit_code=97) for Druid analytics cluster: Reboot Druid nodes
• 15:45 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99)
• 15:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm
• 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2003.codfw.wmnet with OS bookworm
• 15:21 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 15:20 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 15:04 bking@cumin2003: START - Cookbook sre.wdqs.reboot
• 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage
• 15:00 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage
• 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 14:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 14:47 btullis@cumin1004: START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes
• 14:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet
• 14:44 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm
• 14:39 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet
• 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet
• 14:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet
• 14:32 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache2003.codfw.wmnet with OS bookworm
• 14:31 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm
• 14:30 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet
• 14:25 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 14:25 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet
• 14:23 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 14:23 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
• 14:17 btullis@cumin1004: END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons.
• 14:10 btullis@cumin1004: START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons.
• 14:03 bking@cumin2003: START - Cookbook sre.wdqs.reboot
• 13:58 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0)
• 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2002.codfw.wmnet with OS bookworm
• 13:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 13:36 bking@cumin2003: START - Cookbook sre.wdqs.reboot
• 12:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 12:36 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 12:27 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 12:20 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage
• 12:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 12:19 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only)
• 12:19 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only)
• 12:16 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage
• 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:15 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004"
• 12:15 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004"
• 12:14 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only)
• 12:14 elukey@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only)
• 12:14 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 12:11 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 12:11 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 12:01 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 12:00 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2002.codfw.wmnet with OS bookworm
• 12:00 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 11:58 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 11:56 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 11:40 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:37 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2001.codfw.wmnet with OS bookworm
• 11:08 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 10:58 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 10:55 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004"
• 10:54 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004"
• 10:47 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 10:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage
• 10:39 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage
• 10:29 moritzm: installing Linux 6.12.111 on Trixie hosts
• 10:23 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache2001.codfw.wmnet with OS bookworm
• 10:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 10:14 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
• 10:12 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
• 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
• 10:11 dpogorzelski@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
• 09:53 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
• 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:45 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004"
• 09:41 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004"
• 09:30 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 09:06 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 08:56 elukey@cumin1004: START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 08:49 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 08:48 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 08:48 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 08:47 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 07:58 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
• 07:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
• 07:52 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
• 07:52 oblivian@cumin1004: END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
• 07:47 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically)
• 06:55 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dima koushha out of all services on: 2448 hosts
• 02:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1238.eqiad.wmnet with reason: maint
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
• 00:54 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 00:54 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 00:54 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 00:53 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 00:53 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 00:52 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
• 00:39 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp11[08,10,12,14].eqiad.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()

2026-10-01

• 23:38 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp1108.*
• 23:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp11[08,10,12,14].eqiad.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 21:56 dreamyjazz@deploy1003: Finished scap sync-world: Backport for WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928) (duration: 10m 19s)
• 21:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
• 21:50 dreamyjazz@deploy1003: dreamyjazz: Backport for WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:46 dreamyjazz@deploy1003: Started scap sync-world: Backport for WikimediaAntiAbuse: Enable personal info content policy on dewiki (T439928)
• 21:33 tappof: Rebalancing has been completed on kafka-logging in codfw: kafka-logging200[1-3] are no longer assigned any partitions — T432444
• 21:24 egardner@deploy1003: Finished scap sync-world: Backport for Preferred Sources: Move all eligibility checks to the client (T436692) (duration: 11m 02s)
• 21:19 egardner@deploy1003: egardner: Continuing with deployment
• 21:17 egardner@deploy1003: egardner: Backport for Preferred Sources: Move all eligibility checks to the client (T436692) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:13 egardner@deploy1003: Started scap sync-world: Backport for Preferred Sources: Move all eligibility checks to the client (T436692)
• 21:06 brennen@deploy1003: Finished scap sync-world: Backport for Fix UTF-8 corruption in newline sanitization (T431960 T426631) (duration: 10m 43s)
• 21:00 brennen@deploy1003: brennen, srapoj: Continuing with deployment
• 20:59 brennen@deploy1003: brennen, srapoj: Backport for Fix UTF-8 corruption in newline sanitization (T431960 T426631) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:55 brennen@deploy1003: Started scap sync-world: Backport for Fix UTF-8 corruption in newline sanitization (T431960 T426631)
• 20:55 tappof: Rebalancing is in progress on kafka-logging in codfw to evacuate kafka-logging2001, kafka-logging2002 and kafka-logging2003 - T432444
• 20:53 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:48 mlitn@deploy1003: Finished scap sync-world: Backport for Lose DB call for page prop (T439182), Extract images in parser post-processing (T439182), Make carousel open MMV directly (T439611), Removes image preview from Treatment 1-2 (T439611) (duration: 10m 56s)
• 20:46 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:45 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:44 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:43 mlitn@deploy1003: mlitn, ksarabia: Continuing with deployment
• 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 20:42 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 20:41 mlitn@deploy1003: mlitn, ksarabia: Backport for Lose DB call for page prop (T439182), Extract images in parser post-processing (T439182), Make carousel open MMV directly (T439611), Removes image preview from Treatment 1-2 (T439611) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:37 mlitn@deploy1003: Started scap sync-world: Backport for Lose DB call for page prop (T439182), Extract images in parser post-processing (T439182), Make carousel open MMV directly (T439611), Removes image preview from Treatment 1-2 (T439611)
• 20:35 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:34 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:33 cjming@deploy1003: Finished scap sync-world: Backport for fix(ReadingRecommendations): Update button style to primary (T436332), instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887), instrument(AccountSetup): Record setup motivation on `page_visit` (T439217) (duration: 18m 28s)
• 20:31 bking@cumin2003: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:27 cjming@deploy1003: cjming, volker-e: Continuing with deployment
• 20:27 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:26 bking@cumin2003: END (ERROR) - Cookbook sre.hadoop.roll-restart-workers (exit_code=97) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:21 bking@cumin2003: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
• 20:18 cjming@deploy1003: cjming, volker-e: Backport for fix(ReadingRecommendations): Update button style to primary (T436332), instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887), instrument(AccountSetup): Record setup motivation on `page_visit` (T439217) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can
• 20:14 cjming@deploy1003: Started scap sync-world: Backport for fix(ReadingRecommendations): Update button style to primary (T436332), instrument(WelcomeSurvey): align motivation event with AccountSetup (T439887), instrument(AccountSetup): Record setup motivation on `page_visit` (T439217)
• 20:02 jhathaway: flipping flag to disable legacy facts in puppet prod, attempt two
• 19:52 zabe@deploy1003: Finished scap sync-world: Backport for Add missing select of fr_timestamp (duration: 09m 40s)
• 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 19:51 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 19:46 zabe@deploy1003: zabe: Continuing with deployment
• 19:46 zabe@deploy1003: zabe: Backport for Add missing select of fr_timestamp synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 19:42 zabe@deploy1003: Started scap sync-world: Backport for Add missing select of fr_timestamp
• 19:42 zabe@deploy1003: Finished scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753) (duration: 10m 01s)
• 19:36 zabe@deploy1003: zabe: Continuing with deployment
• 19:36 zabe@deploy1003: zabe: Backport for Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 19:32 zabe@deploy1003: Started scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for s5 wikis (T439753)
• 19:19 jhathaway: puppet re-enabled
• 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 19:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 18:42 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 ()
• 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.22 refs T438218
• 18:36 jhathaway: disabling puppet fleet wide to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/1347689
• 18:25 brennen: 1.47.0-wmf.22 train status (T438218): logs kind of a mess, but no current blockers, rolling to all wikis
• 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 18:16 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 18:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 ()
• 17:55 swfrench@deploy1003: Finished scap deploy-service shellbox: Test deploy-service configuration for Shellbox (duration: 03m 28s)
• 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 17:55 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
• 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
• 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
• 17:54 spiderpig@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
• 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
• 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
• 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
• 17:54 spiderpig@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
• 17:53 spiderpig@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply
• 17:52 spiderpig@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 17:52 swfrench@deploy1003: Started scap deploy-service shellbox: Test deploy-service configuration for Shellbox
• 17:49 kamila@deploy1003: Finished scap sync-world: rebuild for T439100 (duration: 33m 25s)
• 17:16 kamila@deploy1003: Started scap sync-world: rebuild for T439100
• 17:13 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
• 17:12 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
• 17:12 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
• 17:12 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
• 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 17:01 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 16:25 cdobbins@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum and A:durum
• 16:12 cdobbins@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum and A:durum
• 15:40 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqiad - 7.1.1-2~bpo13+wmf3 ()
• 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqiad - 7.1.1-2~bpo13+wmf3 ()
• 15:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy (exit_code=0) rolling restart_daemons on A:tcpproxy and A:tcpproxy
• 15:32 moritzm: installing socat security updates
• 15:29 dpogorzelski@dns1004: END - running authdns-update
• 15:26 cdobbins@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-tcp-proxy rolling restart_daemons on A:tcpproxy and A:tcpproxy
• 15:25 dpogorzelski@dns1004: START - running authdns-update
• 15:20 zabe@deploy1003: Finished scap sync-world: Backport for Add missing alias for filerevision, Add missing alias for filerevision (duration: 10m 12s)
• 15:14 zabe@deploy1003: zabe: Continuing with deployment
• 15:14 zabe@deploy1003: zabe: Backport for Add missing alias for filerevision, Add missing alias for filerevision synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:11 moritzm: installing postgresql security updates
• 15:09 zabe@deploy1003: Started scap sync-world: Backport for Add missing alias for filerevision, Add missing alias for filerevision
• 15:08 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=tcp-proxy.*
• 15:08 cdobbins@cumin1004: conftool action : set/pooled=no; selector: name=tcp-proxy.*
• 14:56 moritzm: installing initramfs-tools bugfix updates from trixie point release
• 14:48 moritzm: installing busybox bugfix updates from trixie point release
• 14:47 urbanecm@deploy1003: Finished scap sync-world: Backport for NotificationController: Use lazyPush() for the delete job (T439626), NotificationController: Use lazyPush() for the delete job (T439626) (duration: 10m 05s)
• 14:42 urbanecm@deploy1003: urbanecm: Continuing with deployment
• 14:42 moritzm: installing zsh bugfix update from trixie point release
• 14:41 urbanecm@deploy1003: urbanecm: Backport for NotificationController: Use lazyPush() for the delete job (T439626), NotificationController: Use lazyPush() for the delete job (T439626) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:37 urbanecm@deploy1003: Started scap sync-world: Backport for NotificationController: Use lazyPush() for the delete job (T439626), NotificationController: Use lazyPush() for the delete job (T439626)
• 14:32 moritzm: installing e2fsprogs bugfix update from trixie point release
• 14:31 dpogorzelski@dns1004: END - running authdns-update
• 14:29 dpogorzelski@dns1004: START - running authdns-update
• 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
• 14:19 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 14:18 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 14:14 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
• 14:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 14:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 14:06 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
• 14:04 awight: EU afternoon backport complete
• 14:03 awight@deploy1003: Finished scap sync-world: Backport for Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790) (duration: 10m 26s)
• 14:01 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 13:58 awight@deploy1003: lilients, awight: Continuing with deployment
• 13:57 awight@deploy1003: lilients, awight: Backport for Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:53 awight@deploy1003: Started scap sync-world: Backport for Revert "Use explicit numbers in reference lists and Parsoid rendering" (T439790)
• 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
• 13:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
• 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
• 13:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
• 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
• 13:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
• 13:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
• 13:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
• 13:41 awight@deploy1003: Finished scap sync-world: Backport for instrument(AccountSetup): don't double count finish/skip clicks, stats(AccountSetup): Record onboarding ready time in Prometheus (T432810), instrument(AccountSetup): Send exposure from server, send ready event (T432164) (duration: 22m 53s)
• 13:41 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 13:40 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 13:36 awight@deploy1003: awight, migr: Continuing with deployment
• 13:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 13:34 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1055.eqiad.wmnet with OS bookworm
• 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
• 13:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
• 13:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
• 13:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
• 13:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
• 13:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
• 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
• 13:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
• 13:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
• 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
• 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
• 13:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 13:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 13:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
• 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 13:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 13:26 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply
• 13:26 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply
• 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply
• 13:25 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply
• 13:25 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
• 13:25 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
• 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
• 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
• 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
• 13:24 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
• 13:24 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: sync
• 13:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
• 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply
• 13:23 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply
• 13:23 awight@deploy1003: awight, migr: Backport for instrument(AccountSetup): don't double count finish/skip clicks, stats(AccountSetup): Record onboarding ready time in Prometheus (T432810), instrument(AccountSetup): Send exposure from server, send ready event (T432164) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be ve
• 13:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
• 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
• 13:22 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
• 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
• 13:21 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
• 13:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
• 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
• 13:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
• 13:18 awight@deploy1003: Started scap sync-world: Backport for instrument(AccountSetup): don't double count finish/skip clicks, stats(AccountSetup): Record onboarding ready time in Prometheus (T432810), instrument(AccountSetup): Send exposure from server, send ready event (T432164)
• 13:18 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply
• 13:17 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply
• 13:12 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
• 13:08 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
• 13:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
• 13:07 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
• 13:07 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
• 13:06 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
• 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:05 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003"
• 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
• 12:59 mszwarc@deploy1003: Synchronized private/PrivateSettings.php: Sync PrivateSettings.php (duration: 10m 02s)
• 12:59 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply
• 12:54 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm
• 12:54 mszwarc@deploy1003: mszwarc: Continuing with deployment
• 12:53 mszwarc@deploy1003: mszwarc: Sync PrivateSettings.php synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:45 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
• 12:44 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 12:44 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
• 12:40 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
• 12:34 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 12:34 moritzm: installing libtasn1-6 security updates
• 12:33 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:32 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:27 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply
• 12:27 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply
• 12:25 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939] (duration: 04m 27s)
• 12:24 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:23 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:22 moritzm: installing glibc security updates
• 12:21 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993]: Regular analytics weekly train [analytics/refinery@13379939]
• 12:20 a-pizzata@deploy1003: Finished deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939] (duration: 00m 46s)
• 12:19 a-pizzata@deploy1003: Started deploy [analytics/refinery@1337993] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@13379939]
• 12:15 swfrench@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply
• 12:14 swfrench@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply
• 12:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:05 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:05 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqsin and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 11:58 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad
• 11:55 jforrester@deploy1003: Finished scap sync-world: Backport for RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533) (duration: 12m 27s)
• 11:50 jforrester@deploy1003: jforrester: Continuing with deployment
• 11:50 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad
• 11:47 jforrester@deploy1003: jforrester: Backport for RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 11:43 jforrester@deploy1003: Started scap sync-world: Backport for RepoHooks: Fix fresh-result rate limit blocking autoconfirmed users (T437533)
• 11:36 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add VIPs for ldap-ro-test in eqiad/codfw - jmm@cumin2003"
• 11:32 jmm@cumin2003: START - Cookbook sre.dns.netbox
• 11:31 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 11:30 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 11:30 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 11:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'.
• 11:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'.
• 11:06 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw
• 10:57 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw
• 10:50 javiermonton@deploy1003: Finished scap sync-world: Backport for stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875) (duration: 10m 51s)
• 10:45 javiermonton@deploy1003: javiermonton: Continuing with deployment
• 10:44 javiermonton@deploy1003: javiermonton: Backport for stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 10:40 javiermonton@deploy1003: Started scap sync-world: Backport for stream: mediawiki.job.pageviewTrendingRelativeUpdate (T437875)
• 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
• 10:30 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
• 10:24 ammarpad@deploy1003: mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=metawiki Jakehekd Ivyeclipse # T439805
• 10:23 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
• 10:22 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
• 09:59 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
• 09:47 jelto@deploy1003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
• 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
• 09:35 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
• 08:39 hashar@deploy1003: Finished deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7 (duration: 00m 10s)
• 08:38 hashar@deploy1003: Started deploy [integration/docroot@5cf09c8]: build: Updating brace-expansion to 1.1.21, 2.1.7
• 08:35 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 08:22 moritzm: restart FPM on phab1005 to pick up openssl/PCRE security updates
• 08:05 moritzm: installing pcre2/openssl security updates
• 07:54 vgutierrez: upload purged 0.25 to trixie-wikimedia (apt.wm.o)
• 07:31 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
• 07:29 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
• 07:28 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
• 07:24 kharlan@deploy1003: Finished scap sync-world: Backport for CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808), CheckUser: Enable ClientHintsInEditRequest on group1 (T401808) (duration: 14m 37s)
• 07:23 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
• 07:16 kharlan@deploy1003: kharlan: Continuing with deployment
• 07:15 kharlan@deploy1003: kharlan: Backport for CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808), CheckUser: Enable ClientHintsInEditRequest on group1 (T401808) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:11 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
• 07:09 kharlan@deploy1003: Started scap sync-world: Backport for CheckUser: Enable ClientHintsInEditRequest on all wikis (T401808), CheckUser: Enable ClientHintsInEditRequest on group1 (T401808)
• 07:07 jmm@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
• 07:06 kharlan@deploy1003: Finished scap sync-world: Backport for Add extra UI elements to Special:AbuseReview following user feedback (T439694) (duration: 33m 18s)
• 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1002.eqiad.wmnet
• 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 06:57 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 06:57 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 06:54 kharlan@deploy1003: kharlan: Continuing with deployment
• 06:53 tappof@cumin1004: START - Cookbook sre.dns.netbox
• 06:53 kharlan@deploy1003: kharlan: Backport for Add extra UI elements to Special:AbuseReview following user feedback (T439694) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 06:47 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1002.eqiad.wmnet
• 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1001.eqiad.wmnet
• 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 06:46 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 06:45 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 06:41 tappof@cumin1004: START - Cookbook sre.dns.netbox
• 06:35 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1001.eqiad.wmnet
• 06:33 kharlan@deploy1003: Started scap sync-world: Backport for Add extra UI elements to Special:AbuseReview following user feedback (T439694)
• 06:29 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Send the abuse review tag in interaction events (T439238), AbuseReview: Instrument when content policy match adds tags (T439242) (duration: 10m 33s)
• 06:23 kharlan@deploy1003: kharlan: Continuing with deployment
• 06:22 kharlan@deploy1003: kharlan: Backport for AbuseReview: Send the abuse review tag in interaction events (T439238), AbuseReview: Instrument when content policy match adds tags (T439242) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 06:18 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Send the abuse review tag in interaction events (T439238), AbuseReview: Instrument when content policy match adds tags (T439242)

2026-09-30

• 22:23 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 ()
• 21:22 sbassett: Deployed updated mitigation for T435455
• 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 21:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 21:00 arlolra@deploy1003: Finished scap sync-world: Backport for Set a default for wgSplitParsoidParserCache in prod (T432760) (duration: 12m 28s)
• 21:00 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 20:58 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 20:57 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 20:52 arlolra@deploy1003: arlolra: Continuing with deployment
• 20:52 arlolra@deploy1003: arlolra: Backport for Set a default for wgSplitParsoidParserCache in prod (T432760) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:47 arlolra@deploy1003: Started scap sync-world: Backport for Set a default for wgSplitParsoidParserCache in prod (T432760)
• 20:44 mstyles@deploy1003: Finished scap sync-world: Backport for Email confirmation: implement delayed enforcement experiment (T439283), Email confirmation: Add server-side enforcement (T439283), Email confirmation: Prepare for next experiment (T439283) (duration: 32m 47s)
• 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 20:38 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 20:32 mstyles@deploy1003: mstyles, catrope: Continuing with deployment
• 20:31 mstyles@deploy1003: mstyles, catrope: Backport for Email confirmation: implement delayed enforcement experiment (T439283), Email confirmation: Add server-side enforcement (T439283), Email confirmation: Prepare for next experiment (T439283) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 20:20 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 20:11 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 20:11 mstyles@deploy1003: Started scap sync-world: Backport for Email confirmation: implement delayed enforcement experiment (T439283), Email confirmation: Add server-side enforcement (T439283), Email confirmation: Prepare for next experiment (T439283)
• 20:06 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
• 20:05 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
• 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 ()
• 19:45 jhathaway: reverting legacy facts disablement, discovered some remaining usage
• 18:56 dzahn@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 18:56 dzahn@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 18:56 dzahn@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 18:55 dzahn@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 18:55 dzahn@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 18:55 dzahn@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
• 18:52 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs T438218
• 18:40 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs T438218
• 18:37 jhathaway: flipping flag to disable legacy facts in puppet prod
• 18:24 brennen@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs T438218
• 18:22 brennen@deploy1003: Rolling back deployment
• 18:10 brennen: 1.47.0-wmf.22 train status (T438218): no current blockers, rolling to group1
• 17:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
• 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 17:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 17:33 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 17:28 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
• 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 ()
• 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 17:21 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 17:21 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 ()
• 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart
• 17:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.restart
• 17:15 bking@dns2004: END - running authdns-update
• 17:13 bking@dns2004: START - running authdns-update
• 17:09 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 17:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:52 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:51 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:38 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1043.eqiad.wmnet with OS bookworm
• 16:14 aaron@deploy1003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
• 16:14 aaron@deploy1003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
• 16:13 aaron@deploy1003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
• 16:13 aaron@deploy1003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
• 16:12 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage
• 16:11 aaron@deploy1003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
• 16:11 aaron@deploy1003: helmfile [staging] START helmfile.d/services/developer-portal: apply
• 16:08 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage
• 15:49 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm
• 15:47 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm
• 15:47 andrew@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage
• 15:42 andrew@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage
• 15:41 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
• 15:41 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
• 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 15:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 15:15 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm
• 15:11 jmm@cumin2003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir
• 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker236[5-9].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 15:09 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2369.codfw.wmnet
• 15:09 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2369.codfw.wmnet
• 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 15:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 15:03 jmm@cumin2003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir
• 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2369.codfw.wmnet
• 15:02 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker133[0-4].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 15:02 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1334.eqiad.wmnet
• 15:02 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1334.eqiad.wmnet
• 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2369.codfw.wmnet
• 15:01 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2368.codfw.wmnet
• 15:01 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2368.codfw.wmnet
• 15:01 jmm@cumin2003: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry
• 14:58 jmm@cumin2003: START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry
• 14:54 zabe@deploy1003: Finished scap sync-world: Backport for Add script to drop duplicate oldimage/filerevision rows (T67264 T428406), Add script to drop duplicate oldimage/filerevision rows (T67264 T428406) (duration: 09m 54s)
• 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2368.codfw.wmnet
• 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1334.eqiad.wmnet
• 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1334.eqiad.wmnet
• 14:53 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1333.eqiad.wmnet
• 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1333.eqiad.wmnet
• 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2368.codfw.wmnet
• 14:53 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2367.codfw.wmnet
• 14:53 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2367.codfw.wmnet
• 14:51 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 14:49 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 14:49 zabe@deploy1003: zabe: Continuing with deployment
• 14:49 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 14:48 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:48 zabe@deploy1003: zabe: Backport for Add script to drop duplicate oldimage/filerevision rows (T67264 T428406), Add script to drop duplicate oldimage/filerevision rows (T67264 T428406) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:48 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:47 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2367.codfw.wmnet
• 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2367.codfw.wmnet
• 14:45 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 14:45 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 14:45 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2366.codfw.wmnet
• 14:45 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2366.codfw.wmnet
• 14:45 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 14:45 oblivian@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1333.eqiad.wmnet
• 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1333.eqiad.wmnet
• 14:44 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1332.eqiad.wmnet
• 14:44 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1332.eqiad.wmnet
• 14:44 zabe@deploy1003: Started scap sync-world: Backport for Add script to drop duplicate oldimage/filerevision rows (T67264 T428406), Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)
• 14:44 oblivian@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 14:44 oblivian@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 14:43 oblivian@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 14:43 oblivian@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 14:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 14:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2366.codfw.wmnet
• 14:38 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm
• 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2366.codfw.wmnet
• 14:37 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:37 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2365.codfw.wmnet
• 14:37 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2365.codfw.wmnet
• 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 14:37 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:37 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 14:36 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1332.eqiad.wmnet
• 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1332.eqiad.wmnet
• 14:36 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1331.eqiad.wmnet
• 14:36 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1331.eqiad.wmnet
• 14:35 jforrester@deploy1003: Finished scap sync-world: Backport for mlwiki: Enable Wikifunctions client mode (duration: 09m 51s)
• 14:32 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 14:31 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 14:31 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2365.codfw.wmnet
• 14:30 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:30 jforrester@deploy1003: jforrester: Continuing with deployment
• 14:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2365.codfw.wmnet
• 14:30 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:29 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker236[5-9].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 14:29 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:29 jforrester@deploy1003: jforrester: Backport for mlwiki: Enable Wikifunctions client mode synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1331.eqiad.wmnet
• 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1331.eqiad.wmnet
• 14:27 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1330.eqiad.wmnet
• 14:27 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1330.eqiad.wmnet
• 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 14:26 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 14:25 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 14:25 jforrester@deploy1003: Started scap sync-world: Backport for mlwiki: Enable Wikifunctions client mode
• 14:25 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 14:25 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 14:24 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 14:24 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 14:24 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 14:24 jforrester@deploy1003: Finished scap sync-world: Backport for mlwiki: Enable Abstract Wikimedia client mode (duration: 11m 31s)
• 14:23 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 14:22 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 14:22 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:22 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 14:21 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 14:21 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 14:21 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 14:21 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 14:21 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:20 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:18 jforrester@deploy1003: jforrester: Continuing with deployment
• 14:17 oblivian@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1330.eqiad.wmnet
• 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1330.eqiad.wmnet
• 14:17 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker133[0-4].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 14:16 ecarg@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 14:16 jforrester@deploy1003: jforrester: Backport for mlwiki: Enable Abstract Wikimedia client mode synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:15 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 14:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 14:15 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 14:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 14:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 14:14 ecarg@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 14:13 ecarg@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:12 jforrester@deploy1003: Started scap sync-world: Backport for mlwiki: Enable Abstract Wikimedia client mode
• 14:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 14:09 ecarg@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:09 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 14:08 ecarg@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:07 ecarg@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:06 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 14:06 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for Fix xsd xml version (T439093) (duration: 11m 33s)
• 14:05 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 14:04 Amir1: mwscript-k8s --comment="T358266" --local_dblist=<(expanddblist all | grep -v commons) -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --key '360p.mpeg4.mov' --throttle --video --missing (T358266)
• 14:04 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 14:04 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 14:01 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 14:00 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 14:00 inflatador: bking@krb2002 disable wmf_auto_restart krb admin timers/services as this host is no longer primary T437937
• 14:00 seanleong-wmde@deploy1003: awight, seanleong-wmde: Continuing with deployment
• 13:58 seanleong-wmde@deploy1003: awight, seanleong-wmde: Backport for Fix xsd xml version (T439093) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 13:54 seanleong-wmde@deploy1003: Started scap sync-world: Backport for Fix xsd xml version (T439093)
• 13:53 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 13:51 seanleong-wmde@deploy1003: Finished scap sync-world: Backport for Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093) (duration: 12m 01s)
• 13:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 13:46 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Continuing with deployment
• 13:46 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm
• 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker237[0-5].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2374.codfw.wmnet
• 13:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2374.codfw.wmnet
• 13:43 seanleong-wmde@deploy1003: seanleong-wmde, lucamauri: Backport for Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:41 andrew@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm
• 13:39 seanleong-wmde@deploy1003: Started scap sync-world: Backport for Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)
• 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2374.codfw.wmnet
• 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2374.codfw.wmnet
• 13:38 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2373.codfw.wmnet
• 13:38 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2373.codfw.wmnet
• 13:32 jgiannelos@deploy1003: Finished scap sync-world: Backport for prv: Enable parsoid rendering for 5 wikis (T439444) (duration: 11m 13s)
• 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1003.eqiad.wmnet
• 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:32 tappof@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 13:31 tappof@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004"
• 13:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2373.codfw.wmnet
• 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2373.codfw.wmnet
• 13:30 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2372.codfw.wmnet
• 13:30 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2372.codfw.wmnet
• 13:27 jgiannelos@deploy1003: jgiannelos: Continuing with deployment
• 13:26 tappof@cumin1004: START - Cookbook sre.dns.netbox
• 13:25 jgiannelos@deploy1003: jgiannelos: Backport for prv: Enable parsoid rendering for 5 wikis (T439444) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
• 13:23 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 13:23 andrew@cumin2003: START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm
• 13:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2372.codfw.wmnet
• 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2372.codfw.wmnet
• 13:22 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2371.codfw.wmnet
• 13:22 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2371.codfw.wmnet
• 13:21 jgiannelos@deploy1003: Started scap sync-world: Backport for prv: Enable parsoid rendering for 5 wikis (T439444)
• 13:20 tappof@cumin1004: START - Cookbook sre.hosts.decommission for hosts kafka-logging1003.eqiad.wmnet
• 13:18 samtar@deploy1003: Finished scap sync-world: Backport for fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872) (duration: 12m 23s)
• 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2371.codfw.wmnet
• 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2371.codfw.wmnet
• 13:15 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2370.codfw.wmnet
• 13:15 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2370.codfw.wmnet
• 13:13 samtar@deploy1003: migr, samtar: Continuing with deployment
• 13:12 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 13:10 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 13:10 samtar@deploy1003: migr, samtar: Backport for fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:07 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2370.codfw.wmnet
• 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2370.codfw.wmnet
• 13:07 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker237[0-5].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 13:06 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 13:06 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 13:06 samtar@deploy1003: Started scap sync-world: Backport for fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)
• 13:05 tappof@deploy1003: Finished scap sync-world: T432444 - remove kafka-logging100[12] (duration: 10m 41s)
• 13:05 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 13:05 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 13:04 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 13:04 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
• 13:02 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 12:56 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 12:56 tappof@deploy1003: Started scap sync-world: T432444 - remove kafka-logging100[12]
• 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 12:54 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 12:54 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
• 12:53 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
• 12:53 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
• 12:53 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
• 12:48 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 12:47 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker138[0-3].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 12:46 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1383.eqiad.wmnet
• 12:46 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1383.eqiad.wmnet
• 12:45 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 12:44 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 12:43 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 12:43 gkyziridis@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 12:42 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 12:41 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667)
• 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1383.eqiad.wmnet
• 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1383.eqiad.wmnet
• 12:39 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1382.eqiad.wmnet
• 12:39 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1382.eqiad.wmnet
• 12:37 krinkle@deploy1003: Finished scap sync-world: Backport for Profiler: Fix excimer component regex skipping the last frame in each stack (duration: 10m 56s)
• 12:33 dkertesz: fetching new haproxy packages on reprepro (3.2.23-1~bpo13+1 -> 3.2.25-1~bpo13+1) - T439667
• 12:32 krinkle@deploy1003: somerandomdeveloper, krinkle: Continuing with deployment
• 12:32 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1382.eqiad.wmnet
• 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1382.eqiad.wmnet
• 12:31 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1381.eqiad.wmnet
• 12:31 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1381.eqiad.wmnet
• 12:30 krinkle@deploy1003: somerandomdeveloper, krinkle: Backport for Profiler: Fix excimer component regex skipping the last frame in each stack synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:26 krinkle@deploy1003: Started scap sync-world: Backport for Profiler: Fix excimer component regex skipping the last frame in each stack
• 12:24 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1381.eqiad.wmnet
• 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1381.eqiad.wmnet
• 12:23 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1380.eqiad.wmnet
• 12:23 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1380.eqiad.wmnet
• 12:16 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1380.eqiad.wmnet
• 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1380.eqiad.wmnet
• 12:16 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker138[0-3].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 12:15 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 12:14 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 12:14 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 12:13 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 12:12 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging[1001-1003].eqiad.wmnet with reason: stop service on kafka-logging100[1-3]
• 12:02 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet
• 12:00 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet
• 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1044.eqiad.wmnet
• 11:55 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1043.eqiad.wmnet
• 11:41 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 11:39 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet
• 11:36 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 11:34 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1042.eqiad.wmnet
• 11:33 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet
• 11:33 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet
• 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet
• 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet
• 11:32 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet
• 11:32 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet
• 11:29 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 11:27 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 11:25 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply
• 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-experimental: apply
• 11:24 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
• 11:24 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 11:24 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 11:24 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
• 11:23 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 11:22 cgoubert@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply
• 11:22 cgoubert@deploy1003: helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply
• 11:22 cgoubert@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply
• 11:22 cgoubert@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply
• 11:07 cgoubert@deploy1003: Finished scap sync-world: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547 (duration: 11m 12s)
• 11:01 cgoubert@deploy1003: cgoubert: Continuing with deployment
• 10:59 cgoubert@deploy1003: cgoubert: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 10:56 cgoubert@deploy1003: Started scap sync-world: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547
• 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker1384.eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 10:40 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1384.eqiad.wmnet
• 10:40 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1384.eqiad.wmnet
• 10:38 elukey: `sudo ipvsadm --delete-service --tcp-service 10.2.1.96:443` on lvs201[34] to remove the pki service
• 10:33 oblivian@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1384.eqiad.wmnet
• 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1384.eqiad.wmnet
• 10:33 oblivian@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker1384.eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad)
• 10:32 tappof: Grafana upgrade from 13.2.0 to 13.2.3 - T438454
• 10:27 oblivian@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 10:27 oblivian@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
• 10:06 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 10:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 10:03 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 09:39 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad
• 09:38 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
• 09:33 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 09:32 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 09:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
• 09:08 btullis@dns1004: END - running authdns-update
• 09:06 btullis@dns1004: START - running authdns-update
• 09:04 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service'
• 09:01 elukey: elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service'
• 08:27 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad
• 08:25 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad
• 08:24 elukey@cumin1004: conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca
• 08:24 jmm@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw
• 08:22 jmm@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw
• 07:49 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 07:46 elukey@dns1004: END - running authdns-update
• 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
• 07:44 kevinbazira@deploy1003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 07:44 elukey@dns1004: START - running authdns-update
• 07:35 moritzm: installing pcre2/openssl security updates
• 07:18 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 07:18 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 07:15 dcausse@deploy1003: Finished scap sync-world: Backport for Search: optimize morelike queries (duration: 11m 27s)
• 07:13 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 07:12 filippo@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 07:09 dcausse@deploy1003: dcausse: Continuing with deployment
• 07:07 dcausse@deploy1003: dcausse: Backport for Search: optimize morelike queries synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:03 dcausse@deploy1003: Started scap sync-world: Backport for Search: optimize morelike queries
• 06:51 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm
• 06:48 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet
• 06:33 filippo@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet
• 06:32 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 40s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-09-29

• 22:42 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm
• 22:39 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm
• 22:26 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage
• 22:23 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage
• 22:21 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage
• 22:19 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage
• 22:07 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm
• 22:06 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1003 to dse-k8s-worker1044
• 22:06 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1044
• 22:05 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1044
• 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1044 on all recursors
• 22:05 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1044 on all recursors
• 22:05 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 22:05 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003"
• 22:05 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003"
• 22:03 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm
• 22:03 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1002 to dse-k8s-worker1043
• 22:02 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1043
• 22:01 bking@cumin2003: START - Cookbook sre.dns.netbox
• 22:01 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1043
• 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1043 on all recursors
• 22:01 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1043 on all recursors
• 22:01 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 22:01 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003"
• 22:01 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003"
• 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1003 to dse-k8s-worker1044
• 21:56 bking@cumin2003: START - Cookbook sre.dns.netbox
• 21:56 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1002 to dse-k8s-worker1043
• 21:56 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts
• 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts
• 21:55 bking@cumin2003: END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts
• 21:55 bking@cumin2003: START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts
• 21:29 mutante: decom'ing VM zuul1002 - replaced by physical machine zuul1005 - T427353 T395938
• 21:27 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm
• 21:20 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
• 21:20 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
• 21:16 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
• 21:16 eevans@deploy1003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
• 21:11 bking@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage
• 21:08 cklimas@deploy1003: Finished scap sync-world: Backport for Beta Cluster: Use different Redis instance for LockManager (T436480) (duration: 12m 45s)
• 21:06 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet']
• 21:06 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet']
• 21:06 bking@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage
• 21:04 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet']
• 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet']
• 21:03 bking@cumin2003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet']
• 21:03 bking@cumin2003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet']
• 21:02 cklimas@deploy1003: cklimas, southparkfan: Continuing with deployment
• 20:59 cklimas@deploy1003: cklimas, southparkfan: Backport for Beta Cluster: Use different Redis instance for LockManager (T436480) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:55 cklimas@deploy1003: Started scap sync-world: Backport for Beta Cluster: Use different Redis instance for LockManager (T436480)
• 20:51 bking@cumin2003: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm
• 20:48 cklimas@deploy1003: Finished scap sync-world: Backport for InitialiseSettings: Enable return-to-app banner on testwiki (T434258) (duration: 14m 32s)
• 20:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1001 to dse-k8s-worker1042
• 20:46 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1042
• 20:45 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1042
• 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1042 on all recursors
• 20:45 bking@cumin2003: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1042 on all recursors
• 20:45 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 20:45 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003"
• 20:43 cklimas@deploy1003: cklimas: Continuing with deployment
• 20:42 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003"
• 20:38 cklimas@deploy1003: cklimas: Backport for InitialiseSettings: Enable return-to-app banner on testwiki (T434258) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:33 cklimas@deploy1003: Started scap sync-world: Backport for InitialiseSettings: Enable return-to-app banner on testwiki (T434258)
• 20:33 bking@cumin2003: START - Cookbook sre.dns.netbox
• 20:32 bking@cumin2003: START - Cookbook sre.hosts.rename from ml-cache1001 to dse-k8s-worker1042
• 20:22 sbassett@deploy1003: Finished scap sync-world: Backport for Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424), Remove foundationwiki's CSP report-only header (T423691) (duration: 14m 07s)
• 20:19 sbassett@deploy1003: sbassett: Rolling back deployment
• 20:12 sbassett@deploy1003: sbassett: Backport for Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424), Remove foundationwiki's CSP report-only header (T423691) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1002.eqiad.wmnet with OS bookworm
• 20:11 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 20:09 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 20:08 sbassett@deploy1003: Started scap sync-world: Backport for Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424), Remove foundationwiki's CSP report-only header (T423691)
• 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS bookworm
• 20:04 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 19:57 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 19:56 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 19:56 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 19:55 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1003.eqiad.wmnet with OS bookworm
• 19:55 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 19:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage
• 19:53 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 ()
• 19:52 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004"
• 19:48 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage
• 19:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage
• 19:40 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 19:39 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 19:38 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage
• 19:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage
• 19:33 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm
• 19:33 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage
• 19:30 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 19:29 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 19:27 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:24 kemayo@deploy1003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
• 19:23 kemayo@deploy1003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
• 19:22 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm
• 19:19 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm
• 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:15 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:15 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:10 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm
• 19:10 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm
• 19:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 19:09 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:08 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:08 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:07 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1003.eqiad.wmnet with OS bookworm
• 19:06 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:06 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:05 btullis@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1002.eqiad.wmnet with OS bookworm
• 19:05 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:04 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm
• 19:03 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm
• 19:03 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm
• 19:02 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors
• 19:02 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors
• 18:57 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5004.*
• 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004"
• 18:56 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004"
• 18:56 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm
• 18:50 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004"
• 18:49 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004"
• 18:45 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:41 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004"
• 18:39 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004"
• 18:35 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:34 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004"
• 18:31 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004"
• 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
• 18:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
• 18:27 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1033.eqiad.wmnet
• 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1034.eqiad.wmnet
• 18:26 ryankemper@cumin2003: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1035.eqiad.wmnet
• 18:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 18:24 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 18:19 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 18:18 brennen@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs T438218
• 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 18:15 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 18:08 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards
• 18:06 brennen: 1.47.0-wmf.22 train status (T438218) - no current blockers, rolling to group0
• 18:05 sandraebele: Deployed Refinery as part of weekly deployment train
• 18:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 ()
• 18:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards
• 17:59 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards
• 17:58 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] (duration: 04m 54s)
• 17:57 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 17:55 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
• 17:54 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
• 17:53 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9]
• 17:53 sandraebele: Deploying Refinery as part of weekly deployment train
• 17:52 ebysans@deploy1003: Finished deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] (duration: 00m 46s)
• 17:51 ebysans@deploy1003: Started deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9]
• 17:51 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5004.eqsin.wmnet with OS trixie
• 17:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 17:46 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 17:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 17:44 sandraebele: Deployed refinery-source using jenkins as part of weekly deployment train
• 17:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards
• 17:01 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards
• 17:00 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards
• 16:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp5028.*
• 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s)
• 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host
• 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s)
• 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host
• 16:54 ryankemper@deploy1003: Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s)
• 16:54 ryankemper@deploy1003: Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host
• 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028.eqsin.wmnet
• 16:36 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5028
• 16:35 inflatador: bking@cephosd1001 `sudo radosgw-admin user create --uid=matomo --display-name="Matomo on K8s"` T436186
• 16:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 16:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 16:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:19 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Blacephalon --performer=Blacephalon --as-job # T418194
• 16:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 16:19 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 16:18 jhathaway@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply
• 16:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 16:15 jhathaway@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-cron: apply
• 15:59 jhathaway: updating puppet stdlib, should be a noop
• 15:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 ()
• 15:39 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 ()
• 15:23 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
• 15:14 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage
• 15:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage
• 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 15:08 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 14:32 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues
• 14:23 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
• 14:21 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
• 14:21 moritzm: installing rsyslog security updates
• 14:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
• 14:16 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet
• 14:16 klausman@cumin1004: START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet
• 14:12 bking@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw
• 14:12 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 14:11 krinkle@deploy1003: Finished scap sync-world: Backport for File: Implement LocalFile::getUrlForPurge workaround for caching (T425216), ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209) (duration: 14m 26s)
• 14:07 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004"
• 14:06 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004"
• 14:05 krinkle@deploy1003: krinkle: Continuing with deployment
• 14:02 btullis@cumin1004: START - Cookbook sre.dns.netbox
• 14:00 krinkle@deploy1003: krinkle: Backport for File: Implement LocalFile::getUrlForPurge workaround for caching (T425216), ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:59 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
• 13:58 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
• 13:56 krinkle@deploy1003: Started scap sync-world: Backport for File: Implement LocalFile::getUrlForPurge workaround for caching (T425216), ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)
• 13:54 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 13:51 krinkle@deploy1003: Finished deploy [performance/arc-lamp@35c590e]: T331941 (duration: 00m 09s)
• 13:51 krinkle@deploy1003: Started deploy [performance/arc-lamp@35c590e]: T331941
• 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet
• 13:47 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie
• 13:47 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 13:46 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for fix: prevent AccountSetup from being scrollable (T436872), fix: prevent AccountSetup from being scrollable (T436872) (duration: 11m 39s)
• 13:40 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Continuing with deployment
• 13:38 lucaswerkmeister-wmde@deploy1003: migr, lucaswerkmeister-wmde: Backport for fix: prevent AccountSetup from being scrollable (T436872), fix: prevent AccountSetup from being scrollable (T436872) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:34 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for fix: prevent AccountSetup from being scrollable (T436872), fix: prevent AccountSetup from being scrollable (T436872)
• 13:30 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 13:30 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 13:25 samtar@deploy1003: Finished scap sync-world: Backport for CommonSettings: Load ModeratorToolkit extension (T438334) (duration: 11m 17s)
• 13:21 btullis@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 13:20 samtar@deploy1003: samtar: Continuing with deployment
• 13:19 samtar@deploy1003: samtar: Backport for CommonSettings: Load ModeratorToolkit extension (T438334) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:16 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 13:16 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 13:15 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 13:15 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 13:14 samtar@deploy1003: Started scap sync-world: Backport for CommonSettings: Load ModeratorToolkit extension (T438334)
• 13:10 samtar@deploy1003: Finished scap sync-world: Backport for InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334) (duration: 10m 21s)
• 13:05 samtar@deploy1003: samtar: Continuing with deployment
• 13:04 samtar@deploy1003: samtar: Backport for InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:00 samtar@deploy1003: Started scap sync-world: Backport for InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)
• 12:54 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 12:54 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 12:53 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging
• 12:52 urbanecm@deploy1003: Finished scap sync-world: Backport for ReassignMentees: Make the jobReleaseTimestamp work (T418194) (duration: 12m 26s)
• 12:47 urbanecm@deploy1003: urbanecm: Continuing with deployment
• 12:44 urbanecm@deploy1003: urbanecm: Backport for ReassignMentees: Make the jobReleaseTimestamp work (T418194) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:40 urbanecm@deploy1003: Started scap sync-world: Backport for ReassignMentees: Make the jobReleaseTimestamp work (T418194)
• 12:39 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm
• 12:31 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:30 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 12:06 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 12:06 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 12:06 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 12:06 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 12:03 osleger@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 12:03 osleger@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 12:03 osleger@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 12:02 btullis@cumin1004: START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 12:02 osleger@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:47 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:46 tappof: Rebalancing has been completed on kafka-logging in eqiad: kafka-logging1001 and kafka-logging1002 are no longer assigned any partitions — T432444
• 10:43 atsukoito: shutting down https://airflow-test-k8s.wikimedia.org for upgrade, https://phabricator.wikimedia.org/T433383#12363530
• 10:38 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
• 10:37 blake@deploy1003: Finished scap sync-world: add sidecar flag for T417800 (duration: 04m 14s)
• 10:33 blake@deploy1003: Started scap sync-world: add sidecar flag for T417800
• 10:32 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage
• 10:19 claime: deploy2002:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232
• 10:17 claime: deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232
• 10:17 claime: deploy2003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232
• 10:15 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie
• 10:15 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003"
• 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003"
• 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2001.codfw.wmnet on all recursors
• 10:14 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache testvm2001.codfw.wmnet on all recursors
• 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 10:14 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003"
• 10:14 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003"
• 10:10 ihurbain@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 10:10 jmm@cumin2003: START - Cookbook sre.dns.netbox
• 10:10 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet
• 10:09 ihurbain@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 09:36 urbanecm@deploy1003: mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Ikan --performer=Ikan --as-job # T418194
• 08:59 urbanecm@deploy1003: Finished scap sync-world: Backport for ReassignMentees: Actually deduplicate reassignMenteesJob (T418194) (duration: 11m 07s)
• 08:54 urbanecm@deploy1003: urbanecm: Continuing with deployment
• 08:52 urbanecm@deploy1003: urbanecm: Backport for ReassignMentees: Actually deduplicate reassignMenteesJob (T418194) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 08:50 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 08:49 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 08:48 urbanecm@deploy1003: Started scap sync-world: Backport for ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)
• 08:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-eqiad
• 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-eqiad
• 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f5-eqiad
• 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f5-eqiad
• 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad
• 08:41 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f6-eqiad
• 08:41 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e7-eqiad
• 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e7-eqiad
• 08:40 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e6-eqiad
• 08:40 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e6-eqiad
• 08:39 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f7-eqiad
• 08:39 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-f7-eqiad
• 08:18 isaranto@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 08:10 moritzm: installing PHP 8.4 security updates
• 07:49 matthiasmullie: UTC morning backport window done
• 07:49 mlitn@deploy1003: Finished scap sync-world: Backport for carousel: keep the hide button attached to the carousel (T439507), carousel: keep the hide button attached to the carousel (T439507) (duration: 09m 57s)
• 07:44 mlitn@deploy1003: mlitn: Continuing with deployment
• 07:43 mlitn@deploy1003: mlitn: Backport for carousel: keep the hide button attached to the carousel (T439507), carousel: keep the hide button attached to the carousel (T439507) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:39 mlitn@deploy1003: Started scap sync-world: Backport for carousel: keep the hide button attached to the carousel (T439507), carousel: keep the hide button attached to the carousel (T439507)
• 07:35 kharlan@deploy1003: Finished scap sync-world: Backport for CheckUser: Enable ClientHintsInEditRequest on group0 (T401808) (duration: 12m 41s)
• 07:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet
• 07:30 kharlan@deploy1003: kharlan: Continuing with deployment
• 07:27 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet
• 07:27 kharlan@deploy1003: kharlan: Backport for CheckUser: Enable ClientHintsInEditRequest on group0 (T401808) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:26 kart_: Updated cxserver to 2026-09-25-114231-production
• 07:24 kartik@deploy1003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
• 07:24 kartik@deploy1003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
• 07:23 kartik@deploy1003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
• 07:23 kharlan@deploy1003: Started scap sync-world: Backport for CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)
• 07:22 kartik@deploy1003: helmfile [codfw] START helmfile.d/services/cxserver: apply
• 07:18 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org
• 07:18 mlitn@deploy1003: Finished scap sync-world: Backport for Skip expensive work if known to be useless (T439182), Cache extracted carousel data (T439182), Make carousel images are visible in the margins (T439422), carousel: fix carousel image preload observer root (T439422) (duration: 13m 39s)
• 07:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org
• 07:11 mlitn@deploy1003: mlitn: Continuing with deployment
• 07:09 mlitn@deploy1003: mlitn: Backport for Skip expensive work if known to be useless (T439182), Cache extracted carousel data (T439182), Make carousel images are visible in the margins (T439422), carousel: fix carousel image preload observer root (T439422) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now
• 07:04 mlitn@deploy1003: Started scap sync-world: Backport for Skip expensive work if known to be useless (T439182), Cache extracted carousel data (T439182), Make carousel images are visible in the margins (T439422), carousel: fix carousel image preload observer root (T439422)
• 07:02 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
• 07:01 kartik@deploy1003: helmfile [staging] START helmfile.d/services/cxserver: apply
• 06:57 jelto@deploy1003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
• 06:56 jelto@deploy1003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
• 05:54 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm
• 05:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm
• 05:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage
• 05:34 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage
• 05:32 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage
• 05:31 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage
• 05:21 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035
• 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1035
• 05:21 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm
• 05:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034
• 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1034
• 05:19 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm
• 05:03 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm
• 04:51 fabfur: manually restarting dump_cloud_ip_range on puppetserver1001 after failure
• 04:47 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage
• 04:43 ryankemper@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage
• 04:32 ryankemper@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033
• 04:32 ryankemper@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1033
• 04:31 ryankemper@cumin2003: START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm
• 04:01 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s)
• 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs T438218 (duration: 36m 03s)
• 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.22 refs T438218
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 28s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
• 01:49 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 ()
• 00:22 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage
• 00:17 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage
• 00:01 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie

2026-09-28

• 23:44 dzahn@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
• 23:35 zabe: run script to fix fr_metadata drifts # T420341
• 22:22 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 22:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 21:47 sbassett: Deployed security mitigation for T435455
• 21:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 ()
• 21:43 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 ()
• 21:42 eevans@deploy1003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: sync
• 21:42 eevans@deploy1003: helmfile [eqiad] START helmfile.d/services/sessionstore: sync
• 21:41 eevans@deploy1003: helmfile [staging] DONE helmfile.d/services/sessionstore: sync
• 21:41 eevans@deploy1003: helmfile [staging] START helmfile.d/services/sessionstore: sync
• 21:24 swfrench-wmf: deleting suspect coredns pods showing upstream resolution health check failures in eqiad - T439201
• 21:13 cjming@deploy1003: Finished scap sync-world: Backport for fix(ReadingRecommendations): reposition personalize button (T439158), fix(Homepage): apply the body transform only when an overlay is open (T439038) (duration: 12m 47s)
• 21:12 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp601[3-6].drmrs.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 21:08 arlolra@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 21:08 cjming@deploy1003: volker-e, cjming: Continuing with deployment
• 21:07 arlolra@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 21:07 arlolra@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 21:07 arlolra@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 21:04 cjming@deploy1003: volker-e, cjming: Backport for fix(ReadingRecommendations): reposition personalize button (T439158), fix(Homepage): apply the body transform only when an overlay is open (T439038) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:00 cjming@deploy1003: Started scap sync-world: Backport for fix(ReadingRecommendations): reposition personalize button (T439158), fix(Homepage): apply the body transform only when an overlay is open (T439038)
• 20:51 kemayo@deploy1003: Finished scap sync-world: Backport for BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506), EditCheck: only log a session if checks are actually possible (T438916) (duration: 10m 28s)
• 20:47 herron: uploaded rsyslog 8.2608.0-4~wmf13+1 to trixie-wikimedia T438375
• 20:47 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
• 20:46 kemayo@deploy1003: kemayo: Continuing with deployment
• 20:45 kemayo@deploy1003: kemayo: Backport for BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506), EditCheck: only log a session if checks are actually possible (T438916) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:41 kemayo@deploy1003: Started scap sync-world: Backport for BaseEditCheck#inAllowedNamespace: was true if any extra namespaces set (T400506), EditCheck: only log a session if checks are actually possible (T438916)
• 20:32 cjming@deploy1003: Finished scap sync-world: Backport for Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid" (duration: 09m 47s)
• 20:29 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1007.eqiad.wmnet with OS trixie
• 20:26 cjming@deploy1003: ssastry, cjming: Continuing with deployment
• 20:26 cjming@deploy1003: ssastry, cjming: Backport for Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:22 cjming@deploy1003: Started scap sync-world: Backport for Revert "HtmlToContentTransform: Pass DomPageBundle to Parsoid"
• 20:16 aude@deploy1003: Finished scap sync-world: Backport for Enable ReadingLists on all Wikipedia wikis (T434924), Enable Reading Recommendations experiment featured category setting (T437348) (duration: 13m 05s)
• 20:13 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage
• 20:10 aude@deploy1003: aude, volker-e: Continuing with deployment
• 20:08 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1007.eqiad.wmnet with reason: host reimage
• 20:08 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp6013.*
• 20:07 aude@deploy1003: aude, volker-e: Backport for Enable ReadingLists on all Wikipedia wikis (T434924), Enable Reading Recommendations experiment featured category setting (T437348) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:03 aude@deploy1003: Started scap sync-world: Backport for Enable ReadingLists on all Wikipedia wikis (T434924), Enable Reading Recommendations experiment featured category setting (T437348)
• 19:54 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1007.eqiad.wmnet with OS trixie
• 19:44 dancy@deploy1003: Installation of scap version "4.293.1" completed for 3 hosts
• 19:42 dancy@deploy1003: Installing scap version "4.293.1" for 3 host(s)
• 19:31 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp601[3-6].drmrs.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 19:16 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 ()
• 19:03 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_esams - 7.1.1-2~bpo13+wmf3 ()
• 19:03 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
• 18:47 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 ()
• 18:46 jhathaway: puppet lvm upgrade broke ml-serve hosts, looking for a hotfix, or I'll rollback...
• 18:38 dancy@deploy1003: Installation of scap version "4.293.0" completed for 3 hosts
• 18:36 dancy@deploy1003: Installing scap version "4.293.0" for 3 host(s)
• 18:24 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1006.eqiad.wmnet with OS trixie
• 18:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
• 18:08 dzahn@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage
• 18:04 dzahn@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1006.eqiad.wmnet with reason: host reimage
• 17:48 dzahn@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1006.eqiad.wmnet with OS trixie
• 16:20 zabe@deploy1003: Finished scap sync-world: Backport for wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319) (duration: 13m 50s)
• 16:15 zabe@deploy1003: zabe: Continuing with deployment
• 16:10 zabe@deploy1003: zabe: Backport for wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 16:06 zabe@deploy1003: Started scap sync-world: Backport for wikibase: Set virtual-wikibase-entityusage to x1 for test wikis (T439319)
• 16:05 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_drmrs - 7.1.1-2~bpo13+wmf3 ()
• 16:04 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_drmrs - 7.1.1-2~bpo13+wmf3 ()
• 16:03 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
• 16:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
• 16:01 jdrewniak@deploy1003: Finished scap sync-world: Backport for Bumping portals to master (T128546) (duration: 19m 54s)
• 15:56 jdrewniak@deploy1003: jdrewniak: Continuing with deployment
• 15:46 zabe: clone wbc_entity_usage to x1 for testwikidatawiki clients # T439319
• 15:45 jdrewniak@deploy1003: jdrewniak: Backport for Bumping portals to master (T128546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetboard,name=codfw
• 15:41 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=puppetdb-api,name=codfw
• 15:41 jdrewniak@deploy1003: Started scap sync-world: Backport for Bumping portals to master (T128546)
• 15:40 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=codfw
• 15:06 filippo@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm
• 14:55 Lucas_WMDE: UTC afternoon backport+config window done
• 14:54 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332), feat(AccountSetup), styles: Apply correct background and button styles (T436332) (duration: 33m 44s)
• 14:49 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Continuing with deployment
• 14:24 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, volker-e: Backport for feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332), feat(AccountSetup), styles: Apply correct background and button styles (T436332) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:20 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations (T436332), feat(AccountSetup), styles: Apply correct background and button styles (T436332)
• 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 14:17 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 14:10 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:10 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:09 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004"
• 14:09 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding krb1002 to eqiad - jclark@cumin1004"
• 14:09 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776) (duration: 10m 40s)
• 14:05 jclark@cumin1004: START - Cookbook sre.dns.netbox
• 14:03 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Continuing with deployment
• 14:02 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:02 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde: Backport for Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:58 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Revert "Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org" (T386776)
• 13:58 vgutierrez: repool cp7001 - T439361
• 13:57 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:57 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776) (duration: 19m 38s)
• 13:54 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:54 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Rolling back deployment
• 13:53 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:51 tappof: Rebalancing is in progress on kafka-logging in eqiad to evacuate kafka-logging1001 and kafka-logging1002 - T432444
• 13:47 bking@dns2004: END - running authdns-update
• 13:44 bking@dns2004: START - running authdns-update
• 13:42 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:41 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:41 lucaswerkmeister-wmde@deploy1003: lucaswerkmeister-wmde, ameisenigel: Backport for Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:37 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org (T386776)
• 13:36 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-staging,name=codfw
• 13:35 lucaswerkmeister-wmde@deploy1003: Finished scap sync-world: Backport for Enable FlaggedRevs for the Author NS on ruwikisource (T439159) (duration: 12m 02s)
• 13:33 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm
• 13:33 vgutierrez: depool cp7001 for some purged tests - T439361
• 13:32 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=codfw
• 13:30 dpogorzelski@cumin1004: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw
• 13:30 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Continuing with deployment
• 13:27 lucaswerkmeister-wmde@deploy1003: ameisenigel, lucaswerkmeister-wmde: Backport for Enable FlaggedRevs for the Author NS on ruwikisource (T439159) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:23 lucaswerkmeister-wmde@deploy1003: Started scap sync-world: Backport for Enable FlaggedRevs for the Author NS on ruwikisource (T439159)
• 13:22 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:21 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:18 jclark@cumin1004: START - Cookbook sre.hosts.provision for host krb1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:16 sbisson@deploy1003: Finished scap sync-world: Backport for Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643) (duration: 12m 58s)
• 13:11 sbisson@deploy1003: sbisson: Continuing with deployment
• 13:07 sbisson@deploy1003: sbisson: Backport for Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:03 sbisson@deploy1003: Started scap sync-world: Backport for Remove unused config: ArticleGuidanceRedirectEntryPointTitles (T437643)
• 12:33 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2008.codfw.wmnet
• 12:33 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2008.codfw.wmnet
• 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet
• 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 12:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 12:28 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 12:27 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2008.codfw.wmnet
• 12:21 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2008.codfw.wmnet with OS bookworm
• 12:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 12:19 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade (T438828)
• 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:18 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 12:17 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 12:15 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
• 12:14 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
• 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply
• 12:14 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: apply
• 12:04 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage
• 11:57 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2008.codfw.wmnet with reason: host reimage
• 11:54 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 11:53 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 11:53 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 11:53 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 11:53 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
• 11:53 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply
• 11:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade (T438828)
• 11:46 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2008.codfw.wmnet with OS bookworm
• 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2003 to dse-k8s-worker2008
• 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2008
• 11:43 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2008
• 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2008 on all recursors
• 11:43 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2008 on all recursors
• 11:43 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:42 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004"
• 11:42 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2003 to dse-k8s-worker2008 - brouberol@cumin1004"
• 11:28 zabe@deploy1003: Finished scap sync-world: Backport for reconcileTables: Add sleep option (T438750) (duration: 10m 52s)
• 11:23 brouberol@cumin1004: START - Cookbook sre.dns.netbox
• 11:22 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2003 to dse-k8s-worker2008
• 11:22 zabe@deploy1003: zabe: Continuing with deployment
• 11:22 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet
• 11:21 zabe@deploy1003: zabe: Backport for reconcileTables: Add sleep option (T438750) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 11:20 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2007.codfw.wmnet
• 11:20 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2007.codfw.wmnet
• 11:17 zabe@deploy1003: Started scap sync-world: Backport for reconcileTables: Add sleep option (T438750)
• 11:17 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2007.codfw.wmnet
• 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - 3.2.23 upgrade (T438828)
• 11:15 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - 3.2.23 upgrade (T438828)
• 11:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade (T438828)
• 10:49 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade (T438828)
• 10:43 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
• 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.roll-reimage-nodes (exit_code=0) rolling reimage on P{wikikube-worker2041.codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 10:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet
• 10:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet
• 10:42 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2041.codfw.wmnet with OS trixie
• 10:42 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply
• 10:41 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply
• 10:30 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
• 10:30 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply
• 10:28 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
• 10:28 blake@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
• 10:27 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
• 10:26 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
• 10:25 blake@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply
• 10:25 moritzm: installing chromium security updates
• 10:23 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage
• 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - 3.2.23 upgrade (T438828)
• 10:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - 3.2.23 upgrade (T438828)
• 10:18 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade (T438828)
• 10:17 jayme@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2041.codfw.wmnet with reason: host reimage
• 10:12 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2007.codfw.wmnet with OS bookworm
• 10:00 jayme@cumin1004: START - Cookbook sre.hosts.reimage for host wikikube-worker2041.codfw.wmnet with OS trixie
• 10:00 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet
• 09:54 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet
• 09:54 jayme@cumin1004: START - Cookbook sre.k8s.roll-reimage-nodes rolling reimage on P{wikikube-worker2041.codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw)
• 09:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage
• 09:50 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2007.codfw.wmnet with reason: host reimage
• 09:46 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
• 09:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade (T438828)
• 09:45 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
• 09:39 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2007.codfw.wmnet with OS bookworm
• 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2002 to dse-k8s-worker2007
• 09:36 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2007
• 09:35 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2007
• 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2007 on all recursors
• 09:35 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2007 on all recursors
• 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:35 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004"
• 09:35 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2002 to dse-k8s-worker2007 - brouberol@cumin1004"
• 09:31 brouberol@cumin1004: START - Cookbook sre.dns.netbox
• 09:31 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2002 to dse-k8s-worker2007
• 09:29 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 09:27 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker2006.codfw.wmnet
• 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
• 09:27 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 09:27 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker2006.codfw.wmnet
• 09:26 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 09:26 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet
• 09:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 09:20 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2006.codfw.wmnet
• 09:19 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
• 09:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - 3.2.23 upgrade (T438828)
• 09:10 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - 3.2.23 upgrade (T438828)
• 09:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 09:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
• 09:04 dkertesz: running ncmonitor manually on ncmonitor1001 since last execution failed due to a 504 from MarkMonitor
• 09:03 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 09:00 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync
• 08:52 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-liftwing-studio: sync
• 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync
• 08:51 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-liftwing-studio: sync
• 08:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm
• 08:44 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
• 08:36 dpogorzelski@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
• 08:33 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage
• 08:29 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2006.codfw.wmnet with reason: host reimage
• 08:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and not P{cp2059.*} and A:cp - 3.2.23 upgrade (T438828)
• 08:18 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm
• 08:18 brouberol@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dse-k8s-worker2006.codfw.wmnet with OS bookworm
• 08:16 moritzm: installing Exim security updates
• 08:04 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade (T438828)
• 07:59 matthiasmullie: UTC morning backport window complete
• 07:59 mlitn@deploy1003: Finished scap sync-world: Backport for Growth: Set minimum registration date for A/B test (T432126), EarlyOnboarding: record account setup motivation in `action_context` (T437339) (duration: 14m 12s)
• 07:58 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2006.codfw.wmnet with OS bookworm
• 07:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo2001 to dse-k8s-worker2006
• 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2006
• 07:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2006
• 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2006 on all recursors
• 07:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2006 on all recursors
• 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 07:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004"
• 07:52 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo2001 to dse-k8s-worker2006 - brouberol@cumin1004"
• 07:51 mlitn@deploy1003: migr, mlitn: Continuing with deployment
• 07:49 mlitn@deploy1003: migr, mlitn: Backport for Growth: Set minimum registration date for A/B test (T432126), EarlyOnboarding: record account setup motivation in `action_context` (T437339) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:48 brouberol@cumin1004: START - Cookbook sre.dns.netbox
• 07:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo2001 to dse-k8s-worker2006
• 07:44 mlitn@deploy1003: Started scap sync-world: Backport for Growth: Set minimum registration date for A/B test (T432126), EarlyOnboarding: record account setup motivation in `action_context` (T437339)
• 07:40 mlitn@deploy1003: Finished scap sync-world: Backport for [MultimediaViewer] Testwiki use same carousel config as other wikis, Fix jump to image break in carousel retest (T439252), Remove reliance on ParserOutput (T439182), Remove reliance on file repo (T439182) (duration: 32m 06s)
• 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - 3.2.23 upgrade (T438828)
• 07:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and not P{cp2059.*} and A:cp - 3.2.23 upgrade (T438828)
• 07:27 mlitn@deploy1003: mlitn: Continuing with deployment
• 07:26 mlitn@deploy1003: mlitn: Backport for [MultimediaViewer] Testwiki use same carousel config as other wikis, Fix jump to image break in carousel retest (T439252), Remove reliance on ParserOutput (T439182), Remove reliance on file repo (T439182) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verifi
• 07:08 mlitn@deploy1003: Started scap sync-world: Backport for [MultimediaViewer] Testwiki use same carousel config as other wikis, Fix jump to image break in carousel retest (T439252), Remove reliance on ParserOutput (T439182), Remove reliance on file repo (T439182)
• 06:28 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 06:27 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply
• 06:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook-next: apply
• 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
• 06:09 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 48s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-09-27

• 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 32s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-09-26

• 21:26 krinkle@deploy1003: Finished deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296 (duration: 00m 09s)
• 21:26 krinkle@deploy1003: Started deploy [performance/arc-lamp@68349ee]: https://gerrit.wikimedia.org/r/c/performance/arc-lamp/+/1345296
• 16:37 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 16:37 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 16:37 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 16:37 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 16:30 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 16:30 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 16:30 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 16:29 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 08:07 oblivian@deploy1003: Finished scap sync-world: Backport for Revert "Disable Score exec" (duration: 10m 53s)
• 08:02 oblivian@deploy1003: oblivian: Continuing with deployment
• 08:00 oblivian@deploy1003: oblivian: Backport for Revert "Disable Score exec" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:56 oblivian@deploy1003: Started scap sync-world: Backport for Revert "Disable Score exec"
• 07:52 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
• 07:50 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
• 07:46 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
• 07:44 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply
• 07:42 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
• 07:42 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply
• 06:30 oblivian@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
• 06:30 oblivian@deploy1003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
• 06:29 oblivian@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
• 06:29 oblivian@deploy1003: helmfile [staging] START helmfile.d/services/shellbox: apply
• 06:28 oblivian@deploy1003: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
• 06:27 oblivian@deploy1003: helmfile [codfw] START helmfile.d/services/shellbox: apply
• 03:37 ladsgroup@deploy1003: Finished scap sync-world: Backport for Disable Score exec (T439297 T438443) (duration: 11m 01s)
• 03:31 ladsgroup@deploy1003: ladsgroup: Continuing with deployment
• 03:30 ladsgroup@deploy1003: ladsgroup: Backport for Disable Score exec (T439297 T438443) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 03:26 ladsgroup@deploy1003: Started scap sync-world: Backport for Disable Score exec (T439297 T438443)
• 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 13s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-09-25

• 23:15 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm
• 22:51 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 22:51 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 22:51 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 22:51 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 22:47 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 22:46 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 22:46 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 22:46 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 22:39 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm
• 18:27 krinkle@deploy1003: Finished deploy [statsv/statsv@df3ebff]: T439183: Accept dot, plus, hyphen in label values (duration: 00m 11s)
• 18:27 krinkle@deploy1003: Started deploy [statsv/statsv@df3ebff]: T439183: Accept dot, plus, hyphen in label values
• 17:59 cdanis@dns1004: END - running authdns-update
• 17:57 cdanis@dns1004: START - running authdns-update
• 15:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2001.*
• 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 15:03 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 15:02 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
• 15:01 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
• 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
• 15:01 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
• 14:57 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2001.codfw.wmnet with OS trixie
• 14:56 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1017.eqiad.wmnet
• 14:56 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1017.eqiad.wmnet
• 14:51 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet
• 14:51 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1040.eqiad.wmnet
• 14:51 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1040.eqiad.wmnet
• 14:49 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1041.eqiad.wmnet
• 14:49 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1041.eqiad.wmnet
• 14:49 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet
• 14:46 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1040.eqiad.wmnet
• 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 14:44 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 14:43 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1041.eqiad.wmnet
• 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 14:41 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 14:38 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage
• 14:33 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2001.codfw.wmnet with reason: host reimage
• 14:32 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm
• 14:30 dkertesz: moved haproxy stat file from /var/lib/haproxy/stats-file to /run/haproxy/ in cp7001,cp7011 - T343000
• 14:29 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm
• 14:23 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,name=cp2059.*
• 14:18 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
• 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
• 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
• 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
• 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
• 14:17 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
• 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
• 14:15 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
• 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 14:14 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 14:14 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2001.codfw.wmnet with OS trixie
• 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
• 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
• 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
• 14:13 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
• 14:06 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage
• 14:06 brouberol@cumin1004: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage
• 14:05 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1041.eqiad.wmnet with reason: host reimage
• 14:02 brouberol@cumin1004: conftool action : set/weight=10; selector: name=dse-k8s-worker1039.eqiad.wmnet
• 14:01 brouberol@cumin1004: conftool action : set/pooled=yes; selector: name=dse-k8s-worker1039.eqiad.wmnet
• 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-main,name=codfw
• 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-logging-external,name=codfw
• 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics-external,name=codfw
• 14:00 atsuko@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=eventgate-analytics,name=codfw
• 14:00 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 13:59 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1040.eqiad.wmnet with reason: host reimage
• 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 13:58 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 13:55 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 13:54 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1041.eqiad.wmnet with OS bookworm
• 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1003 to dse-k8s-worker1041
• 13:53 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1041
• 13:52 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1041
• 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1041 on all recursors
• 13:52 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1041 on all recursors
• 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:52 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004"
• 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 13:52 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 13:51 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1003 to dse-k8s-worker1041 - brouberol@cumin1004"
• 13:51 zabe: clone wbc_entity_usage from local cluster to x1 for all wikidata client wikis # T438750
• 13:50 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet
• 13:49 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
• 13:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
• 13:47 brouberol@cumin1004: START - Cookbook sre.dns.netbox
• 13:47 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1003 to dse-k8s-worker1041
• 13:46 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1019.*} and A:lvs
• 13:46 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1019.*} and A:lvs
• 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1039.eqiad.wmnet
• 13:45 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1040.eqiad.wmnet with OS bookworm
• 13:44 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs1020.*} and A:lvs
• 13:44 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs1020.*} and A:lvs
• 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
• 13:42 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
• 13:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1002 to dse-k8s-worker1040
• 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1040
• 13:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1040
• 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1040 on all recursors
• 13:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1040 on all recursors
• 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004"
• 13:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1002 to dse-k8s-worker1040 - brouberol@cumin1004"
• 13:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox
• 13:34 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1002 to dse-k8s-worker1040
• 13:29 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in codfw: return to active/active
• 13:24 Emperor: repool sessionstore in codfw
• 13:24 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in codfw: return to active/active
• 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 13:24 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
• 13:22 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
• 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
• 13:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
• 13:15 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm
• 13:03 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:59 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp2049.codfw.wmnet
• 12:58 jclark@cumin1004: START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage
• 12:52 brouberol@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1039.eqiad.wmnet with reason: host reimage
• 12:47 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance
• 12:47 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance
• 12:45 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 12:44 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 12:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:42 brouberol@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1039.eqiad.wmnet with OS bookworm
• 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ganeti-jumbo1001 to dse-k8s-worker1039
• 12:40 brouberol@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1039
• 12:39 brouberol@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1039
• 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1039 on all recursors
• 12:39 brouberol@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1039 on all recursors
• 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004"
• 12:38 brouberol@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ganeti-jumbo1001 to dse-k8s-worker1039 - brouberol@cumin1004"
• 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin1003.eqiad.wmnet
• 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:34 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
• 12:34 brouberol@cumin1004: START - Cookbook sre.dns.netbox
• 12:33 brouberol@cumin1004: START - Cookbook sre.hosts.rename from ganeti-jumbo1001 to dse-k8s-worker1039
• 12:26 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
• 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
• 12:21 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
• 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
• 12:20 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
• 12:18 jmm@cumin2003: START - Cookbook sre.dns.netbox
• 12:13 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin1003.eqiad.wmnet
• 11:41 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1002.eqiad.wmnet
• 11:40 urbanecm@deploy1003: mwscript-k8s job started: foreachwikiindblist growthexperiments GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366
• 11:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1002.eqiad.wmnet
• 11:20 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 11:19 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 11:17 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet
• 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:54 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 10:53 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet
• 10:52 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues
• 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4
• 09:49 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4
• 09:49 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4
• 09:48 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4
• 09:43 tappof: reset modified_attributes for hosts and services that fully match the Puppet configuration in Icinga - T439105
• 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4
• 09:36 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4
• 09:36 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4
• 09:35 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4
• 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts build2001.codfw.wmnet
• 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 09:28 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
• 09:11 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1207.eqiad.wmnet
• 09:01 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: build2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
• 08:57 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1207.eqiad.wmnet
• 08:57 jmm@cumin2003: START - Cookbook sre.dns.netbox
• 08:52 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts build2001.codfw.wmnet
• 08:24 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
• 08:23 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
• 08:23 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
• 08:22 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
• 08:20 vgutierrez@puppetserver1001: conftool action : set/weight=1; selector: dc=codfw,name=cp2059.*
• 08:15 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,name=cp2059.*
• 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 05:58 dcausse@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 05:21 ryankemper: [Cirrus] Stumble across orphaned index `sawikisource_content_1784136042`, deleted. The real index is `sawikisource_content_1784136826` which I've obviously left untouched
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 38s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
• 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker1*.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 01:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
• 01:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
• 01:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
• 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
• 01:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
• 01:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
• 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
• 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
• 00:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
• 00:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
• 00:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
• 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
• 00:14 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
• 00:14 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
• 00:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
• 00:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
• 00:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
• 00:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet

2026-09-24

• 23:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
• 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
• 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
• 23:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
• 23:50 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
• 23:32 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp404[1-6].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
• 23:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
• 23:20 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
• 23:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
• 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
• 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
• 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
• 22:28 ryankemper: [WDQS] Expanding match in https://requestctl.wikimedia.org/pattern/ua/rocks to test a likely block candidate
• {{safesubst:SAL entry|1=22:27 egardner@deploy1003: Finished scap sync-world: Backport for ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692), ReaderExperiments: Drop the stale ShareHighlight config var (T424764), Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779), [[gerrit:1343560|Revert "Enable Reading Recommendations experiment on t}}
• 22:22 egardner@deploy1003: volker-e, egardner, jdlrobson: Continuing with deployment
• 22:21 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet
• 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
• 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
• 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
• {{safesubst:SAL entry|1=22:14 egardner@deploy1003: volker-e, egardner, jdlrobson: Backport for ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692), ReaderExperiments: Drop the stale ShareHighlight config var (T424764), Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779), [[gerrit:1343560|Revert "Enable Reading Recommendations experiment}}
• {{safesubst:SAL entry|1=22:10 egardner@deploy1003: Started scap sync-world: Backport for ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692), ReaderExperiments: Drop the stale ShareHighlight config var (T424764), Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779), [[gerrit:1343560|Revert "Enable Reading Recommendations experiment on te}}
• 22:04 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_magru and not P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 22:02 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
• 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
• 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
• 22:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
• 21:58 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4052.*
• 21:57 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.21 refs T438217
• 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet
• 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
• 21:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
• 21:53 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
• 21:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp4052.ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet
• 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
• 21:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
• 21:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
• 21:45 catrope@deploy1003: Finished scap sync-world: Backport for Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545) (duration: 17m 05s)
• 21:42 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp4052.ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 21:40 catrope@deploy1003: catrope: Continuing with deployment
• 21:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
• 21:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
• 21:34 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
• 21:34 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
• 21:34 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 21:33 catrope@deploy1003: catrope: Backport for Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:28 catrope@deploy1003: Started scap sync-world: Backport for Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)
• 21:28 catrope@deploy1003: Finished scap sync-world: Backport for ext.wikimediaEvents.testKitchen: Add withContext helper (T438898), ReaderExperiments: add dewiki and svwiki (T438072), Image Browsing carousel: taps outside the preview dialog should close it (T439006), Cap the dialog viewport (T439007) (duration: 19m 27s)
• 21:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
• 21:22 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Continuing with deployment
• 21:12 catrope@deploy1003: cjming, mfossati, catrope, mlitn: Backport for ext.wikimediaEvents.testKitchen: Add withContext helper (T438898), ReaderExperiments: add dewiki and svwiki (T438072), Image Browsing carousel: taps outside the preview dialog should close it (T439006), Cap the dialog viewport (T439007) synced to the testservers (see https://wi
• 21:08 catrope@deploy1003: Started scap sync-world: Backport for ext.wikimediaEvents.testKitchen: Add withContext helper (T438898), ReaderExperiments: add dewiki and svwiki (T438072), Image Browsing carousel: taps outside the preview dialog should close it (T439006), Cap the dialog viewport (T439007)
• 21:04 catrope@deploy1003: Finished scap sync-world: Backport for Revert "cirrus: Send more_like traffic to eqiad", prv: Enable parsoid rendering for 5 wikis (T438998) (duration: 10m 45s)
• 21:03 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4051.*
• 21:02 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp4041.*
• 20:58 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Continuing with deployment
• 20:57 catrope@deploy1003: catrope, ebernhardson, jgiannelos: Backport for Revert "cirrus: Send more_like traffic to eqiad", prv: Enable parsoid rendering for 5 wikis (T438998) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:57 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
• 20:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
• 20:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
• 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 20:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 20:53 catrope@deploy1003: Started scap sync-world: Backport for Revert "cirrus: Send more_like traffic to eqiad", prv: Enable parsoid rendering for 5 wikis (T438998)
• 20:50 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 20:49 catrope@deploy1003: Finished scap sync-world: Backport for HookHandler: Guard against recovery code expiry being null (T438593) (duration: 10m 19s)
• 20:49 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
• 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
• 20:48 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
• 20:47 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
• 20:44 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 20:44 catrope@deploy1003: catrope: Continuing with deployment
• 20:43 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp404[1-6].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 20:43 catrope@deploy1003: catrope: Backport for HookHandler: Guard against recovery code expiry being null (T438593) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
• 20:39 catrope@deploy1003: Started scap sync-world: Backport for HookHandler: Guard against recovery code expiry being null (T438593)
• 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 20:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 ()
• 20:25 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 ()
• 20:19 kemayo@deploy1003: Finished scap sync-world: Backport for EditCheck: add some statsv tracking of check/suggestion actions (T438916) (duration: 11m 23s)
• 20:14 kemayo@deploy1003: kemayo: Continuing with deployment
• 20:12 kemayo@deploy1003: kemayo: Backport for EditCheck: add some statsv tracking of check/suggestion actions (T438916) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
• 20:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
• 20:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
• 20:08 kemayo@deploy1003: Started scap sync-world: Backport for EditCheck: add some statsv tracking of check/suggestion actions (T438916)
• 20:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet
• 19:57 cjming@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
• 19:56 cjming@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
• 19:56 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
• 19:50 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_magru and not P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 19:46 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
• 19:36 ryankemper: [Cirrus] All cirrus pools are serving again. Actively monitoring while the system returns to equilibrium, but all initial indications are that things are as they should be
• 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 19:34 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 19:33 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-omega in codfw: maintenance
• 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet
• 19:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet
• 19:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet
• 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 19:29 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 19:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-omega in codfw: maintenance
• 19:27 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=k8s-ingress-aux-ro
• 19:26 ryankemper: [Cirrus] nevermind, that's just the cookbook assuming the DNS record should exist, which it doesn't because chi/psi/omega all share `search.svc.$DC.wmnet`
• 19:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet
• 19:24 ryankemper: [Cirrus] `dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: search-psi.svc.eqiad.wmnet` checking briefly if this is real failure or just some TTL wonkiness
• 19:23 ryankemper@cumin2003: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-psi in codfw: maintenance
• 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 19:20 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 19:18 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
• 19:18 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search-psi in codfw: maintenance
• 19:17 ryankemper: [Cirrus] codfw chi (big cluster) repooled; metrics are already improving, I see poolcounter rejections dropping significantly
• 19:17 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool search in codfw: maintenance
• 19:17 cdanis@cumin1004: conftool action : set/ttl=300; selector: name=codfw
• 19:13 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
• 19:12 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool search in codfw: maintenance
• 19:11 ryankemper: [Cirrus] Repooling codfw, chi first followed by the small clusters
• 19:11 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=(kartotherian|tegola-vector-tiles)
• 19:07 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
• 19:02 jhuneidi@deploy1003: Finished scap sync-world: Backport for REST: restore PageContentHelper::checkAccess (fix live breakage) (duration: 10m 15s)
• 18:57 jhuneidi@deploy1003: daniel, jhuneidi: Continuing with deployment
• 18:56 jhuneidi@deploy1003: daniel, jhuneidi: Backport for REST: restore PageContentHelper::checkAccess (fix live breakage) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet
• 18:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet
• 18:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet
• 18:52 jhuneidi@deploy1003: Started scap sync-world: Backport for REST: restore PageContentHelper::checkAccess (fix live breakage)
• 18:49 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-scholarly in codfw: maintenance
• 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet
• 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet
• 18:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet
• 18:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet
• 18:44 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-scholarly in codfw: maintenance
• 18:44 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-main in codfw: maintenance
• 18:42 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=codfw
• 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
• 18:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
• 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet
• 18:39 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=codfw
• 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
• 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet
• 18:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet
• 18:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet
• 18:39 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
• 18:39 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-internal-main in codfw: maintenance
• 18:38 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wcqs in codfw: maintenance
• 18:37 herron@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=thanos-web.*,name=codfw
• 18:36 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 18:34 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 18:33 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wcqs in codfw: maintenance
• 18:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 18:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-scholarly in codfw: maintenance
• 18:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet
• 18:30 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 18:28 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-scholarly in codfw: maintenance
• 18:25 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in codfw: maintenance
• 18:25 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
• 18:20 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in codfw: maintenance
• 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
• 18:19 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.21 refs T438217
• 18:19 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
• 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
• 18:18 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
• 18:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
• 18:16 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
• 18:15 ryankemper: [WDQS] Preparing to repool codfw WDQS shortly; it's been operating single DC so this second DC should restore proper service availability
• 18:13 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
• 18:12 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
• 18:11 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 ()
• 18:11 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 ()
• 18:10 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 18:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 18:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 18:06 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 18:04 taavi@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions (duration: 109m 23s)
• 18:04 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
• 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 18:02 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
• 18:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
• 18:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
• 17:59 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool codfw [reason: no reason specified, no task ID specified]
• 17:59 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool codfw [reason: no reason specified, no task ID specified]
• 17:58 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw
• 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
• 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
• 17:54 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
• 17:54 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
• 17:52 cdanis@cumin1004: conftool action : set/pooled=false; selector: name=codfw,dnsdisc=mwdebug.*
• 17:52 swfrench@cumin1004: conftool action : set/pooled=false; selector: dnsdisc=mwdebug.*,name=codfw
• 17:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-.*-ro
• 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apus
• 17:47 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug.*
• 17:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
• 17:44 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=swift
• 17:42 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=config-master|device-analytics|echostore|helm-charts|k8s-ingress-wikikube-ro|linkrecommendation|mathoid|restbase|restbase-async|rest-gateway-ro|mobileapps|mwdebug.*|push-notifications|recommendation-api|releases|wikifeeds
• 17:38 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
• 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_magru and not P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 17:37 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_magru and not P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 17:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
• 17:32 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=citoid|zotero
• 17:30 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apertium|schema|termbox|proton|cxserver
• 17:22 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
• 17:19 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=thumbor
• 17:18 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=shellbox.*
• 17:17 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=urldownloader
• 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
• 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
• 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
• 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
• 17:05 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1001.*
• 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 16:55 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 16:54 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 16:49 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-next-ro
• 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
• 16:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
• 16:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
• 16:40 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
• 16:37 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-ro
• 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
• 16:33 cdanis@cumin1004: conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-api-int-ro
• 16:33 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS trixie
• 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
• 16:23 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
• 16:20 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw
• 16:19 hnowlan@cumin1004: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw
• 16:15 taavi@deploy1003: Locking from deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions
• 16:14 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
• 16:14 dreamyjazz@deploy1003: Finished scap sync-world: Backport for AbuseReview: Enable on enwiki (T439149), Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467) (duration: 33m 52s)
• 16:08 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
• 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
• 16:03 swfrench@cumin1004: END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo
• 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
• 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
• 16:01 hnowlan@cumin1004: START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw
• 16:01 swfrench@cumin1004: START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo
• 16:01 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment
• 16:00 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for AbuseReview: Enable on enwiki (T439149), Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 15:57 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 15:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
• 15:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet
• 15:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS trixie
• 15:51 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3005.*
• 15:51 swfrench-wmf: begin rolling restarts of confds in eqsin, codfw, ulsfo to reflect etcd SRV record changes
• 15:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1017.eqiad.wmnet
• 15:40 dreamyjazz@deploy1003: Started scap sync-world: Backport for AbuseReview: Enable on enwiki (T439149), Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)
• 15:35 vgutierrez@dns1004: END - running authdns-update
• 15:33 vgutierrez@dns1004: START - running authdns-update
• 15:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for EventMapper::fetchByPage: Allow filtering by type (T438031) (duration: 12m 33s)
• 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 15:30 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
• 15:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 15:27 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
• 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
• 15:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
• 15:25 dreamyjazz@deploy1003: kharlan, dreamyjazz: Continuing with deployment
• 15:24 dreamyjazz@deploy1003: kharlan, dreamyjazz: Backport for EventMapper::fetchByPage: Allow filtering by type (T438031) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
• 15:20 dreamyjazz@deploy1003: Started scap sync-world: Backport for EventMapper::fetchByPage: Allow filtering by type (T438031)
• 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3005.esams.wmnet with OS trixie
• 15:12 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox
• 15:06 vgutierrez@dns1004: END - running authdns-update
• 15:04 vgutierrez@dns1004: START - running authdns-update
• 15:03 vgutierrez@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns2.*,service=authdns-update
• 14:59 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Add local CheckUsers to vandalism alpha test (T438467), AbuseReview: Inidicate if the queue hides recent edits (T438235) (duration: 32m 20s)
• 14:57 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7011.*
• 14:54 dkertesz@cumin1004: conftool action : set/pooled=yes; selector: name=cp7001.*
• 14:54 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
• 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
• 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
• 14:53 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
• 14:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 14:51 dkertesz: repooling cp7001|7011 after successful testing (T343000)
• 14:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
• 14:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
• 14:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
• 14:47 kharlan@deploy1003: kharlan: Continuing with deployment
• 14:46 kharlan@deploy1003: kharlan: Backport for AbuseReview: Add local CheckUsers to vandalism alpha test (T438467), AbuseReview: Inidicate if the queue hides recent edits (T438235) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:43 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage
• 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet
• 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet
• 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet
• 14:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet
• 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet
• 14:40 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet
• 14:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet
• 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet
• 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet
• 14:39 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[5025,5026].*} and A:cp - 3.2.23 upgrade (T438828)
• 14:39 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage
• 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
• 14:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
• 14:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
• 14:34 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7011.*
• 14:33 dkertesz@cumin1004: conftool action : set/pooled=no; selector: name=cp7001.*
• 14:32 dkertesz: depooling cp7001|7011 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222 (context: https://phabricator.wikimedia.org/T343000)
• 14:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
• 14:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
• 14:30 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker1*.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 14:27 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Add local CheckUsers to vandalism alpha test (T438467), AbuseReview: Inidicate if the queue hides recent edits (T438235)
• 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 14:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
• 14:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
• 14:22 btullis@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 14:22 elukey: elukey@rdb2013:/srv/redis/appendonlydir$ sudo -u redis redis-check-aof --fix rdb2013-6380.aof.22039.incr.aof
• 14:21 btullis@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 14:21 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[5025,5026].*} and A:cp - 3.2.23 upgrade (T438828)
• 14:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
• 14:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
• 14:19 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 14:17 moritzm: installing Bird security updates
• 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 14:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
• 14:13 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
• 14:11 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade (T438828)
• 14:09 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 14:09 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3005.esams.wmnet with OS trixie
• 14:08 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
• 14:07 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir4004.*
• 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
• 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
• 14:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
• 14:07 urbanecm@deploy1003: Finished scap sync-world: Backport for fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872) (duration: 12m 27s)
• 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 14:05 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 14:03 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
• 14:01 urbanecm@deploy1003: migr, urbanecm: Continuing with deployment
• 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
• 14:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
• 14:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
• 14:00 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir4004.ulsfo.wmnet with OS trixie
• 13:58 urbanecm@deploy1003: migr, urbanecm: Backport for fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
• 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 13:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
• 13:55 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 13:55 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 13:54 urbanecm@deploy1003: Started scap sync-world: Backport for fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)
• 13:40 awight@deploy1003: Finished scap sync-world: Backport for Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687) (duration: 10m 38s)
• 13:39 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 13:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage
• 13:35 moritzm: installing nghttp2 security updates
• 13:35 awight@deploy1003: awight: Continuing with deployment
• 13:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage
• 13:33 awight@deploy1003: awight: Backport for Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:29 awight@deploy1003: Started scap sync-world: Backport for Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)
• 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
• 13:26 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
• 13:26 elukey@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
• 13:26 elukey@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
• 13:25 elukey@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
• 13:25 elukey@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'sync'.
• 13:25 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
• 13:24 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
• 13:18 mlitn@deploy1003: Finished scap sync-world: Backport for Instrument five-arm image carousel retest (T431362), Wire image carousel retest instrumentation (T431362), ThumbExtractor: trim nbsp and dangling colons from caption text (T435672), ThumbExtractor: exclude lead infobox images from the carousel (T438907) (duration: 12m 25s)
• 13:13 mlitn@deploy1003: mfossati, mlitn: Continuing with deployment
• 13:10 mlitn@deploy1003: mfossati, mlitn: Backport for Instrument five-arm image carousel retest (T431362), Wire image carousel retest instrumentation (T431362), ThumbExtractor: trim nbsp and dangling colons from caption text (T435672), ThumbExtractor: exclude lead infobox images from the carousel (T438907) synced to the testservers (see https://wiki
• 13:08 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir4004.ulsfo.wmnet with OS trixie
• 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 13:07 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 13:06 mlitn@deploy1003: Started scap sync-world: Backport for Instrument five-arm image carousel retest (T431362), Wire image carousel retest instrumentation (T431362), ThumbExtractor: trim nbsp and dangling colons from caption text (T435672), ThumbExtractor: exclude lead infobox images from the carousel (T438907)
• 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 13:06 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 13:02 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 13:02 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade (T438828)
• 13:01 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=cp2059.*
• 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 12:59 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 12:58 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 12:52 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 12:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
• 12:34 urbanecm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
• 12:34 urbanecm@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
• 12:04 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:03 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 11:21 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[5031,5032].*} and A:cp - 3.2.23 upgrade (T438828)
• 11:13 hnowlan: restarted restbase on restbase2029
• 11:04 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[5031,5032].*} and A:cp - 3.2.23 upgrade (T438828)
• 10:50 hnowlan: deleting stuck mw-web pods in eqiad
• 10:45 dreamyjazz@deploy1003: Finished scap sync-world: Backport for AbuseReview: Let specific users and suppressors see vandalism tag (T438860) (duration: 10m 09s)
• 10:44 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
• 10:42 vgutierrez@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade (T438828)
• 10:40 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
• 10:39 dreamyjazz@deploy1003: dreamyjazz: Backport for AbuseReview: Let specific users and suppressors see vandalism tag (T438860) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 10:36 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision
• 10:35 dreamyjazz@deploy1003: Started scap sync-world: Backport for AbuseReview: Let specific users and suppressors see vandalism tag (T438860)
• 10:34 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
• 10:32 dreamyjazz@deploy1003: Finished scap sync-world: Backport for WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860) (duration: 10m 34s)
• 10:29 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 10:26 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
• 10:26 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 10:25 dreamyjazz@deploy1003: dreamyjazz: Backport for WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 10:23 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision
• 10:22 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
• 10:21 dreamyjazz@deploy1003: Started scap sync-world: Backport for WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)
• 10:17 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident T439010. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s)
• 10:17 hnowlan@deploy1003: Forcefully removing global lock: Unlocking scap after restoration of power in codfw
• 10:12 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
• 10:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade (T438828)
• 10:08 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade (T438828)
• 10:03 moritzm: installing apr-util security updates
• 09:46 moritzm: installing bind9 security updates (client-side tools/libs only)
• 09:40 vgutierrez@puppetserver1001: conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet
• 09:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, T437984]
• 09:27 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, T437984]
• 09:26 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13
• 09:25 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13
• 09:23 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
• 09:22 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
• 09:22 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
• 09:22 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
• 09:21 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
• 09:21 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
• 09:10 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet
• 09:05 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade (T438828)
• 09:04 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1016.eqiad.wmnet
• 09:01 XioNoX: asw1-b13-drmrs> request system reboot - T437984
• 09:00 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner
• 09:00 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B13
• 08:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 08:58 moritzm: installing node-lodash security updates
• 08:56 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B13
• 08:55 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1078.eqiad.wmnet with reason: provision
• 08:54 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 08:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B12
• 08:47 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B12
• 08:46 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 19 hosts with reason: Switches upgrade
• 08:46 moritzm: uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia T438866
• 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt
• 08:45 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt
• 08:45 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b13-drmrs,asw1-b13-drmrs IPv6,asw1-b13-drmrs.mgmt with reason: Switch upgrade
• 08:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker1015.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 08:37 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host dse-k8s-worker1015.eqiad.wmnet
• 08:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
• 08:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
• 08:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
• 08:31 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker1015.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 08:22 XioNoX: asw1-b12-drmrs> request system reboot - T437984
• 08:20 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B12
• 08:13 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B12
• 08:06 jelto@cumin1004: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner
• 08:02 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt with reason: Switch upgrade
• 07:53 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switches upgrade
• 07:52 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: switch upgrade, T437984]
• 07:52 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: switch upgrade, T437984]
• 07:48 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
• 07:19 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
• 07:16 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
• 07:06 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
• 07:02 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
• 06:51 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
• 06:41 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production (T437213)
• 06:41 kartik@deploy1003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
• 06:39 kart_: staging: Update machinetranslation/MinT to 2026-09-21-112314-production
• 06:38 kartik@deploy1003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
• 06:07 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-e5-codfw
• 06:06 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device lsw1-e5-codfw
• 05:07 ryankemper@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - T439010
• 01:21 ryankemper@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - T439010
• 01:19 ryankemper: [Cirrus] Reverted `node_concurrent_recoveries` to 5 from 10, now that we're back to green
• 01:16 ryankemper: [Cirrus] With the restart of `cirrussearch2115`, the codfw cluster has officially reached green status!!! Still working on full verification, but we're almost done here
• 01:14 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2115.codfw.wmnet with reason: Codfw survivor recovery on 2115; temporary chi red expected (T439010)
• 01:11 brett@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2059.codfw.wmnet with reason: failing services but not in service yet
• 01:10 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2109.codfw.wmnet with reason: Codfw survivor recovery on 2109; temporary chi red expected (T439010)
• 01:04 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2104.codfw.wmnet with reason: Codfw survivor recovery on 2104; temporary chi red expected (T439010)
• 01:03 ryankemper: [Cirrus] grr, I'd missed some hosts. restarting the last few dangling ones, we're really close to back to green, prob 3-ish more hosts
• 00:40 ryankemper: [Cirrus] Great news, we briefly dipped red (same as previous restarts) but went back to yellow almost immediately. AFAICT election went fine, still checking though
• 00:38 ryankemper: [Cirrus] Preparing to restart cirrussearch2084 (active cluster manager). With luck, this should restore updater availability (and general cluster green status, after some reshuffling)
• 00:35 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 55 hosts with reason: Codfw chi elected-manager recovery on 2084; expected brief failover and red state (T439010)
• 00:10 brett@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.*
• 00:05 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 00:00 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()

2026-09-23

• 23:58 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
• 23:56 brett: Switching acme-chief primary from codfw to eqiad - T439010
• 23:54 brett@cumin1004: END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 23:49 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 23:48 brett@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 23:48 ryankemper: [Cirrus] Every host except 2084, which is the current elected chi master, has now been restarted, and shard recoveries healed accordingly. AFAICT we will not be able to revive the updater until we restart this host. Pausing for a few mins to mull things over and get my bearings though, because this restart would be higher-touch than the previous ones
• 23:38 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2108.codfw.wmnet with reason: Codfw survivor recovery on 2108; sequential chi and psi restarts (T439010)
• 23:38 brett@cumin1004: START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
• 23:35 brett: import varnish 7.1.1-2~bpo13+wmf3 into trixie-wikimedia (T438293)
• 23:34 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2107.codfw.wmnet with reason: Codfw survivor recovery on 2107; sequential chi and psi restarts (T439010)
• 23:27 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2085.codfw.wmnet with reason: Codfw survivor recovery on 2085; sequential chi and psi restarts (T439010)
• 23:23 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident T439010. Thursday UTC morning at the earliest, but please ask SRE oncall.
• 23:23 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress T439010 (duration: 121m 40s)
• 23:20 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2072.codfw.wmnet with reason: Codfw survivor recovery on 2072; sequential chi and psi restarts (T439010)
• 23:09 ryankemper: [Cirrus] rolling cirrussearch2086 next
• 23:08 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2086.codfw.wmnet with reason: Codfw survivor recovery on 2086; sequential chi and omega restarts (T439010)
• 23:01 ryankemper: [Cirrus] Doing cirrussearch2114 next
• 22:59 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2114.codfw.wmnet with reason: Codfw survivor recovery on 2114; sequential chi and omega restarts (T439010)
• 22:44 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2106.codfw.wmnet with reason: Codfw chi survivor recovery on 2106; temporary red expected (T439010)
• 22:29 ryankemper: [Cirrus] proceeding with manual restart of cirrussearch2105; red status expected, hopefully brief but we'll see
• 22:28 ryankemper@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2105.codfw.wmnet with reason: Codfw chi recovery canary on 2105; temporary service interruption expected (T439010)
• 22:24 ryankemper: [Cirrus] s/expected/expect
• 22:23 ryankemper: [Cirrus] Alright, I'm getting increasingly convinced that there's no way to restore healthy cluster state without inevitably having to restart sole-shard-holder hosts, which will put the cluster into red status. going to start with just `cirrussearch2105`; I expected red status. silencing alerts first so I don't blow out the channel
• 22:08 ryankemper: [Cirrus] (to be clear the cluster is not serving live traffic, but if I can avoid red I will)
• 22:08 ryankemper: [Cirrus] updater still failing in codfw cirrussearch; i've restarted the directly-impacted hosts but not the others. some bulk updates appear to be getting rejected, going to do some targeted restarts and assess impact before considering a broader operation. first up is `cirrussearch2071.codfw.wmnet` which is not the sole holder of any shards therefore should not plunge the cluster into red status
• 21:49 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
• 21:22 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress T439010
• 21:22 rzl@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress T439010 (duration: 51m 29s)
• 21:21 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
• 21:18 Emperor: ceph mgr fail on apus-be2005
• 21:18 Emperor: reset-failed then restart ceph-mon on moss-be2003
• 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2235].codfw.wmnet with reason: needs fixing
• 21:08 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2234].codfw.wmnet with reason: needs fixing
• 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2233].codfw.wmnet with reason: needs fixing
• 21:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2232].codfw.wmnet with reason: needs fixing
• 20:57 ryankemper: [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51%
• 20:55 ryankemper: [Cirrus] Bump codfw cirrussearch shard recoveries from 5 to 10; cluster not serving live traffic so I'm hoping we have headroom to recover faster
• 20:49 swfrench@dns1004: END - running authdns-update
• 20:46 swfrench@dns1004: START - running authdns-update
• 20:41 ryankemper: [Cirrus] Been restarting all impacted codfw opensearch hosts one at a time (they didn't rejoin the cluster naturally)
• 20:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
• 20:38 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
• 20:30 rzl@deploy1003: Locking from deployment [ALL REPOSITORIES]: incident recovery in progress T439010
• 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 20:27 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 20:06 dzahn@dns1004: END - running authdns-update
• 20:03 dzahn@dns1004: START - running authdns-update
• 19:52 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
• 19:34 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2059.codfw.wmnet with OS trixie
• 19:33 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 19:33 volans: rebooting arclamp2001.codfw.wmnet
• 19:32 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 19:20 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 979 hosts with reason: power is still coming back on
• 19:17 taavi@dns1004: END - running authdns-update
• 19:14 taavi@dns1004: START - running authdns-update
• 19:10 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org
• 19:10 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org
• 19:10 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6001.*
• 19:08 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: dc=codfw,cluster=dnsbox,service=authdns-update
• 18:59 sukhe@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on 980 hosts with reason: power is still coming back on
• 18:58 taavi@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.discovery.wmnet on all recursors
• 18:58 taavi@cumin1004: START - Cookbook sre.dns.wipe-cache gerrit.discovery.wmnet on all recursors
• 18:50 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit2003.wikimedia.org
• 18:45 sukhe@dns1004: END - running authdns-update
• 18:43 sukhe@dns1004: START - running authdns-update
• 18:43 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org
• 18:42 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox,service=authdns-update
• 18:42 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org
• 18:42 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org
• 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org
• 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org
• 18:40 dzahn@cumin2003: END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org
• 18:40 dzahn@cumin2003: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org
• 18:40 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org
• 18:38 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors
• 18:38 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors
• 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org
• 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org
• 18:36 taavi@cumin1004: END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org
• 18:36 taavi@cumin1004: START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org
• 18:30 taavi@cumin1004: START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org
• 18:29 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 18:28 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 18:14 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie
• 18:08 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) idp.wikimedia.org on all recursors
• 18:08 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors
• 18:05 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors
• 18:05 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors
• 18:03 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors
• 18:03 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors
• 18:02 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors
• 18:02 cdanis@cumin1004: START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors
• 18:01 cdanis@dns1005: END - running authdns-update
• 17:58 cdanis@dns1005: START - running authdns-update
• 17:57 vriley@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage
• 17:54 taavi@dns1004: END - running authdns-update
• 17:53 vriley@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage
• 17:51 taavi@dns1004: START - running authdns-update
• 17:46 taavi@dns1004: END - running authdns-update
• 17:43 taavi@dns1004: START - running authdns-update
• 17:37 vriley@cumin1004: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
• 17:35 rzl@cumin1004: START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: maintenance - T439010
• 17:35 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6001.drmrs.wmnet with OS trixie
• 17:35 cdanis@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified]
• 17:35 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified]
• 17:24 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool codfw [reason: no reason specified, no task ID specified]
• 17:23 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified]
• 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
• 17:21 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
• 17:18 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
• 17:17 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
• 17:16 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 17:14 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage
• 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049.codfw.wmnet
• 17:11 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp2049
• 17:10 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage
• 17:07 mutante: cloudcontrol2005-dev, cloudcontrol2006-dev, cloudcontrol2010-dev: restart zookeeper, enabled logging (/var/log/zookeeper/zookeeper.log) after gerrit:1342354
• 17:02 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage
• 16:59 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage
• 16:51 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie
• 16:51 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2059.codfw.wmnet with OS trixie
• 16:48 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie
• 16:39 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie
• 16:35 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
• 16:34 dzahn@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
• 16:30 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 16:29 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6001.drmrs.wmnet with OS trixie
• 16:10 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie
• 16:10 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie
• 15:55 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade (T438828)
• 15:54 moritzm: installing cjose security updates
• 15:54 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir7004.*
• 15:53 dancy@deploy1003: Finished scap sync-world: testing (duration: 07m 06s)
• 15:52 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie
• 15:52 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie
• 15:51 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 15:50 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 15:46 dancy@deploy1003: Started scap sync-world: testing
• 15:43 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie
• 15:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7004.magru.wmnet with OS trixie
• 15:42 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 15:41 sukhe: homer "lsw1-e4-codfw.*" commit 'pending from cookbook'
• 15:41 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 T438961
• 15:39 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from sretest2013 to cp2059
• 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp2059
• 15:38 sukhe@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host cp2059
• 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp2059 on all recursors
• 15:38 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ T438961
• 15:38 sukhe@cumin1004: START - Cookbook sre.dns.wipe-cache cp2059 on all recursors
• 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:38 sukhe@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004"
• 15:37 sukhe@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004"
• 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 15:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 15:35 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d T438961
• 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 15:35 mutante: zuul1005 - reimage - should not have had nftables on it before T438786
• 15:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 15:34 dzahn@cumin2003: START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
• 15:34 sukhe@cumin1004: START - Cookbook sre.dns.netbox
• 15:33 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 15:32 Emperor: rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 T438961
• 15:23 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
• 15:23 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade (T438828)
• 15:21 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie
• 15:21 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
• 15:21 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
• 15:20 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 15:20 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 15:19 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 15:18 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage
• 15:14 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage
• 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad
• 15:12 jayme@deploy1003: conftool action : set/pooled=true; selector: dnsdisc=mw-web-next-ro,name=eqiad
• 15:12 moritzm: removed buster-wikimedia and all related components from apt.wikimedia.org following the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247618
• 15:06 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
• 15:02 dancy@deploy1003: Installation of scap version "4.292.0" completed for 3 hosts
• 15:02 sukhe@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage
• 15:02 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
• 15:01 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
• 15:01 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
• 15:01 jayme@deploy1003: conftool action : set/pooled=false; selector: dnsdisc=mw-web-next-ro,name=eqiad
• 15:01 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
• 15:01 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
• 15:00 dancy@deploy1003: Installing scap version "4.292.0" for 3 host(s)
• 15:00 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
• 14:58 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
• 14:58 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
• 14:57 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
• 14:57 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
• 14:57 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
• 14:57 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
• 14:56 sukhe@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage
• 14:55 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance
• 14:55 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance
• 14:55 jayme@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: maintenance
• 14:55 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4
• 14:54 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4
• 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance
• 14:54 jayme@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance
• 14:54 jayme@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance
• 14:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4
• 14:53 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 14:53 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4
• 14:47 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 14:47 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:47 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4
• 14:46 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4
• 14:45 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4
• 14:45 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:45 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 14:44 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 14:44 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4
• 14:43 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4
• 14:43 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:42 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:42 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 12s)
• 14:42 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4
• 14:42 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566]
• 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir7004.magru.wmnet with OS trixie
• 14:40 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4
• 14:40 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4
• 14:39 moritzm: upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia T438866
• 14:38 gengh@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 14:38 gengh@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 14:37 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:37 urbanecm@deploy1003: Finished scap sync-world: Backport for feat(AddLink): Do not resuggest an already reviewed page (T429417), feat(AddLink): Do not resuggest an already reviewed page (T429417) (duration: 14m 34s)
• 14:37 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
• 14:37 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:36 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:36 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
• 14:28 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie
• 14:28 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir1002.*
• 14:26 gengh@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 14:26 gengh@deploy1003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 14:24 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:23 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:23 urbanecm@deploy1003: Started scap sync-world: Backport for feat(AddLink): Do not resuggest an already reviewed page (T429417), feat(AddLink): Do not resuggest an already reviewed page (T429417)
• 14:17 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS trixie
• 14:10 gengh@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 14:09 gengh@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
• 14:07 ebernhardson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply
• 13:58 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage
• 13:56 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie
• 13:53 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage
• 13:38 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
• 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
• 13:37 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
• 13:35 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS trixie
• 13:30 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
• 13:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
• 13:29 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 13:25 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4
• 13:24 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4
• 13:24 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues
• 13:18 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie
• 13:13 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4
• 13:10 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4
• 13:09 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4
• 13:04 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4
• 12:57 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 12:57 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
• 12:56 awight: manually run puppet agent
• 12:56 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 12:56 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
• 12:55 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0)
• 12:55 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
• 12:45 awight: add seanleong-wmde to deployment-prep
• 12:44 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-eqiad,ssw1-d[1,8]-eqiad with reason: re-rack ssw1-a1-eqiad
• 12:39 cwilliams@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4
• 12:39 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0)
• 12:38 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
• 12:34 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Add warning indicating alpha test to vandalism queue (T438467) (duration: 33m 33s)
• 12:33 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0)
• 12:33 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
• 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
• 12:32 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 12:32 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
• 12:30 brouberol@cumin1004: END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0)
• 12:30 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
• 12:29 cwilliams@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4
• 12:22 kharlan@deploy1003: kharlan: Continuing with deployment
• 12:21 kharlan@deploy1003: kharlan: Backport for AbuseReview: Add warning indicating alpha test to vandalism queue (T438467) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:15 cdanis@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified]
• 12:15 cdanis@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified]
• 12:01 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)
• 11:51 Dreamy_Jazz: Deployed patch for T438729
• 11:31 mvolz@deploy1003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
• 11:28 mvolz@deploy1003: helmfile [eqiad] START helmfile.d/services/citoid: apply
• 11:27 mvolz@deploy1003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
• 11:27 mvolz@deploy1003: helmfile [codfw] START helmfile.d/services/citoid: apply
• 11:25 mvolz@deploy1003: helmfile [staging] DONE helmfile.d/services/citoid: apply
• 11:25 mvolz@deploy1003: helmfile [staging] START helmfile.d/services/citoid: apply
• 10:38 jayme: sudo confctl --quiet --object-type discovery select 'dnsdisc=mw-web-ro' set/ttl=10 - T438896
• 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply
• 10:31 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply
• 10:25 blake@deploy1003: Finished scap sync-world: Upsize mw-web T438896 (duration: 04m 20s)
• 10:22 blake@deploy1003: Started scap sync-world: Upsize mw-web T438896
• 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 10:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
• 10:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
• 10:04 ayounsi@cumin1004: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004
• 09:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
• 09:59 ayounsi@cumin1004: START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004
• 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
• 09:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
• 09:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
• 09:57 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 09:55 brouberol@cumin1004: DONE (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0)
• 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 09:54 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 09:53 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 09:52 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 09:51 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
• 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
• 09:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
• 09:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
• 09:50 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 09:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
• 09:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
• 09:43 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 09:38 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 09:34 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 08:45 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 08:44 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 08:41 brouberol@cumin1004: DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 08:27 brouberol@cumin1004: END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99)
• 08:27 brouberol@cumin1004: START - Cookbook sre.ceph.remove-osd
• 08:25 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 08:24 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 08:13 tappof@deploy1003: Finished scap sync-world: T432444 - Provision kafka-logging100[6-8] (duration: 12m 52s)
• 08:05 moritzm: installing grub2 bugfix updates on Bookworm hosts
• 08:04 tappof@deploy1003: Started scap sync-world: T432444 - Provision kafka-logging100[6-8]
• 08:00 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
• 07:59 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
• 07:59 moritzm: installing giflib security updates
• 07:58 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
• 07:58 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
• 07:29 moritzm: installing python-idna security updates
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 39s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image
• 00:50 ryankemper@cumin2003: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance
• 00:46 ryankemper: [WDQS] T435443 Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture
• 00:45 ryankemper@cumin2003: START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance

2026-09-22

• 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker10[02-28].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 23:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
• 23:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
• 23:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
• 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
• 22:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
• 22:45 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
• 22:36 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
• 22:30 ryankemper: [WDQS] codfw wdqs-main is struggling under the switchover load, fiddling with some auto-restart knobs to see if it helps or hurts
• 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
• 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
• 22:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
• 21:58 rzl@deploy1003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1339694 T437403 (duration: 13m 43s)
• 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
• 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
• 21:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
• 21:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
• 21:53 rzl@deploy1003: rzl: Continuing with deployment
• 21:51 rzl@deploy1003: rzl: https://gerrit.wikimedia.org/r/1339694 T437403 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
• 21:47 rzl@deploy1003: Started scap sync-world: https://gerrit.wikimedia.org/r/1339694 T437403
• 21:25 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 01m 09s)
• 21:24 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566]
• 21:24 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 24m 20s)
• 21:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
• 21:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
• 21:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
• 21:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
• 21:05 sukhe@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie
• 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566]
• 20:59 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 30s)
• 20:59 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566]
• 20:55 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566] (duration: 06m 59s)
• 20:48 aqu@deploy1003: Started deploy [analytics/refinery@58c9356]: Regular analytics weekly train [analytics/refinery@58c93566]
• 20:46 aqu@deploy1003: Finished deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566] (duration: 00m 40s)
• 20:45 bking@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 20:45 sbisson@deploy1003: Finished scap sync-world: Backport for Keep Article Guidance on where it is on today (T433293) (duration: 09m 53s)
• 20:45 aqu@deploy1003: Started deploy [analytics/refinery@58c9356] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@58c93566]
• 20:44 bking@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 20:44 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 20:43 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 20:40 sbisson@deploy1003: sbisson: Continuing with deployment
• 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
• 20:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
• 20:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
• 20:40 sbisson@deploy1003: sbisson: Backport for Keep Article Guidance on where it is on today (T433293) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:35 sbisson@deploy1003: Started scap sync-world: Backport for Keep Article Guidance on where it is on today (T433293)
• 20:33 ebernhardson@deploy1003: Finished scap sync-world: Backport for eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652) (duration: 13m 35s)
• 20:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
• 20:28 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Continuing with deployment
• 20:24 ebernhardson@deploy1003: ebernhardson, codenamenoreste: Backport for eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:20 ebernhardson@deploy1003: Started scap sync-world: Backport for eswiki: Add abusefilter-access-protected-vars to abusefilter user group (T436652)
• 20:17 ebernhardson@deploy1003: Finished scap sync-world: Backport for cirrus: Send more_like traffic to eqiad (duration: 10m 29s)
• 20:15 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 20:13 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir2002.*
• 20:12 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 20:12 ebernhardson@deploy1003: ebernhardson: Continuing with deployment
• 20:11 ebernhardson@deploy1003: ebernhardson: Backport for cirrus: Send more_like traffic to eqiad synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:06 ebernhardson@deploy1003: Started scap sync-world: Backport for cirrus: Send more_like traffic to eqiad
• 20:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
• 20:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
• 20:02 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
• 20:02 sukhe@cumin1004: START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie
• 19:59 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir2002.codfw.wmnet with OS trixie
• 19:44 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1022.eqiad.wmnet
• 19:42 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage
• 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
• 19:42 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
• 19:42 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
• 19:38 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir2002.codfw.wmnet with reason: host reimage
• 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1016.eqiad.wmnet with OS trixie
• 19:34 jclark@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004"
• 19:33 jclark@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1004"
• 19:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 19:24 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 19:23 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet
• 19:22 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 19:21 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 19:19 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir2002.codfw.wmnet with OS trixie
• 19:19 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage
• 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
• 19:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
• 19:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
• 19:15 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1016.eqiad.wmnet with reason: host reimage
• 19:01 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-codfw to update to opensearch 3.8.0
• 18:58 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
• 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
• 18:56 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
• 18:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
• 18:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
• 18:55 dancy@deploy1003: Installation of scap version "4.291.0" completed for 2 hosts
• 18:53 dancy@deploy1003: Installing scap version "4.291.0" for 2 host(s)
• 18:53 dancy@deploy1003: Installation of scap version "4.291.0" completed for 3 hosts
• 18:51 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s)
• 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet
• 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
• 18:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
• 18:49 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
• 18:47 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s)
• 18:44 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s)
• 18:43 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s)
• 18:41 dancy@deploy1003: install-world aborted: (no justification provided) (duration: 00m 48s)
• 18:41 dancy@deploy1003: Installing scap version "4.291.0" for 3 host(s)
• 18:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet
• 18:36 jhuneidi@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.21 refs T438217
• 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
• 18:35 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
• 18:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
• 18:18 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet
• 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
• 18:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
• 18:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
• 18:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet
• 18:07 ebernhardson: Rolling restart opensearch-semantic-search in dse-k8s-eqiad to update to opensearch 3.8.0
• 17:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad (duration: 10m 09s)
• 17:54 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1030
• 17:54 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1030
• 17:53 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1030
• 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:53 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1030.eqiad.wmnet 8.32.64.10.in-addr.arpa 8.0.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:53 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:53 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003"
• 17:53 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1030 - bking@cumin2003"
• 17:51 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2218: Optimizer issues fixed
• 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Continuing with deployment
• 17:50 dreamyjazz@deploy1003: dreamyjazz, isaranto: Backport for fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 17:47 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 17:46 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 17:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for fix(WikimediaAntiAbuse): use correct endpoint for LiftWing in eqiad
• 17:45 bking@cumin2003: START - Cookbook sre.dns.netbox
• 17:43 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1030
• 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet
• 17:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet
• 17:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet
• 17:37 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1029
• 17:37 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1029
• 17:36 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host wdqs1029
• 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:36 bking@cumin2003: START - Cookbook sre.dns.wipe-cache wdqs1029.eqiad.wmnet 8.48.64.10.in-addr.arpa 8.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 17:36 bking@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:36 bking@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003"
• 17:36 bking@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1029 - bking@cumin2003"
• 17:33 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 17:32 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 17:31 bking@cumin2003: START - Cookbook sre.dns.netbox
• 17:31 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host wdqs1029
• 17:26 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1012.eqiad.wmnet
• 17:25 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 17:25 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 17:24 dzahn@dns1004: END - running authdns-update
• 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet
• 17:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet
• 17:24 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet
• 17:22 dzahn@dns1004: START - running authdns-update
• 17:17 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet
• 17:17 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet
• 17:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet
• 17:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet
• 17:15 oblivian@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=rest-gateway,name=codfw
• 17:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet
• 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet
• 17:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet
• 17:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet
• 17:06 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db2218: Optimizer issues fixed
• 17:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet
• 17:02 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059.codfw.wmnet
• 17:01 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059.codfw.wmnet
• 17:01 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 17:00 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 16:59 dreamyjazz@deploy1003: Finished scap sync-world: Backport for Enable AbuseReview on jawiki for likely PII (T438867) (duration: 13m 13s)
• 16:54 oblivian@cumin1004: END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool 2 services in eqiad: maintenance
• 16:51 dreamyjazz@deploy1003: dreamyjazz: Continuing with deployment
• 16:50 dreamyjazz@deploy1003: dreamyjazz: Backport for Enable AbuseReview on jawiki for likely PII (T438867) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 16:48 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 2 services in eqiad: maintenance
• 16:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: fixing
• 16:45 dreamyjazz@deploy1003: Started scap sync-world: Backport for Enable AbuseReview on jawiki for likely PII (T438867)
• 16:42 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2218.codfw.wmnet with reason: fixing
• 16:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 16:41 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
• 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
• 16:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
• 16:26 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 16:26 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 16:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
• 16:19 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool 4 services in eqiad: maintenance
• 16:13 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool 4 services in eqiad: maintenance
• 16:04 sukhe@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 16:04 sukhe@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 15:58 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2218: optimizer issues
• 15:57 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool db2218: optimizer issues
• 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
• 15:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
• 15:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
• 15:50 moritzm: installing libhtml-parser-perl security updates
• 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
• 15:40 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool mw-web-ro in eqiad: maintenance
• 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:36 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004"
• 15:36 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cirrussearch1120 move vlan - ayounsi@cumin1004"
• 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route pool mw-web-ro in eqiad: maintenance
• 15:35 oblivian@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance
• 15:35 oblivian@cumin1004: START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance
• 15:27 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
• 15:22 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) depool all services in eqiad: Datacenter services switchover - T435443
• 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
• 15:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
• 15:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
• 15:16 bking@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch1120
• 15:16 bking@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch1120
• 15:14 bking@cumin2003: END (FAIL) - Cookbook sre.hosts.move-vlan (exit_code=99) for host cirrussearch1120
• 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
• 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
• 15:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
• 15:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
• 15:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
• 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
• 15:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
• 15:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
• 15:01 dancy@deploy1003: Installation of scap version "4.290.0" completed for 3 hosts
• 14:59 dancy@deploy1003: Installing scap version "4.290.0" for 3 host(s)
• 14:55 slyngshede@cumin1004: START - Cookbook sre.discovery.datacenter depool all services in eqiad: Datacenter services switchover - T435443
• 14:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
• 14:54 dancy@deploy1003: Installing scap version "4.290.0" for 155 host(s)
• 14:54 slyngshede@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqiad [reason: no reason specified, no task ID specified]
• 14:54 slyngshede@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqiad [reason: no reason specified, no task ID specified]
• 14:53 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120
• 14:51 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch1120.eqiad.wmnet with reason: migrate VLAN T436571
• 14:47 bking@cumin2003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cirrussearch1120
• 14:47 bking@cumin2003: START - Cookbook sre.hosts.move-vlan for host cirrussearch1120
• 14:42 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 14:42 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 14:36 cdobbins@cumin1004: END (FAIL) - Cookbook sre.hosts.rename (exit_code=93) from sretest2013 to cp2059
• 14:35 cdobbins@cumin1004: START - Cookbook sre.hosts.rename from sretest2013 to cp2059
• 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
• 14:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
• 14:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
• 14:17 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1003.eqiad.wmnet
• 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
• 14:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
• 14:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
• 13:59 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet
• 13:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
• 13:57 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker10[02-28].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 13:57 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
• 13:56 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
• 13:56 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
• 13:55 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
• 13:53 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host pki1002
• 13:51 elukey@cumin1004: START - Cookbook sre.hosts.powercycle for host pki1002
• 13:23 tappof@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'sync'.
• 13:22 tappof@deploy1003: helmfile [codfw] START helmfile.d/admin 'sync'.
• 13:21 tappof@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'sync'.
• 13:21 tappof@deploy1003: helmfile [eqiad] START helmfile.d/admin 'sync'.
• 12:53 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet
• 12:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet
• 12:44 marostegui: Stop mariadb on db2250:s5 T437411 T437279
• 12:43 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: preparations
• 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 12:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
• 12:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
• 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
• 12:19 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Hide recently saved revisions from the vandalism queue (T438235) (duration: 33m 01s)
• 12:17 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 12:16 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 12:16 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:15 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:08 kharlan@deploy1003: kharlan: Continuing with deployment
• 12:06 kharlan@deploy1003: kharlan: Backport for AbuseReview: Hide recently saved revisions from the vandalism queue (T438235) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 11:54 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): T435791 to production host (duration: 00m 54s)
• 11:54 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): T435791 to production host
• 11:52 jnuche@deploy1003: Finished deploy [releng/jenkins-deploy@ddb3f1a] (releasing): T435791 to backup host (duration: 01m 01s)
• 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
• 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 11:52 jnuche@deploy1003: Started deploy [releng/jenkins-deploy@ddb3f1a] (releasing): T435791 to backup host
• 11:46 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Hide recently saved revisions from the vandalism queue (T438235)
• 11:41 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Hide Echo banner when user cannot see personal info (T438477) (duration: 13m 46s)
• 11:34 kharlan@deploy1003: kharlan: Continuing with deployment
• 11:33 kharlan@deploy1003: kharlan: Backport for AbuseReview: Hide Echo banner when user cannot see personal info (T438477) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 11:27 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Hide Echo banner when user cannot see personal info (T438477)
• 11:24 kharlan@deploy1003: Finished scap sync-world: Backport for AbuseReview: Allow interaction with verdict buttons on closed rows (T438808) (duration: 33m 09s)
• 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
• 11:24 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 11:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 11:19 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 11:13 kharlan@deploy1003: kharlan: Continuing with deployment
• 11:12 kharlan@deploy1003: kharlan: Backport for AbuseReview: Allow interaction with verdict buttons on closed rows (T438808) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 10:54 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 10:53 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 10:52 topranks: enable rule cache-upload/eqsin_originals_scraper_20260922
• 10:51 kharlan@deploy1003: Started scap sync-world: Backport for AbuseReview: Allow interaction with verdict buttons on closed rows (T438808)
• 10:20 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2005.codfw.wmnet with OS trixie
• 10:13 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s1
• 10:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s1
• 10:10 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s4
• 10:10 gmodena@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 10:09 gmodena@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 10:09 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s4
• 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 10:09 gmodena@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 10:08 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s8
• 10:06 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s8
• 10:06 moritzm: installing libcap2 security updates
• 10:05 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s7
• 10:03 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s7
• 10:02 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage
• 10:02 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s3
• 10:01 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s3
• 10:00 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s2
• 09:58 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s2
• 09:58 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2005.codfw.wmnet with reason: host reimage
• 09:57 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s5
• 09:56 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
• 09:55 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s5
• 09:53 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section s6
• 09:51 elukey: install spicerack 13.3.0 on cumin1004 and cumin2003
• 09:50 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section s6
• 09:47 elukey: uploaded spicerack_13.3.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia
• 09:47 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es1035: issues
• 09:46 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool es1035: issues
• 09:44 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7
• 09:44 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828)
• 09:44 kevinbazira@deploy1003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 09:43 kevinbazira@deploy1003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 09:42 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7
• 09:41 kevinbazira@deploy1003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
• 09:41 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2005.codfw.wmnet with OS trixie
• 09:40 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es7
• 09:39 vgutierrez: fetch haproxy 3.2.23 on thirdparty/haproxy32 for trixie (apt.wm.o) - T438828
• 09:32 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es1035: issues
• 09:32 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad
• 09:32 marostegui@cumin1004: START - Cookbook sre.mysql.depool depool es1035: issues
• 09:31 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 8 hosts with reason: dc preparations
• 09:30 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 09:28 jelto@cumin1004: conftool action : set/pooled=inactive; selector: name=wikikube-worker1152.eqiad.wmnet
• 09:28 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad
• 09:26 btullis@dns1004: END - running authdns-update
• 09:24 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 09:23 btullis@dns1004: START - running authdns-update
• 09:23 jelto@cumin1004: conftool action : set/pooled=no; selector: name=wikikube-worker1152.eqiad.wmnet
• 09:20 jelto@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues
• 09:16 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es7
• 09:15 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section es6
• 09:14 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section es6
• 09:12 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x4
• 09:11 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x4
• 09:11 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x3
• 09:10 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 52320
• 09:09 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 52320
• 09:05 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x3
• 09:04 fceratto@cumin1004: END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section x1
• 09:02 fceratto@cumin1004: START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section x1
• 08:58 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
• 08:55 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
• 08:52 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
• 08:49 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
• 08:45 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: switch reboot, T437984]
• 08:45 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: switch reboot, T437984]
• 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt
• 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt
• 08:44 ayounsi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 13 hosts
• 08:44 ayounsi@cumin1004: START - Cookbook sre.hosts.remove-downtime for 13 hosts
• 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 08:39 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 08:37 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
• 08:32 moritzm: installig zip security updates
• 08:30 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad
• 08:29 XioNoX: asw1-bw27-esams> request system reboot - T437984
• 08:28 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 08:27 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for esams rack BW27
• 08:26 jelto@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad
• 08:26 ayounsi@cumin1004: START - Cookbook sre.network.depool-rack with action 'depool' for esams rack BW27
• 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 08:24 dcausse@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 08:22 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 08:18 moritzm: installing gst-plugins-base1.0 security updates
• 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
• 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 08:09 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 08:05 arthurtaylor@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
• 08:05 arthurtaylor@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
• 08:05 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
• 08:04 arthurtaylor@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
• 08:04 arthurtaylor@deploy1003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
• 08:01 arthurtaylor@deploy1003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
• 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 13 hosts with reason: Switch reboot
• 08:01 arthurtaylor@deploy1003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
• 08:01 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch reboot
• 07:59 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: switch reboot, T437984]
• 07:59 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: switch reboot, T437984]
• 07:23 awight: UTC morning deployment window complete
• 07:22 awight@deploy1003: Finished scap sync-world: Backport for Config change for launch of stopping sending LL notifications. (T438463), Change feedback URLs for EditCheck TextMatch on ruwiki (T426271) (duration: 17m 46s)
• 07:15 awight@deploy1003: seanleong-wmde, esanders, awight: Continuing with deployment
• 07:09 awight@deploy1003: seanleong-wmde, esanders, awight: Backport for Config change for launch of stopping sending LL notifications. (T438463), Change feedback URLs for EditCheck TextMatch on ruwiki (T426271) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 07:05 awight@deploy1003: Started scap sync-world: Backport for Config change for launch of stopping sending LL notifications. (T438463), Change feedback URLs for EditCheck TextMatch on ruwiki (T426271)
• 07:02 moritzm: installing pyasn1 security updates
• 04:02 mwpresync@deploy1003: Pruned MediaWiki: 1.47.0-wmf.18 (duration: 02m 28s)
• 03:39 mwpresync@deploy1003: Finished scap sync-world: testwikis to 1.47.0-wmf.21 refs T438217 (duration: 35m 52s)
• 03:03 mwpresync@deploy1003: Started scap sync-world: testwikis to 1.47.0-wmf.21 refs T438217
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 30s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

2026-09-21

• 22:11 rzl@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 22:10 rzl@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 22:09 rzl@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 22:08 rzl@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
• 22:08 rzl@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 22:07 rzl@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 22:06 rzl@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 22:05 rzl@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 21:18 maryum: Deployed security fix for T437708
• 20:35 cjming@deploy1003: Finished scap sync-world: Backport for Disable wgMFCustomSiteModules on German Wikipedia (T403380) (duration: 15m 56s)
• 20:30 cjming@deploy1003: ameisenigel, cjming: Continuing with deployment
• 20:26 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 20:25 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 20:25 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 20:25 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 20:23 cjming@deploy1003: ameisenigel, cjming: Backport for Disable wgMFCustomSiteModules on German Wikipedia (T403380) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 20:19 cjming@deploy1003: Started scap sync-world: Backport for Disable wgMFCustomSiteModules on German Wikipedia (T403380)
• 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
• 19:02 sfaci@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
• 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 18:59 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 18:35 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool sessionstore in eqiad: sessionstore1005 repaired
• 18:32 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir5003.*
• 18:30 Emperor: repool eqiad sessionstore T437915
• 18:30 mvernon@cumin1004: START - Cookbook sre.discovery.service-route pool sessionstore in eqiad: sessionstore1005 repaired
• 18:27 mvernon@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check sessionstore: maintenance
• 18:27 mvernon@cumin1004: START - Cookbook sre.discovery.service-route check sessionstore: maintenance
• 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 18:25 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 18:24 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5003.eqsin.wmnet with OS trixie
• 17:54 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage
• 17:50 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5003.eqsin.wmnet with reason: host reimage
• 17:40 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sessionstore1005.eqiad.wmnet with OS bookworm
• 17:30 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:29 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
• 17:26 jclark@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sessionstore1005.eqiad.wmnet with reason: host reimage
• 17:12 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 17:00 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host sessionstore1005.eqiad.wmnet with OS bookworm
• 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 16:56 ebernhardson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply
• 16:54 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir5003.eqsin.wmnet with OS trixie
• 16:46 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir6002.*
• 16:44 jclark@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:44 tappof@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging1003.eqiad.wmnet with reason: migrating to kafka-logging1006
• 16:36 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6002.drmrs.wmnet with OS trixie
• 16:32 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:27 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:27 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:23 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:22 jclark@cumin1004: START - Cookbook sre.hosts.provision for host sessionstore1005.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 16:16 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:15 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004"
• 16:15 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004"
• 16:13 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage
• 16:10 cmooney@cumin1004: START - Cookbook sre.dns.netbox
• 16:09 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6002.drmrs.wmnet with reason: host reimage
• 16:01 cklimas@deploy1003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
• 16:00 cklimas@deploy1003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
• 16:00 cklimas@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
• 16:00 cklimas@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
• 16:00 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host registry2004.codfw.wmnet with OS trixie
• 15:55 cklimas@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
• 15:54 cklimas@deploy1003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
• 15:49 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 15:45 jdlrobson@deploy1003: Finished scap sync-world: Backport for Fixes: '.action_context' should be string (T437122) (duration: 12m 40s)
• 15:42 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage
• 15:39 jdlrobson@deploy1003: jdlrobson: Continuing with deployment
• 15:39 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir6002.drmrs.wmnet with OS trixie
• 15:38 elukey@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on registry2004.codfw.wmnet with reason: host reimage
• 15:36 jdlrobson@deploy1003: jdlrobson: Backport for Fixes: '.action_context' should be string (T437122) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:33 slyngshede@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
• 15:32 slyngshede@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
• 15:32 jdlrobson@deploy1003: Started scap sync-world: Backport for Fixes: '.action_context' should be string (T437122)
• 15:19 elukey@puppetserver1001: conftool action : set/pooled=false; selector: name=registry2004.*
• 15:18 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host registry2004.codfw.wmnet with OS trixie
• 15:16 cdobbins@cumin1004: conftool action : set/pooled=yes; selector: name=ncredir3006.*
• 15:11 fceratto@deploy1003: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 15:07 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
• 15:07 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply
• 15:03 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3006.esams.wmnet with OS trixie
• 15:01 slyngshede@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
• 15:01 slyngshede@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
• 14:47 elukey@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-serve1016.eqiad.wmnet with OS trixie
• 14:39 cdobbins@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage
• 14:36 elukey@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
• 14:34 cdobbins@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3006.esams.wmnet with reason: host reimage
• 14:26 elukey@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:20 elukey@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:13 zabe@deploy1003: Finished scap sync-world: Backport for Move wbc_entity_usage to x1 for mediawikiwiki (T438716), Set db explicitly to false for virtual-wikibase-entityusage (duration: 08m 09s)
• 14:08 zabe@deploy1003: zabe: Continuing with deployment
• 14:08 zabe@deploy1003: zabe: Backport for Move wbc_entity_usage to x1 for mediawikiwiki (T438716), Set db explicitly to false for virtual-wikibase-entityusage synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:07 cdobbins@cumin1004: START - Cookbook sre.hosts.reimage for host ncredir3006.esams.wmnet with OS trixie
• 14:05 zabe@deploy1003: Started scap sync-world: Backport for Move wbc_entity_usage to x1 for mediawikiwiki (T438716), Set db explicitly to false for virtual-wikibase-entityusage
• 14:01 zabe@deploy1003: Started scap sync-world: Backport for Move wbc_entity_usage to x1 for mediawikiwiki (T438716), Set db explicitly to false for virtual-wikibase-entityusage
• 13:55 dreamyjazz@deploy1003: Finished scap sync-world: Backport for nlwiki: enable SecurePoll local elections (T434045) (duration: 12m 30s)
• 13:51 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Continuing with deployment
• 13:47 dreamyjazz@deploy1003: dreamyjazz, novemlinguae: Backport for nlwiki: enable SecurePoll local elections (T434045) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004"
• 13:45 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add entries for new eqiad links - cmooney@cumin1004"
• 13:43 zabe: reconcile wbc_entity_usage from local cluster to x1 for mediawikiwiki # T438716
• 13:43 dreamyjazz@deploy1003: Started scap sync-world: Backport for nlwiki: enable SecurePoll local elections (T434045)
• 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-pageview: apply
• 13:41 javiermonton@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-pageview: apply
• 13:41 cmooney@cumin1004: START - Cookbook sre.dns.netbox
• 13:40 samtar@deploy1003: Finished scap sync-world: Backport for arywiki: Create patroller and autopatrolled user groups (T438421) (duration: 11m 40s)
• 13:36 samtar@deploy1003: samtar, tryvix1509: Continuing with deployment
• 13:33 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 13:33 samtar@deploy1003: samtar, tryvix1509: Backport for arywiki: Create patroller and autopatrolled user groups (T438421) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:29 samtar@deploy1003: Started scap sync-world: Backport for arywiki: Create patroller and autopatrolled user groups (T438421)
• 13:22 mfossati@deploy1003: Finished scap sync-world: Backport for Let AA measure eligible readers w/o beta opt-in (T437076) (duration: 14m 19s)
• 13:15 mfossati@deploy1003: mfossati: Continuing with deployment
• 13:14 mfossati@deploy1003: mfossati: Backport for Let AA measure eligible readers w/o beta opt-in (T437076) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:10 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1063.eqiad.wmnet
• 13:07 mfossati@deploy1003: Started scap sync-world: Backport for Let AA measure eligible readers w/o beta opt-in (T437076)
• 13:01 brouberol@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM archiva1002.wikimedia.org
• 12:59 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1063.eqiad.wmnet
• 12:57 brouberol@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM archiva1002.wikimedia.org
• 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 12:54 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-serve1016.eqiad.wmnet with OS trixie
• 12:54 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
• 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
• 12:53 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 12:51 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:48 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 12:48 btullis@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 12:36 XioNoX: delete BGP sessions to 15305 in Equinix Ashburn (peer leaving the IX)
• 12:30 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
• 12:29 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
• 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
• 12:28 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 12:27 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 12:23 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
• 12:05 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2005.codfw.wmnet
• 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 11:45 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
• 11:45 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2005.codfw.wmnet
• 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
• 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
• 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
• 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
• 11:44 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
• 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
• 11:43 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
• 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
• 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
• 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
• 11:42 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
• 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
• 11:41 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
• 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
• 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
• 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
• 11:40 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
• 11:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2004.codfw.wmnet
• 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
• 11:39 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
• 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
• 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
• 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
• 11:38 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
• 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
• 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
• 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
• 11:37 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
• 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
• 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
• 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
• 11:35 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
• 11:34 jclark@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1016.eqiad.wmnet with OS trixie
• 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
• 11:34 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
• 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
• 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
• 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
• 11:33 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
• 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
• 11:32 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
• 11:31 jclark@cumin1004: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 11:31 jclark@cumin1004: START - Cookbook sre.hosts.provision for host ml-serve1016.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 11:31 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
• 11:13 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2004.codfw.wmnet
• 11:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2003.codfw.wmnet
• 11:04 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' 'Martin Urbanec' --reason 'per request T438704'
• 10:59 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2003.codfw.wmnet
• 10:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2002.codfw.wmnet
• 10:50 urbanecm@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request T438704'
• 10:38 zabe: create wbc_entity_usage table in x1 for all wikidata client wikis # T438499
• 10:36 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2002.codfw.wmnet
• 10:36 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2001.codfw.wmnet
• 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request T438704'
• 10:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2001.codfw.wmnet
• 10:21 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request T438704'
• 10:20 zabe@deploy1003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki metawiki 'Wikimedia Apps/Team/Android/Customizable Donation Reminder Experiment' 'Wikimedia Apps/Team/Customizable Donation Reminder/Android' Zabe --reason 'per request T438704'
• 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad
• 10:17 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 10:16 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
• 10:12 jmm@deploy1003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
• 10:11 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad
• 10:09 jmm@deploy1003: helmfile [eqiad] START helmfile.d/services/proton: apply
• 10:04 jmm@deploy1003: helmfile [codfw] DONE helmfile.d/services/proton: apply
• 10:02 jmm@deploy1003: helmfile [codfw] START helmfile.d/services/proton: apply
• 10:01 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
• 10:00 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
• 10:00 jmm@deploy1003: helmfile [staging] DONE helmfile.d/services/proton: apply
• 09:59 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1063.eqiad.wmnet with OS trixie
• 09:59 jmm@deploy1003: helmfile [staging] START helmfile.d/services/proton: apply
• 09:56 klausman@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply
• 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw
• 09:55 jelto@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 09:54 jelto@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
• 09:54 klausman@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply
• 09:50 jelto@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw
• 09:35 moritzm: installing chromium security updates
• 09:22 tappof: bump space for prometheus k8s-dse in eqiad
• 09:11 ihurbain@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 09:07 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage
• 09:04 ihurbain@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 09:04 ihurbain@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 09:01 urbanecm@deploy1003: Finished scap sync-world: Backport for [Growth] Remove unused config variables (T392944) (duration: 32m 54s)
• 09:01 filippo@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1063.eqiad.wmnet with reason: host reimage
• 08:58 ihurbain@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 08:45 filippo@cumin1004: START - Cookbook sre.hosts.reimage for host cloudvirt1063.eqiad.wmnet with OS trixie
• 08:29 urbanecm@deploy1003: Started scap sync-world: Backport for [Growth] Remove unused config variables (T392944)
• 08:15 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:05 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1063.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 08:04 filippo@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1063.eqiad.wmnet with reason: provision
• 08:01 XioNoX: restart gnmic on all netflow servers except 2005 and 1004 to pickup the new version - T438291
• 07:59 XioNoX: install gnmic 0.49 on all netflow hosts - T438291
• 07:57 XioNoX: add gnmic 0.49 to trixie-wikimedia - T438291
• 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5a-codfw
• 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5a-codfw
• 07:53 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device fasw1-f5b-codfw
• 07:53 ayounsi@cumin1004: START - Cookbook sre.network.tls for network device fasw1-f5b-codfw
• 07:45 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 07:39 filippo@cumin1004: START - Cookbook sre.hosts.provision for host cloudvirt1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
• 07:37 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1077.eqiad.wmnet
• 07:23 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirt1077.eqiad.wmnet
• 07:13 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 07:12 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
• 07:11 brouberol@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 07:10 brouberol@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 07:00 jmm@cumin2003: DONE (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for krb1002.eqiad.wmnet: Renew puppet certificate - jmm@cumin2003
• 05:24 moritzm: upgrade docker-report on build2004 to 0.0.20 T435314
• 05:14 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org

2026-09-20

• 20:08 dani@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 20:08 dani@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 20:08 dani@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 20:08 dani@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 20:08 dani@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 20:07 dani@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply

2026-09-19

• 16:55 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 16:55 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 16:55 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 16:55 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 14:11 urbanecm: Attach SHB@commonswiki to the SUL account manually (T438591, see T438591#12341750 for what I did exactly)
• 04:08 ssastry@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
• 04:08 ssastry@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
• 04:08 ssastry@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
• 04:07 ssastry@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
• 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 36s)
• 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image

Other archives

See Server Admin Log/Archives.