Server Admin Log

2026-01-24

• 00:06 hnowlan: setting batphone for SRE escalations

2026-01-23

• 23:15 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2165 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87898 and previous config saved to /var/cache/conftool/dbconfig/20260123-231537-marostegui.json
• 23:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
• 23:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87897 and previous config saved to /var/cache/conftool/dbconfig/20260123-231511-marostegui.json
• 23:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P87895 and previous config saved to /var/cache/conftool/dbconfig/20260123-230502-marostegui.json
• 22:54 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P87894 and previous config saved to /var/cache/conftool/dbconfig/20260123-225454-marostegui.json
• 22:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87893 and previous config saved to /var/cache/conftool/dbconfig/20260123-224446-marostegui.json
• 20:31 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T410589)', diff saved to https://phabricator.wikimedia.org/P87892 and previous config saved to /var/cache/conftool/dbconfig/20260123-203134-ladsgroup.json
• 20:21 jforrester@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: sync
• 20:21 jforrester@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: sync
• 20:21 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P87891 and previous config saved to /var/cache/conftool/dbconfig/20260123-202126-ladsgroup.json
• 20:11 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P87890 and previous config saved to /var/cache/conftool/dbconfig/20260123-201118-ladsgroup.json
• 20:01 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T410589)', diff saved to https://phabricator.wikimedia.org/P87889 and previous config saved to /var/cache/conftool/dbconfig/20260123-200109-ladsgroup.json
• 19:01 inflatador: bking@clouddumps1002 remove non-puppet-managed file `/srv/dumps/xmldatadumps/public/dvd.html`
• 18:31 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 18:31 xcollazo@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 18:23 cgoubert@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 18:21 cgoubert@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 18:20 cgoubert@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 18:19 cgoubert@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 17:29 kamila@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 17:28 kamila@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 17:28 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host tools-k8s-worker1002.eqiad.wmnet with OS trixie
• 17:28 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 17:28 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 17:18 greg-g: test where does this go?
• 17:16 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host tools-k8s-worker1003.eqiad.wmnet with OS trixie
• 17:16 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 17:15 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 17:14 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on tools-k8s-worker1002.eqiad.wmnet with reason: host reimage
• 17:08 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on tools-k8s-worker1002.eqiad.wmnet with reason: host reimage
• 17:01 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
• 16:59 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on tools-k8s-worker1003.eqiad.wmnet with reason: host reimage
• 16:57 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host tools-k8s-worker1002.eqiad.wmnet with OS trixie
• 16:56 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
• 16:56 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 16:55 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on tools-k8s-worker1003.eqiad.wmnet with reason: host reimage
• 16:50 dancy@deploy2002: Installation of scap version "4.235.0" completed for 2 hosts
• 16:49 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 16:48 dancy@deploy2002: Installing scap version "4.235.0" for 2 host(s)
• 16:43 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 16:43 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host tools-k8s-worker1003.eqiad.wmnet with OS trixie
• 16:41 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 16:39 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 16:38 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 16:31 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 16:31 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 16:30 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:30 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt tools-k8 - jclark@cumin1003"
• 16:30 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt tools-k8 - jclark@cumin1003"
• 16:25 jclark@cumin1003: START - Cookbook sre.dns.netbox
• 16:25 jclark@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 16:23 jclark@cumin1003: START - Cookbook sre.dns.netbox
• 16:21 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
• 16:21 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:20 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt tools-k8 - jclark@cumin1003"
• 16:20 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt tools-k8 - jclark@cumin1003"
• 16:20 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/services/thumbor: apply
• 16:20 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
• 16:20 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/services/thumbor: apply
• 16:19 daniel@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 16:18 daniel@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 16:17 jclark@cumin1003: START - Cookbook sre.dns.netbox
• 16:05 vgutierrez@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on cp5022.eqsin.wmnet with reason: cp5022 is unreacheable
• 15:25 jgiannelos@deploy2002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
• 15:24 jgiannelos@deploy2002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 15:24 jgiannelos@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
• 15:23 jgiannelos@deploy2002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
• 15:23 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 15:22 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 14:42 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host tools-k8s-ctrl1002.eqiad.wmnet with OS trixie
• 14:42 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:35 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:28 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host tools-k8s-ctrl1001.eqiad.wmnet with OS trixie
• 14:28 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:27 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:27 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:26 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host tools-k8s-worker1001.eqiad.wmnet with OS trixie
• 14:26 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:24 jclark@cumin1003: START - Cookbook sre.dns.netbox
• 14:23 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:22 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host tools-k8s-worker1004.eqiad.wmnet with OS trixie
• 14:22 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:21 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:20 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:20 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:19 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:19 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:19 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on tools-k8s-ctrl1002.eqiad.wmnet with reason: host reimage
• 14:11 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on tools-k8s-ctrl1001.eqiad.wmnet with reason: host reimage
• 14:11 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:10 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:10 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:10 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:09 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:08 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host tools-k8s-worker1002.eqiad.wmnet with OS trixie
• 14:07 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:07 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on tools-k8s-worker1001.eqiad.wmnet with reason: host reimage
• 14:04 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on tools-k8s-worker1004.eqiad.wmnet with reason: host reimage
• 14:00 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on tools-k8s-ctrl1002.eqiad.wmnet with reason: host reimage
• 14:00 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on tools-k8s-ctrl1001.eqiad.wmnet with reason: host reimage
• 13:59 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on tools-k8s-worker1001.eqiad.wmnet with reason: host reimage
• 13:59 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on tools-k8s-worker1004.eqiad.wmnet with reason: host reimage
• 13:55 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:51 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host tools-k8s-worker1002.eqiad.wmnet with OS trixie
• 13:49 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:48 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host tools-k8s-ctrl1002.eqiad.wmnet with OS trixie
• 13:48 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host tools-k8s-worker1004.eqiad.wmnet with OS trixie
• 13:48 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host tools-k8s-ctrl1001.eqiad.wmnet with OS trixie
• 13:48 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host tools-k8s-worker1001.eqiad.wmnet with OS trixie
• 13:44 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:44 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host tools-k8s-worker1003
• 13:44 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:44 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host tools-k8s-worker1003
• 13:42 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-worker1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:41 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-worker1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:40 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:40 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-ctrl1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:40 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host tools-k8s-ctrl1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:34 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1004.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:33 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:33 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:33 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-worker1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:32 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-ctrl1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:32 jclark@cumin1003: START - Cookbook sre.hosts.provision for host tools-k8s-ctrl1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 13:29 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:29 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt tools-k8 - jclark@cumin1003"
• 13:29 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt tools-k8 - jclark@cumin1003"
• 13:26 jclark@cumin1003: START - Cookbook sre.dns.netbox
• 12:55 cgoubert@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 12:53 cgoubert@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 12:32 moritzm: uploaded dnsmasq 2.92-1~wmf12u to bookworm-wikimedia/main T396864
• 12:18 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Update
• 11:07 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Update
• 10:07 jgiannelos@deploy2002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
• 10:07 jgiannelos@deploy2002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 10:06 jgiannelos@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
• 10:06 jgiannelos@deploy2002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
• 10:05 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 10:05 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 10:03 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet
• 09:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet
• 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet
• 09:46 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet
• 09:07 moritzm: installing Linux 6.1.159 on Bookworm hosts
• 08:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 08:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87887 and previous config saved to /var/cache/conftool/dbconfig/20260123-081240-marostegui.json
• 08:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P87886 and previous config saved to /var/cache/conftool/dbconfig/20260123-080232-marostegui.json
• 07:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P87885 and previous config saved to /var/cache/conftool/dbconfig/20260123-075223-marostegui.json
• 07:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87884 and previous config saved to /var/cache/conftool/dbconfig/20260123-074215-marostegui.json
• 06:29 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2238 (T410589)', diff saved to https://phabricator.wikimedia.org/P87883 and previous config saved to /var/cache/conftool/dbconfig/20260123-062948-ladsgroup.json
• 06:29 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
• 06:29 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T410589)', diff saved to https://phabricator.wikimedia.org/P87882 and previous config saved to /var/cache/conftool/dbconfig/20260123-062924-ladsgroup.json
• 06:26 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy2008.codfw.wmnet with OS trixie
• 06:19 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P87881 and previous config saved to /var/cache/conftool/dbconfig/20260123-061915-ladsgroup.json
• 06:09 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P87880 and previous config saved to /var/cache/conftool/dbconfig/20260123-060908-ladsgroup.json
• 06:03 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy2008.codfw.wmnet with reason: host reimage
• 05:59 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T410589)', diff saved to https://phabricator.wikimedia.org/P87879 and previous config saved to /var/cache/conftool/dbconfig/20260123-055859-ladsgroup.json
• 05:57 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy2008.codfw.wmnet with reason: host reimage
• 05:57 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on dbstore1009.eqiad.wmnet with reason: long schema change
• 05:42 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host dbproxy2008.codfw.wmnet with OS trixie
• 03:50 eileen: civicrm upgraded from 9e754954 to f7064a46
• 02:50 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1177 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87878 and previous config saved to /var/cache/conftool/dbconfig/20260123-025018-marostegui.json
• 02:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 02:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87877 and previous config saved to /var/cache/conftool/dbconfig/20260123-025005-marostegui.json
• 02:39 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P87876 and previous config saved to /var/cache/conftool/dbconfig/20260123-023957-marostegui.json
• 02:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P87875 and previous config saved to /var/cache/conftool/dbconfig/20260123-022948-marostegui.json
• 02:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87874 and previous config saved to /var/cache/conftool/dbconfig/20260123-021940-marostegui.json
• 02:04 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2164 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87873 and previous config saved to /var/cache/conftool/dbconfig/20260123-020453-marostegui.json
• 02:04 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
• 02:04 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87872 and previous config saved to /var/cache/conftool/dbconfig/20260123-020427-marostegui.json
• 01:54 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P87871 and previous config saved to /var/cache/conftool/dbconfig/20260123-015419-marostegui.json
• 01:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P87870 and previous config saved to /var/cache/conftool/dbconfig/20260123-014411-marostegui.json
• 01:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87869 and previous config saved to /var/cache/conftool/dbconfig/20260123-013402-marostegui.json
• 01:32 zabe@deploy2002: Finished scap sync-world: Backport for Start reading from il_target_id from s5 and s8 wikis (T413669) (duration: 07m 16s)
• 01:28 zabe@deploy2002: zabe: Continuing with sync
• 01:27 zabe@deploy2002: zabe: Backport for Start reading from il_target_id from s5 and s8 wikis (T413669) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 01:25 zabe@deploy2002: Started scap sync-world: Backport for Start reading from il_target_id from s5 and s8 wikis (T413669)
• 01:01 cjming@deploy2002: Finished scap sync-world: Backport for Remove problematic logging for now (T415309) (duration: 06m 27s)
• 00:56 cjming@deploy2002: cjming: Continuing with sync
• 00:56 cjming@deploy2002: cjming: Backport for Remove problematic logging for now (T415309) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 00:54 cjming@deploy2002: Started scap sync-world: Backport for Remove problematic logging for now (T415309)
• 00:47 cjming@deploy2002: Finished scap sync-world: Backport for Remove problematic logging for now (T415309) (duration: 07m 12s)
• 00:43 cjming@deploy2002: cjming: Continuing with sync
• 00:42 cjming@deploy2002: cjming: Backport for Remove problematic logging for now (T415309) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 00:40 cjming@deploy2002: Started scap sync-world: Backport for Remove problematic logging for now (T415309)
• 00:20 jasmine@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
• 00:19 jasmine@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
• 00:14 jasmine@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/sophroid: apply
• 00:13 jasmine@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/sophroid: apply
• 00:01 sukhe@cumin1003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on P{ms-fe2009*} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad)
• 00:01 sukhe@cumin1003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on P{ms-fe2009*} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad)

2026-01-22

• 23:56 jasmine@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/sophroid: apply
• 23:48 cjming@deploy2002: Finished scap sync-world: Backport for Remove problematic logging for now (T415309) (duration: 06m 27s)
• 23:45 jasmine@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/sophroid: apply
• 23:44 cjming@deploy2002: cjming: Continuing with sync
• 23:44 cjming@deploy2002: cjming: Backport for Remove problematic logging for now (T415309) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 23:42 cjming@deploy2002: Started scap sync-world: Backport for Remove problematic logging for now (T415309)
• 23:36 jasmine@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/sophroid: apply
• 23:29 jasmine@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/sophroid: apply
• 23:28 jasmine@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/sophroid: apply
• 23:27 jasmine@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/sophroid: apply
• 23:16 cjming@deploy2002: Finished scap sync-world: Backport for Remove problematic logging for now (T415309) (duration: 06m 29s)
• 23:12 cjming@deploy2002: cjming: Continuing with sync
• 23:11 cjming@deploy2002: cjming: Backport for Remove problematic logging for now (T415309) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 23:10 jasmine@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/sophroid: apply
• 23:09 cjming@deploy2002: Started scap sync-world: Backport for Remove problematic logging for now (T415309)
• 23:03 larssandergreen: civicrm upgraded from 24bca0d2 to 9e75495
• 23:02 jasmine@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/sophroid: apply
• 23:01 dreamyjazz@deploy2002: Finished scap sync-world: Backport for CheckUser: Read new for user agent table migration on group0 (T361199) (duration: 09m 39s)
• 22:57 dreamyjazz@deploy2002: dreamyjazz: Continuing with sync
• 22:56 jasmine@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 22:55 jasmine@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 22:54 jasmine@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 22:54 dreamyjazz@deploy2002: dreamyjazz: Backport for CheckUser: Read new for user agent table migration on group0 (T361199) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:51 dreamyjazz@deploy2002: Started scap sync-world: Backport for CheckUser: Read new for user agent table migration on group0 (T361199)
• 22:51 jasmine@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
• 22:43 maryum: Deployed security fix for T406088
• 22:40 maryum: Deployed security fix for T411305
• 22:15 kemayo@deploy2002: Finished scap sync-world: Backport for Revert "Toggler: Update heading toggler to match WAI ARIA pattern" (T415303 T407908) (duration: 06m 43s)
• 22:14 kamila@deploy2002: helmfile [staging] DONE helmfile.d/services/failoid-ng: apply
• 22:12 kamila@deploy2002: helmfile [staging] START helmfile.d/services/failoid-ng: apply
• 22:11 kemayo@deploy2002: kemayo: Continuing with sync
• 22:10 kemayo@deploy2002: kemayo: Backport for Revert "Toggler: Update heading toggler to match WAI ARIA pattern" (T415303 T407908) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:08 kemayo@deploy2002: Started scap sync-world: Backport for Revert "Toggler: Update heading toggler to match WAI ARIA pattern" (T415303 T407908)
• 21:41 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2226 (T410589)', diff saved to https://phabricator.wikimedia.org/P87868 and previous config saved to /var/cache/conftool/dbconfig/20260122-214143-ladsgroup.json
• 21:41 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
• 21:41 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T410589)', diff saved to https://phabricator.wikimedia.org/P87867 and previous config saved to /var/cache/conftool/dbconfig/20260122-214119-ladsgroup.json
• 21:38 jsn@deploy2002: Finished scap sync-world: Backport for [itwiki] Change tagline for Wikipedia 25 (T414320), [hawiki] Add a temporary wordmark and tagline for Wikipedia 25 (T414736), [tgwiki] Add a temporary logo for Wikipedia 25 (T415307) (duration: 12m 18s)
• 21:33 jsn@deploy2002: superpes, jsn: Continuing with sync
• 21:31 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P87866 and previous config saved to /var/cache/conftool/dbconfig/20260122-213110-ladsgroup.json
• 21:27 jsn@deploy2002: superpes, jsn: Backport for [itwiki] Change tagline for Wikipedia 25 (T414320), [hawiki] Add a temporary wordmark and tagline for Wikipedia 25 (T414736), [tgwiki] Add a temporary logo for Wikipedia 25 (T415307) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:25 jsn@deploy2002: Started scap sync-world: Backport for [itwiki] Change tagline for Wikipedia 25 (T414320), [hawiki] Add a temporary wordmark and tagline for Wikipedia 25 (T414736), [tgwiki] Add a temporary logo for Wikipedia 25 (T415307)
• 21:21 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P87865 and previous config saved to /var/cache/conftool/dbconfig/20260122-212102-ladsgroup.json
• 21:18 jsn@deploy2002: Finished scap sync-world: Backport for When filtering for edits with high Revert Risk, Recent Changes shouldn't display edits from non-main namespaces (T404200) (duration: 13m 05s)
• 21:13 jsn@deploy2002: kgraessle, jsn: Continuing with sync
• 21:10 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T410589)', diff saved to https://phabricator.wikimedia.org/P87864 and previous config saved to /var/cache/conftool/dbconfig/20260122-211054-ladsgroup.json
• 21:06 jsn@deploy2002: kgraessle, jsn: Backport for When filtering for edits with high Revert Risk, Recent Changes shouldn't display edits from non-main namespaces (T404200) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:04 jsn@deploy2002: Started scap sync-world: Backport for When filtering for edits with high Revert Risk, Recent Changes shouldn't display edits from non-main namespaces (T404200)
• 20:44 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 20:42 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 18:59 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2356.codfw.wmnet with OS bookworm
• 18:59 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 18:36 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 18:18 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2356.codfw.wmnet with reason: host reimage
• 18:13 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1251 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87863 and previous config saved to /var/cache/conftool/dbconfig/20260122-181347-marostegui.json
• 18:13 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1251.eqiad.wmnet with reason: Maintenance
• 18:13 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 18:11 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2356.codfw.wmnet with reason: host reimage
• 18:09 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2355.codfw.wmnet with OS bookworm
• 18:09 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 18:07 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 18:00 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2356.codfw.wmnet with OS bookworm
• 17:49 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2355.codfw.wmnet with reason: host reimage
• 17:44 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2355.codfw.wmnet with reason: host reimage
• 17:20 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2355.codfw.wmnet with OS bookworm
• 17:17 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2354.codfw.wmnet with OS bookworm
• 17:17 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 17:17 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 17:17 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2356.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:59 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2354.codfw.wmnet with reason: host reimage
• 16:52 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2354.codfw.wmnet with reason: host reimage
• 16:40 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2354.codfw.wmnet with OS bookworm
• 16:36 ladsgroup@deploy2002: Finished scap sync-world: Backport for TimedMediaThumbnail: Set physical width and height (T402792) (duration: 09m 04s)
• 16:34 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2354.codfw.wmnet with OS bookworm
• 16:33 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2354.codfw.wmnet with reason: host reimage
• 16:32 ladsgroup@deploy2002: ladsgroup: Continuing with sync
• 16:29 ladsgroup@deploy2002: ladsgroup: Backport for TimedMediaThumbnail: Set physical width and height (T402792) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 16:29 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2354.codfw.wmnet with reason: host reimage
• 16:27 ladsgroup@deploy2002: Started scap sync-world: Backport for TimedMediaThumbnail: Set physical width and height (T402792)
• 16:18 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2356.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:17 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2354.codfw.wmnet with OS bookworm
• 16:00 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/failoid-ng: apply
• 15:59 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/failoid-ng: apply
• 15:52 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/failoid-ng: apply
• 15:52 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/failoid-ng: apply
• 15:47 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/failoid-ng: apply
• 15:46 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/failoid-ng: apply
• 15:46 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/failoid-ng: apply
• 15:46 jgiannelos@deploy2002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
• 15:46 jgiannelos@deploy2002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 15:37 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/failoid-ng: apply
• 15:37 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/failoid-ng: apply
• 15:36 jgiannelos@deploy2002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
• 15:36 jgiannelos@deploy2002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
• 15:35 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/failoid-ng: apply
• 15:35 jgiannelos@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
• 15:35 jgiannelos@deploy2002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
• 15:28 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/failoid-ng: apply
• 15:20 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/failoid-ng: apply
• 15:20 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/failoid-ng: apply
• 15:16 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/failoid-ng: apply
• 14:59 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 thumbsize (T406724)
• 14:57 bwojtowicz@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
• 14:21 ihurbain@deploy2002: Finished scap sync-world: Backport for Explicitly disable postprocessing cache for wikidata and commons (T415111) (duration: 10m 12s)
• 14:17 ihurbain@deploy2002: ihurbain: Continuing with sync
• 14:13 ihurbain@deploy2002: ihurbain: Backport for Explicitly disable postprocessing cache for wikidata and commons (T415111) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:11 ihurbain@deploy2002: Started scap sync-world: Backport for Explicitly disable postprocessing cache for wikidata and commons (T415111)
• 14:02 moritzm: installing krb5 security updates
• 14:02 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx-envoy (exit_code=0) rolling restart_daemons on A:wdqs-all
• 13:51 kamila@deploy2002: helmfile [staging] DONE helmfile.d/services/failoid-ng: apply
• 13:50 kamila@deploy2002: helmfile [staging] START helmfile.d/services/failoid-ng: apply
• 13:48 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx-envoy rolling restart_daemons on A:wdqs-all
• 13:46 jmm@cumin2002: END (PASS) - Cookbook sre.wdqs.restart-nginx-envoy (exit_code=0) rolling restart_daemons on A:wcqs-public
• 13:44 jmm@cumin2002: START - Cookbook sre.wdqs.restart-nginx-envoy rolling restart_daemons on A:wcqs-public
• 13:41 moritzm: installing libgd2 security updates
• 13:18 kamila@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
• 13:18 kamila@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
• 13:14 kamila@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply
• 13:13 kamila@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply
• 13:13 moritzm: installing e2fsprogs updates from Bookworm point release
• 12:31 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 12:30 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 12:30 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2077.codfw.wmnet with OS bullseye
• 12:29 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 12:28 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 12:27 cgoubert@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 12:26 cgoubert@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 12:25 cgoubert@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 12:23 cgoubert@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 12:22 kamila@deploy2002: Finished deploy [restbase/deploy@dcc15be]: Add kaiwiki, kajwiki & pplwiki - T414238, T415039, T415047 (duration: 16m 14s)
• 12:09 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2077.codfw.wmnet with reason: host reimage
• 12:06 kamila@deploy2002: Started deploy [restbase/deploy@dcc15be]: Add kaiwiki, kajwiki & pplwiki - T414238, T415039, T415047
• 12:05 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2077.codfw.wmnet with reason: host reimage
• 12:01 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab
• 12:00 Dreamy_Jazz: Run of script for T413868 has finished on s1
• 12:00 Dreamy_Jazz: Run of script for T413868 has finished on s4
• 11:56 blake@cumin1003: END (PASS) - Cookbook sre.switchdc.mediawiki.09-unlock-scap (exit_code=0) for datacenter switchover from codfw to eqiad
• 11:56 root@deploy2002: Forcefully removing global lock: Datacenter switchover from codfw to eqiad - T330996
• 11:56 blake@cumin1003: START - Cookbook sre.switchdc.mediawiki.09-unlock-scap for datacenter switchover from codfw to eqiad
• 11:55 Dreamy_Jazz: Run of script for T413868 has finished on s8
• 11:55 blake@cumin1003: END (PASS) - Cookbook sre.switchdc.mediawiki.09-unlock-scap (exit_code=0) for datacenter switchover from codfw to eqiad
• 11:55 root@deploy2002: Unlocked for deployment [ALL REPOSITORIES]: Datacenter switchover from codfw to eqiad - T330996 (duration: 00m 39s)
• 11:55 root@deploy2002: Forcefully removing global lock: Datacenter switchover from codfw to eqiad - T12345
• 11:54 blake@cumin1003: START - Cookbook sre.switchdc.mediawiki.09-unlock-scap for datacenter switchover from codfw to eqiad
• 11:54 blake@cumin1003: END (PASS) - Cookbook sre.switchdc.mediawiki.00-lock-scap (exit_code=0) for datacenter switchover from codfw to eqiad
• 11:54 root@deploy2002: Locking from deployment [ALL REPOSITORIES]: Datacenter switchover from codfw to eqiad - T330996
• 11:54 blake@cumin1003: START - Cookbook sre.switchdc.mediawiki.00-lock-scap for datacenter switchover from codfw to eqiad
• 11:52 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab
• 11:51 arnaudb@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab
• 11:47 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2077
• 11:47 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2077
• 11:47 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2077.codfw.wmnet with OS bullseye
• 11:46 mvernon@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2077.codfw.wmnet with OS bullseye
• 11:42 arnaudb@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab
• 11:32 daniel@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/redioscope: apply
• 11:31 daniel@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/redioscope: apply
• 11:28 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2077
• 11:28 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2077
• 11:28 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2077
• 11:28 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2077.codfw.wmnet 238.32.192.10.in-addr.arpa 8.3.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:28 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2077.codfw.wmnet 238.32.192.10.in-addr.arpa 8.3.2.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:28 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:28 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2077 - mvernon@cumin2002"
• 11:28 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2077 - mvernon@cumin2002"
• 11:22 mvernon@cumin2002: START - Cookbook sre.dns.netbox
• 11:21 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker1006.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 11:21 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2077
• 11:20 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2077.codfw.wmnet with OS bullseye
• 11:08 a-pizzata@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 11:08 a-pizzata@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich: apply
• 10:43 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker1006.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 10:35 moritzm: installing systemd bugfix updates from Bookworm point release
• 10:14 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy2007.codfw.wmnet with OS trixie
• 09:50 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy2007.codfw.wmnet with reason: host reimage
• 09:48 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 09:48 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 09:46 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy2007.codfw.wmnet with reason: host reimage
• 09:44 aklapper@deploy2002: rebuilt and synchronized wikiversions files: group2 to 1.46.0-wmf.12 refs T413803
• 09:31 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host dbproxy2007.codfw.wmnet with OS trixie
• 09:26 aklapper@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.12 refs T413803
• 09:18 aklapper@deploy2002: Finished scap sync-world: Backport for Revert "Fix DivisionByZeroError when calculating bitrate" (T415169) (duration: 07m 13s)
• 09:13 aklapper@deploy2002: jforrester, aklapper: Continuing with sync
• 09:13 aklapper@deploy2002: jforrester, aklapper: Backport for Revert "Fix DivisionByZeroError when calculating bitrate" (T415169) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 09:10 aklapper@deploy2002: Started scap sync-world: Backport for Revert "Fix DivisionByZeroError when calculating bitrate" (T415169)
• 08:06 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2225 (T410589)', diff saved to https://phabricator.wikimedia.org/P87861 and previous config saved to /var/cache/conftool/dbconfig/20260122-080653-ladsgroup.json
• 08:06 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
• 08:06 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T410589)', diff saved to https://phabricator.wikimedia.org/P87860 and previous config saved to /var/cache/conftool/dbconfig/20260122-080629-ladsgroup.json
• 07:56 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P87859 and previous config saved to /var/cache/conftool/dbconfig/20260122-075620-ladsgroup.json
• 07:46 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P87858 and previous config saved to /var/cache/conftool/dbconfig/20260122-074612-ladsgroup.json
• 07:36 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T410589)', diff saved to https://phabricator.wikimedia.org/P87857 and previous config saved to /var/cache/conftool/dbconfig/20260122-073604-ladsgroup.json
• 07:15 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy2006.codfw.wmnet with OS trixie
• 06:55 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy2006.codfw.wmnet with reason: host reimage
• 06:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 06:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87856 and previous config saved to /var/cache/conftool/dbconfig/20260122-065138-marostegui.json
• 06:49 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy2006.codfw.wmnet with reason: host reimage
• 06:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P87855 and previous config saved to /var/cache/conftool/dbconfig/20260122-064131-marostegui.json
• 06:33 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host dbproxy2006.codfw.wmnet with OS trixie
• 06:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P87854 and previous config saved to /var/cache/conftool/dbconfig/20260122-063122-marostegui.json
• 06:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87853 and previous config saved to /var/cache/conftool/dbconfig/20260122-062114-marostegui.json
• 06:06 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1240.eqiad.wmnet with reason: long schema change
• 06:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1239.eqiad.wmnet with reason: long schema change
• 04:20 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2163 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87852 and previous config saved to /var/cache/conftool/dbconfig/20260122-041956-marostegui.json
• 04:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
• 04:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87851 and previous config saved to /var/cache/conftool/dbconfig/20260122-041931-marostegui.json
• 04:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P87850 and previous config saved to /var/cache/conftool/dbconfig/20260122-040922-marostegui.json
• 03:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P87849 and previous config saved to /var/cache/conftool/dbconfig/20260122-035914-marostegui.json
• 03:49 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87848 and previous config saved to /var/cache/conftool/dbconfig/20260122-034905-marostegui.json
• 03:43 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1172 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87847 and previous config saved to /var/cache/conftool/dbconfig/20260122-034302-marostegui.json
• 03:42 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 01:58 larssandergreen: civicrm upgraded from 02a558a2 to 24bca0d2
• 01:47 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mwlog1003.eqiad.wmnet with OS bookworm
• 01:47 pt1979@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1003"
• 01:46 pt1979@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1003"
• 01:27 pt1979@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mwlog1003.eqiad.wmnet with reason: host reimage
• 01:23 pt1979@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on mwlog1003.eqiad.wmnet with reason: host reimage
• 01:07 zabe@deploy2002: Finished scap sync-world: Backport for Start reading from il_target_id on small wikis (T413669) (duration: 07m 12s)
• 01:02 zabe@deploy2002: zabe: Continuing with sync
• 01:02 zabe@deploy2002: zabe: Backport for Start reading from il_target_id on small wikis (T413669) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 00:59 zabe@deploy2002: Started scap sync-world: Backport for Start reading from il_target_id on small wikis (T413669)

2026-01-21

• 23:40 eileen: config revision changed from 5a1bd9b8 to 8d6fc61d
• 23:00 eileen: config revision changed from 485f8e69 to 5a1bd9b8
• 21:14 catrope@deploy2002: Finished scap sync-world: Backport for Enable OATHAuth passkey features in production (T415146), Don't directly serialize PublicKeyCredential objects, Don't directly serialize PublicKeyCredential objects (duration: 09m 42s)
• 21:10 catrope@deploy2002: catrope: Continuing with sync
• 21:06 catrope@deploy2002: catrope: Backport for Enable OATHAuth passkey features in production (T415146), Don't directly serialize PublicKeyCredential objects, Don't directly serialize PublicKeyCredential objects synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:04 catrope@deploy2002: Started scap sync-world: Backport for Enable OATHAuth passkey features in production (T415146), Don't directly serialize PublicKeyCredential objects, Don't directly serialize PublicKeyCredential objects
• 20:39 eileen: config revision changed from 8d56295b to 485f8e69
• 20:36 eileen: config revision changed from db79212a to 8d56295b - disable jobs so I can run settle queue heavily
• 20:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87846 and previous config saved to /var/cache/conftool/dbconfig/20260121-202951-marostegui.json
• 20:22 eileen: civicrm upgraded from d137ca87 to 02a558a2
• 20:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P87845 and previous config saved to /var/cache/conftool/dbconfig/20260121-201943-marostegui.json
• 20:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P87844 and previous config saved to /var/cache/conftool/dbconfig/20260121-200935-marostegui.json
• 20:06 btullis@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker[1009-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 19:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87843 and previous config saved to /var/cache/conftool/dbconfig/20260121-195927-marostegui.json
• 19:30 sukhe: re-enable puppet on A:dnsbox: T81605
• 19:28 sukhe@dns1004: END - running authdns-update
• 19:27 sukhe@dns1004: START - running authdns-update
• 19:26 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.wikimedia.org [reason: END testing authdns IPv6 change]
• 19:21 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dns7001.wikimedia.org
• 19:07 sukhe@cumin1003: START - Cookbook sre.hosts.reboot-single for host dns7001.wikimedia.org
• 19:07 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2356.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:03 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2355.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:03 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2356.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:02 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2355.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:02 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2354.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:01 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2354.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:00 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mwlog2003
• 19:00 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host mwlog2003
• 19:00 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2356
• 18:59 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2356
• 18:59 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2355
• 18:59 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2355
• 18:59 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2354
• 18:59 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2354
• 18:48 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:48 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: corrections to codfw - jhancock@cumin1003"
• 18:48 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: corrections to codfw - jhancock@cumin1003"
• 18:44 jhancock@cumin1003: START - Cookbook sre.dns.netbox
• 18:39 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:39 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: corrections to codfw - jhancock@cumin1003"
• 18:39 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: corrections to codfw - jhancock@cumin1003"
• 18:35 jhancock@cumin1003: START - Cookbook sre.dns.netbox
• 18:34 daniel@deploy2002: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
• 18:34 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:34 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: corrections to codfw - jhancock@cumin1003"
• 18:34 daniel@deploy2002: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
• 18:33 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: corrections to codfw - jhancock@cumin1003"
• 18:31 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2207 (T410589)', diff saved to https://phabricator.wikimedia.org/P87838 and previous config saved to /var/cache/conftool/dbconfig/20260121-183104-ladsgroup.json
• 18:30 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
• 18:23 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Update Homer plugin to enable IPv6 peering to authdns boxes - cmooney@cumin1003
• 18:23 daniel@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
• 18:22 jhancock@cumin1003: START - Cookbook sre.dns.netbox
• 18:22 daniel@deploy2002: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
• 18:22 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:21 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Update Homer plugin to enable IPv6 peering to authdns boxes - cmooney@cumin1003
• 18:19 jhancock@cumin1003: START - Cookbook sre.dns.netbox
• 18:18 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2076.codfw.wmnet with OS bullseye
• 18:11 aklapper@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.12 refs T413803
• 18:09 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2354.codfw.wmnet with OS bookworm
• 18:09 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 17:59 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2076.codfw.wmnet with reason: host reimage
• 17:56 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2076.codfw.wmnet with reason: host reimage
• 17:45 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2356.codfw.wmnet with OS bookworm
• 17:43 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.wikimedia.org [reason: testing authdns IPv6 change]
• 17:38 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2076
• 17:38 jhathaway@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2076
• 17:38 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2076.codfw.wmnet with OS bullseye
• 17:37 sukhe: sudo cumin "A:dnsbox" "disable-puppet 'merging CR 1226928'": T81605
• 17:34 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2074.codfw.wmnet with OS bullseye
• 17:27 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 17:16 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2074.codfw.wmnet with reason: host reimage
• 17:14 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1235 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87837 and previous config saved to /var/cache/conftool/dbconfig/20260121-171401-marostegui.json
• 17:13 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance
• 17:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87836 and previous config saved to /var/cache/conftool/dbconfig/20260121-171336-marostegui.json
• 17:11 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2074.codfw.wmnet with reason: host reimage
• 17:09 jhancock@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:09 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2354.codfw.wmnet with reason: host reimage
• 17:08 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host mwlog1003.eqiad.wmnet with OS bookworm
• 17:07 jhancock@cumin1003: START - Cookbook sre.dns.netbox
• 17:07 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:07 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update IP for mwlog1003 - pt1979@cumin2002"
• 17:06 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update IP for mwlog1003 - pt1979@cumin2002"
• 17:06 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2355.codfw.wmnet with OS bookworm
• 17:06 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 17:05 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2354.codfw.wmnet with reason: host reimage
• 17:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P87835 and previous config saved to /var/cache/conftool/dbconfig/20260121-170328-marostegui.json
• 17:01 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 16:59 pt1979@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host mwlog1003.eqiad.wmnet with OS bookworm
• 16:59 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 16:53 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2074
• 16:53 jhathaway@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2074
• 16:53 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2074.codfw.wmnet with OS bullseye
• 16:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P87834 and previous config saved to /var/cache/conftool/dbconfig/20260121-165319-marostegui.json
• 16:53 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2354.codfw.wmnet with OS bookworm
• 16:51 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker2354.codfw.wmnet with OS bookworm
• 16:44 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2355.codfw.wmnet with reason: host reimage
• 16:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87833 and previous config saved to /var/cache/conftool/dbconfig/20260121-164311-marostegui.json
• 16:40 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:40 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2354.codfw.wmnet with reason: host reimage
• 16:37 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 16:36 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2355.codfw.wmnet with reason: host reimage
• 16:36 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2354.codfw.wmnet with reason: host reimage
• 16:31 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:28 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 16:28 dancy@deploy2002: Installation of scap version "4.234.0" completed for 2 hosts
• 16:26 dancy@deploy2002: Installing scap version "4.234.0" for 2 host(s)
• 16:25 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2356.codfw.wmnet with OS bookworm
• 16:25 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2355.codfw.wmnet with OS bookworm
• 16:24 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2354.codfw.wmnet with OS bookworm
• 16:22 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host mwlog1003.eqiad.wmnet with OS bookworm
• 16:21 pt1979@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mwlog1003.eqiad.wmnet with OS bookworm
• 16:17 zabe: start populating il_target_id on s1, s2, s4, s7 and s8 wikis # T413668
• 16:01 moritzm: installing docker.io updates from Bookworm point release
• 15:59 cmooney@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Update Homer plugin to enable IPv6 peering to authdns boxes - cmooney@cumin1003
• 15:57 cmooney@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet,cumin1003.eqiad.wmnet with reason: Update Homer plugin to enable IPv6 peering to authdns boxes - cmooney@cumin1003
• 15:38 TheresNoTime: foreachwiki ... completed for T406843
• 15:33 TheresNoTime: started `[samtar@deploy2002 ~]$ foreachwiki sql.php /srv/mediawiki/php-1.46.0-wmf.12/sql/mysql/patch-watchlist_label.sql` for T406843
• 15:26 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker[1009-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 15:20 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1008.eqiad.wmnet
• 15:19 daniel@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 15:18 daniel@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 15:15 jforrester@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 15:15 jforrester@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 15:15 jforrester@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 15:14 jforrester@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 15:12 kamila@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
• 15:12 jforrester@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 15:12 jforrester@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 15:09 jforrester@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 15:08 jforrester@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 15:08 jforrester@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 15:07 jforrester@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 15:07 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1008.eqiad.wmnet
• 15:07 jforrester@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 15:06 jforrester@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 15:02 kamila@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
• 15:01 kamila@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
• 14:58 kamila@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
• 14:57 kamila@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
• 14:50 TheresNoTime: `[samtar@deploy2002 ~]$ mwscript sql.php --wiki=testwiki /srv/mediawiki/php-1.46.0-wmf.12/sql/mysql/patch-watchlist_label.sql` for T406843
• 14:47 kamila@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
• 14:47 kamila@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
• 14:47 kamila@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
• 14:45 daniel@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
• 14:42 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.reboot-nodes (exit_code=1) rolling reboot on P{dse-k8s-worker[1008-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 14:41 daniel@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
• 14:40 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker[1008-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 14:40 daniel@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
• 14:37 Dreamy_Jazz: Run of script for T413868 has finished on s7
• 14:37 Dreamy_Jazz: Run of script for T413868 has finished on s6
• 14:37 Dreamy_Jazz: Run of script for T413868 has finished on s5
• 14:34 daniel@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
• 14:33 daniel@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/redioscope: apply
• 14:33 daniel@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/redioscope: apply
• 14:33 Lucas_WMDE: UTC afternoon backport+config window done
• 14:31 jsn@deploy2002: Finished scap sync-world: Backport for Use page creation date as offset only if the legacy ordering is specified (T414892) (duration: 17m 24s)
• 14:27 jsn@deploy2002: jsn: Continuing with sync
• 14:21 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host mwlog1003.eqiad.wmnet with OS bookworm
• 14:16 jsn@deploy2002: jsn: Backport for Use page creation date as offset only if the legacy ordering is specified (T414892) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:13 jsn@deploy2002: Started scap sync-world: Backport for Use page creation date as offset only if the legacy ordering is specified (T414892)
• 14:08 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1032.eqiad.wmnet with OS trixie
• 14:02 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet
• 13:56 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet
• 13:55 jgiannelos@deploy2002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
• 13:50 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1032.eqiad.wmnet with reason: host reimage
• 13:48 moritzm: installing qemu security updates
• 13:47 bwojtowicz@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
• 13:46 jgiannelos@deploy2002: helmfile [staging] START helmfile.d/services/mobileapps: apply
• 13:45 Dreamy_Jazz: Running `foreachwikiindblist s8.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 3000 --sleep 2` for T413868
• 13:44 Dreamy_Jazz: Running `foreachwikiindblist s7.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 5000 --sleep 1` for T413868
• 13:44 bwojtowicz@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
• 13:43 Dreamy_Jazz: Running `foreachwikiindblist s6.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 5000 --sleep 1` for T413868
• 13:42 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1032.eqiad.wmnet with reason: host reimage
• 13:42 Dreamy_Jazz: Run of script for T413868 has finished on s3
• 13:41 Dreamy_Jazz: Run of script for T413868 has finished on s2
• 13:40 Dreamy_Jazz: Running `foreachwikiindblist s5.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 5000 --sleep 1` for T413868
• 13:25 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1032.eqiad.wmnet with OS trixie
• 13:22 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1032.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:22 Msz2001: Finished deploying for T414547
• 13:21 logmsgbot: mszwarc Deployed security patch for T414547
• 13:18 bwojtowicz@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
• 13:15 logmsgbot: mszwarc Deployed security patch for T414547
• 13:14 bwojtowicz@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
• 13:14 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1032.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:11 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 11 skin|thumbsize
• 13:08 Msz2001: Deploying mitigation for T414547
• 13:07 bwojtowicz@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
• 13:02 bwojtowicz@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
• 12:58 bwojtowicz@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
• 12:41 mvernon@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2076.codfw.wmnet with OS bullseye
• 12:41 mvernon@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ms-be2074.codfw.wmnet with OS bullseye
• 12:20 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1007.eqiad.wmnet with OS bookworm
• 12:18 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1004.eqiad.wmnet with OS bookworm
• 12:16 mvolz@deploy2002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
• 12:16 mvolz@deploy2002: helmfile [eqiad] START helmfile.d/services/citoid: apply
• 12:15 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1003.eqiad.wmnet with OS bookworm
• 12:15 mvolz@deploy2002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
• 12:15 mvolz@deploy2002: helmfile [codfw] START helmfile.d/services/citoid: apply
• 12:11 Dreamy_Jazz: Running `foreachwikiindblist s3.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 5000 --sleep 1` for T413868
• 12:10 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1005.eqiad.wmnet with OS bookworm
• 12:09 Dreamy_Jazz: Running `foreachwikiindblist s2.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 5000 --sleep 1` for T413868
• 12:04 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-druid1006.eqiad.wmnet with OS bookworm
• 12:04 Dreamy_Jazz: Running `foreachwikiindblist s1.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 5000 --sleep 2` for T413868
• 12:03 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage
• 12:03 Dreamy_Jazz: Running `foreachwikiindblist s1.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 10000 --sleep 5` for T413868
• 12:01 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/citoid: apply
• 12:01 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/citoid: apply
• 11:59 Dreamy_Jazz: Running `foreachwikiindblist s4.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 3000 --sleep 2` for T413868
• 11:59 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage
• 11:58 Dreamy_Jazz: Stopped run of script for T413868 on large.dblist
• 11:58 Dreamy_Jazz: Run of medium.dblist for T413868 has completed
• 11:58 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1007.eqiad.wmnet with reason: host reimage
• 11:55 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2076
• 11:55 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2076
• 11:55 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2076
• 11:55 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2076.codfw.wmnet 248.16.192.10.in-addr.arpa 8.4.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:55 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2076.codfw.wmnet 248.16.192.10.in-addr.arpa 8.4.2.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:55 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:55 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2076 - mvernon@cumin2002"
• 11:55 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2076 - mvernon@cumin2002"
• 11:55 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage
• 11:52 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1004.eqiad.wmnet with reason: host reimage
• 11:51 dreamyjazz@deploy2002: Finished scap sync-world: Backport for Remove unused LoginNotify config (T412939) (duration: 08m 39s)
• 11:51 mvernon@cumin2002: START - Cookbook sre.dns.netbox
• 11:51 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2076
• 11:50 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1003.eqiad.wmnet with reason: host reimage
• 11:50 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2076.codfw.wmnet with OS bullseye
• 11:49 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage
• 11:47 dreamyjazz@deploy2002: dreamyjazz, tstarling: Continuing with sync
• 11:46 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm
• 11:46 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-druid1007.eqiad.wmnet with OS bookworm
• 11:45 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1005.eqiad.wmnet with reason: host reimage
• 11:45 dreamyjazz@deploy2002: dreamyjazz, tstarling: Backport for Remove unused LoginNotify config (T412939) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 11:44 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage
• 11:44 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-be2074
• 11:44 mvernon@cumin2002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-be2074
• 11:43 dreamyjazz@deploy2002: Started scap sync-world: Backport for Remove unused LoginNotify config (T412939)
• 11:40 btullis@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-druid1006.eqiad.wmnet with reason: host reimage
• 11:34 mvernon@cumin2002: START - Cookbook sre.network.configure-switch-interfaces for host ms-be2074
• 11:34 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-be2074.codfw.wmnet 137.0.192.10.in-addr.arpa 7.3.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:34 mvernon@cumin2002: START - Cookbook sre.dns.wipe-cache ms-be2074.codfw.wmnet 137.0.192.10.in-addr.arpa 7.3.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors
• 11:34 mvernon@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 11:34 mvernon@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2074 - mvernon@cumin2002"
• 11:34 mvernon@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-be2074 - mvernon@cumin2002"
• 11:30 mvernon@cumin2002: START - Cookbook sre.dns.netbox
• 11:29 mvernon@cumin2002: START - Cookbook sre.hosts.move-vlan for host ms-be2074
• 11:29 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host ms-be2074.codfw.wmnet with OS bullseye
• 11:28 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-druid1007.eqiad.wmnet with OS bookworm
• 11:28 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-druid1006.eqiad.wmnet with OS bookworm
• 11:27 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-druid1005.eqiad.wmnet with OS bookworm
• 11:27 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-druid1004.eqiad.wmnet with OS bookworm
• 11:26 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm
• 11:14 moritzm: installing curl bugfix updates from Bookworm point release
• 10:49 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1032.eqiad.wmnet with OS trixie
• 10:43 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1032.eqiad.wmnet with OS trixie
• 10:34 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on dbstore1008.eqiad.wmnet with reason: long schema change
• 10:18 moritzm: installing setuptools security updates
• 10:13 elukey@deploy2002: helmfile [staging] DONE helmfile.d/services/kartotherian: sync
• 10:12 elukey@deploy2002: helmfile [staging] START helmfile.d/services/kartotherian: sync
• 09:09 aklapper@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.12 refs T413803
• 08:19 a-pizzata@deploy2002: Finished deploy [analytics/refinery@4f6560f] (thin): Regular analytics weekly train THIN [analytics/refinery@4f6560f9] (duration: 01m 16s)
• 08:18 a-pizzata@deploy2002: Started deploy [analytics/refinery@4f6560f] (thin): Regular analytics weekly train THIN [analytics/refinery@4f6560f9]
• 08:11 a-pizzata@deploy2002: Finished deploy [analytics/refinery@4f6560f]: Regular analytics weekly train [analytics/refinery@4f6560f9] (duration: 02m 43s)
• 08:08 a-pizzata@deploy2002: Started deploy [analytics/refinery@4f6560f]: Regular analytics weekly train [analytics/refinery@4f6560f9]
• 08:08 a-pizzata@deploy2002: Finished deploy [analytics/refinery@4f6560f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4f6560f9] (duration: 01m 05s)
• 08:07 a-pizzata@deploy2002: Started deploy [analytics/refinery@4f6560f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4f6560f9]
• 07:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2216 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87830 and previous config saved to /var/cache/conftool/dbconfig/20260121-072446-marostegui.json
• 07:24 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance
• 07:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2203 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87829 and previous config saved to /var/cache/conftool/dbconfig/20260121-072422-marostegui.json
• 07:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2203', diff saved to https://phabricator.wikimedia.org/P87828 and previous config saved to /var/cache/conftool/dbconfig/20260121-071414-marostegui.json
• 07:04 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2203', diff saved to https://phabricator.wikimedia.org/P87827 and previous config saved to /var/cache/conftool/dbconfig/20260121-070405-marostegui.json
• 06:54 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2203 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87826 and previous config saved to /var/cache/conftool/dbconfig/20260121-065357-marostegui.json
• 06:48 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 06:48 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T410589)', diff saved to https://phabricator.wikimedia.org/P87825 and previous config saved to /var/cache/conftool/dbconfig/20260121-064817-ladsgroup.json
• 06:47 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool db2179: After schema change
• 06:47 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool db1160: After schema change
• 06:44 samwilson@deploy2002: Finished scap sync-world: Backport for Revert "jquery.wikiEditor: enable resizing drag bar without RTP" (duration: 09m 37s)
• 06:40 samwilson@deploy2002: samwilson: Continuing with sync
• 06:38 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P87822 and previous config saved to /var/cache/conftool/dbconfig/20260121-063809-ladsgroup.json
• 06:37 samwilson@deploy2002: samwilson: Backport for Revert "jquery.wikiEditor: enable resizing drag bar without RTP" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 06:34 samwilson@deploy2002: Started scap sync-world: Backport for Revert "jquery.wikiEditor: enable resizing drag bar without RTP"
• 06:28 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P87819 and previous config saved to /var/cache/conftool/dbconfig/20260121-062801-ladsgroup.json
• 06:17 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T410589)', diff saved to https://phabricator.wikimedia.org/P87818 and previous config saved to /var/cache/conftool/dbconfig/20260121-061752-ladsgroup.json
• 06:14 eileen: civicrm upgraded from 9cdb1915 to d137ca87
• 06:01 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db2179: After schema change
• 06:01 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.newpool (exit_code=97) pool db2179: After schema change
• 06:01 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db1160: After schema change
• 06:01 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.newpool (exit_code=97) pool db1160: After schema change
• 06:01 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db2179: After schema change
• 06:00 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db1160: After schema change
• 05:57 marostegui@dns1006: END - running authdns-update
• 05:57 marostegui: Promote dbproxy1024 to m1-master T414656
• 05:56 marostegui@dns1006: START - running authdns-update
• 05:53 eileen: config revision changed from d75e49b5 to db79212a
• 05:45 eileen: config revision changed from 92dff26b to d75e49b5
• 05:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2154 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87813 and previous config saved to /var/cache/conftool/dbconfig/20260121-052432-marostegui.json
• 05:24 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
• 05:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87812 and previous config saved to /var/cache/conftool/dbconfig/20260121-052408-marostegui.json
• 05:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 05:18 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87811 and previous config saved to /var/cache/conftool/dbconfig/20260121-051854-marostegui.json
• 05:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P87810 and previous config saved to /var/cache/conftool/dbconfig/20260121-051359-marostegui.json
• 05:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P87809 and previous config saved to /var/cache/conftool/dbconfig/20260121-050845-marostegui.json
• 05:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P87808 and previous config saved to /var/cache/conftool/dbconfig/20260121-050351-marostegui.json
• 04:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P87807 and previous config saved to /var/cache/conftool/dbconfig/20260121-045837-marostegui.json
• 04:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87806 and previous config saved to /var/cache/conftool/dbconfig/20260121-045342-marostegui.json
• 04:48 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87805 and previous config saved to /var/cache/conftool/dbconfig/20260121-044828-marostegui.json
• 04:40 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1234 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87804 and previous config saved to /var/cache/conftool/dbconfig/20260121-044039-marostegui.json
• 04:40 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance
• 04:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87803 and previous config saved to /var/cache/conftool/dbconfig/20260121-044015-marostegui.json
• 04:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P87802 and previous config saved to /var/cache/conftool/dbconfig/20260121-043006-marostegui.json
• 04:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P87801 and previous config saved to /var/cache/conftool/dbconfig/20260121-041958-marostegui.json
• 04:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87800 and previous config saved to /var/cache/conftool/dbconfig/20260121-040949-marostegui.json
• 02:19 zabe@deploy2002: Finished scap sync-world: Backport for Removed dropped special page from disabled query pages (T414202) (duration: 07m 03s)
• 02:15 zabe@deploy2002: zabe: Continuing with sync
• 02:14 zabe@deploy2002: zabe: Backport for Removed dropped special page from disabled query pages (T414202) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 02:12 zabe@deploy2002: Started scap sync-world: Backport for Removed dropped special page from disabled query pages (T414202)
• 01:32 zabe@deploy2002: Finished scap sync-world: Backport for Start reading from il_target_id on cebwiki (T413669) (duration: 09m 18s)
• 01:28 zabe@deploy2002: zabe: Continuing with sync
• 01:25 zabe@deploy2002: zabe: Backport for Start reading from il_target_id on cebwiki (T413669) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 01:23 zabe@deploy2002: Started scap sync-world: Backport for Start reading from il_target_id on cebwiki (T413669)
• 01:14 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 13m 08s)
• 01:03 zabe: start populating il_target_id on s3 and s6 wikis # T413668
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

2026-01-20

• 23:34 zabe@deploy2002: Finished scap sync-world: Backport for Start writing il_target_id on commonswiki (T413526) (duration: 07m 11s)
• 23:30 zabe@deploy2002: zabe: Continuing with sync
• 23:29 zabe@deploy2002: zabe: Backport for Start writing il_target_id on commonswiki (T413526) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 23:27 zabe@deploy2002: Started scap sync-world: Backport for Start writing il_target_id on commonswiki (T413526)
• 22:34 tgr_: UTC late deploys done
• 22:32 tgr@deploy2002: Finished scap sync-world: Backport for Enable WikimediaCustomizations (T410515) (duration: 12m 03s)
• 22:28 tgr@deploy2002: tgr: Continuing with sync
• 22:26 tgr@deploy2002: tgr: Backport for Enable WikimediaCustomizations (T410515) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:20 tgr@deploy2002: Started scap sync-world: Backport for Enable WikimediaCustomizations (T410515)
• 22:14 tgr@deploy2002: Finished scap sync-world: Backport for Add WikimediaCustomizations to extension-list (T410515) (duration: 40m 32s)
• 22:00 tgr@deploy2002: tgr: Continuing with sync
• 21:58 tgr@deploy2002: tgr: Backport for Add WikimediaCustomizations to extension-list (T410515) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:33 tgr@deploy2002: Started scap sync-world: Backport for Add WikimediaCustomizations to extension-list (T410515)
• 21:19 ebernhardson@deploy2002: Finished scap sync-world: Backport for Repair deepcat matching against spaced titles (T414859), Repair deepcat matching against spaced titles (T414859) (duration: 12m 51s)
• 21:15 ebernhardson@deploy2002: ebernhardson: Continuing with sync
• 21:10 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2353.codfw.wmnet with OS bookworm
• 21:10 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:09 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:09 ebernhardson@deploy2002: ebernhardson: Backport for Repair deepcat matching against spaced titles (T414859), Repair deepcat matching against spaced titles (T414859) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:08 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2352.codfw.wmnet with OS bookworm
• 21:08 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:07 ebernhardson@deploy2002: Started scap sync-world: Backport for Repair deepcat matching against spaced titles (T414859), Repair deepcat matching against spaced titles (T414859)
• 21:06 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:03 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2351.codfw.wmnet with OS bookworm
• 21:03 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:02 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:52 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2353.codfw.wmnet with reason: host reimage
• 20:48 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2352.codfw.wmnet with reason: host reimage
• 20:44 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2351.codfw.wmnet with reason: host reimage
• 20:41 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2353.codfw.wmnet with reason: host reimage
• 20:41 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2352.codfw.wmnet with reason: host reimage
• 20:40 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2351.codfw.wmnet with reason: host reimage
• 20:30 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2353.codfw.wmnet with OS bookworm
• 20:30 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2352.codfw.wmnet with OS bookworm
• 20:29 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2351.codfw.wmnet with OS bookworm
• 20:16 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2350.codfw.wmnet with OS bookworm
• 20:15 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:15 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:13 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2349.codfw.wmnet with OS bookworm
• 20:12 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:12 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:08 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2345.codfw.wmnet with OS bookworm
• 20:08 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:07 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 19:58 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2350.codfw.wmnet with reason: host reimage
• 19:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2349.codfw.wmnet with reason: host reimage
• 19:49 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2345.codfw.wmnet with reason: host reimage
• 19:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2203 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87798 and previous config saved to /var/cache/conftool/dbconfig/20260120-194626-marostegui.json
• 19:46 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2203.codfw.wmnet with reason: Maintenance
• 19:43 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2350.codfw.wmnet with reason: host reimage
• 19:42 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2349.codfw.wmnet with reason: host reimage
• 19:42 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2345.codfw.wmnet with reason: host reimage
• 19:31 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2350.codfw.wmnet with OS bookworm
• 19:31 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2349.codfw.wmnet with OS bookworm
• 19:31 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2345.codfw.wmnet with OS bookworm
• 19:27 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2345.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:23 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.reboot-nodes (exit_code=1) rolling reboot on P{dse-k8s-worker[1007-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 19:20 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2345.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:10 jhathaway@dns1004: END - running authdns-update
• 19:09 jhathaway@dns1004: START - running authdns-update
• 19:06 andrew@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/services/mw-debug: apply
• 19:05 andrew@deploy2002: helmfile [staging-eqiad] START helmfile.d/services/mw-debug: apply
• 18:54 pt1979@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mwlog1003.eqiad.wmnet with OS bookworm
• 18:43 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker[1007-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 18:30 pt1979@cumin1003: START - Cookbook sre.hosts.reimage for host mwlog1003.eqiad.wmnet with OS bookworm
• 18:15 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.reboot-nodes (exit_code=1) rolling reboot on P{dse-k8s-worker[1006-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 18:15 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mwlog1003.eqiad.wmnet with OS bookworm
• 18:13 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker[1006-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 18:04 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host mwlog1003.eqiad.wmnet with OS bookworm
• 17:59 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mwlog1003.eqiad.wmnet with OS bookworm
• 17:39 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host mwlog1003.eqiad.wmnet with OS bookworm
• 17:26 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 17:25 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 17:25 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
• 17:24 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
• 17:24 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 17:24 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 17:23 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
• 17:23 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
• 17:22 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
• 17:22 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
• 17:21 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
• 17:21 hnowlan: rotated statograph API key
• 17:21 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox: apply
• 17:18 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2189 (T410589)', diff saved to https://phabricator.wikimedia.org/P87797 and previous config saved to /var/cache/conftool/dbconfig/20260120-171821-ladsgroup.json
• 17:18 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
• 17:18 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T410589)', diff saved to https://phabricator.wikimedia.org/P87796 and previous config saved to /var/cache/conftool/dbconfig/20260120-171757-ladsgroup.json
• 17:08 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 17:07 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P87795 and previous config saved to /var/cache/conftool/dbconfig/20260120-170748-ladsgroup.json
• 17:07 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 17:06 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
• 17:06 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
• 17:05 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 17:05 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 17:04 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
• 17:04 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
• 17:03 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
• 17:03 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
• 17:02 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
• 17:02 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox: apply
• 16:59 dancy@deploy2002: Installation of scap version "4.233.0" completed for 2 hosts
• 16:58 dancy@deploy2002: Installing scap version "4.233.0" for 2 host(s)
• 16:57 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P87794 and previous config saved to /var/cache/conftool/dbconfig/20260120-165740-ladsgroup.json
• 16:55 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
• 16:54 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-video: apply
• 16:54 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
• 16:53 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
• 16:53 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 16:53 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 16:52 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
• 16:52 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-media: apply
• 16:51 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
• 16:51 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
• 16:50 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox: apply
• 16:50 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox: apply
• 16:47 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T410589)', diff saved to https://phabricator.wikimedia.org/P87793 and previous config saved to /var/cache/conftool/dbconfig/20260120-164731-ladsgroup.json
• offsite: payments-wiki upgraded from 249105bf to 258fda1c
• 16:37 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1232 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87792 and previous config saved to /var/cache/conftool/dbconfig/20260120-163748-marostegui.json
• 16:37 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance
• 16:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87791 and previous config saved to /var/cache/conftool/dbconfig/20260120-163724-marostegui.json
• 16:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P87790 and previous config saved to /var/cache/conftool/dbconfig/20260120-162715-marostegui.json
• 16:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P87789 and previous config saved to /var/cache/conftool/dbconfig/20260120-161707-marostegui.json
• 16:10 blake@cumin1003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
• 16:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87788 and previous config saved to /var/cache/conftool/dbconfig/20260120-160659-marostegui.json
• 15:50 sukhe: sudo cumin -b31 "A:cp" "run-puppet-agent --enable 'merging CR 1059423'": T117618
• 15:47 sukhe: enable puppet on cp110[15].eqiad.wmnet: T117618
• 15:43 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.reboot-nodes (exit_code=1) rolling reboot on P{dse-k8s-worker[1006-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 15:41 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker[1006-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 15:38 sukhe: sudo cumin "A:cp" "disable-puppet 'merging CR 1059423'": T117618
• 15:32 inflatador: bking@deploy2002 renewing TLS certificates on opensearch-ipoid codfw (AKA deleting pods one-by-one)
• 15:30 sukhe@dns1004: END - running authdns-update
• 15:29 sukhe: ran authdns-update for CR 1226904 to match zone file TTLs with registrar for wikimedia.org/wikipedia.org NS/glue records
• 15:29 sukhe@dns1004: START - running authdns-update
• 15:28 sukhe@dns1004: START - running authdns-update
• 15:19 blake@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
• 15:17 blake@cumin1003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
• 15:00 zabe: running `migrateLinksTable.php --table imagelinks` on s5 wikis # T413668
• 14:54 zabe@deploy2002: Finished scap sync-world: Backport for Start writing to il_target_id everywhere except commons (T413526) (duration: 06m 39s)
• 14:50 zabe@deploy2002: zabe: Continuing with sync
• 14:49 zabe@deploy2002: zabe: Backport for Start writing to il_target_id everywhere except commons (T413526) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:47 zabe@deploy2002: Started scap sync-world: Backport for Start writing to il_target_id everywhere except commons (T413526)
• 14:27 blake@cumin1003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
• 14:21 gehel: switching off Blazegraph on wdqs2009 (legacy full graph endpoint is end of life) - T411410
• 14:16 btullis@cumin1003: END (FAIL) - Cookbook sre.k8s.reboot-nodes (exit_code=1) rolling reboot on P{dse-k8s-worker[1002-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 13:18 cgoubert@deploy2002: Unlocked for deployment [ALL REPOSITORIES]: Testing scap bg lock (duration: 00m 20s)
• 13:18 cgoubert@deploy2002: Forcefully removing global lock: Testing scap bg lock
• 13:18 cgoubert@deploy2002: Forcefully removing global lock: Testing scap bg lock
• 13:17 cgoubert@deploy2002: Locking from deployment [ALL REPOSITORIES]: Testing scap bg lock
• 13:10 root@deploy2002: Unlocked for deployment [ALL REPOSITORIES]: Testing scap bg lock (duration: 00m 47s)
• 13:10 cgoubert@deploy2002: Forcefully removing global lock: Testing scap bg lock
• 13:10 root@deploy2002: Forcefully removing global lock: Testing scap bg lock
• 13:10 root@deploy2002: Forcefully removing global lock: Testing scap bg lock
• 13:09 root@deploy2002: Locking from deployment [ALL REPOSITORIES]: Testing scap bg lock
• 13:06 cgoubert@cumin1003: END (FAIL) - Cookbook sre.switchdc.mediawiki.00-lock-scap (exit_code=99) for datacenter switchover from codfw to eqiad
• 13:06 cgoubert@cumin1003: START - Cookbook sre.switchdc.mediawiki.00-lock-scap for datacenter switchover from codfw to eqiad
• 13:04 cgoubert@deploy2002: Unlocked for deployment [ALL REPOSITORIES]: Testing scap bg lock (duration: 00m 40s)
• 13:04 cgoubert@deploy2002: Forcefully removing global lock: Testing scap bg lock
• 13:04 cgoubert@deploy2002: Forcefully removing global lock: Testing scap bg lock
• 13:03 cgoubert@deploy2002: Locking from deployment [ALL REPOSITORIES]: Testing scap bg lock
• 12:49 moritzm: installing git security updates
• 12:36 bwojtowicz@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 12:34 bwojtowicz@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 12:32 bwojtowicz@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 12:14 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 12:13 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 11:44 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker[1002-1019].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
• 11:42 btullis@cumin1003: END (ERROR) - Cookbook sre.k8s.reboot-nodes (exit_code=97) rolling reboot on A:dse-k8s-worker-eqiad
• 11:41 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "all.dblist - mediamoderation-continuous-scan.dblist - preinstall.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose`
• 11:36 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
• 11:36 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
• 11:35 Dreamy_Jazz: Running `foreachwikiindblist medium.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 1000 -sleep 1` for T413868
• 11:33 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
• 11:33 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
• 11:30 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 11:30 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 11:15 btullis@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
• 11:00 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1024.eqiad.wmnet with OS trixie
• 10:44 Dreamy_Jazz: Running `foreachwikiindblist small.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 1000` for T413868
• 10:44 Dreamy_Jazz: Running `foreachwikiindblist large.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 1000 --sleep 1` for T413868
• 10:43 Dreamy_Jazz: Stopped all maintenance script runs for T413868
• 10:41 Dreamy_Jazz: Restarted group1 run with `foreachwikiindblist group1.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 1000 --sleep 1` for T413868
• 10:38 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1024.eqiad.wmnet with reason: host reimage
• 10:33 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1024.eqiad.wmnet with reason: host reimage
• 10:23 Dreamy_Jazz: Running `foreachwikiindblist group2.dblist extensions/CheckUser/maintenance/populateUserAgentTable.php --batch-size 500` for T413868
• 10:17 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host dbproxy1024.eqiad.wmnet with OS trixie
• 10:16 marostegui@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dbproxy1024.eqiad.wmnet with OS trixie
• 10:08 moritzm: installing postgresql-15 security updates
• 09:54 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host dbproxy1024.eqiad.wmnet with OS trixie
• 09:50 kevinbazira@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 09:46 kevinbazira@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 09:40 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for registry[2004-2005].codfw.wmnet,registry[1004-1005].eqiad.wmnet
• 09:40 dpogorzelski@cumin1003: START - Cookbook sre.hosts.remove-downtime for registry[2004-2005].codfw.wmnet,registry[1004-1005].eqiad.wmnet
• 09:36 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:35 ayounsi@cumin1003: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 09:21 aklapper@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.12 refs T413803
• 09:07 dpogorzelski@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on registry[2004-2005].codfw.wmnet,registry[1004-1005].eqiad.wmnet with reason: testing ml changes
• 08:56 slyngshede@dns1004: END - running authdns-update
• 08:55 kharlan@deploy2002: Finished scap sync-world: Backport for Hooks: Log the security log context for edit errors (T410877) (duration: 07m 24s)
• 08:55 slyngshede@dns1004: START - running authdns-update
• 08:55 slyngshede@dns1004: START - running authdns-update
• 08:54 dpogorzelski@deploy2002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
• 08:53 dpogorzelski@deploy2002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
• 08:52 dpogorzelski@deploy2002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
• 08:51 kharlan@deploy2002: kharlan: Continuing with sync
• 08:51 dpogorzelski@deploy2002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
• 08:50 kharlan@deploy2002: kharlan: Backport for Hooks: Log the security log context for edit errors (T410877) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 08:48 kharlan@deploy2002: Started scap sync-world: Backport for Hooks: Log the security log context for edit errors (T410877)
• 08:45 kharlan@deploy2002: Finished scap sync-world: Backport for IPReputation: Enable OpenSearch IPoid provider on testwiki (T410615) (duration: 11m 43s)
• 08:44 mvernon@cumin2002: END (PASS) - Cookbook sre.swift.remove-ghost-objects (exit_code=0) from container wikipedia-commons-local-public.0c in codfw
• 08:41 mvernon@cumin2002: START - Cookbook sre.swift.remove-ghost-objects from container wikipedia-commons-local-public.0c in codfw
• 08:40 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2202.codfw.wmnet with reason: Maintenance
• 08:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87786 and previous config saved to /var/cache/conftool/dbconfig/20260120-084001-marostegui.json
• 08:39 kharlan@deploy2002: kharlan: Continuing with sync
• 08:36 kharlan@deploy2002: kharlan: Backport for IPReputation: Enable OpenSearch IPoid provider on testwiki (T410615) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 08:33 kharlan@deploy2002: Started scap sync-world: Backport for IPReputation: Enable OpenSearch IPoid provider on testwiki (T410615)
• 08:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P87785 and previous config saved to /var/cache/conftool/dbconfig/20260120-082952-marostegui.json
• 08:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P87784 and previous config saved to /var/cache/conftool/dbconfig/20260120-081944-marostegui.json
• 08:10 bwojtowicz@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 08:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87783 and previous config saved to /var/cache/conftool/dbconfig/20260120-080935-marostegui.json
• 08:08 bwojtowicz@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 07:53 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
• 07:50 moritzm: installing unbound security updates
• 07:49 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/admin 'apply'.
• 07:48 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
• 07:46 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/admin 'apply'.
• 07:45 cgoubert@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
• 07:42 cgoubert@deploy2002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
• 07:42 cgoubert@deploy2002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
• 07:41 cgoubert@deploy2002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
• 06:49 marostegui: Deploy schema change on s8 sanitarium master - s8 wikireplicas will be lagging for many hours T411164 T411163
• 06:48 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2152 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87781 and previous config saved to /var/cache/conftool/dbconfig/20260120-064801-marostegui.json
• 06:47 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance
• 06:47 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1167 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87780 and previous config saved to /var/cache/conftool/dbconfig/20260120-064708-marostegui.json
• 06:47 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 06:46 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 06:37 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Schema change
• 06:29 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2179.codfw.wmnet with reason: Maintenance
• 06:27 marostegui@dns1006: END - running authdns-update
• 06:26 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db2179 T414543', diff saved to https://phabricator.wikimedia.org/P87779 and previous config saved to /var/cache/conftool/dbconfig/20260120-062653-marostegui.json
• 06:26 marostegui@dns1006: START - running authdns-update
• 06:25 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db2240 to s4 primary and set section read-write T414543', diff saved to https://phabricator.wikimedia.org/P87778 and previous config saved to /var/cache/conftool/dbconfig/20260120-062551-marostegui.json
• 06:25 marostegui@cumin1003: dbctl commit (dc=all): 'Set s4 codfw as read-only for maintenance - T414543', diff saved to https://phabricator.wikimedia.org/P87777 and previous config saved to /var/cache/conftool/dbconfig/20260120-062527-marostegui.json
• 06:25 marostegui: Starting s4 codfw failover from db2179 to db2240 - T414543
• 06:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 42 hosts with reason: Primary switchover s4 T414543
• 06:18 marostegui@cumin1003: dbctl commit (dc=all): 'Set db2240 with weight 0 T414543', diff saved to https://phabricator.wikimedia.org/P87776 and previous config saved to /var/cache/conftool/dbconfig/20260120-061840-marostegui.json
• 05:03 mwpresync@deploy2002: Pruned MediaWiki: 1.46.0-wmf.7 (duration: 03m 23s)
• 04:47 mwpresync@deploy2002: Finished scap sync-world: testwikis to 1.46.0-wmf.12 refs T413803 (duration: 44m 14s)
• 04:31 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1219 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87775 and previous config saved to /var/cache/conftool/dbconfig/20260120-043145-marostegui.json
• 04:31 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance
• 04:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87774 and previous config saved to /var/cache/conftool/dbconfig/20260120-043120-marostegui.json
• 04:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P87773 and previous config saved to /var/cache/conftool/dbconfig/20260120-042112-marostegui.json
• 04:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P87772 and previous config saved to /var/cache/conftool/dbconfig/20260120-041104-marostegui.json
• 04:03 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.46.0-wmf.12 refs T413803
• 04:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87771 and previous config saved to /var/cache/conftool/dbconfig/20260120-040055-marostegui.json
• 03:01 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2175 (T410589)', diff saved to https://phabricator.wikimedia.org/P87770 and previous config saved to /var/cache/conftool/dbconfig/20260120-030136-ladsgroup.json
• 03:01 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
• 03:01 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T410589)', diff saved to https://phabricator.wikimedia.org/P87769 and previous config saved to /var/cache/conftool/dbconfig/20260120-030112-ladsgroup.json
• 02:51 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P87768 and previous config saved to /var/cache/conftool/dbconfig/20260120-025103-ladsgroup.json
• 02:40 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P87767 and previous config saved to /var/cache/conftool/dbconfig/20260120-024056-ladsgroup.json
• 02:30 ladsgroup@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T410589)', diff saved to https://phabricator.wikimedia.org/P87766 and previous config saved to /var/cache/conftool/dbconfig/20260120-023047-ladsgroup.json
• 01:01 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 01m 03s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

2026-01-19

• 21:24 tgr@deploy2002: Finished scap sync-world: Backport for enwikiquote: Add autopatroller protection option (T414711), debug: Add X-Provenance header to Logstash (T412396), Urwikiquote: restore flipped icon (T413592) (duration: 09m 58s)
• 21:20 tgr@deploy2002: seawolf35gerrit, pppery, tgr: Continuing with sync
• 21:16 tgr@deploy2002: seawolf35gerrit, pppery, tgr: Backport for enwikiquote: Add autopatroller protection option (T414711), debug: Add X-Provenance header to Logstash (T412396), Urwikiquote: restore flipped icon (T413592) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:14 tgr@deploy2002: Started scap sync-world: Backport for enwikiquote: Add autopatroller protection option (T414711), debug: Add X-Provenance header to Logstash (T412396), Urwikiquote: restore flipped icon (T413592)
• 21:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2188 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87763 and previous config saved to /var/cache/conftool/dbconfig/20260119-210153-marostegui.json
• 21:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance
• 21:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87762 and previous config saved to /var/cache/conftool/dbconfig/20260119-210128-marostegui.json
• 20:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P87761 and previous config saved to /var/cache/conftool/dbconfig/20260119-205120-marostegui.json
• 20:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P87760 and previous config saved to /var/cache/conftool/dbconfig/20260119-204112-marostegui.json
• 20:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87759 and previous config saved to /var/cache/conftool/dbconfig/20260119-203104-marostegui.json
• 20:21 brett@dns1006: END - running authdns-update
• 20:20 brett@dns1006: START - running authdns-update
• 18:28 ammarpad@deploy2002: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki --reason 'Requested at phab:T414529' --skip-redirect 'Extension:DynamicPageList3 ' Extension:DynamicPageList4 Ammarpad # T414529
• 17:58 sukhe: sudo systemctl restart pybal.service on lvs2014: T414940
• 17:08 ammarpad@deploy2002: mwscript-k8s job started: refreshImageMetadata.php --wiki=commonswiki --mediatype=AUDIO --mime=audio/midi --force --start=Honeysuckle_Rose_for_wikipedia.mid --end=Honeysuckle_Rose_for_wikipedia.mid # T414642
• 17:03 kevinbazira@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 17:02 kevinbazira@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 16:55 ammarpad@deploy2002: mwscript-k8s job started: refreshImageMetadata.php --wiki=commonswiki --mediatype=AUDIO --mime=audio/mid --force --start=Segne_du,_Maria.mid --end=Segne_du,_Maria.mid # T414642
• 16:36 ammarpad@deploy2002: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=metawiki --reason 'Requested at phab:T414808' 'Celebrate Women/Events' 'Celebrate Women/Events/2025' Ammarpad # T414808
• 16:12 brett@dns1006: END - running authdns-update
• 16:11 dpogorzelski@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 16:11 brett@dns1006: START - running authdns-update
• 16:08 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
• 16:07 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
• 16:06 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
• 16:06 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
• 16:00 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 16:00 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 15:53 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1218 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87758 and previous config saved to /var/cache/conftool/dbconfig/20260119-155338-marostegui.json
• 15:53 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance
• 15:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87757 and previous config saved to /var/cache/conftool/dbconfig/20260119-155324-marostegui.json
• 15:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P87756 and previous config saved to /var/cache/conftool/dbconfig/20260119-154316-marostegui.json
• 15:38 kevinbazira@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 15:37 kevinbazira@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 15:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P87755 and previous config saved to /var/cache/conftool/dbconfig/20260119-153308-marostegui.json
• 15:24 Dreamy_Jazz: Running populateUserAgentTable.php on group1 wikis for T413868
• 15:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87754 and previous config saved to /var/cache/conftool/dbconfig/20260119-152300-marostegui.json
• 15:16 arnaudb@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-high-traffic1-eqiad and A:lvs (T414940)
• 15:16 arnaudb@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-high-traffic1-eqiad and A:lvs (T414940)
• 15:15 arnaudb@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-high-traffic1-codfw and A:lvs (T414940)
• 15:14 arnaudb@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-high-traffic1-codfw and A:lvs (T414940)
• 15:13 arnaudb@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-eqiad and A:lvs (T414940)
• 15:12 arnaudb@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad and A:lvs (T414940)
• 15:09 arnaudb@cumin1003: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-codfw and A:lvs (T414940)
• 15:09 arnaudb@cumin1003: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-codfw and A:lvs (T414940)
• 15:04 arnaudb@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading A:liberica and A:liberica
• 14:53 arnaudb@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading A:liberica and A:liberica
• 14:51 arnaudb@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3010*} and A:liberica
• 14:51 kharlan@deploy2002: Finished scap sync-world: Backport for IPReputation: Define data provider, URL and developer mode config (T410615) (duration: 12m 09s)
• 14:50 arnaudb@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3010*} and A:liberica
• 14:48 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
• 14:48 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
• 14:47 kharlan@deploy2002: kharlan: Continuing with sync
• 14:40 kharlan@deploy2002: kharlan: Backport for IPReputation: Define data provider, URL and developer mode config (T410615) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:38 kharlan@deploy2002: Started scap sync-world: Backport for IPReputation: Define data provider, URL and developer mode config (T410615)
• 14:30 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
• 14:30 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
• 14:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240 (T413525)', diff saved to https://phabricator.wikimedia.org/P87753 and previous config saved to /var/cache/conftool/dbconfig/20260119-142627-marostegui.json
• 14:26 zabe@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
• 14:25 zabe@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
• 14:20 kevinbazira@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 14:17 kevinbazira@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 14:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P87752 and previous config saved to /var/cache/conftool/dbconfig/20260119-141619-marostegui.json
• 14:14 urbanecm@deploy2002: Finished scap sync-world: Backport for [itwiki] Change the temporary logo for Vector legacy and fix tagline (T414320), GrowthExperiments: cleanup unnecessary GEUseMetricsPlatformExtension (T411479) (duration: 07m 28s)
• 14:10 urbanecm@deploy2002: urbanecm, sgimeno, superpes: Continuing with sync
• 14:09 urbanecm@deploy2002: urbanecm, sgimeno, superpes: Backport for [itwiki] Change the temporary logo for Vector legacy and fix tagline (T414320), GrowthExperiments: cleanup unnecessary GEUseMetricsPlatformExtension (T411479) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:07 urbanecm@deploy2002: Started scap sync-world: Backport for [itwiki] Change the temporary logo for Vector legacy and fix tagline (T414320), GrowthExperiments: cleanup unnecessary GEUseMetricsPlatformExtension (T411479)
• 14:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P87751 and previous config saved to /var/cache/conftool/dbconfig/20260119-140611-marostegui.json
• 13:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240 (T413525)', diff saved to https://phabricator.wikimedia.org/P87750 and previous config saved to /var/cache/conftool/dbconfig/20260119-135602-marostegui.json
• 13:55 vgutierrez: clean up nft prom file on tcp-proxy instances
• 13:54 zabe@deploy2002: Finished scap sync-world: Backport for Start writing to il_target_id on large s6 wikis (T413526) (duration: 10m 31s)
• 13:52 Dreamy_Jazz: Running populateUserAgentTable.php on group0 wikis for T413868
• 13:48 zabe@deploy2002: zabe: Continuing with sync
• 13:48 zabe@deploy2002: zabe: Backport for Start writing to il_target_id on large s6 wikis (T413526) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:44 zabe@deploy2002: Started scap sync-world: Backport for Start writing to il_target_id on large s6 wikis (T413526)
• 13:37 dreamyjazz@deploy2002: Finished scap sync-world: Backport for Write new for CheckUser user agent table migration everywhere (T361196) (duration: 36m 53s)
• 13:31 moritzm: upgrade hcaptcha-proxy nodes to Bird 2.18 T413740
• 13:24 dreamyjazz@deploy2002: dreamyjazz: Continuing with sync
• 13:23 dreamyjazz@deploy2002: dreamyjazz: Backport for Write new for CheckUser user agent table migration everywhere (T361196) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:11 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:03 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wikikube-worker[2019-2032].codfw.wmnet
• 13:03 cgoubert@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 13:03 cgoubert@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[2019-2032].codfw.wmnet decommissioned, removing all IPs except the asset tag one - cgoubert@cumin1003"
• 13:02 cgoubert@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[2019-2032].codfw.wmnet decommissioned, removing all IPs except the asset tag one - cgoubert@cumin1003"
• 13:01 ayounsi@cumin1003: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 13:00 dreamyjazz@deploy2002: Started scap sync-world: Backport for Write new for CheckUser user agent table migration everywhere (T361196)
• 12:59 cgoubert@cumin1003: START - Cookbook sre.dns.netbox
• 12:53 ladsgroup@cumin1003: dbctl commit (dc=all): 'Depooling db2148 (T410589)', diff saved to https://phabricator.wikimedia.org/P87749 and previous config saved to /var/cache/conftool/dbconfig/20260119-125317-ladsgroup.json
• 12:53 ladsgroup@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2148.codfw.wmnet with reason: Maintenance
• 12:52 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:40 ayounsi@cumin1003: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:37 kamila@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/services/mw-debug: apply
• 12:36 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:34 kamila@deploy2002: helmfile [staging-eqiad] START helmfile.d/services/mw-debug: apply
• 12:30 kamila@deploy2002: helmfile [staging-eqiad] DONE helmfile.d/services/mw-debug: apply
• 12:30 kamila@deploy2002: helmfile [staging-eqiad] START helmfile.d/services/mw-debug: apply
• 12:27 ayounsi@cumin1003: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:23 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:18 cgoubert@cumin1003: START - Cookbook sre.hosts.decommission for hosts wikikube-worker[2019-2032].codfw.wmnet
• 12:16 ayounsi@cumin1003: START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 12:16 cgoubert@cumin1003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wikikube-worker[2003-2004,2007-2010,2040,2043,2045,2048].codfw.wmnet
• 12:16 cgoubert@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 12:16 cgoubert@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[2003-2004,2007-2010,2040,2043,2045,2048].codfw.wmnet decommissioned, removing all IPs except the asset tag one - cgoubert@cumin1003"
• 12:16 cgoubert@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[2003-2004,2007-2010,2040,2043,2045,2048].codfw.wmnet decommissioned, removing all IPs except the asset tag one - cgoubert@cumin1003"
• 12:12 cgoubert@cumin1003: START - Cookbook sre.dns.netbox
• 12:08 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2003.codfw.wmnet
• 11:59 ayounsi@cumin1003: START - Cookbook sre.hosts.reboot-single for host sretest2003.codfw.wmnet
• 11:51 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1193.eqiad.wmnet
• 11:50 cgoubert@cumin1003: START - Cookbook sre.hosts.decommission for hosts wikikube-worker[2003-2004,2007-2010,2040,2043,2045,2048].codfw.wmnet
• 11:46 moritzm: intalling openjpeg2 security updates
• 11:44 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-worker1193.eqiad.wmnet
• 11:43 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1206.eqiad.wmnet
• 11:43 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-search: apply
• 11:43 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-search: apply
• 11:38 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1160.eqiad.wmnet with reason: Maintenance
• 11:37 marostegui@cumin1003: dbctl commit (dc=all): 'Depool db1160 T414542', diff saved to https://phabricator.wikimedia.org/P87748 and previous config saved to /var/cache/conftool/dbconfig/20260119-113722-marostegui.json
• 11:35 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-worker1206.eqiad.wmnet
• 11:35 marostegui@cumin1003: dbctl commit (dc=all): 'Promote db1244 to s4 primary T414542', diff saved to https://phabricator.wikimedia.org/P87747 and previous config saved to /var/cache/conftool/dbconfig/20260119-113518-marostegui.json
• 11:34 marostegui: Starting s4 eqiad failover from db1160 to db1244 - T414542
• 11:34 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
• 11:34 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
• 11:33 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1187.eqiad.wmnet
• 11:31 vgutierrez@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp7004.magru.wmnet
• 11:31 vgutierrez@cumin1003: START - Cookbook sre.hosts.remove-downtime for cp7004.magru.wmnet
• 11:30 vgutierrez@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 111 hosts
• 11:29 vgutierrez@cumin1003: START - Cookbook sre.hosts.remove-downtime for 111 hosts
• 11:28 marostegui@cumin1003: dbctl commit (dc=all): 'Set db1244 with weight 0 T414542', diff saved to https://phabricator.wikimedia.org/P87746 and previous config saved to /var/cache/conftool/dbconfig/20260119-112825-marostegui.json
• 11:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 42 hosts with reason: Primary switchover s4 T414542
• 11:26 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-worker1187.eqiad.wmnet
• 10:58 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
• 10:56 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
• 10:55 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 10:54 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 10:52 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 10:51 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 10:39 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 (T413525)', diff saved to https://phabricator.wikimedia.org/P87745 and previous config saved to /var/cache/conftool/dbconfig/20260119-103917-marostegui.json
• 10:29 Emperor: restart apus rgws in eqiad
• 10:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P87744 and previous config saved to /var/cache/conftool/dbconfig/20260119-102909-marostegui.json
• 10:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2003.codfw.wmnet
• 10:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P87743 and previous config saved to /var/cache/conftool/dbconfig/20260119-101901-marostegui.json
• 10:12 ayounsi@cumin1003: START - Cookbook sre.hosts.reboot-single for host sretest2003.codfw.wmnet
• 10:11 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2240 (T413525)', diff saved to https://phabricator.wikimedia.org/P87742 and previous config saved to /var/cache/conftool/dbconfig/20260119-101136-marostegui.json
• 10:11 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2240.codfw.wmnet with reason: Maintenance
• 10:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 (T413525)', diff saved to https://phabricator.wikimedia.org/P87741 and previous config saved to /var/cache/conftool/dbconfig/20260119-101111-marostegui.json
• 10:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 (T413525)', diff saved to https://phabricator.wikimedia.org/P87740 and previous config saved to /var/cache/conftool/dbconfig/20260119-100852-marostegui.json
• 10:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P87739 and previous config saved to /var/cache/conftool/dbconfig/20260119-100103-marostegui.json
• 09:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P87738 and previous config saved to /var/cache/conftool/dbconfig/20260119-095055-marostegui.json
• 09:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 (T413525)', diff saved to https://phabricator.wikimedia.org/P87737 and previous config saved to /var/cache/conftool/dbconfig/20260119-094048-marostegui.json
• 09:35 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-sre: apply
• 09:35 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-sre: apply
• 09:16 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 09:15 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 08:50 dpogorzelski@deploy2002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
• 08:49 dpogorzelski@deploy2002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
• 08:47 XioNoX: continue asw1-b12-drmrs troubleshooting - T413181
• 08:46 dpogorzelski@deploy2002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
• 08:45 dpogorzelski@deploy2002: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
• 08:20 brouberol@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 08:19 brouberol@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
• 08:16 brouberol@dns1004: END - running authdns-update
• 08:15 brouberol@dns1004: START - running authdns-update
• 08:11 brouberol@dns1004: START - running authdns-update
• 07:44 bwojtowicz@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
• 06:45 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2176 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87736 and previous config saved to /var/cache/conftool/dbconfig/20260119-064555-marostegui.json
• 06:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance
• 06:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87735 and previous config saved to /var/cache/conftool/dbconfig/20260119-064531-marostegui.json
• 06:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P87734 and previous config saved to /var/cache/conftool/dbconfig/20260119-063522-marostegui.json
• 06:29 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2248 (T413525)', diff saved to https://phabricator.wikimedia.org/P87733 and previous config saved to /var/cache/conftool/dbconfig/20260119-062926-marostegui.json
• 06:29 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance
• 06:28 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1262 (T413525)', diff saved to https://phabricator.wikimedia.org/P87732 and previous config saved to /var/cache/conftool/dbconfig/20260119-062813-marostegui.json
• 06:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1262.eqiad.wmnet with reason: Maintenance
• 06:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P87731 and previous config saved to /var/cache/conftool/dbconfig/20260119-062514-marostegui.json
• 06:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87730 and previous config saved to /var/cache/conftool/dbconfig/20260119-061506-marostegui.json
• 05:33 eileen: civicrm upgraded from fbd59466 to 9cdb1915
• 04:22 eileen: civicrm upgraded from 8a0e4548 to fbd59466
• 03:38 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1206 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87729 and previous config saved to /var/cache/conftool/dbconfig/20260119-033806-marostegui.json
• 03:37 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance
• 03:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87728 and previous config saved to /var/cache/conftool/dbconfig/20260119-033742-marostegui.json
• 03:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P87727 and previous config saved to /var/cache/conftool/dbconfig/20260119-032733-marostegui.json
• 03:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P87726 and previous config saved to /var/cache/conftool/dbconfig/20260119-031725-marostegui.json
• 03:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87725 and previous config saved to /var/cache/conftool/dbconfig/20260119-030716-marostegui.json
• 01:14 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 13m 28s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

2026-01-18

• 17:06 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2174 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87724 and previous config saved to /var/cache/conftool/dbconfig/20260118-170604-marostegui.json
• 17:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance
• 17:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87723 and previous config saved to /var/cache/conftool/dbconfig/20260118-170540-marostegui.json
• 16:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P87722 and previous config saved to /var/cache/conftool/dbconfig/20260118-165531-marostegui.json
• 16:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P87721 and previous config saved to /var/cache/conftool/dbconfig/20260118-164523-marostegui.json
• 16:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87720 and previous config saved to /var/cache/conftool/dbconfig/20260118-163514-marostegui.json
• 15:51 sukhe: downtime cp5022 as host is down: T414411
• 15:51 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on cp5022.eqsin.wmnet with reason: host down
• 12:59 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1196 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87719 and previous config saved to /var/cache/conftool/dbconfig/20260118-125913-marostegui.json
• 12:59 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 12:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance
• 12:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87718 and previous config saved to /var/cache/conftool/dbconfig/20260118-125829-marostegui.json
• 12:48 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P87717 and previous config saved to /var/cache/conftool/dbconfig/20260118-124820-marostegui.json
• 12:38 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P87716 and previous config saved to /var/cache/conftool/dbconfig/20260118-123812-marostegui.json
• 12:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87715 and previous config saved to /var/cache/conftool/dbconfig/20260118-122804-marostegui.json
• 08:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 08:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 (T413525)', diff saved to https://phabricator.wikimedia.org/P87714 and previous config saved to /var/cache/conftool/dbconfig/20260118-080510-marostegui.json
• 07:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P87713 and previous config saved to /var/cache/conftool/dbconfig/20260118-075502-marostegui.json
• 07:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P87712 and previous config saved to /var/cache/conftool/dbconfig/20260118-074453-marostegui.json
• 07:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 (T413525)', diff saved to https://phabricator.wikimedia.org/P87711 and previous config saved to /var/cache/conftool/dbconfig/20260118-073445-marostegui.json
• 03:42 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1263 (T413525)', diff saved to https://phabricator.wikimedia.org/P87710 and previous config saved to /var/cache/conftool/dbconfig/20260118-034228-marostegui.json
• 03:42 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1263.eqiad.wmnet with reason: Maintenance
• 03:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 (T413525)', diff saved to https://phabricator.wikimedia.org/P87709 and previous config saved to /var/cache/conftool/dbconfig/20260118-034214-marostegui.json
• 03:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P87708 and previous config saved to /var/cache/conftool/dbconfig/20260118-033205-marostegui.json
• 03:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P87707 and previous config saved to /var/cache/conftool/dbconfig/20260118-032157-marostegui.json
• 03:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 (T413525)', diff saved to https://phabricator.wikimedia.org/P87706 and previous config saved to /var/cache/conftool/dbconfig/20260118-031149-marostegui.json
• 02:47 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2173 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87705 and previous config saved to /var/cache/conftool/dbconfig/20260118-024705-marostegui.json
• 02:46 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance
• 02:46 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87704 and previous config saved to /var/cache/conftool/dbconfig/20260118-024640-marostegui.json
• 02:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P87703 and previous config saved to /var/cache/conftool/dbconfig/20260118-023632-marostegui.json
• 02:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P87702 and previous config saved to /var/cache/conftool/dbconfig/20260118-022624-marostegui.json
• 02:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87701 and previous config saved to /var/cache/conftool/dbconfig/20260118-021615-marostegui.json
• 01:14 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 13m 31s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

2026-01-17

• 23:14 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1261 (T413525)', diff saved to https://phabricator.wikimedia.org/P87700 and previous config saved to /var/cache/conftool/dbconfig/20260117-231441-marostegui.json
• 23:14 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1261.eqiad.wmnet with reason: Maintenance
• 23:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 (T413525)', diff saved to https://phabricator.wikimedia.org/P87699 and previous config saved to /var/cache/conftool/dbconfig/20260117-231416-marostegui.json
• 23:04 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P87698 and previous config saved to /var/cache/conftool/dbconfig/20260117-230407-marostegui.json
• 22:56 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1195 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87697 and previous config saved to /var/cache/conftool/dbconfig/20260117-225600-marostegui.json
• 22:55 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance
• 22:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87696 and previous config saved to /var/cache/conftool/dbconfig/20260117-225536-marostegui.json
• 22:54 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P87695 and previous config saved to /var/cache/conftool/dbconfig/20260117-225359-marostegui.json
• 22:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P87694 and previous config saved to /var/cache/conftool/dbconfig/20260117-224528-marostegui.json
• 22:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 (T413525)', diff saved to https://phabricator.wikimedia.org/P87693 and previous config saved to /var/cache/conftool/dbconfig/20260117-224351-marostegui.json
• 22:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P87692 and previous config saved to /var/cache/conftool/dbconfig/20260117-223519-marostegui.json
• 22:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87691 and previous config saved to /var/cache/conftool/dbconfig/20260117-222511-marostegui.json
• 18:42 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1260 (T413525)', diff saved to https://phabricator.wikimedia.org/P87690 and previous config saved to /var/cache/conftool/dbconfig/20260117-184203-marostegui.json
• 18:41 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1260.eqiad.wmnet with reason: Maintenance
• 18:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T413525)', diff saved to https://phabricator.wikimedia.org/P87689 and previous config saved to /var/cache/conftool/dbconfig/20260117-184138-marostegui.json
• 18:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P87688 and previous config saved to /var/cache/conftool/dbconfig/20260117-183130-marostegui.json
• 18:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P87687 and previous config saved to /var/cache/conftool/dbconfig/20260117-182122-marostegui.json
• 18:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T413525)', diff saved to https://phabricator.wikimedia.org/P87686 and previous config saved to /var/cache/conftool/dbconfig/20260117-181113-marostegui.json
• 14:09 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1252 (T413525)', diff saved to https://phabricator.wikimedia.org/P87685 and previous config saved to /var/cache/conftool/dbconfig/20260117-140950-marostegui.json
• 14:09 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1252.eqiad.wmnet with reason: Maintenance
• 14:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T413525)', diff saved to https://phabricator.wikimedia.org/P87684 and previous config saved to /var/cache/conftool/dbconfig/20260117-140926-marostegui.json
• 13:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P87683 and previous config saved to /var/cache/conftool/dbconfig/20260117-135917-marostegui.json
• 13:49 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P87682 and previous config saved to /var/cache/conftool/dbconfig/20260117-134909-marostegui.json
• 13:39 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T413525)', diff saved to https://phabricator.wikimedia.org/P87681 and previous config saved to /var/cache/conftool/dbconfig/20260117-133901-marostegui.json
• 12:34 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2170 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87680 and previous config saved to /var/cache/conftool/dbconfig/20260117-123414-marostegui.json
• 12:34 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance
• 12:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87679 and previous config saved to /var/cache/conftool/dbconfig/20260117-123350-marostegui.json
• 12:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P87678 and previous config saved to /var/cache/conftool/dbconfig/20260117-122342-marostegui.json
• 12:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P87677 and previous config saved to /var/cache/conftool/dbconfig/20260117-121333-marostegui.json
• 12:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87676 and previous config saved to /var/cache/conftool/dbconfig/20260117-120324-marostegui.json
• 09:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1249 (T413525)', diff saved to https://phabricator.wikimedia.org/P87675 and previous config saved to /var/cache/conftool/dbconfig/20260117-094637-marostegui.json
• 09:46 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1249.eqiad.wmnet with reason: Maintenance
• 09:46 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T413525)', diff saved to https://phabricator.wikimedia.org/P87674 and previous config saved to /var/cache/conftool/dbconfig/20260117-094612-marostegui.json
• 09:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P87673 and previous config saved to /var/cache/conftool/dbconfig/20260117-093604-marostegui.json
• 09:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P87672 and previous config saved to /var/cache/conftool/dbconfig/20260117-092555-marostegui.json
• 09:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T413525)', diff saved to https://phabricator.wikimedia.org/P87671 and previous config saved to /var/cache/conftool/dbconfig/20260117-091547-marostegui.json
• 09:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1186 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87670 and previous config saved to /var/cache/conftool/dbconfig/20260117-090151-marostegui.json
• 09:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance
• 09:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87669 and previous config saved to /var/cache/conftool/dbconfig/20260117-090127-marostegui.json
• 08:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P87668 and previous config saved to /var/cache/conftool/dbconfig/20260117-085119-marostegui.json
• 08:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P87667 and previous config saved to /var/cache/conftool/dbconfig/20260117-084111-marostegui.json
• 08:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87666 and previous config saved to /var/cache/conftool/dbconfig/20260117-083103-marostegui.json
• 05:37 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1248 (T413525)', diff saved to https://phabricator.wikimedia.org/P87665 and previous config saved to /var/cache/conftool/dbconfig/20260117-053725-marostegui.json
• 05:37 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1248.eqiad.wmnet with reason: Maintenance
• 05:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T413525)', diff saved to https://phabricator.wikimedia.org/P87664 and previous config saved to /var/cache/conftool/dbconfig/20260117-053700-marostegui.json
• 05:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P87663 and previous config saved to /var/cache/conftool/dbconfig/20260117-052652-marostegui.json
• 05:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P87662 and previous config saved to /var/cache/conftool/dbconfig/20260117-051643-marostegui.json
• 05:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T413525)', diff saved to https://phabricator.wikimedia.org/P87661 and previous config saved to /var/cache/conftool/dbconfig/20260117-050635-marostegui.json
• 01:23 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1247 (T413525)', diff saved to https://phabricator.wikimedia.org/P87660 and previous config saved to /var/cache/conftool/dbconfig/20260117-012303-marostegui.json
• 01:22 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1247.eqiad.wmnet with reason: Maintenance
• 01:13 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 13m 05s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

2026-01-16

• 23:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 (T413525)', diff saved to https://phabricator.wikimedia.org/P87659 and previous config saved to /var/cache/conftool/dbconfig/20260116-233200-marostegui.json
• 23:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P87658 and previous config saved to /var/cache/conftool/dbconfig/20260116-232151-marostegui.json
• 23:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P87657 and previous config saved to /var/cache/conftool/dbconfig/20260116-231143-marostegui.json
• 23:04 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2347.codfw.wmnet with OS bookworm
• 23:04 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 23:04 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 23:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 (T413525)', diff saved to https://phabricator.wikimedia.org/P87656 and previous config saved to /var/cache/conftool/dbconfig/20260116-230134-marostegui.json
• 22:59 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2348.codfw.wmnet with OS bookworm
• 22:59 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 22:59 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 22:57 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2344.codfw.wmnet with OS bookworm
• 22:57 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 22:56 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 22:54 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2153 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87655 and previous config saved to /var/cache/conftool/dbconfig/20260116-225416-marostegui.json
• 22:54 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance
• 22:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87654 and previous config saved to /var/cache/conftool/dbconfig/20260116-225351-marostegui.json
• 22:46 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2347.codfw.wmnet with reason: host reimage
• 22:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P87653 and previous config saved to /var/cache/conftool/dbconfig/20260116-224343-marostegui.json
• 22:42 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2348.codfw.wmnet with reason: host reimage
• 22:38 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2344.codfw.wmnet with reason: host reimage
• 22:34 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2348.codfw.wmnet with reason: host reimage
• 22:34 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2347.codfw.wmnet with reason: host reimage
• 22:34 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2344.codfw.wmnet with reason: host reimage
• 22:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P87652 and previous config saved to /var/cache/conftool/dbconfig/20260116-223334-marostegui.json
• 22:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87651 and previous config saved to /var/cache/conftool/dbconfig/20260116-222326-marostegui.json
• 22:23 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2348.codfw.wmnet with OS bookworm
• 22:23 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2347.codfw.wmnet with OS bookworm
• 22:22 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2344.codfw.wmnet with OS bookworm
• 22:15 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2343.codfw.wmnet with OS bookworm
• 22:15 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 22:15 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 22:15 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2342.codfw.wmnet with OS bookworm
• 22:15 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 22:13 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 22:09 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2341.codfw.wmnet with OS bookworm
• 22:09 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 22:08 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:58 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2343.codfw.wmnet with reason: host reimage
• 21:54 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 21:54 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T413525)', diff saved to https://phabricator.wikimedia.org/P87650 and previous config saved to /var/cache/conftool/dbconfig/20260116-215436-marostegui.json
• 21:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2342.codfw.wmnet with reason: host reimage
• 21:50 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2341.codfw.wmnet with reason: host reimage
• 21:46 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2343.codfw.wmnet with reason: host reimage
• 21:46 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2342.codfw.wmnet with reason: host reimage
• 21:46 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2341.codfw.wmnet with reason: host reimage
• 21:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P87649 and previous config saved to /var/cache/conftool/dbconfig/20260116-214427-marostegui.json
• 21:35 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2343.codfw.wmnet with OS bookworm
• 21:35 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2342.codfw.wmnet with OS bookworm
• 21:35 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2341.codfw.wmnet with OS bookworm
• 21:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P87648 and previous config saved to /var/cache/conftool/dbconfig/20260116-213419-marostegui.json
• 21:32 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2340.codfw.wmnet with OS bookworm
• 21:32 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:31 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:24 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2339.codfw.wmnet with OS bookworm
• 21:24 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T413525)', diff saved to https://phabricator.wikimedia.org/P87647 and previous config saved to /var/cache/conftool/dbconfig/20260116-212411-marostegui.json
• 21:24 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:20 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2338.codfw.wmnet with OS bookworm
• 21:20 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:20 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:13 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2340.codfw.wmnet with reason: host reimage
• 21:06 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2339.codfw.wmnet with reason: host reimage
• 21:03 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2338.codfw.wmnet with reason: host reimage
• 21:00 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2339.codfw.wmnet with reason: host reimage
• 21:00 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2340.codfw.wmnet with reason: host reimage
• 20:59 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2338.codfw.wmnet with reason: host reimage
• 20:49 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2340.codfw.wmnet with OS bookworm
• 20:48 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2339.codfw.wmnet with OS bookworm
• 20:48 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2338.codfw.wmnet with OS bookworm
• 20:45 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2335.codfw.wmnet with OS bookworm
• 20:45 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:44 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:42 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2337.codfw.wmnet with OS bookworm
• 20:42 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:40 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:37 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2336.codfw.wmnet with OS bookworm
• 20:37 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:36 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:26 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2335.codfw.wmnet with reason: host reimage
• 20:22 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2337.codfw.wmnet with reason: host reimage
• 20:18 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2336.codfw.wmnet with reason: host reimage
• 20:14 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2337.codfw.wmnet with reason: host reimage
• 20:14 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2335.codfw.wmnet with reason: host reimage
• 20:14 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2336.codfw.wmnet with reason: host reimage
• 20:03 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2337.codfw.wmnet with OS bookworm
• 20:03 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2336.codfw.wmnet with OS bookworm
• 20:02 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2335.codfw.wmnet with OS bookworm
• 20:01 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2334.codfw.wmnet with OS bookworm
• 20:01 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 20:00 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 19:55 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2333.codfw.wmnet with OS bookworm
• 19:55 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 19:55 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 19:53 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2332.codfw.wmnet with OS bookworm
• 19:53 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 19:52 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 19:42 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2334.codfw.wmnet with reason: host reimage
• 19:38 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2333.codfw.wmnet with reason: host reimage
• 19:35 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker2332.codfw.wmnet with reason: host reimage
• 19:29 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2334.codfw.wmnet with reason: host reimage
• 19:29 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2333.codfw.wmnet with reason: host reimage
• 19:29 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker2332.codfw.wmnet with reason: host reimage
• 19:18 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2334.codfw.wmnet with OS bookworm
• 19:18 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2333.codfw.wmnet with OS bookworm
• 19:17 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker2332.codfw.wmnet with OS bookworm
• 19:14 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2345.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:12 jhancock@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker2345
• 19:12 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2344.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:12 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2343.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:12 jhancock@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker2345
• 19:11 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2342.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:06 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2345.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:05 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1184 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87646 and previous config saved to /var/cache/conftool/dbconfig/20260116-190539-marostegui.json
• 19:05 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2341.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance
• 19:05 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2344.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87645 and previous config saved to /var/cache/conftool/dbconfig/20260116-190510-marostegui.json
• 19:05 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2343.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:04 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2340.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:04 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2339.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:04 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2342.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:04 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2338.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:03 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 19:02 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2247 (T413525)', diff saved to https://phabricator.wikimedia.org/P87644 and previous config saved to /var/cache/conftool/dbconfig/20260116-190235-marostegui.json
• 19:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance
• 19:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 (T413525)', diff saved to https://phabricator.wikimedia.org/P87643 and previous config saved to /var/cache/conftool/dbconfig/20260116-190220-marostegui.json
• 18:58 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2341.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:57 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2340.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:57 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2339.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:56 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2338.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:56 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P87642 and previous config saved to /var/cache/conftool/dbconfig/20260116-185502-marostegui.json
• 18:54 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2333.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:53 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2336.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:52 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2335.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P87641 and previous config saved to /var/cache/conftool/dbconfig/20260116-185212-marostegui.json
• 18:49 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:46 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2336.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:45 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2335.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:45 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2333.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P87640 and previous config saved to /var/cache/conftool/dbconfig/20260116-184454-marostegui.json
• 18:44 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2333.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:44 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2333.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:42 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 18:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P87639 and previous config saved to /var/cache/conftool/dbconfig/20260116-184203-marostegui.json
• 18:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87638 and previous config saved to /var/cache/conftool/dbconfig/20260116-183445-marostegui.json
• 18:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 (T413525)', diff saved to https://phabricator.wikimedia.org/P87637 and previous config saved to /var/cache/conftool/dbconfig/20260116-183155-marostegui.json
• 18:15 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host an-worker1187.eqiad.wmnet
• 18:11 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2347.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:56 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2347.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:54 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2348.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:52 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-worker1187.eqiad.wmnet
• 17:47 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2348.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:43 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2349.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:41 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1244 (T413525)', diff saved to https://phabricator.wikimedia.org/P87636 and previous config saved to /var/cache/conftool/dbconfig/20260116-174107-marostegui.json
• 17:40 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 17:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T413525)', diff saved to https://phabricator.wikimedia.org/P87635 and previous config saved to /var/cache/conftool/dbconfig/20260116-174042-marostegui.json
• 17:36 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2349.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P87634 and previous config saved to /var/cache/conftool/dbconfig/20260116-173033-marostegui.json
• 17:28 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:28 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:27 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:27 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:25 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2350.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:20 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P87633 and previous config saved to /var/cache/conftool/dbconfig/20260116-172025-marostegui.json
• 17:20 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:19 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:17 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2350.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:16 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:16 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:14 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2351.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:10 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T413525)', diff saved to https://phabricator.wikimedia.org/P87632 and previous config saved to /var/cache/conftool/dbconfig/20260116-171016-marostegui.json
• 17:08 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:08 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:07 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:06 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:06 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:05 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:05 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2333.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:05 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2333.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:05 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:05 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2351.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:04 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 17:00 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2352.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:57 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2336.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:56 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2334.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:54 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:54 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:54 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2335.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:53 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2336.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:53 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2333.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:52 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:50 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2335.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:50 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2352.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:49 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2334.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:49 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2333.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:48 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker2332.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:48 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2353.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:40 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2353.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:28 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2354.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:21 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2354.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:17 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2355.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:09 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2355.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2356.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 15:58 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host wikikube-worker2356.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 15:18 ejegg: fundraising civicrm upgraded from 54fb7415 to 8a0e4548
• 14:50 dzahn@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 14:50 dzahn@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 14:49 dzahn@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 14:49 dzahn@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 14:48 dzahn@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 14:48 dzahn@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
• 14:44 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'clear' for AS: 4800
• 14:43 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'clear' for AS: 4800
• 14:31 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2246 (T413525)', diff saved to https://phabricator.wikimedia.org/P87629 and previous config saved to /var/cache/conftool/dbconfig/20260116-143147-marostegui.json
• 14:31 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance
• 14:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 (T413525)', diff saved to https://phabricator.wikimedia.org/P87628 and previous config saved to /var/cache/conftool/dbconfig/20260116-143122-marostegui.json
• 14:28 XioNoX: asw1-b12-drmrs> restart statistics-service - T413181
• 14:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P87627 and previous config saved to /var/cache/conftool/dbconfig/20260116-142114-marostegui.json
• 14:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P87626 and previous config saved to /var/cache/conftool/dbconfig/20260116-141105-marostegui.json
• 14:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 (T413525)', diff saved to https://phabricator.wikimedia.org/P87625 and previous config saved to /var/cache/conftool/dbconfig/20260116-140057-marostegui.json
• 13:29 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1243 (T413525)', diff saved to https://phabricator.wikimedia.org/P87624 and previous config saved to /var/cache/conftool/dbconfig/20260116-132930-marostegui.json
• 13:29 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1243.eqiad.wmnet with reason: Maintenance
• 13:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T413525)', diff saved to https://phabricator.wikimedia.org/P87623 and previous config saved to /var/cache/conftool/dbconfig/20260116-132905-marostegui.json
• 13:18 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P87622 and previous config saved to /var/cache/conftool/dbconfig/20260116-131857-marostegui.json
• 13:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P87621 and previous config saved to /var/cache/conftool/dbconfig/20260116-130848-marostegui.json
• 12:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T413525)', diff saved to https://phabricator.wikimedia.org/P87620 and previous config saved to /var/cache/conftool/dbconfig/20260116-125840-marostegui.json
• 11:46 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mwlog1003.eqiad.wmnet with OS bookworm
• 10:39 moritzm: installing Linux 6.12.63 on trixie hosts
• 10:03 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2245 (T413525)', diff saved to https://phabricator.wikimedia.org/P87619 and previous config saved to /var/cache/conftool/dbconfig/20260116-100322-marostegui.json
• 10:03 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance
• 10:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240 (T413525)', diff saved to https://phabricator.wikimedia.org/P87618 and previous config saved to /var/cache/conftool/dbconfig/20260116-100257-marostegui.json
• 09:56 topranks: remove static routes for magru ranges on cr1-eqiad to revert load-balance of transport traffic T414473 (https://phabricator.wikimedia.org/P87617)
• 09:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P87616 and previous config saved to /var/cache/conftool/dbconfig/20260116-095248-marostegui.json
• 09:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P87615 and previous config saved to /var/cache/conftool/dbconfig/20260116-094240-marostegui.json
• 09:34 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2146 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87614 and previous config saved to /var/cache/conftool/dbconfig/20260116-093444-marostegui.json
• 09:34 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2146.codfw.wmnet with reason: Maintenance
• 09:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87613 and previous config saved to /var/cache/conftool/dbconfig/20260116-093418-marostegui.json
• 09:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240 (T413525)', diff saved to https://phabricator.wikimedia.org/P87612 and previous config saved to /var/cache/conftool/dbconfig/20260116-093232-marostegui.json
• 09:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P87611 and previous config saved to /var/cache/conftool/dbconfig/20260116-092410-marostegui.json
• 09:16 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1242 (T413525)', diff saved to https://phabricator.wikimedia.org/P87609 and previous config saved to /var/cache/conftool/dbconfig/20260116-091649-marostegui.json
• 09:16 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1242.eqiad.wmnet with reason: Maintenance
• 09:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T413525)', diff saved to https://phabricator.wikimedia.org/P87608 and previous config saved to /var/cache/conftool/dbconfig/20260116-091623-marostegui.json
• 09:15 mutante: attempting soft reboot of instance codesearch9.codesearch - down and can't connect - T414776
• 09:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P87607 and previous config saved to /var/cache/conftool/dbconfig/20260116-091401-marostegui.json
• 09:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P87606 and previous config saved to /var/cache/conftool/dbconfig/20260116-090614-marostegui.json
• 09:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87605 and previous config saved to /var/cache/conftool/dbconfig/20260116-090353-marostegui.json
• 08:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P87604 and previous config saved to /var/cache/conftool/dbconfig/20260116-085605-marostegui.json
• 08:50 tappof: depool titan2001, cleaning up block 01K88XDMJ9S0T2DR5K00VG9CFE (T410152)
• 08:46 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T413525)', diff saved to https://phabricator.wikimedia.org/P87603 and previous config saved to /var/cache/conftool/dbconfig/20260116-084557-marostegui.json
• 08:32 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2240 (T413525)', diff saved to https://phabricator.wikimedia.org/P87602 and previous config saved to /var/cache/conftool/dbconfig/20260116-083206-marostegui.json
• 08:31 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2240.codfw.wmnet with reason: Maintenance
• 07:58 mutante: phabricator - adding kimpham to WMF-NDA (T414157)
• 07:58 mutante: phabricator - adding Martyn.ranyard to WMF-NDA (T413994)
• 07:54 mutante: phabricator - addign Johannes_Richter_WMDE to WMF-NDA T414678
• 07:02 larssandergreen: civicrm upgraded from 43808405 to 54fb7415
• 06:26 marostegui@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0)
• 06:21 marostegui@cumin1003: START - Cookbook sre.wikireplicas.update-views
• 06:21 marostegui@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0)
• 06:16 marostegui@cumin1003: START - Cookbook sre.wikireplicas.update-views
• 05:48 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1169 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87601 and previous config saved to /var/cache/conftool/dbconfig/20260116-054831-marostegui.json
• 05:48 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 05:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2239.codfw.wmnet with reason: Maintenance
• 05:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T413525)', diff saved to https://phabricator.wikimedia.org/P87600 and previous config saved to /var/cache/conftool/dbconfig/20260116-050536-marostegui.json
• 05:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1241 (T413525)', diff saved to https://phabricator.wikimedia.org/P87599 and previous config saved to /var/cache/conftool/dbconfig/20260116-050102-marostegui.json
• 05:00 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1241.eqiad.wmnet with reason: Maintenance
• 05:00 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T413525)', diff saved to https://phabricator.wikimedia.org/P87598 and previous config saved to /var/cache/conftool/dbconfig/20260116-050038-marostegui.json
• 04:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P87597 and previous config saved to /var/cache/conftool/dbconfig/20260116-045527-marostegui.json
• 04:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P87596 and previous config saved to /var/cache/conftool/dbconfig/20260116-045028-marostegui.json
• 04:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P87595 and previous config saved to /var/cache/conftool/dbconfig/20260116-044519-marostegui.json
• 04:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P87594 and previous config saved to /var/cache/conftool/dbconfig/20260116-044020-marostegui.json
• 04:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T413525)', diff saved to https://phabricator.wikimedia.org/P87593 and previous config saved to /var/cache/conftool/dbconfig/20260116-043511-marostegui.json
• 04:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T413525)', diff saved to https://phabricator.wikimedia.org/P87592 and previous config saved to /var/cache/conftool/dbconfig/20260116-043012-marostegui.json
• 02:38 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 02:38 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87591 and previous config saved to /var/cache/conftool/dbconfig/20260116-023806-marostegui.json
• 02:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P87590 and previous config saved to /var/cache/conftool/dbconfig/20260116-022758-marostegui.json
• 02:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P87589 and previous config saved to /var/cache/conftool/dbconfig/20260116-021748-marostegui.json
• 02:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1263 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87588 and previous config saved to /var/cache/conftool/dbconfig/20260116-020740-marostegui.json
• 01:14 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 13m 05s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image
• 00:45 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2237 (T413525)', diff saved to https://phabricator.wikimedia.org/P87587 and previous config saved to /var/cache/conftool/dbconfig/20260116-004540-marostegui.json
• 00:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance
• 00:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 (T413525)', diff saved to https://phabricator.wikimedia.org/P87586 and previous config saved to /var/cache/conftool/dbconfig/20260116-004514-marostegui.json
• 00:41 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1238 (T413525)', diff saved to https://phabricator.wikimedia.org/P87585 and previous config saved to /var/cache/conftool/dbconfig/20260116-004117-marostegui.json
• 00:41 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1238.eqiad.wmnet with reason: Maintenance
• 00:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T413525)', diff saved to https://phabricator.wikimedia.org/P87584 and previous config saved to /var/cache/conftool/dbconfig/20260116-004052-marostegui.json
• 00:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P87583 and previous config saved to /var/cache/conftool/dbconfig/20260116-003506-marostegui.json
• 00:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P87582 and previous config saved to /var/cache/conftool/dbconfig/20260116-003044-marostegui.json
• 00:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P87581 and previous config saved to /var/cache/conftool/dbconfig/20260116-002457-marostegui.json
• 00:20 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P87580 and previous config saved to /var/cache/conftool/dbconfig/20260116-002036-marostegui.json
• 00:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 (T413525)', diff saved to https://phabricator.wikimedia.org/P87579 and previous config saved to /var/cache/conftool/dbconfig/20260116-001449-marostegui.json
• 00:10 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T413525)', diff saved to https://phabricator.wikimedia.org/P87578 and previous config saved to /var/cache/conftool/dbconfig/20260116-001027-marostegui.json

2026-01-15

• 23:02 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 23:02 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add new wikikube-worker nodes - pt1979@cumin2002"
• 23:02 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add new wikikube-worker nodes - pt1979@cumin2002"
• 22:59 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 22:02 cjming@deploy2002: Finished scap sync-world: Backport for Update experiment code for JS, PHP SDKs testing of TK (T414528 T414530) (duration: 09m 23s)
• 21:58 cjming@deploy2002: cjming: Continuing with sync
• 21:55 cjming@deploy2002: cjming: Backport for Update experiment code for JS, PHP SDKs testing of TK (T414528 T414530) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:53 cjming@deploy2002: Started scap sync-world: Backport for Update experiment code for JS, PHP SDKs testing of TK (T414528 T414530)
• 21:41 jhuneidi@deploy2002: Finished scap sync-world: Backport for ukwiki: Move assignments of FlaggedRevs permissions to flaggedrevs.php (T414277 T414684), ukwiki: Add "changetags" to sysop user group. (T414277) (duration: 07m 46s)
• 21:37 jhuneidi@deploy2002: asmartkitten, seawolf35gerrit, jhuneidi: Continuing with sync
• 21:35 jhuneidi@deploy2002: asmartkitten, seawolf35gerrit, jhuneidi: Backport for ukwiki: Move assignments of FlaggedRevs permissions to flaggedrevs.php (T414277 T414684), ukwiki: Add "changetags" to sysop user group. (T414277) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:33 jhuneidi@deploy2002: Started scap sync-world: Backport for ukwiki: Move assignments of FlaggedRevs permissions to flaggedrevs.php (T414277 T414684), ukwiki: Add "changetags" to sysop user group. (T414277)
• 21:30 jhuneidi@deploy2002: Finished scap sync-world: Backport for Deploy PersonalDashboard to testwiki (T403982) (duration: 09m 18s)
• 21:26 jhuneidi@deploy2002: jhuneidi, jsn: Continuing with sync
• 21:23 jhuneidi@deploy2002: jhuneidi, jsn: Backport for Deploy PersonalDashboard to testwiki (T403982) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:21 jhuneidi@deploy2002: Started scap sync-world: Backport for Deploy PersonalDashboard to testwiki (T403982)
• 21:16 derick@deploy2002: Finished scap sync-world: Backport for Control: When saving grants, ensure array has no gaps, Control: Keep irrevocable grants when accepting new OAuth 2 consumers (T413947) (duration: 09m 41s)
• 21:12 derick@deploy2002: derick, d3r1ck01: Continuing with sync
• 21:08 derick@deploy2002: derick, d3r1ck01: Backport for Control: When saving grants, ensure array has no gaps, Control: Keep irrevocable grants when accepting new OAuth 2 consumers (T413947) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:06 derick@deploy2002: Started scap sync-world: Backport for Control: When saving grants, ensure array has no gaps, Control: Keep irrevocable grants when accepting new OAuth 2 consumers (T413947)
• 20:38 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2236 (T413525)', diff saved to https://phabricator.wikimedia.org/P87576 and previous config saved to /var/cache/conftool/dbconfig/20260115-203759-marostegui.json
• 20:37 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance
• 20:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T413525)', diff saved to https://phabricator.wikimedia.org/P87575 and previous config saved to /var/cache/conftool/dbconfig/20260115-203746-marostegui.json
• 20:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P87574 and previous config saved to /var/cache/conftool/dbconfig/20260115-202738-marostegui.json
• 20:23 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1221 (T413525)', diff saved to https://phabricator.wikimedia.org/P87573 and previous config saved to /var/cache/conftool/dbconfig/20260115-202305-marostegui.json
• 20:22 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
• 20:22 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1221.eqiad.wmnet with reason: Maintenance
• 20:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T413525)', diff saved to https://phabricator.wikimedia.org/P87572 and previous config saved to /var/cache/conftool/dbconfig/20260115-202218-marostegui.json
• 20:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P87571 and previous config saved to /var/cache/conftool/dbconfig/20260115-201730-marostegui.json
• 20:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P87570 and previous config saved to /var/cache/conftool/dbconfig/20260115-201210-marostegui.json
• 20:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T413525)', diff saved to https://phabricator.wikimedia.org/P87569 and previous config saved to /var/cache/conftool/dbconfig/20260115-200721-marostegui.json
• 20:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P87568 and previous config saved to /var/cache/conftool/dbconfig/20260115-200202-marostegui.json
• 20:00 eileen: civicrm upgraded from 6775b0eb to 43808405
• 19:54 jasmine_: “homer run T409102”
• 19:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T413525)', diff saved to https://phabricator.wikimedia.org/P87567 and previous config saved to /var/cache/conftool/dbconfig/20260115-195153-marostegui.json
• 19:50 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2003-2004,2007-2010,2019-2032,2040,2043,2045,2048].codfw.wmnet
• 19:49 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2003-2004,2007-2010,2019-2032,2040,2043,2045,2048].codfw.wmnet
• 19:30 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2145 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87566 and previous config saved to /var/cache/conftool/dbconfig/20260115-193040-marostegui.json
• 19:30 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2145.codfw.wmnet with reason: Maintenance
• 19:11 jhuneidi@deploy2002: rebuilt and synchronized wikiversions files: group2 to 1.46.0-wmf.11 refs T413802
• 18:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:53 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add new wikikube-worker nodes - pt1979@cumin2002"
• 18:53 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add new wikikube-worker nodes - pt1979@cumin2002"
• 18:50 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 18:46 pt1979@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 18:44 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 17:45 cdanis: 💙cdanis@cumin1003.eqiad.wmnet ~ 🕧☕ sudo cumin 'A:lvs-codfw or A:lvs-eqiad' 'enable-puppet T411895'
• 17:41 cdanis: 💙cdanis@cumin1003.eqiad.wmnet ~ 🕧☕ sudo cumin A:lvs-high-traffic1-codfw 'systemctl restart pybal.service'
• 17:37 cdanis: 💙cdanis@cumin1003.eqiad.wmnet ~ 🕧☕ sudo cumin A:lvs-secondary-codfw 'systemctl restart pybal.service'
• 17:34 cdanis: 💙cdanis@cumin1003.eqiad.wmnet ~ 🕧☕ sudo cumin 'A:lvs-codfw' 'disable-puppet T411895'
• 17:17 cdanis: 💙cdanis@cumin1003.eqiad.wmnet ~ 🕛☕ sudo cumin A:lvs-high-traffic1-eqiad 'systemctl restart pybal.service'
• 17:11 mutante: [cumin2002:~] $ sudo cumin -b 15 'tcp-proxy*' 'systemctl reset-failed'
• 17:10 mutante: [cumin2002:~] $ sudo cumin -b 15 'tcp-proxy*' 'rm /lib/systemd/system/prometheus-node-textfile-check-nft*'
• 17:09 cdanis: 💙cdanis@cumin1003.eqiad.wmnet ~ 🕛☕ sudo cumin A:lvs-secondary-eqiad 'systemctl restart pybal.service'
• 17:02 cdanis: 💙cdanis@cumin1003.eqiad.wmnet ~ 🕛☕ sudo cumin 'A:lvs-eqiad' 'disable-puppet T411895'
• 16:55 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:55 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns entries for arelion link ips - cmooney@cumin1003"
• 16:55 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns entries for arelion link ips - cmooney@cumin1003"
• 16:22 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2219 (T413525)', diff saved to https://phabricator.wikimedia.org/P87565 and previous config saved to /var/cache/conftool/dbconfig/20260115-162249-marostegui.json
• 16:22 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance
• 16:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T413525)', diff saved to https://phabricator.wikimedia.org/P87564 and previous config saved to /var/cache/conftool/dbconfig/20260115-162224-marostegui.json
• 16:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P87563 and previous config saved to /var/cache/conftool/dbconfig/20260115-161216-marostegui.json
• 16:07 dcausse@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
• 16:06 dcausse@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
• 16:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P87562 and previous config saved to /var/cache/conftool/dbconfig/20260115-160208-marostegui.json
• 15:53 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1263 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87561 and previous config saved to /var/cache/conftool/dbconfig/20260115-155351-marostegui.json
• 15:53 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1263.eqiad.wmnet with reason: Maintenance
• 15:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87560 and previous config saved to /var/cache/conftool/dbconfig/20260115-155326-marostegui.json
• 15:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T413525)', diff saved to https://phabricator.wikimedia.org/P87558 and previous config saved to /var/cache/conftool/dbconfig/20260115-155159-marostegui.json
• 15:47 cdanis@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3*} and A:liberica
• 15:45 cdanis@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5*} and A:liberica
• 15:45 cdanis@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3*} and A:liberica
• 15:45 dancy@deploy2002: Installation of scap version "4.232.0" completed for 2 hosts
• 15:43 cdanis@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5*} and A:liberica
• 15:43 dancy@deploy2002: Installing scap version "4.232.0" for 2 host(s)
• 15:43 cdanis@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4*} and A:liberica
• 15:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P87557 and previous config saved to /var/cache/conftool/dbconfig/20260115-154317-marostegui.json
• 15:41 cdanis@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4*} and A:liberica
• 15:33 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 15:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P87556 and previous config saved to /var/cache/conftool/dbconfig/20260115-153309-marostegui.json
• 15:28 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1199 (T413525)', diff saved to https://phabricator.wikimedia.org/P87555 and previous config saved to /var/cache/conftool/dbconfig/20260115-152817-marostegui.json
• 15:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1199.eqiad.wmnet with reason: Maintenance
• 15:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T413525)', diff saved to https://phabricator.wikimedia.org/P87554 and previous config saved to /var/cache/conftool/dbconfig/20260115-152752-marostegui.json
• 15:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1262 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87553 and previous config saved to /var/cache/conftool/dbconfig/20260115-152301-marostegui.json
• 15:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P87552 and previous config saved to /var/cache/conftool/dbconfig/20260115-151744-marostegui.json
• 15:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P87551 and previous config saved to /var/cache/conftool/dbconfig/20260115-150735-marostegui.json
• 15:06 cdanis@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6001.drmrs.wmnet} and A:liberica
• 15:06 cdanis@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6001.drmrs.wmnet} and A:liberica
• 15:05 cdanis@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6003.drmrs.wmnet} and A:liberica
• 15:05 cdanis@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6003.drmrs.wmnet} and A:liberica
• 15:05 phuedx@deploy2002: Finished scap sync-world: Backport for Enable Test Kitchen on all prod wikis (T407806) (duration: 13m 46s)
• 15:01 phuedx@deploy2002: cjming, phuedx: Continuing with sync
• 14:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T413525)', diff saved to https://phabricator.wikimedia.org/P87549 and previous config saved to /var/cache/conftool/dbconfig/20260115-145727-marostegui.json
• 14:53 phuedx@deploy2002: cjming, phuedx: Backport for Enable Test Kitchen on all prod wikis (T407806) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:51 cdanis@cumin1003: conftool action : set/pooled=yes; selector: cluster=tcp-proxy,service=gerrit
• 14:51 phuedx@deploy2002: Started scap sync-world: Backport for Enable Test Kitchen on all prod wikis (T407806)
• 14:47 sbisson@deploy2002: Finished scap sync-world: Backport for CX3 Build 1.0.0+20260114 (T413646), Fallback to source title if target title is not provided by cxserver (T414558) (duration: 08m 07s)
• 14:44 moritzm: installing net-snmp security updates
• 14:43 sbisson@deploy2002: sbisson: Continuing with sync
• 14:41 sbisson@deploy2002: sbisson: Backport for CX3 Build 1.0.0+20260114 (T413646), Fallback to source title if target title is not provided by cxserver (T414558) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:39 sbisson@deploy2002: Started scap sync-world: Backport for CX3 Build 1.0.0+20260114 (T413646), Fallback to source title if target title is not provided by cxserver (T414558)
• 14:28 jsn@deploy2002: Finished scap sync-world: Backport for Deploy PersonalDashboard to testwiki (T403982) (duration: 09m 41s)
• 14:25 btullis@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1159.eqiad.wmnet
• 14:25 btullis@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1159.eqiad.wmnet
• 14:23 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:23 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns entries for arelion link ips - cmooney@cumin1003"
• 14:22 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns entries for arelion link ips - cmooney@cumin1003"
• 14:22 jsn@deploy2002: jsn: Continuing with sync
• 14:21 jsn@deploy2002: jsn: Backport for Deploy PersonalDashboard to testwiki (T403982) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:19 jmm@deploy2002: helmfile [eqiad] DONE helmfile.d/services/proton: apply
• 14:19 jsn@deploy2002: Started scap sync-world: Backport for Deploy PersonalDashboard to testwiki (T403982)
• 14:19 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 14:18 jmm@deploy2002: helmfile [eqiad] START helmfile.d/services/proton: apply
• 14:17 jmm@deploy2002: helmfile [codfw] DONE helmfile.d/services/proton: apply
• 14:16 jmm@deploy2002: helmfile [codfw] START helmfile.d/services/proton: apply
• 14:16 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for ukwiki: Various changes to user rights. (T414277) (duration: 13m 13s)
• 14:13 jmm@deploy2002: helmfile [staging] DONE helmfile.d/services/proton: apply
• 14:11 jmm@deploy2002: helmfile [staging] START helmfile.d/services/proton: apply
• 14:11 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, seawolf35gerrit: Continuing with sync
• 14:05 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, seawolf35gerrit: Backport for ukwiki: Various changes to user rights. (T414277) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:02 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for ukwiki: Various changes to user rights. (T414277)
• 13:42 moritzm: upgrade wikidough to Bird 2.18 T413740
• 13:41 jdrewniak@deploy2002: Finished scap sync-world: Backport for Bumping portals to master (T128546) (duration: 07m 58s)
• 13:37 jdrewniak@deploy2002: jdrewniak: Continuing with sync
• 13:35 jdrewniak@deploy2002: jdrewniak: Backport for Bumping portals to master (T128546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:33 jdrewniak@deploy2002: Started scap sync-world: Backport for Bumping portals to master (T128546)
• 13:27 moritzm: installing squid security updates
• 13:27 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host mwlog1003.eqiad.wmnet with OS bookworm
• 13:25 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mwlog1003.eqiad.wmnet with OS bookworm
• 12:53 topranks: drainin Arelion transit circuit on cr1-codfw in advance of adding second 10G port to bundle
• 12:45 ihurbain@deploy2002: Finished scap sync-world: Backport for Turn on debugging for unsafe postproc cache entries logging (T412803) (duration: 08m 24s)
• 12:41 ihurbain@deploy2002: ihurbain: Continuing with sync
• 12:39 ihurbain@deploy2002: ihurbain: Backport for Turn on debugging for unsafe postproc cache entries logging (T412803) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:37 ihurbain@deploy2002: Started scap sync-world: Backport for Turn on debugging for unsafe postproc cache entries logging (T412803)
• 12:11 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2210 (T413525)', diff saved to https://phabricator.wikimedia.org/P87546 and previous config saved to /var/cache/conftool/dbconfig/20260115-121105-marostegui.json
• 12:10 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance
• 12:10 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T413525)', diff saved to https://phabricator.wikimedia.org/P87545 and previous config saved to /var/cache/conftool/dbconfig/20260115-121040-marostegui.json
• 12:00 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P87544 and previous config saved to /var/cache/conftool/dbconfig/20260115-120032-marostegui.json
• 11:52 gkyziridis@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 11:52 gkyziridis@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 11:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P87543 and previous config saved to /var/cache/conftool/dbconfig/20260115-115023-marostegui.json
• 11:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T413525)', diff saved to https://phabricator.wikimedia.org/P87542 and previous config saved to /var/cache/conftool/dbconfig/20260115-114015-marostegui.json
• 11:33 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - haproxy 2.8.18 upgrade (T414318)
• 11:29 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - haproxy 2.8.18 upgrade (T414318)
• 11:21 moritzm: installing nginx security updates
• 11:16 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host an-worker1200.eqiad.wmnet
• 11:10 jynus: force dbprov1004 restart
• 10:51 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqiad and A:cp - haproxy 2.8.18 upgrade (T414318)
• 10:51 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqiad and A:cp - haproxy 2.8.18 upgrade (T414318)
• 10:30 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1190 (T413525)', diff saved to https://phabricator.wikimedia.org/P87541 and previous config saved to /var/cache/conftool/dbconfig/20260115-103053-marostegui.json
• 10:30 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1190.eqiad.wmnet with reason: Maintenance
• 10:11 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host tcp-proxy1001.eqiad.wmnet
• 10:07 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-single for host tcp-proxy1001.eqiad.wmnet
• 10:06 dzahn@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
• 10:06 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-cluster
• 10:05 dzahn@cumin2002: END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99)
• 10:05 dzahn@cumin2002: START - Cookbook sre.hosts.reboot-cluster
• 09:58 dzahn@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on tcp-proxy1001.eqiad.wmnet with reason: remove nftables
• 09:56 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-worker1200.eqiad.wmnet
• 09:36 kharlan@deploy2002: Finished scap sync-world: Backport for WebRequest::getSecurityLogContext: Log if user is a bot (T395204) (duration: 09m 04s)
• 09:32 kharlan@deploy2002: kharlan: Continuing with sync
• 09:29 kharlan@deploy2002: kharlan: Backport for WebRequest::getSecurityLogContext: Log if user is a bot (T395204) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 09:27 kharlan@deploy2002: Started scap sync-world: Backport for WebRequest::getSecurityLogContext: Log if user is a bot (T395204)
• 08:52 hashar: https://www.wikipedia.org/ and click that orange button! # T414533
• 08:52 hashar: purged portals URLs using: `cat /srv/mediawiki-staging/portals/urls-to-purge.txt | MEDIAWIKI_STAGING_DIR=/srv/mediawiki-staging mwscript purgeList.php` # T414533
• 08:50 hashar@deploy2002: Finished scap sync-world: Backport for Update portals submodule for WP25 birthday preview [2] (T128546 T414533) (duration: 06m 42s)
• 08:45 hashar@deploy2002: hashar: Continuing with sync
• 08:45 hashar@deploy2002: hashar: Backport for Update portals submodule for WP25 birthday preview [2] (T128546 T414533) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 08:44 dzahn@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 08:44 dzahn@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 08:43 dzahn@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 08:43 dzahn@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 08:43 hashar@deploy2002: Started scap sync-world: Backport for Update portals submodule for WP25 birthday preview [2] (T128546 T414533)
• 08:43 dzahn@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 08:42 dzahn@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
• 08:40 dreamyjazz@deploy2002: Finished scap sync-world: Backport for Write new for CheckUser user agent table migration on group1 (T361196) (duration: 07m 54s)
• 08:36 dreamyjazz@deploy2002: dreamyjazz: Continuing with sync
• 08:34 dreamyjazz@deploy2002: dreamyjazz: Backport for Write new for CheckUser user agent table migration on group1 (T361196) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 08:32 dreamyjazz@deploy2002: Started scap sync-world: Backport for Write new for CheckUser user agent table migration on group1 (T361196)
• 08:30 hashar@deploy2002: Finished scap sync-world: Backport for Revert "Update portals submodule for WP25 birthday preview." (T128546 T414533) (duration: 07m 04s)
• 08:25 hashar@deploy2002: hashar: Continuing with sync
• 08:25 hashar@deploy2002: hashar: Backport for Revert "Update portals submodule for WP25 birthday preview." (T128546 T414533) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 08:22 hashar@deploy2002: Started scap sync-world: Backport for Revert "Update portals submodule for WP25 birthday preview." (T128546 T414533)
• 08:16 hashar@deploy2002: Sync cancelled.
• 08:10 hashar@deploy2002: hashar, jdrewniak, superpes: Backport for [slwiki] Fix temporary logo for Wikipedia 25 (T414265), Update portals submodule for WP25 birthday preview. (T128546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 08:07 hashar@deploy2002: Started scap sync-world: Backport for [slwiki] Fix temporary logo for Wikipedia 25 (T414265), Update portals submodule for WP25 birthday preview. (T128546)
• 07:54 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2206 (T413525)', diff saved to https://phabricator.wikimedia.org/P87540 and previous config saved to /var/cache/conftool/dbconfig/20260115-075444-marostegui.json
• 07:54 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance
• 07:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
• 07:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1169 gradually with 4 steps - After schema change
• 07:01 XioNoX: restart snmp and MIB processes on asw1-b12-drmrs - T413181
• 06:33 marostegui@cumin1003: START - Cookbook sre.mysql.pool db1169 gradually with 4 steps - After schema change
• 06:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 06:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T413525)', diff saved to https://phabricator.wikimedia.org/P87536 and previous config saved to /var/cache/conftool/dbconfig/20260115-063011-marostegui.json
• 06:29 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1169 (T413525)', diff saved to https://phabricator.wikimedia.org/P87535 and previous config saved to /var/cache/conftool/dbconfig/20260115-062902-marostegui.json
• 06:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 05:35 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1262 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87534 and previous config saved to /var/cache/conftool/dbconfig/20260115-053537-marostegui.json
• 05:35 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1262.eqiad.wmnet with reason: Maintenance
• 05:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87533 and previous config saved to /var/cache/conftool/dbconfig/20260115-053512-marostegui.json
• 05:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P87532 and previous config saved to /var/cache/conftool/dbconfig/20260115-052504-marostegui.json
• 05:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P87530 and previous config saved to /var/cache/conftool/dbconfig/20260115-051455-marostegui.json
• 05:04 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1261 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87529 and previous config saved to /var/cache/conftool/dbconfig/20260115-050448-marostegui.json
• 04:33 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2199.codfw.wmnet with reason: Maintenance
• 04:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T413525)', diff saved to https://phabricator.wikimedia.org/P87528 and previous config saved to /var/cache/conftool/dbconfig/20260115-043242-marostegui.json
• 04:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P87527 and previous config saved to /var/cache/conftool/dbconfig/20260115-042233-marostegui.json
• 04:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P87526 and previous config saved to /var/cache/conftool/dbconfig/20260115-041225-marostegui.json
• 04:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T413525)', diff saved to https://phabricator.wikimedia.org/P87525 and previous config saved to /var/cache/conftool/dbconfig/20260115-040216-marostegui.json
• 01:33 wfan: payments-wiki revision is a4102ce7
• 01:13 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 12m 57s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image
• 00:57 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster
• 00:10 eileen: civicrm upgraded from 3d70b84d to 6775b0eb
• 00:09 zabe@deploy2002: Finished scap sync-world: Backport for Start reading from il_target_id on testwiki (T413669) (duration: 08m 13s)
• 00:05 zabe@deploy2002: zabe: Continuing with sync
• 00:03 zabe@deploy2002: zabe: Backport for Start reading from il_target_id on testwiki (T413669) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 00:01 zabe@deploy2002: Started scap sync-world: Backport for Start reading from il_target_id on testwiki (T413669)

2026-01-14

• 23:56 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2172 (T413525)', diff saved to https://phabricator.wikimedia.org/P87524 and previous config saved to /var/cache/conftool/dbconfig/20260114-235619-marostegui.json
• 23:56 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
• 23:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T413525)', diff saved to https://phabricator.wikimedia.org/P87523 and previous config saved to /var/cache/conftool/dbconfig/20260114-235606-marostegui.json
• 23:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P87522 and previous config saved to /var/cache/conftool/dbconfig/20260114-234557-marostegui.json
• 23:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P87521 and previous config saved to /var/cache/conftool/dbconfig/20260114-233549-marostegui.json
• 23:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T413525)', diff saved to https://phabricator.wikimedia.org/P87520 and previous config saved to /var/cache/conftool/dbconfig/20260114-232541-marostegui.json
• 23:25 zabe: zabe@deploy2002:~$ mwscript migrateLinksTable.php testwiki --table imagelinks # T413668
• 23:03 rzl@deploy2002: helmfile [codfw] DONE helmfile.d/services/mw-cron: apply
• 23:02 rzl@deploy2002: helmfile [codfw] START helmfile.d/services/mw-cron: apply
• 22:26 ryankemper@cumin2002: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
• 22:22 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster
• 22:12 jeena: Updating development images on contint primary for T412259
• 22:06 jhuneidi@deploy2002: Finished scap sync-world: Backport for Revert "Do not use deprecated menu" (T414630) (duration: 09m 16s)
• 22:01 jhuneidi@deploy2002: jhuneidi, jdlrobson: Continuing with sync
• 21:59 jhuneidi@deploy2002: jhuneidi, jdlrobson: Backport for Revert "Do not use deprecated menu" (T414630) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:56 jhuneidi@deploy2002: Started scap sync-world: Backport for Revert "Do not use deprecated menu" (T414630)
• 21:51 zabe@deploy2002: Finished scap sync-world: Backport for Start writing to il_target_id on non-large wikis (T413526) (duration: 06m 30s)
• 21:47 zabe@deploy2002: zabe: Continuing with sync
• 21:47 zabe@deploy2002: zabe: Backport for Start writing to il_target_id on non-large wikis (T413526) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:45 zabe@deploy2002: Started scap sync-world: Backport for Start writing to il_target_id on non-large wikis (T413526)
• 21:41 cdanis@cumin1003: conftool action : set/pooled=yes; selector: service=gerrit,dc=magru
• 21:40 zabe@deploy2002: Finished scap sync-world: Backport for [itwiki] Add a temporary logo for Wikipedia 25 (T414320), [slwiki] Add a temporary logo for Wikipedia 25 (T414265), [kkwiki] Add a temporary logo for Wikipedia 25 (T414267) (duration: 06m 36s)
• 21:36 cdanis@cumin1003: conftool action : set/weight=1; selector: service=gerrit
• 21:36 zabe@deploy2002: zabe, superpes: Continuing with sync
• 21:36 zabe@deploy2002: zabe, superpes: Backport for [itwiki] Add a temporary logo for Wikipedia 25 (T414320), [slwiki] Add a temporary logo for Wikipedia 25 (T414265), [kkwiki] Add a temporary logo for Wikipedia 25 (T414267) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:34 zabe@deploy2002: Started scap sync-world: Backport for [itwiki] Add a temporary logo for Wikipedia 25 (T414320), [slwiki] Add a temporary logo for Wikipedia 25 (T414265), [kkwiki] Add a temporary logo for Wikipedia 25 (T414267)
• 21:33 zabe@deploy2002: Finished scap sync-world: Backport for Revert "Shorten 'close' cookie wait period for enwiki banners" (T411800) (duration: 08m 40s)
• 21:29 zabe@deploy2002: zabe, ejegg: Continuing with sync
• 21:26 zabe@deploy2002: zabe, ejegg: Backport for Revert "Shorten 'close' cookie wait period for enwiki banners" (T411800) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:24 zabe@deploy2002: Started scap sync-world: Backport for Revert "Shorten 'close' cookie wait period for enwiki banners" (T411800)
• 21:22 taavi: manually purge 17 URLs for enwiki and zhwiki 25 year anniversary logos
• 21:17 cdanis@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs7001.magru.wmnet} and A:liberica
• 21:17 cdanis@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs7001.magru.wmnet} and A:liberica
• 21:16 zabe@deploy2002: Finished scap sync-world: Backport for enwiki: change to Wikipedia 25 logo (T414271), zhwiki: Temporary Logo Change for WP25 (T414299) (duration: 09m 03s)
• 21:11 zabe@deploy2002: chlod, zhaofjx, zabe: Continuing with sync
• 21:11 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1004.eqiad.wmnet with OS trixie
• 21:10 cdanis@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs7003.magru.wmnet} and A:liberica
• 21:09 cdanis@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs7003.magru.wmnet} and A:liberica
• 21:09 zabe@deploy2002: chlod, zhaofjx, zabe: Backport for enwiki: change to Wikipedia 25 logo (T414271), zhwiki: Temporary Logo Change for WP25 (T414299) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:07 zabe@deploy2002: Started scap sync-world: Backport for enwiki: change to Wikipedia 25 logo (T414271), zhwiki: Temporary Logo Change for WP25 (T414299)
• 20:43 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1004.eqiad.wmnet with reason: host reimage
• 20:36 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1004.eqiad.wmnet with reason: host reimage
• 20:30 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host mwlog1003.eqiad.wmnet with OS bookworm
• 20:29 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mwlog1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 20:23 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS trixie
• 20:22 cdanis: 💔cdanis@cumin1003.eqiad.wmnet ~ 🕞🍵 sudo cumin 'A:cp-text' 'disable-puppet "cdanis deploy Ie99c64c48d"'
• 20:21 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudbackup1004.eqiad.wmnet with OS trixie
• 20:06 jclark@cumin1003: START - Cookbook sre.hosts.provision for host mwlog1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 19:14 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup1004.eqiad.wmnet with OS trixie
• 19:12 cmooney@dns2005: END - running authdns-update
• 19:11 cmooney@dns2005: START - running authdns-update
• 19:11 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 19:10 jhuneidi@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.11 refs T413802
• 19:08 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 19:08 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 19:08 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox entries beofre dns patch - cmooney@cumin1003"
• 19:08 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox entries beofre dns patch - cmooney@cumin1003"
• 19:07 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1261 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87518 and previous config saved to /var/cache/conftool/dbconfig/20260114-190711-marostegui.json
• 19:07 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1261.eqiad.wmnet with reason: Maintenance
• 19:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87517 and previous config saved to /var/cache/conftool/dbconfig/20260114-190647-marostegui.json
• 19:04 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 19:01 pt1979@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mr1-ulsfo,mr1-ulsfo IPv6 with reason: loopback IPV4 change on ulsfo core router
• 19:00 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 19:00 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox entries beofre dns patch - cmooney@cumin1003"
• 19:00 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox entries beofre dns patch - cmooney@cumin1003"
• 19:00 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2155 (T413525)', diff saved to https://phabricator.wikimedia.org/P87516 and previous config saved to /var/cache/conftool/dbconfig/20260114-190033-marostegui.json
• 19:00 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
• 19:00 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T413525)', diff saved to https://phabricator.wikimedia.org/P87515 and previous config saved to /var/cache/conftool/dbconfig/20260114-190008-marostegui.json
• 18:59 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup2004.codfw.wmnet with OS trixie
• 18:57 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 18:56 cmooney@cumin1003: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97)
• 18:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P87514 and previous config saved to /var/cache/conftool/dbconfig/20260114-185638-marostegui.json
• 18:56 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 18:55 cmooney@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 18:55 cmooney@cumin1003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox entries beofre dns patch - cmooney@cumin1003"
• 18:55 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update netbox entries beofre dns patch - cmooney@cumin1003"
• 18:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P87513 and previous config saved to /var/cache/conftool/dbconfig/20260114-185001-marostegui.json
• 18:49 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 18:48 cmooney@cumin1003: END (ERROR) - Cookbook sre.dns.netbox (exit_code=97)
• 18:48 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 18:47 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:46 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260', diff saved to https://phabricator.wikimedia.org/P87512 and previous config saved to /var/cache/conftool/dbconfig/20260114-184630-marostegui.json
• 18:44 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 18:42 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:42 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update cr3-cr4-ulsfo loopback - pt1979@cumin2002"
• 18:42 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update cr3-cr4-ulsfo loopback - pt1979@cumin2002"
• 18:41 cmooney@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 18:40 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host an-worker1148.eqiad.wmnet
• 18:39 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P87511 and previous config saved to /var/cache/conftool/dbconfig/20260114-183951-marostegui.json
• 18:38 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 18:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1260 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87509 and previous config saved to /var/cache/conftool/dbconfig/20260114-183621-marostegui.json
• 18:33 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 18:33 cmooney@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 18:29 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 18:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T413525)', diff saved to https://phabricator.wikimedia.org/P87508 and previous config saved to /var/cache/conftool/dbconfig/20260114-182942-marostegui.json
• 18:21 cmooney@dns2005: START - running authdns-update
• 18:16 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:16 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update cr3-cr4-ulsfo loopback - pt1979@cumin2002"
• 18:16 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update cr3-cr4-ulsfo loopback - pt1979@cumin2002"
• 18:05 cdobbins@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site ulsfo [reason: switch work, T408510]
• 18:05 cdobbins@cumin1003: START - Cookbook sre.dns.admin DNS admin: pool site ulsfo [reason: switch work, T408510]
• 18:01 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 17:55 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:53 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:53 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update cr3-cr4-ulsfo loopback - pt1979@cumin2002"
• 17:52 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update cr3-cr4-ulsfo loopback - pt1979@cumin2002"
• 17:52 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 17:49 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup2004.codfw.wmnet with reason: host reimage
• 17:43 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 17:43 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup2004.codfw.wmnet with reason: host reimage
• 17:34 btullis@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1148.eqiad.wmnet
• 17:34 btullis@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1148.eqiad.wmnet
• 17:33 btullis@cumin1003: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-worker1148.eqiad.wmnet
• 17:33 btullis@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-worker1148.eqiad.wmnet
• 17:29 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4008*} and A:liberica (T408510)
• 17:29 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4008*} and A:liberica (T408510)
• 17:29 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4009*} and A:liberica (T408510)
• 17:29 btullis@deploy2002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply
• 17:29 btullis@deploy2002: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply
• 17:28 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4009*} and A:liberica (T408510)
• 17:28 btullis@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply
• 17:27 btullis@deploy2002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply
• 17:27 btullis@deploy2002: helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply
• 17:27 btullis@deploy2002: helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply
• 17:26 sukhe@cumin1003: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs4010*} and A:liberica (T408510)
• 17:26 sukhe@cumin1003: START - Cookbook sre.loadbalancer.admin config_reloading P{lvs4010*} and A:liberica (T408510)
• 17:24 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup2004.codfw.wmnet with OS trixie
• 17:20 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-worker1148.eqiad.wmnet
• 17:14 dzahn@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 17:14 dzahn@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 17:14 tappof: pool titan1002 (T411273)
• 17:13 tappof: pool titan1002
• 17:13 dzahn@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 17:13 dzahn@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 17:12 dzahn@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 17:12 dzahn@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
• 16:55 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1013.eqiad.wmnet
• 16:54 herron@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM grafana2001.codfw.wmnet
• 16:50 herron@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM grafana2001.codfw.wmnet
• 16:49 herron@cumin1003: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM grafana1002.eqiad.wmnet
• 16:48 brouberol@cumin1003: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1013.eqiad.wmnet
• 16:44 stran@deploy2002: Finished scap sync-world: Backport for Only validate IRS configs on writes; skip validations for reads (T414582), Only validate IRS configs on writes; skip validations for reads (T414582) (duration: 08m 59s)
• 16:44 herron@cumin1003: START - Cookbook sre.ganeti.reboot-vm for VM grafana1002.eqiad.wmnet
• 16:42 herron: restarting grafana1002 for memory increase T414604
• 16:40 stran@deploy2002: dreamyjazz, stran: Continuing with sync
• 16:38 stran@deploy2002: dreamyjazz, stran: Backport for Only validate IRS configs on writes; skip validations for reads (T414582), Only validate IRS configs on writes; skip validations for reads (T414582) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 16:35 stran@deploy2002: Started scap sync-world: Backport for Only validate IRS configs on writes; skip validations for reads (T414582), Only validate IRS configs on writes; skip validations for reads (T414582)
• 16:27 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2003-dev.codfw.wmnet with OS trixie
• 16:21 fnegri@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0)
• 16:18 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_esams and A:cp - haproxy 2.8.18 upgrade (T414318)
• 16:14 fnegri@cumin1003: START - Cookbook sre.wikireplicas.update-views
• 16:09 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
• 16:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87507 and previous config saved to /var/cache/conftool/dbconfig/20260114-160836-marostegui.json
• 16:07 papaul: ongoing loopback ip's change on cr3/cr4-ulsfo
• 16:06 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2003-dev.codfw.wmnet with reason: host reimage
• 16:05 pt1979@cumin2002: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 8 hosts with reason: loopback IPV4 change on ulsfo core router
• 16:04 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - haproxy 2.8.18 upgrade (T414318)
• 15:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P87506 and previous config saved to /var/cache/conftool/dbconfig/20260114-155828-marostegui.json
• 15:49 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudgw2003-dev.codfw.wmnet with OS trixie
• 15:49 XioNoX: drain eqsin-ulsfo transport
• 15:48 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248', diff saved to https://phabricator.wikimedia.org/P87505 and previous config saved to /var/cache/conftool/dbconfig/20260114-154820-marostegui.json
• 15:43 cdobbins@cumin1003: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site ulsfo [reason: switch work, T408510]
• 15:43 cdobbins@cumin1003: START - Cookbook sre.dns.admin DNS admin: depool site ulsfo [reason: switch work, T408510]
• 15:38 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2248 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87504 and previous config saved to /var/cache/conftool/dbconfig/20260114-153811-marostegui.json
• 15:33 dcausse@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
• 15:32 dcausse@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
• 15:30 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_esams and A:cp - haproxy 2.8.18 upgrade (T414318)
• 15:20 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - haproxy 2.8.18 upgrade (T414318)
• 15:19 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
• 15:19 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
• 15:17 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_esams and A:cp - haproxy 2.8.18 upgrade (T414318)
• 15:15 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - haproxy 2.8.18 upgrade (T414318)
• 15:05 jforrester@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 15:05 jforrester@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 15:04 jforrester@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 15:04 jforrester@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 15:04 jforrester@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 15:03 jforrester@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:49 Lucas_WMDE: UTC afternoon backport+config window done
• 14:49 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for Deploy TestKitchen to testwiki (T407806) (duration: 09m 49s)
• 14:45 lucaswerkmeister-wmde@deploy2002: cjming, lucaswerkmeister-wmde: Continuing with sync
• 14:42 lucaswerkmeister-wmde@deploy2002: cjming, lucaswerkmeister-wmde: Backport for Deploy TestKitchen to testwiki (T407806) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:39 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for Deploy TestKitchen to testwiki (T407806)
• 14:36 tgr@deploy2002: Finished scap sync-world: Backport for debug: Add some CDN Backend API headers to Logstash (T412396) (duration: 10m 21s)
• 14:32 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "State, commit to database - oblivian@cumin1003"
• 14:32 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: State, commit to database - oblivian@cumin1003
• 14:32 tgr@deploy2002: tgr: Continuing with sync
• 14:31 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: State, commit to database - oblivian@cumin1003
• 14:31 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "State, commit to database - oblivian@cumin1003"
• 14:28 tgr@deploy2002: tgr: Backport for debug: Add some CDN Backend API headers to Logstash (T412396) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:27 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_drmrs and A:cp - haproxy 2.8.18 upgrade (T414318)
• 14:27 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_drmrs and A:cp - haproxy 2.8.18 upgrade (T414318)
• 14:25 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - haproxy 2.8.18 upgrade (T414318)
• 14:25 tgr@deploy2002: Started scap sync-world: Backport for debug: Add some CDN Backend API headers to Logstash (T412396)
• 14:21 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_eqsin and not P{cp5022.*} and A:cp - haproxy 2.8.18 upgrade (T414318)
• 14:17 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2002-dev.codfw.wmnet with OS trixie
• 14:11 jsn@deploy2002: Finished scap sync-world: Backport for InitialiseSettings.php: Add wmgUsePersonalDashboard (T412528), InitialiseSettings-labs.php: Deploy PersonalDashboard (T412528), CommonSettings-labs: Load PersonalDashbard extension (T412528) (duration: 07m 19s)
• 14:07 jsn@deploy2002: jsn: Continuing with sync
• 14:06 jsn@deploy2002: jsn: Backport for InitialiseSettings.php: Add wmgUsePersonalDashboard (T412528), InitialiseSettings-labs.php: Deploy PersonalDashboard (T412528), CommonSettings-labs: Load PersonalDashbard extension (T412528) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:04 jsn@deploy2002: Started scap sync-world: Backport for InitialiseSettings.php: Add wmgUsePersonalDashboard (T412528), InitialiseSettings-labs.php: Deploy PersonalDashboard (T412528), CommonSettings-labs: Load PersonalDashbard extension (T412528)
• 13:59 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
• 13:58 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2147 (T413525)', diff saved to https://phabricator.wikimedia.org/P87502 and previous config saved to /var/cache/conftool/dbconfig/20260114-135815-marostegui.json
• 13:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
• 13:55 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw2002-dev.codfw.wmnet with reason: host reimage
• 13:44 dreamyjazz@deploy2002: Finished scap sync-world: Backport for Write new for CheckUser user agent table migration on group0 (T361196) (duration: 09m 24s)
• 13:39 dreamyjazz@deploy2002: dreamyjazz: Continuing with sync
• 13:38 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudgw2002-dev.codfw.wmnet with OS trixie
• 13:36 dreamyjazz@deploy2002: dreamyjazz: Backport for Write new for CheckUser user agent table migration on group0 (T361196) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:34 dreamyjazz@deploy2002: Started scap sync-world: Backport for Write new for CheckUser user agent table migration on group0 (T361196)
• 13:30 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and not P{cp5022.*} and A:cp - haproxy 2.8.18 upgrade (T414318)
• 13:30 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - haproxy 2.8.18 upgrade (T414318)
• 13:10 elukey@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry1004.eqiad.wmnet with reason: testing
• 12:37 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_codfw and A:cp - haproxy 2.8.18 upgrade (T414318)
• 12:28 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - haproxy 2.8.18 upgrade (T414318)
• 11:53 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_codfw and A:cp - haproxy 2.8.18 upgrade (T414318)
• 11:45 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_codfw and A:cp - haproxy 2.8.18 upgrade (T414318)
• 11:20 dpogorzelski@cumin1003: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for registry[2004-2005].codfw.wmnet,registry[1004-1005].eqiad.wmnet
• 11:20 dpogorzelski@cumin1003: START - Cookbook sre.hosts.remove-downtime for registry[2004-2005].codfw.wmnet,registry[1004-1005].eqiad.wmnet
• 10:55 dpogorzelski@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on registry[2004-2005].codfw.wmnet,registry[1004-1005].eqiad.wmnet with reason: testing ml changes
• 09:43 XioNoX: configure Arelion LAG on cr1-codfw - T401100
• 09:24 tappof: Depool titan1002; disable Puppet and enable debug log level (T411273)
• 08:43 kart_: Update cxserver to 2026-01-09-231405-production (T414237, T413646, T409998)
• 08:33 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1260 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87499 and previous config saved to /var/cache/conftool/dbconfig/20260114-083332-marostegui.json
• 08:33 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1260.eqiad.wmnet with reason: Maintenance
• 08:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87498 and previous config saved to /var/cache/conftool/dbconfig/20260114-083307-marostegui.json
• 08:31 kartik@deploy2002: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
• 08:30 kartik@deploy2002: helmfile [eqiad] START helmfile.d/services/cxserver: apply
• 08:30 kartik@deploy2002: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
• 08:29 kartik@deploy2002: helmfile [codfw] START helmfile.d/services/cxserver: apply
• 08:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P87497 and previous config saved to /var/cache/conftool/dbconfig/20260114-082259-marostegui.json
• 08:22 kartik@deploy2002: helmfile [staging] DONE helmfile.d/services/cxserver: apply
• 08:20 kartik@deploy2002: helmfile [staging] START helmfile.d/services/cxserver: apply
• 08:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P87496 and previous config saved to /var/cache/conftool/dbconfig/20260114-081251-marostegui.json
• 08:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87495 and previous config saved to /var/cache/conftool/dbconfig/20260114-080242-marostegui.json
• 07:33 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2248 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87494 and previous config saved to /var/cache/conftool/dbconfig/20260114-073321-marostegui.json
• 07:33 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2248.codfw.wmnet with reason: Maintenance
• 07:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87493 and previous config saved to /var/cache/conftool/dbconfig/20260114-073256-marostegui.json
• 07:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P87492 and previous config saved to /var/cache/conftool/dbconfig/20260114-072248-marostegui.json
• 07:17 marostegui@cumin1003: END (PASS) - Cookbook sre.wikireplicas.update-views (exit_code=0)
• 07:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247', diff saved to https://phabricator.wikimedia.org/P87491 and previous config saved to /var/cache/conftool/dbconfig/20260114-071240-marostegui.json
• 07:10 marostegui@cumin1003: START - Cookbook sre.wikireplicas.update-views
• 07:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2247 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87490 and previous config saved to /var/cache/conftool/dbconfig/20260114-070230-marostegui.json
• 01:14 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 12m 54s)
• 01:01 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image
• 00:40 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1370.eqiad.wmnet with OS trixie
• 00:40 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 00:40 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 00:24 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1370.eqiad.wmnet with reason: host reimage
• 00:20 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1370.eqiad.wmnet with reason: host reimage
• 00:15 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1372.eqiad.wmnet with OS trixie
• 00:15 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 00:15 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 00:08 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1370.eqiad.wmnet with OS trixie
• 00:08 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 00:01 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART

2026-01-13

• 23:58 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1372.eqiad.wmnet with reason: host reimage
• 23:55 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1353.eqiad.wmnet with OS bookworm
• 23:55 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 23:55 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 23:52 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1372.eqiad.wmnet with reason: host reimage
• 23:50 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 23:49 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 23:45 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1370
• 23:45 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1370
• 23:42 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 23:40 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1372.eqiad.wmnet with OS trixie
• 23:39 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 23:39 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1372.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 23:39 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 23:38 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1353.eqiad.wmnet with reason: host reimage
• 23:38 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 23:33 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1353.eqiad.wmnet with reason: host reimage
• 23:32 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1372.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 23:31 zabe@deploy2002: Finished scap sync-world: Backport for manage-dblist: Improve generation of db-sections.php (duration: 09m 10s)
• 23:30 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1353.eqiad.wmnet with OS bookworm
• 23:28 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
• 23:27 zabe@deploy2002: zabe: Continuing with sync
• 23:25 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1366.eqiad.wmnet with OS trixie
• 23:24 zabe@deploy2002: zabe: Backport for manage-dblist: Improve generation of db-sections.php synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 23:23 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1365.eqiad.wmnet with OS trixie
• 23:23 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 23:22 zabe@deploy2002: Started scap sync-world: Backport for manage-dblist: Improve generation of db-sections.php
• 23:19 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 23:08 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1366.eqiad.wmnet with reason: host reimage
• 23:04 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1366.eqiad.wmnet with reason: host reimage
• 23:03 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1365.eqiad.wmnet with reason: host reimage
• 23:01 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1371.eqiad.wmnet with OS trixie
• 23:01 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 23:01 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 22:57 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1365.eqiad.wmnet with reason: host reimage
• 22:55 ryankemper@cumin2002: START - Cookbook sre.wdqs.restart
• 22:52 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1366.eqiad.wmnet with OS trixie
• 22:46 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1365.eqiad.wmnet with OS trixie
• 22:45 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1363.eqiad.wmnet with OS trixie
• 22:45 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 22:45 zabe@deploy2002: Finished scap sync-world: Backport for Stop setting import source for crwiki (T411501) (duration: 06m 24s)
• 22:44 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1371.eqiad.wmnet with reason: host reimage
• 22:43 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 22:41 zabe@deploy2002: zabe: Continuing with sync
• 22:40 zabe@deploy2002: zabe: Backport for Stop setting import source for crwiki (T411501) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:38 zabe@deploy2002: Started scap sync-world: Backport for Stop setting import source for crwiki (T411501)
• 22:38 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1371.eqiad.wmnet with reason: host reimage
• 22:28 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1363.eqiad.wmnet with reason: host reimage
• 22:26 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1371.eqiad.wmnet with OS trixie
• 22:24 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1363.eqiad.wmnet with reason: host reimage
• 22:17 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:15 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1365.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:14 zabe@deploy2002: Finished scap sync-world: Backport for Start writing to il_target_id on group0 wikis (T413526) (duration: 06m 55s)
• 22:13 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1252 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87489 and previous config saved to /var/cache/conftool/dbconfig/20260113-221333-marostegui.json
• 22:13 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance
• 22:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87488 and previous config saved to /var/cache/conftool/dbconfig/20260113-221309-marostegui.json
• 22:12 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1363.eqiad.wmnet with OS trixie
• 22:12 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1363.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:11 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1363.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:10 zabe@deploy2002: zabe: Continuing with sync
• 22:09 zabe@deploy2002: zabe: Backport for Start writing to il_target_id on group0 wikis (T413526) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:09 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:08 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1365.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:07 zabe@deploy2002: Started scap sync-world: Backport for Start writing to il_target_id on group0 wikis (T413526)
• 22:06 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:05 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 22:04 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1365
• 22:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P87487 and previous config saved to /var/cache/conftool/dbconfig/20260113-220300-marostegui.json
• 22:01 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1365
• 22:00 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 22:00 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1365 - vriley@cumin1003"
• 22:00 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1365 - vriley@cumin1003"
• 21:57 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:55 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:55 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 21:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P87486 and previous config saved to /var/cache/conftool/dbconfig/20260113-215252-marostegui.json
• 21:52 cjming: end of UTC late backport window
• 21:51 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 21:51 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1352.eqiad.wmnet with OS bookworm
• 21:51 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 21:50 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 21:48 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 21:48 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1371
• 21:48 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1371
• 21:48 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:47 cjming@deploy2002: Finished scap sync-world: Backport for Urwikiquote: Update logo (T413592) (duration: 08m 47s)
• 21:43 cjming@deploy2002: pppery, cjming: Continuing with sync
• 21:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87485 and previous config saved to /var/cache/conftool/dbconfig/20260113-214244-marostegui.json
• 21:41 cjming@deploy2002: pppery, cjming: Backport for Urwikiquote: Update logo (T413592) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:40 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:39 cjming@deploy2002: Started scap sync-world: Backport for Urwikiquote: Update logo (T413592)
• 21:36 cjming@deploy2002: mwscript-k8s job started: namespaceDupes siwiki --fix # T414159
• 21:35 cjming@deploy2002: Finished scap sync-world: Backport for Siwiki: Add MOS namespace (T414159) (duration: 07m 49s)
• 21:34 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1352.eqiad.wmnet with reason: host reimage
• 21:33 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1372.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:32 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1372.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:31 cjming@deploy2002: pppery, cjming: Continuing with sync
• 21:30 cjming@deploy2002: pppery, cjming: Backport for Siwiki: Add MOS namespace (T414159) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:28 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1352.eqiad.wmnet with reason: host reimage
• 21:27 cjming@deploy2002: Started scap sync-world: Backport for Siwiki: Add MOS namespace (T414159)
• 21:25 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1352.eqiad.wmnet with OS bookworm
• 21:24 dani@deploy2002: Finished scap sync-world: Backport for Undeploy Safety survey (T413022) (duration: 06m 59s)
• 21:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2247 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87484 and previous config saved to /var/cache/conftool/dbconfig/20260113-212400-marostegui.json
• 21:23 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2247.codfw.wmnet with reason: Maintenance
• 21:23 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1353.eqiad.wmnet with OS bookworm
• 21:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87483 and previous config saved to /var/cache/conftool/dbconfig/20260113-212333-marostegui.json
• 21:21 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1355.eqiad.wmnet with OS bookworm
• 21:21 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 21:21 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1353.eqiad.wmnet with OS bookworm
• 21:21 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1352.eqiad.wmnet with OS bookworm
• 21:20 dani@deploy2002: dani: Continuing with sync
• 21:20 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 21:20 dani@deploy2002: dani: Backport for Undeploy Safety survey (T413022) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:17 dani@deploy2002: Started scap sync-world: Backport for Undeploy Safety survey (T413022)
• 21:16 cjming@deploy2002: Finished scap sync-world: Backport for Update description of the Math API (T411517), Replaced mpic-next.wikimedia.org with test-kitchen-next.wikimedia.org (T407805) (duration: 09m 18s)
• 21:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P87482 and previous config saved to /var/cache/conftool/dbconfig/20260113-211325-marostegui.json
• 21:12 cjming@deploy2002: aaron, cjming, sfaci: Continuing with sync
• 21:11 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1352.eqiad.wmnet with OS bookworm
• 21:11 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1353.eqiad.wmnet with OS bookworm
• 21:09 cjming@deploy2002: aaron, cjming, sfaci: Backport for Update description of the Math API (T411517), Replaced mpic-next.wikimedia.org with test-kitchen-next.wikimedia.org (T407805) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:06 cjming@deploy2002: Started scap sync-world: Backport for Update description of the Math API (T411517), Replaced mpic-next.wikimedia.org with test-kitchen-next.wikimedia.org (T407805)
• 21:04 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1355.eqiad.wmnet with reason: host reimage
• 21:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246', diff saved to https://phabricator.wikimedia.org/P87481 and previous config saved to /var/cache/conftool/dbconfig/20260113-210317-marostegui.json
• 21:02 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1352.eqiad.wmnet with OS bookworm
• 21:01 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1353.eqiad.wmnet with OS bookworm
• 20:58 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1355.eqiad.wmnet with reason: host reimage
• 20:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2246 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87480 and previous config saved to /var/cache/conftool/dbconfig/20260113-205308-marostegui.json
• 20:49 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1355
• 20:48 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1355
• 20:48 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1353
• 20:48 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1352.eqiad.wmnet with OS bookworm
• 20:48 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1353
• 20:48 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1352
• 20:48 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1352
• 20:47 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 20:47 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt WIKIKUBE - jclark@cumin1003"
• 20:47 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt WIKIKUBE - jclark@cumin1003"
• 20:47 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1355.eqiad.wmnet with OS bookworm
• 20:47 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1353.eqiad.wmnet with OS bookworm
• 20:44 jclark@cumin1003: START - Cookbook sre.dns.netbox
• 19:43 jhuneidi@deploy2002: Finished scap sync-world: test sync for T411516 (duration: 02m 38s)
• 19:40 jhuneidi@deploy2002: Started scap sync-world: test sync for T411516
• 19:22 jhuneidi@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.11 refs T413802
• 16:41 elukey: roll restart docker-registry-swift daemons on registry* to pick up the new settings (apparently the service refresh issued by puppet didn't work as intended) - T390251
• 16:38 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1355.eqiad.wmnet with OS bookworm
• 16:28 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
• 16:27 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
• 16:27 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
• 16:27 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
• 16:26 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
• 16:21 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
• 16:20 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 16:20 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 16:11 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/services/ratelimit: apply
• 16:11 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1358.eqiad.wmnet with OS bookworm
• 16:11 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 16:10 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 16:10 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/services/ratelimit: apply
• 16:10 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/services/ratelimit: apply
• 16:09 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/services/ratelimit: apply
• 16:09 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/ratelimit: apply
• 16:07 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/ratelimit: apply
• 16:07 brennen@deploy2002: Finished deploy [phabricator/deployment@f12e2e1]: deploy phab1004 for T414479 (duration: 02m 09s)
• 16:05 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1347.eqiad.wmnet with OS bookworm
• 16:05 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 16:05 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 16:04 brennen@deploy2002: Started deploy [phabricator/deployment@f12e2e1]: deploy phab1004 for T414479
• 16:04 brennen@deploy2002: Finished deploy [phabricator/deployment@f12e2e1]: deploy phab2002 for T414479 (duration: 00m 31s)
• 16:03 brennen@deploy2002: Started deploy [phabricator/deployment@f12e2e1]: deploy phab2002 for T414479
• 16:03 arnaudb@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:45:00 on phab2002.codfw.wmnet,phab[1004-1005].eqiad.wmnet with reason: T414479
• 16:02 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/services/api-gateway: apply
• 16:02 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/services/api-gateway: apply
• 16:00 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply
• 15:59 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/services/api-gateway: apply
• 15:57 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1356.eqiad.wmnet with OS bookworm
• 15:57 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:57 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:57 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - haproxy 2.8.18 upgrade (T414318)
• 15:56 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
• 15:56 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/api-gateway: apply
• 15:55 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1357.eqiad.wmnet with OS bookworm
• 15:55 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:54 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - haproxy 2.8.18 upgrade (T414318)
• 15:54 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:53 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1358.eqiad.wmnet with reason: host reimage
• 15:51 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1354.eqiad.wmnet with OS bookworm
• 15:51 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:51 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:48 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1347.eqiad.wmnet with reason: host reimage
• 15:46 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1359.eqiad.wmnet with OS bookworm
• 15:46 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:46 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:41 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1347.eqiad.wmnet with reason: host reimage
• 15:41 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1356.eqiad.wmnet with reason: host reimage
• 15:37 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1357.eqiad.wmnet with reason: host reimage
• 15:36 sfaci@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply
• 15:36 sfaci@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply
• 15:36 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 15:35 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 15:33 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1354.eqiad.wmnet with reason: host reimage
• 15:29 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1347.eqiad.wmnet with OS bookworm
• 15:29 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1359.eqiad.wmnet with reason: host reimage
• 15:29 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1356.eqiad.wmnet with reason: host reimage
• 15:28 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1354.eqiad.wmnet with reason: host reimage
• 15:24 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool db2196: Schema change
• 15:22 sfaci@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
• 15:22 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1358.eqiad.wmnet with reason: host reimage
• 15:22 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1357.eqiad.wmnet with reason: host reimage
• 15:22 sfaci@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
• 15:22 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1359.eqiad.wmnet with reason: host reimage
• 15:20 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/api-gateway: apply
• 15:20 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/api-gateway: apply
• 15:19 moritzm: upgrade durum* to Bird 2.18 T413740
• 15:18 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1356.eqiad.wmnet with OS bookworm
• 15:18 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1355.eqiad.wmnet with OS bookworm
• 15:17 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1354.eqiad.wmnet with OS bookworm
• 15:16 marostegui: Deploy schema change on x1 master to fix T414474
• 15:13 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1352.eqiad.wmnet with OS trixie
• 15:13 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:13 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:11 moritzm: revoked legacy restbase discovery certificate T365798
• 15:11 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1357.eqiad.wmnet with OS bookworm
• 15:11 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1358.eqiad.wmnet with OS bookworm
• 15:11 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1359.eqiad.wmnet with OS bookworm
• 15:10 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1348.eqiad.wmnet with OS trixie
• 15:10 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:10 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - haproxy 2.8.18 upgrade (T414318)
• 15:10 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - haproxy 2.8.18 upgrade (T414318)
• 15:08 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:06 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1351.eqiad.wmnet with OS trixie
• 15:06 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:06 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 15:03 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm7001.magru.wmnet with OS bookworm
• 14:59 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1349.eqiad.wmnet with OS trixie
• 14:59 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:58 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:55 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1352.eqiad.wmnet with reason: host reimage
• 14:55 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1350.eqiad.wmnet with OS trixie
• 14:55 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:54 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:52 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1338.eqiad.wmnet with OS trixie
• 14:52 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:52 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1348.eqiad.wmnet with reason: host reimage
• 14:50 ayounsi@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
• 14:50 ayounsi@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
• 14:50 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:49 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1351.eqiad.wmnet with reason: host reimage
• 14:45 jclark@cumin1003: END (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 2:00:00 on wikikube-worker1347.eqiad.wmnet with reason: host reimage
• 14:45 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm7001.magru.wmnet with reason: host reimage
• 14:44 ayounsi@cumin1003: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
• 14:44 ayounsi@cumin1003: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
• 14:42 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7008.*} and A:cp - haproxy 2.8.18 upgrade (T414318)
• 14:41 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1349.eqiad.wmnet with reason: host reimage
• 14:40 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1347.eqiad.wmnet with reason: host reimage
• 14:40 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1348.eqiad.wmnet with reason: host reimage
• 14:39 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm7001.magru.wmnet with reason: host reimage
• 14:39 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P{cp7016.*} and A:cp - haproxy 2.8.18 upgrade (T414318)
• 14:39 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db2196: Schema change
• 14:38 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newdepool (exit_code=0) depool db2196: Schema change
• 14:38 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1350.eqiad.wmnet with reason: host reimage
• 14:38 marostegui@cumin1003: START - Cookbook sre.mysql.newdepool depool db2196: Schema change
• 14:34 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1338.eqiad.wmnet with reason: host reimage
• 14:33 zabe@deploy2002: Finished scap sync-world: Backport for Close kywikibooks (T413845) (duration: 07m 16s)
• 14:33 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1352.eqiad.wmnet with reason: host reimage
• 14:29 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1351.eqiad.wmnet with reason: host reimage
• 14:29 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1349.eqiad.wmnet with reason: host reimage
• 14:29 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1347.eqiad.wmnet with OS trixie
• 14:29 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1348.eqiad.wmnet with OS trixie
• 14:29 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1350.eqiad.wmnet with reason: host reimage
• 14:29 zabe@deploy2002: zabe: Continuing with sync
• 14:28 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1348.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:28 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1347.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:28 zabe@deploy2002: zabe: Backport for Close kywikibooks (T413845) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:27 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1338.eqiad.wmnet with reason: host reimage
• 14:25 zabe@deploy2002: Started scap sync-world: Backport for Close kywikibooks (T413845)
• 14:22 zabe@deploy2002: Finished scap sync-world: Backport for ProofreadPage: Disable flag to render using parsoid temporarily (T408915) (duration: 13m 44s)
• 14:22 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1352.eqiad.wmnet with OS trixie
• 14:21 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1346.eqiad.wmnet with OS trixie
• 14:21 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:21 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1348.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:21 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1347.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 14:20 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:18 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1351.eqiad.wmnet with OS trixie
• 14:18 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1350.eqiad.wmnet with OS trixie
• 14:18 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1349.eqiad.wmnet with OS trixie
• 14:17 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1345.eqiad.wmnet with OS trixie
• 14:17 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:16 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:16 zabe@deploy2002: jgiannelos, zabe: Continuing with sync
• 14:16 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1338.eqiad.wmnet with OS trixie
• 14:13 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1344.eqiad.wmnet with OS trixie
• 14:13 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:13 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:12 zabe@deploy2002: jgiannelos, zabe: Backport for ProofreadPage: Disable flag to render using parsoid temporarily (T408915) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:10 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mwlog1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 14:08 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host testvm7001.magru.wmnet with OS bookworm
• 14:08 zabe@deploy2002: Started scap sync-world: Backport for ProofreadPage: Disable flag to render using parsoid temporarily (T408915)
• 14:08 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1343.eqiad.wmnet with OS trixie
• 14:08 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:07 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:05 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1342.eqiad.wmnet with OS trixie
• 14:05 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:05 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 14:03 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1346.eqiad.wmnet with reason: host reimage
• 14:01 jclark@cumin1003: START - Cookbook sre.hosts.provision for host mwlog1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
• 14:00 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1345.eqiad.wmnet with reason: host reimage
• 14:00 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:00 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt mwlog1003 - jclark@cumin1003"
• 14:00 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt mwlog1003 - jclark@cumin1003"
• 13:56 jclark@cumin1003: START - Cookbook sre.dns.netbox
• 13:56 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1344.eqiad.wmnet with reason: host reimage
• 13:55 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P{cp7016.*} and A:cp - haproxy 2.8.18 upgrade (T414318)
• 13:55 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7008.*} and A:cp - haproxy 2.8.18 upgrade (T414318)
• 13:52 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1343.eqiad.wmnet with reason: host reimage
• 13:49 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1346.eqiad.wmnet with reason: host reimage
• 13:49 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1342.eqiad.wmnet with reason: host reimage
• 13:46 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1337.eqiad.wmnet with OS trixie
• 13:46 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:46 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1345.eqiad.wmnet with reason: host reimage
• 13:46 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:43 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1344.eqiad.wmnet with reason: host reimage
• 13:43 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1343.eqiad.wmnet with reason: host reimage
• 13:43 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1342.eqiad.wmnet with reason: host reimage
• 13:43 ayounsi@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin1003.eqiad.wmnet with reason: Release v0.11.1 - ayounsi@cumin1003
• 13:42 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin1003.eqiad.wmnet with reason: Release v0.11.1 - ayounsi@cumin1003
• 13:38 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1346.eqiad.wmnet with OS trixie
• 13:37 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1341.eqiad.wmnet with OS trixie
• 13:37 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:37 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:36 ayounsi@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) homer to cumin2002.codfw.wmnet with reason: Release v0.11.1 - ayounsi@cumin1003
• 13:35 ayounsi@cumin1003: START - Cookbook sre.deploy.python-code homer to cumin2002.codfw.wmnet with reason: Release v0.11.1 - ayounsi@cumin1003
• 13:35 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1345.eqiad.wmnet with OS trixie
• 13:34 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1340.eqiad.wmnet with OS trixie
• 13:34 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:34 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:32 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1344.eqiad.wmnet with OS trixie
• 13:32 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1343.eqiad.wmnet with OS trixie
• 13:32 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1342.eqiad.wmnet with OS trixie
• 13:29 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1337.eqiad.wmnet with reason: host reimage
• 13:28 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1339.eqiad.wmnet with OS trixie
• 13:28 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:27 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:26 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1336.eqiad.wmnet with OS trixie
• 13:26 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:25 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:25 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1335.eqiad.wmnet with OS trixie
• 13:25 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:22 moritzm: installing squid security updates
• 13:21 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1341.eqiad.wmnet with reason: host reimage
• 13:20 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
• 13:17 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1340.eqiad.wmnet with reason: host reimage
• 13:11 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1339.eqiad.wmnet with reason: host reimage
• 13:08 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1341.eqiad.wmnet with reason: host reimage
• 13:07 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1340.eqiad.wmnet with reason: host reimage
• 13:07 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1336.eqiad.wmnet with reason: host reimage
• 13:07 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1339.eqiad.wmnet with reason: host reimage
• 13:05 moritzm: installing gnup2 security updates
• 13:04 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1335.eqiad.wmnet with reason: host reimage
• 13:04 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1337.eqiad.wmnet with reason: host reimage
• 13:03 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1336.eqiad.wmnet with reason: host reimage
• 13:00 jclark@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1335.eqiad.wmnet with reason: host reimage
• 12:57 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1341.eqiad.wmnet with OS trixie
• 12:56 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1359.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:56 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1339.eqiad.wmnet with OS trixie
• 12:56 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1340.eqiad.wmnet with OS trixie
• 12:54 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1358.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:53 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1357.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:53 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1337.eqiad.wmnet with OS trixie
• 12:52 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1355.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:51 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1336.eqiad.wmnet with OS trixie
• 12:50 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1356.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:50 btullis@dns1004: END - running authdns-update
• 12:49 jclark@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1335.eqiad.wmnet with OS trixie
• 12:49 btullis@dns1004: START - running authdns-update
• 12:49 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1359.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:47 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1354.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:47 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1351.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:47 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1358.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:46 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1352.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:46 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1357.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:46 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1353.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:43 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1356.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:43 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1350.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:42 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1355.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:41 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1349.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:38 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1352.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:37 jclark@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1352.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:37 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1354.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:37 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1353.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:37 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1352.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:37 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1351.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:37 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1350.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:34 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1349.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:31 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1346.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:30 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1344.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:29 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1343.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:28 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1345.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:27 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1342.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:27 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1341.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:21 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1346.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:19 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1342.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:19 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1344.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:19 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1345.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:19 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1343.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:19 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1341.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:19 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1339.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:16 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1337.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:16 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1335.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:16 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1338.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:16 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1336.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:16 jclark@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1340.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:09 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1337.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:09 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1340.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:08 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1338.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:08 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1339.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:08 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1336.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:08 jclark@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1335.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 12:04 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2008.wikimedia.org with OS bookworm
• 11:49 mutante: LDAP - fixed group membership in wmde and nda, kimpham -> pham
• 11:48 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2008.wikimedia.org with reason: host reimage
• 11:42 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2008.wikimedia.org with reason: host reimage
• 11:40 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2007.codfw.wmnet with OS bookworm
• 11:24 ayounsi@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2007.codfw.wmnet with reason: host reimage
• 11:23 mutante: LDAP - add martynranyard to groups wmde and nda (T413994)
• 11:22 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host testvm2008.wikimedia.org with OS bookworm
• 11:22 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1249 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87466 and previous config saved to /var/cache/conftool/dbconfig/20260113-112159-marostegui.json
• 11:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1249.eqiad.wmnet with reason: Maintenance
• 11:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87465 and previous config saved to /var/cache/conftool/dbconfig/20260113-112134-marostegui.json
• 11:18 ayounsi@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2007.codfw.wmnet with reason: host reimage
• 11:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P87464 and previous config saved to /var/cache/conftool/dbconfig/20260113-111127-marostegui.json
• 11:03 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host testvm2007.codfw.wmnet with OS bookworm
• 11:03 elukey: disable HTTP redirects to the Swift backend for all the Docker registries - T390251
• 11:03 ayounsi@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host testvm2007.codfw.wmnet with OS bookworm
• 11:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P87463 and previous config saved to /var/cache/conftool/dbconfig/20260113-110119-marostegui.json
• 10:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87462 and previous config saved to /var/cache/conftool/dbconfig/20260113-105110-marostegui.json
• 10:30 mutante: LDAP - add kimpham to groups wmde and nda (T414157)
• 10:24 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host testvm2007.codfw.wmnet with OS bookworm
• 10:22 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host testvm2007.codfw.wmnet with OS bookworm
• 10:15 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2246 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87461 and previous config saved to /var/cache/conftool/dbconfig/20260113-101552-marostegui.json
• 10:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2246.codfw.wmnet with reason: Maintenance
• 10:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87460 and previous config saved to /var/cache/conftool/dbconfig/20260113-101528-marostegui.json
• 10:06 moritzm: revoked legacy similar-users discovery certificate T365798
• 10:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P87459 and previous config saved to /var/cache/conftool/dbconfig/20260113-100519-marostegui.json
• 09:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245', diff saved to https://phabricator.wikimedia.org/P87458 and previous config saved to /var/cache/conftool/dbconfig/20260113-095510-marostegui.json
• 09:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2245 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87457 and previous config saved to /var/cache/conftool/dbconfig/20260113-094502-marostegui.json
• 09:41 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host testvm2007.codfw.wmnet with OS bookworm
• 09:40 fceratto@deploy2002: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 09:34 fceratto@deploy2002: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 08:43 marostegui: Stop mariadb on db2232 (m5) for testing dbproxy2005 T409398
• 08:42 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db[2160,2232].codfw.wmnet with reason: testing
• 08:31 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy2005.codfw.wmnet with OS trixie
• 08:31 brouberol@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-test-druid1001.eqiad.wmnet with OS bookworm
• 08:08 marostegui@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy2005.codfw.wmnet with reason: host reimage
• 08:04 marostegui@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy2005.codfw.wmnet with reason: host reimage
• 07:53 brouberol@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-test-druid1001.eqiad.wmnet with reason: host reimage
• 07:48 marostegui@cumin1003: START - Cookbook sre.hosts.reimage for host dbproxy2005.codfw.wmnet with OS trixie
• 07:48 brouberol@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on an-test-druid1001.eqiad.wmnet with reason: host reimage
• 07:34 brouberol@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-druid1001.eqiad.wmnet with OS bookworm
• 07:11 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool db2143: After Schema change
• 07:11 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 07:11 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
• 07:11 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db2143: After Schema change
• 07:10 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newdepool (exit_code=0) depool db2143: Schema change
• 07:10 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 07:10 marostegui: Deploy schema change on ms3 T411497
• 07:10 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
• 07:10 marostegui@cumin1003: START - Cookbook sre.mysql.newdepool depool db2143: Schema change
• 07:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool db2144: After Schema change
• 07:06 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 07:06 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
• 07:06 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db2144: After Schema change
• 07:06 marostegui: Deploy schema change on ms2 T411497
• 07:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newdepool (exit_code=0) depool db2144: Schema change
• 07:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 07:05 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
• 07:05 marostegui@cumin1003: START - Cookbook sre.mysql.newdepool depool db2144: Schema change
• 06:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool db2142: After Schema change
• 06:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 06:50 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
• 06:50 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db2142: After Schema change
• 06:50 marostegui: Deploy schema change on ms1 T411497
• 06:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newdepool (exit_code=0) depool db2142: Schema change
• 06:48 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 06:48 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
• 06:48 marostegui@cumin1003: START - Cookbook sre.mysql.newdepool depool db2142: Schema change
• 06:39 XioNoX: push pfw policies - T414393
• 06:13 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
• 06:13 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2212.codfw.wmnet with reason: Maintenance
• 05:04 mwpresync@deploy2002: Pruned MediaWiki: 1.46.0-wmf.5 (duration: 04m 11s)
• 04:47 mwpresync@deploy2002: Finished scap sync-world: testwikis to 1.46.0-wmf.11 refs T413802 (duration: 44m 17s)
• 04:03 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.46.0-wmf.11 refs T413802
• 02:27 sukhe@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on cp5022.eqsin.wmnet with reason: host down
• 02:26 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=cp5022.eqsin.wmnet [reason: host down]
• 01:21 tstarling@deploy2002: Finished scap sync-world: Backport for [metawiki] enable voting on entities with the 'Under review' status (T409613) (duration: 10m 26s)
• 01:17 tstarling@deploy2002: tstarling, hmonroy: Continuing with sync
• 01:12 tstarling@deploy2002: tstarling, hmonroy: Backport for [metawiki] enable voting on entities with the 'Under review' status (T409613) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 01:11 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 01:11 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 01:10 tstarling@deploy2002: Started scap sync-world: Backport for [metawiki] enable voting on entities with the 'Under review' status (T409613)
• 01:02 zabe@deploy2002: Finished scap sync-world: Backport for Start writing to il_target_id on testwiki (T413526) (duration: 08m 19s)
• 00:59 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1248 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87449 and previous config saved to /var/cache/conftool/dbconfig/20260113-005943-marostegui.json
• 00:59 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance
• 00:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87448 and previous config saved to /var/cache/conftool/dbconfig/20260113-005918-marostegui.json
• 00:57 zabe@deploy2002: zabe: Continuing with sync
• 00:55 zabe@deploy2002: zabe: Backport for Start writing to il_target_id on testwiki (T413526) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 00:53 zabe@deploy2002: Started scap sync-world: Backport for Start writing to il_target_id on testwiki (T413526)
• 00:51 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 00:51 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 00:49 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P87447 and previous config saved to /var/cache/conftool/dbconfig/20260113-004910-marostegui.json
• 00:39 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P87446 and previous config saved to /var/cache/conftool/dbconfig/20260113-003901-marostegui.json
• 00:35 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 00:35 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 00:35 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 00:35 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 00:35 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 00:35 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
• 00:31 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 00:31 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 00:30 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 00:30 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 00:29 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 00:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87445 and previous config saved to /var/cache/conftool/dbconfig/20260113-002853-marostegui.json
• 00:09 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply

2026-01-12

• 23:52 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 23:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 (T413525)', diff saved to https://phabricator.wikimedia.org/P87444 and previous config saved to /var/cache/conftool/dbconfig/20260112-235209-marostegui.json
• 23:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P87443 and previous config saved to /var/cache/conftool/dbconfig/20260112-234201-marostegui.json
• 23:38 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2245 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87442 and previous config saved to /var/cache/conftool/dbconfig/20260112-233850-marostegui.json
• 23:38 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2245.codfw.wmnet with reason: Maintenance
• 23:38 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87441 and previous config saved to /var/cache/conftool/dbconfig/20260112-233825-marostegui.json
• 23:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P87440 and previous config saved to /var/cache/conftool/dbconfig/20260112-233152-marostegui.json
• 23:30 cjming@deploy2002: Finished scap sync-world: Backport for Revert^2 "Deploy TestKitchen to Beta Cluster" (duration: 06m 14s)
• 23:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P87439 and previous config saved to /var/cache/conftool/dbconfig/20260112-232817-marostegui.json
• 23:26 cjming@deploy2002: cjming: Continuing with sync
• 23:26 cjming@deploy2002: cjming: Backport for Revert^2 "Deploy TestKitchen to Beta Cluster" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 23:24 cjming@deploy2002: Started scap sync-world: Backport for Revert^2 "Deploy TestKitchen to Beta Cluster"
• 23:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1251 (T413525)', diff saved to https://phabricator.wikimedia.org/P87438 and previous config saved to /var/cache/conftool/dbconfig/20260112-232144-marostegui.json
• 23:20 cjming@deploy2002: Finished scap sync-world: Backport for Revert to `product_metrics` schemas and use `default` as the coordinator value (T407901) (duration: 06m 13s)
• 23:18 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240', diff saved to https://phabricator.wikimedia.org/P87437 and previous config saved to /var/cache/conftool/dbconfig/20260112-231809-marostegui.json
• 23:16 cjming@deploy2002: cjming: Continuing with sync
• 23:16 cjming@deploy2002: cjming: Backport for Revert to `product_metrics` schemas and use `default` as the coordinator value (T407901) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 23:14 cjming@deploy2002: Started scap sync-world: Backport for Revert to `product_metrics` schemas and use `default` as the coordinator value (T407901)
• 23:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2240 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87436 and previous config saved to /var/cache/conftool/dbconfig/20260112-230801-marostegui.json
• 22:56 cjming@deploy2002: Finished scap sync-world: Backport for tests: skip test when WebAuthn is not loaded (T407797) (duration: 06m 25s)
• 22:52 cjming@deploy2002: cjming, zabe: Continuing with sync
• 22:52 cjming@deploy2002: cjming, zabe: Backport for tests: skip test when WebAuthn is not loaded (T407797) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:50 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1251 (T413525)', diff saved to https://phabricator.wikimedia.org/P87435 and previous config saved to /var/cache/conftool/dbconfig/20260112-225015-marostegui.json
• 22:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1251.eqiad.wmnet with reason: Maintenance
• 22:50 cjming@deploy2002: Started scap sync-world: Backport for tests: skip test when WebAuthn is not loaded (T407797)
• 22:50 ryankemper@cumin2002: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
• 22:47 ryankemper@cumin2002: END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop analytics cluster
• 22:47 ryankemper@cumin2002: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
• 22:41 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster
• 22:41 ryankemper@cumin2002: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
• 22:39 ryankemper@cumin2002: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster
• 22:39 ryankemper@cumin2002: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
• 22:34 cjming@deploy2002: Started scap sync-world: Backport for tests: skip test when WebAuthn is not loaded (T407797)
• 22:24 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 22:15 rzl: apt1002# reprepro --noskipold --restrict vopsbot update bookworm-wikimedia
• 22:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T413525)', diff saved to https://phabricator.wikimedia.org/P87434 and previous config saved to /var/cache/conftool/dbconfig/20260112-220104-marostegui.json
• 21:57 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 21:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T413525)', diff saved to https://phabricator.wikimedia.org/P87433 and previous config saved to /var/cache/conftool/dbconfig/20260112-215731-marostegui.json
• 21:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P87432 and previous config saved to /var/cache/conftool/dbconfig/20260112-215056-marostegui.json
• 21:47 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P87431 and previous config saved to /var/cache/conftool/dbconfig/20260112-214723-marostegui.json
• 21:43 cwhite@deploy2002: Finished deploy [statsv/statsv@b935e2d]: T389469 (duration: 00m 09s)
• 21:43 cwhite@deploy2002: Started deploy [statsv/statsv@b935e2d]: T389469
• 21:41 jhathaway@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest2009.codfw.wmnet with reason: reboot
• 21:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P87430 and previous config saved to /var/cache/conftool/dbconfig/20260112-214049-marostegui.json
• 21:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P87429 and previous config saved to /var/cache/conftool/dbconfig/20260112-213714-marostegui.json
• 21:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T413525)', diff saved to https://phabricator.wikimedia.org/P87428 and previous config saved to /var/cache/conftool/dbconfig/20260112-213041-marostegui.json
• 21:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T413525)', diff saved to https://phabricator.wikimedia.org/P87427 and previous config saved to /var/cache/conftool/dbconfig/20260112-212706-marostegui.json
• 21:26 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:19 cjming@deploy2002: Finished scap sync-world: Backport for Revert "Deploy TestKitchen to Beta Cluster" (duration: 11m 13s)
• 21:15 cjming@deploy2002: cjming: Continuing with sync
• 21:10 cjming@deploy2002: cjming: Backport for Revert "Deploy TestKitchen to Beta Cluster" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:08 cjming@deploy2002: Started scap sync-world: Backport for Revert "Deploy TestKitchen to Beta Cluster"
• 21:04 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1372.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:02 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1372.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 21:02 cjming@deploy2002: Started scap sync-world: Backport for Deploy TestKitchen to Beta Cluster (T407806 T407805)
• 21:01 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1372
• 21:01 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1372
• 21:00 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 21:00 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1372 - vriley@cumin1003"
• 21:00 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1372 - vriley@cumin1003"
• 20:59 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2216 (T413525)', diff saved to https://phabricator.wikimedia.org/P87426 and previous config saved to /var/cache/conftool/dbconfig/20260112-205912-marostegui.json
• 20:59 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2216.codfw.wmnet with reason: Maintenance
• 20:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2203 (T413525)', diff saved to https://phabricator.wikimedia.org/P87425 and previous config saved to /var/cache/conftool/dbconfig/20260112-205848-marostegui.json
• 20:57 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:57 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:57 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 20:56 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1235 (T413525)', diff saved to https://phabricator.wikimedia.org/P87424 and previous config saved to /var/cache/conftool/dbconfig/20260112-205602-marostegui.json
• 20:55 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1235.eqiad.wmnet with reason: Maintenance
• 20:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T413525)', diff saved to https://phabricator.wikimedia.org/P87423 and previous config saved to /var/cache/conftool/dbconfig/20260112-205537-marostegui.json
• 20:53 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:50 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:49 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:48 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2203', diff saved to https://phabricator.wikimedia.org/P87422 and previous config saved to /var/cache/conftool/dbconfig/20260112-204840-marostegui.json
• 20:48 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:47 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:47 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1371.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P87421 and previous config saved to /var/cache/conftool/dbconfig/20260112-204529-marostegui.json
• 20:45 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1369.eqiad.wmnet with OS trixie
• 20:45 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 20:44 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1371
• 20:44 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1371
• 20:44 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1370.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:43 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1370
• 20:43 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1370
• 20:42 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1370
• 20:42 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 20:42 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1370
• 20:42 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 20:42 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1371 - vriley@cumin1003"
• 20:41 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1371 - vriley@cumin1003"
• 20:38 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 20:38 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2203', diff saved to https://phabricator.wikimedia.org/P87420 and previous config saved to /var/cache/conftool/dbconfig/20260112-203831-marostegui.json
• 20:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P87419 and previous config saved to /var/cache/conftool/dbconfig/20260112-203521-marostegui.json
• 20:33 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 20:33 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1370 - vriley@cumin1003"
• 20:33 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1370 - vriley@cumin1003"
• 20:30 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1368.eqiad.wmnet with OS trixie
• 20:30 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 20:28 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 20:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2203 (T413525)', diff saved to https://phabricator.wikimedia.org/P87418 and previous config saved to /var/cache/conftool/dbconfig/20260112-202823-marostegui.json
• 20:25 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1369.eqiad.wmnet with reason: host reimage
• 20:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T413525)', diff saved to https://phabricator.wikimedia.org/P87417 and previous config saved to /var/cache/conftool/dbconfig/20260112-202512-marostegui.json
• 20:21 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1369.eqiad.wmnet with reason: host reimage
• 20:20 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 20:10 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1369.eqiad.wmnet with OS trixie
• 20:09 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1369.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:04 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1368.eqiad.wmnet with reason: host reimage
• 20:01 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1369.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 20:01 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1369
• 20:01 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1368.eqiad.wmnet with reason: host reimage
• 20:00 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1369
• 20:00 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 20:00 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1369 - vriley@cumin1003"
• 20:00 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1369 - vriley@cumin1003"
• 19:57 ejegg: fundraising civicrm upgraded from 158b60a0 to 3d70b84d
• 19:57 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2203 (T413525)', diff saved to https://phabricator.wikimedia.org/P87416 and previous config saved to /var/cache/conftool/dbconfig/20260112-195731-marostegui.json
• 19:57 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2203.codfw.wmnet with reason: Maintenance
• 19:57 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 19:54 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1234 (T413525)', diff saved to https://phabricator.wikimedia.org/P87415 and previous config saved to /var/cache/conftool/dbconfig/20260112-195450-marostegui.json
• 19:54 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1234.eqiad.wmnet with reason: Maintenance
• 19:54 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T413525)', diff saved to https://phabricator.wikimedia.org/P87414 and previous config saved to /var/cache/conftool/dbconfig/20260112-195425-marostegui.json
• 19:50 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1368.eqiad.wmnet with OS trixie
• 19:49 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1367.eqiad.wmnet with OS trixie
• 19:49 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 19:46 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 19:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P87413 and previous config saved to /var/cache/conftool/dbconfig/20260112-194417-marostegui.json
• 19:36 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1368.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 19:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P87412 and previous config saved to /var/cache/conftool/dbconfig/20260112-193409-marostegui.json
• 19:32 damilare: payments-wiki upgraded from 14d8501a to 3ffc70f0
• 19:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2202.codfw.wmnet with reason: Maintenance
• 19:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T413525)', diff saved to https://phabricator.wikimedia.org/P87411 and previous config saved to /var/cache/conftool/dbconfig/20260112-193209-marostegui.json
• 19:29 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1367.eqiad.wmnet with reason: host reimage
• 19:29 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1368.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 19:27 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1368
• 19:27 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1368
• 19:26 vriley@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wikikube-worker1366.eqiad.wmnet with OS trixie
• 19:26 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 19:25 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1367.eqiad.wmnet with reason: host reimage
• 19:25 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - vriley@cumin1003"
• 19:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T413525)', diff saved to https://phabricator.wikimedia.org/P87409 and previous config saved to /var/cache/conftool/dbconfig/20260112-192401-marostegui.json
• 19:22 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 19:22 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1368 - vriley@cumin1003"
• 19:22 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1368 - vriley@cumin1003"
• 19:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P87408 and previous config saved to /var/cache/conftool/dbconfig/20260112-192201-marostegui.json
• 19:18 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 19:14 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1367.eqiad.wmnet with OS trixie
• 19:12 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1367.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 19:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P87407 and previous config saved to /var/cache/conftool/dbconfig/20260112-191152-marostegui.json
• 19:09 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1366.eqiad.wmnet with reason: host reimage
• 19:05 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1367.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 19:05 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1367
• 19:05 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1367
• 19:04 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 19:04 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1367 - vriley@cumin1003"
• 19:04 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1367 - vriley@cumin1003"
• 19:04 vriley@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1366.eqiad.wmnet with reason: host reimage
• 19:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T413525)', diff saved to https://phabricator.wikimedia.org/P87406 and previous config saved to /var/cache/conftool/dbconfig/20260112-190144-marostegui.json
• 19:01 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 19:00 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudlb1001.eqiad.wmnet with OS trixie
• 18:55 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1232 (T413525)', diff saved to https://phabricator.wikimedia.org/P87405 and previous config saved to /var/cache/conftool/dbconfig/20260112-185521-marostegui.json
• 18:55 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1232.eqiad.wmnet with reason: Maintenance
• 18:54 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T413525)', diff saved to https://phabricator.wikimedia.org/P87404 and previous config saved to /var/cache/conftool/dbconfig/20260112-185456-marostegui.json
• 18:53 vriley@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1366.eqiad.wmnet with OS trixie
• 18:52 vriley@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1366.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 18:45 vriley@cumin1003: START - Cookbook sre.hosts.provision for host wikikube-worker1366.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
• 18:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P87403 and previous config saved to /var/cache/conftool/dbconfig/20260112-184448-marostegui.json
• 18:44 vriley@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1366
• 18:44 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1366
• 18:43 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:43 vriley@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1366 - vriley@cumin1003"
• 18:43 vriley@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update mgmt wikikube-worker1366 - vriley@cumin1003"
• 18:40 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 18:39 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudlb1001.eqiad.wmnet with reason: host reimage
• 18:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P87402 and previous config saved to /var/cache/conftool/dbconfig/20260112-183440-marostegui.json
• 18:33 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2188 (T413525)', diff saved to https://phabricator.wikimedia.org/P87401 and previous config saved to /var/cache/conftool/dbconfig/20260112-183336-marostegui.json
• 18:33 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2188.codfw.wmnet with reason: Maintenance
• 18:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T413525)', diff saved to https://phabricator.wikimedia.org/P87400 and previous config saved to /var/cache/conftool/dbconfig/20260112-183311-marostegui.json
• 18:33 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudlb1001.eqiad.wmnet with reason: host reimage
• 18:30 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wikikube-worker1365
• 18:30 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1365
• 18:30 vriley@cumin1003: END (FAIL) - Cookbook sre.network.configure-switch-interfaces (exit_code=99) for host wikikube-worker1365
• 18:30 vriley@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1365
• 18:28 vriley@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:26 vriley@cumin1003: START - Cookbook sre.dns.netbox
• 18:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T413525)', diff saved to https://phabricator.wikimedia.org/P87399 and previous config saved to /var/cache/conftool/dbconfig/20260112-182431-marostegui.json
• 18:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P87398 and previous config saved to /var/cache/conftool/dbconfig/20260112-182303-marostegui.json
• 18:17 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudlb1001.eqiad.wmnet with OS trixie
• 18:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P87397 and previous config saved to /var/cache/conftool/dbconfig/20260112-181255-marostegui.json
• 18:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T413525)', diff saved to https://phabricator.wikimedia.org/P87396 and previous config saved to /var/cache/conftool/dbconfig/20260112-180247-marostegui.json
• 17:59 fceratto@deploy2002: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 17:54 fceratto@deploy2002: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 17:53 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1219 (T413525)', diff saved to https://phabricator.wikimedia.org/P87395 and previous config saved to /var/cache/conftool/dbconfig/20260112-175349-marostegui.json
• 17:53 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1219.eqiad.wmnet with reason: Maintenance
• 17:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T413525)', diff saved to https://phabricator.wikimedia.org/P87394 and previous config saved to /var/cache/conftool/dbconfig/20260112-175324-marostegui.json
• 17:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P87393 and previous config saved to /var/cache/conftool/dbconfig/20260112-174316-marostegui.json
• 17:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P87392 and previous config saved to /var/cache/conftool/dbconfig/20260112-173308-marostegui.json
• 17:28 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2176 (T413525)', diff saved to https://phabricator.wikimedia.org/P87391 and previous config saved to /var/cache/conftool/dbconfig/20260112-172808-marostegui.json
• 17:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2176.codfw.wmnet with reason: Maintenance
• 17:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T413525)', diff saved to https://phabricator.wikimedia.org/P87390 and previous config saved to /var/cache/conftool/dbconfig/20260112-172744-marostegui.json
• 17:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T413525)', diff saved to https://phabricator.wikimedia.org/P87389 and previous config saved to /var/cache/conftool/dbconfig/20260112-172259-marostegui.json
• 17:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P87388 and previous config saved to /var/cache/conftool/dbconfig/20260112-171735-marostegui.json
• 17:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P87387 and previous config saved to /var/cache/conftool/dbconfig/20260112-170727-marostegui.json
• 16:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T413525)', diff saved to https://phabricator.wikimedia.org/P87386 and previous config saved to /var/cache/conftool/dbconfig/20260112-165718-marostegui.json
• 16:52 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1218 (T413525)', diff saved to https://phabricator.wikimedia.org/P87385 and previous config saved to /var/cache/conftool/dbconfig/20260112-165253-marostegui.json
• 16:52 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1218.eqiad.wmnet with reason: Maintenance
• 16:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T413525)', diff saved to https://phabricator.wikimedia.org/P87384 and previous config saved to /var/cache/conftool/dbconfig/20260112-165229-marostegui.json
• 16:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P87383 and previous config saved to /var/cache/conftool/dbconfig/20260112-164220-marostegui.json
• 16:35 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudlb1002.eqiad.wmnet with OS trixie
• 16:33 urbanecm@deploy2002: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=plwiki --all --verbose # revalidate-addlink
• 16:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P87382 and previous config saved to /var/cache/conftool/dbconfig/20260112-163212-marostegui.json
• 16:27 fceratto@deploy2002: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 16:27 fceratto@deploy2002: helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' .
• 16:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2174 (T413525)', diff saved to https://phabricator.wikimedia.org/P87381 and previous config saved to /var/cache/conftool/dbconfig/20260112-162445-marostegui.json
• 16:24 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2174.codfw.wmnet with reason: Maintenance
• 16:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T413525)', diff saved to https://phabricator.wikimedia.org/P87380 and previous config saved to /var/cache/conftool/dbconfig/20260112-162421-marostegui.json
• 16:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T413525)', diff saved to https://phabricator.wikimedia.org/P87379 and previous config saved to /var/cache/conftool/dbconfig/20260112-162204-marostegui.json
• 16:16 btullis@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-test-druid1001.eqiad.wmnet with OS bookworm
• 16:16 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-druid1001.eqiad.wmnet with OS bookworm
• 16:15 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudlb1002.eqiad.wmnet with reason: host reimage
• 16:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P87378 and previous config saved to /var/cache/conftool/dbconfig/20260112-161412-marostegui.json
• 16:11 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudlb1002.eqiad.wmnet with reason: host reimage
• 16:04 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P87377 and previous config saved to /var/cache/conftool/dbconfig/20260112-160404-marostegui.json
• 15:58 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1031.eqiad.wmnet with OS trixie
• 15:55 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudlb1002.eqiad.wmnet with OS trixie
• 15:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T413525)', diff saved to https://phabricator.wikimedia.org/P87376 and previous config saved to /var/cache/conftool/dbconfig/20260112-155356-marostegui.json
• 15:51 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1206 (T413525)', diff saved to https://phabricator.wikimedia.org/P87375 and previous config saved to /var/cache/conftool/dbconfig/20260112-155113-marostegui.json
• 15:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1206.eqiad.wmnet with reason: Maintenance
• 15:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T413525)', diff saved to https://phabricator.wikimedia.org/P87374 and previous config saved to /var/cache/conftool/dbconfig/20260112-155049-marostegui.json
• 15:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P87373 and previous config saved to /var/cache/conftool/dbconfig/20260112-154041-marostegui.json
• 15:40 ejegg: donorwiki upgraded from 02dbb1e2 to 3ffc70f0
• 15:39 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1031.eqiad.wmnet with reason: host reimage
• 15:35 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1031.eqiad.wmnet with reason: host reimage
• 15:35 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudlb2004-dev.codfw.wmnet with OS trixie
• 15:33 zabe@deploy2002: Finished scap sync-world: Backport for Disable updates for Special:GloballyUnusedFiles (T414202), Stop updating Deadendpages and Lonelypages on commons (T371662), Correctly disable Special:Deadendpages on commons (T371662) (duration: 06m 47s)
• 15:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P87371 and previous config saved to /var/cache/conftool/dbconfig/20260112-153033-marostegui.json
• 15:29 zabe@deploy2002: zabe: Continuing with sync
• 15:28 zabe@deploy2002: zabe: Backport for Disable updates for Special:GloballyUnusedFiles (T414202), Stop updating Deadendpages and Lonelypages on commons (T371662), Correctly disable Special:Deadendpages on commons (T371662) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:26 zabe@deploy2002: Started scap sync-world: Backport for Disable updates for Special:GloballyUnusedFiles (T414202), Stop updating Deadendpages and Lonelypages on commons (T371662), Correctly disable Special:Deadendpages on commons (T371662)
• 15:20 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T413525)', diff saved to https://phabricator.wikimedia.org/P87370 and previous config saved to /var/cache/conftool/dbconfig/20260112-152025-marostegui.json
• 15:20 zabe@deploy2002: Sync cancelled.
• 15:19 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-test-druid1001.eqiad.wmnet with OS bookworm
• 15:19 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2173 (T413525)', diff saved to https://phabricator.wikimedia.org/P87369 and previous config saved to /var/cache/conftool/dbconfig/20260112-151934-marostegui.json
• 15:19 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-druid1001.eqiad.wmnet with OS bookworm
• 15:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2173.codfw.wmnet with reason: Maintenance
• 15:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T413525)', diff saved to https://phabricator.wikimedia.org/P87368 and previous config saved to /var/cache/conftool/dbconfig/20260112-151908-marostegui.json
• 15:17 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1031.eqiad.wmnet with OS trixie
• 15:16 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-test-druid1001.eqiad.wmnet with OS bookworm
• 15:16 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-druid1001.eqiad.wmnet with OS bookworm
• 15:14 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1030.eqiad.wmnet with OS trixie
• 15:13 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudlb2004-dev.codfw.wmnet with reason: host reimage
• 15:12 zabe@deploy2002: mwscript-k8s job started: foreachwikiindblist all refreshImageMetadata.php --mediatype AUDIO --mime unknown/mpeg --oldimage --force # T414259
• 15:10 zabe@deploy2002: zabe: Backport for Disable updates for Special:GloballyUnusedFiles (T414202), Stop updating Deadendpages and Lonelypages on commons (T371662) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:09 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudlb2004-dev.codfw.wmnet with reason: host reimage
• 15:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P87367 and previous config saved to /var/cache/conftool/dbconfig/20260112-150900-marostegui.json
• 15:08 zabe@deploy2002: Started scap sync-world: Backport for Disable updates for Special:GloballyUnusedFiles (T414202), Stop updating Deadendpages and Lonelypages on commons (T371662)
• 15:06 ejegg: fundraising civicrm upgraded from 197629c6 to 158b60a0
• 15:06 zabe@deploy2002: Finished scap sync-world: Backport for Stop setting $wgBlockTargetMigrationStage (T355034) (duration: 06m 43s)
• 15:05 zabe@deploy2002: mwscript-k8s job started: foreachwikiindblist all refreshImageMetadata.php --mediatype AUDIO --mime unknown/mpeg --force # T414259
• 15:02 zabe@deploy2002: zabe: Continuing with sync
• 15:01 zabe@deploy2002: zabe: Backport for Stop setting $wgBlockTargetMigrationStage (T355034) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:59 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1030.eqiad.wmnet with reason: host reimage
• 14:59 zabe@deploy2002: Started scap sync-world: Backport for Stop setting $wgBlockTargetMigrationStage (T355034)
• 14:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P87366 and previous config saved to /var/cache/conftool/dbconfig/20260112-145852-marostegui.json
• 14:52 zabe@deploy2002: mwscript-k8s job started: foreachwikiindblist all refreshImageMetadata.php --mediatype AUDIO --mime unknown/flac --oldimage --force # T414259
• 14:51 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1030.eqiad.wmnet with reason: host reimage
• 14:51 zabe@deploy2002: mwscript-k8s job started: foreachwikiindblist all refreshImageMetadata.php --mediatype AUDIO --mime unknown/flac --oldimages --force # T414259
• 14:50 zabe: empty machinevision-tester group on commons # T372004
• 14:50 zabe: empty mediasearch-tester group on commons # T372004
• 14:49 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudlb2004-dev.codfw.wmnet with OS trixie
• 14:48 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T413525)', diff saved to https://phabricator.wikimedia.org/P87365 and previous config saved to /var/cache/conftool/dbconfig/20260112-144843-marostegui.json
• 14:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1196 (T413525)', diff saved to https://phabricator.wikimedia.org/P87364 and previous config saved to /var/cache/conftool/dbconfig/20260112-144602-marostegui.json
• 14:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 14:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1196.eqiad.wmnet with reason: Maintenance
• 14:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 (T413525)', diff saved to https://phabricator.wikimedia.org/P87363 and previous config saved to /var/cache/conftool/dbconfig/20260112-144518-marostegui.json
• 14:43 zabe@deploy2002: mwscript-k8s job started: foreachwikiindblist all refreshImageMetadata.php --mediatype AUDIO --mime unknown/flac --force # T414259
• 14:37 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1359
• 14:37 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1359
• 14:37 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1358
• 14:37 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1358
• 14:37 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1358
• 14:37 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1358
• 14:37 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1357
• 14:37 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1357
• 14:36 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1356
• 14:36 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1356
• 14:36 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1355
• 14:36 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1355
• 14:36 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1354
• 14:36 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1354
• 14:36 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1353
• 14:36 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1353
• 14:36 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1352
• 14:36 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1352
• 14:35 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1351
• 14:35 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1351
• 14:35 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1350
• 14:35 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1350
• 14:35 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1346
• 14:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P87362 and previous config saved to /var/cache/conftool/dbconfig/20260112-143510-marostegui.json
• 14:35 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1346
• 14:34 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1345
• 14:34 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1345
• 14:34 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1344
• 14:34 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1344
• 14:34 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1343
• 14:34 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1343
• 14:34 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1030.eqiad.wmnet with OS trixie
• 14:33 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1343
• 14:33 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1343
• 14:33 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1342
• 14:33 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1342
• 14:33 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1341
• 14:33 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1341
• 14:32 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1340
• 14:32 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1340
• 14:32 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1339
• 14:32 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1339
• 14:32 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1338
• 14:32 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1338
• 14:32 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1337
• 14:31 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1337
• 14:31 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1336
• 14:31 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1336
• 14:31 jclark@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1335
• 14:31 jclark@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1335
• 14:29 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:28 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt wikikube1335-59 servers - jclark@cumin1003"
• 14:28 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt wikikube1335-59 servers - jclark@cumin1003"
• 14:26 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1247 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87361 and previous config saved to /var/cache/conftool/dbconfig/20260112-142642-marostegui.json
• 14:26 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance
• 14:25 jclark@cumin1003: START - Cookbook sre.dns.netbox
• 14:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P87360 and previous config saved to /var/cache/conftool/dbconfig/20260112-142502-marostegui.json
• 14:23 jclark@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:23 jclark@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt wikikube1335-59 servers - jclark@cumin1003"
• 14:23 jclark@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt wikikube1335-59 servers - jclark@cumin1003"
• 14:22 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1029.eqiad.wmnet with OS trixie
• 14:22 cscott@deploy2002: Finished scap sync-world: Backport for Increase PRV percentage on fawiki/kowiki/azwiki (T413108), Turn off magic ISBN/RFC/PMID links on iawiki (T414019) (duration: 15m 34s)
• 14:19 jclark@cumin1003: START - Cookbook sre.dns.netbox
• 14:18 cscott@deploy2002: cscott: Continuing with sync
• 14:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1195 (T413525)', diff saved to https://phabricator.wikimedia.org/P87359 and previous config saved to /var/cache/conftool/dbconfig/20260112-141454-marostegui.json
• 14:14 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2170 (T413525)', diff saved to https://phabricator.wikimedia.org/P87358 and previous config saved to /var/cache/conftool/dbconfig/20260112-141443-marostegui.json
• 14:14 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2170.codfw.wmnet with reason: Maintenance
• 14:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T413525)', diff saved to https://phabricator.wikimedia.org/P87357 and previous config saved to /var/cache/conftool/dbconfig/20260112-141429-marostegui.json
• 14:10 cscott@deploy2002: cscott: Backport for Increase PRV percentage on fawiki/kowiki/azwiki (T413108), Turn off magic ISBN/RFC/PMID links on iawiki (T414019) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:06 cscott@deploy2002: Started scap sync-world: Backport for Increase PRV percentage on fawiki/kowiki/azwiki (T413108), Turn off magic ISBN/RFC/PMID links on iawiki (T414019)
• 14:04 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1029.eqiad.wmnet with reason: host reimage
• 14:04 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P87356 and previous config saved to /var/cache/conftool/dbconfig/20260112-140421-marostegui.json
• 14:00 ayounsi@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host testvm2007.codfw.wmnet with OS bookworm
• 13:59 ayounsi@cumin1003: START - Cookbook sre.hosts.reimage for host testvm2007.codfw.wmnet with OS bookworm
• 13:58 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1029.eqiad.wmnet with reason: host reimage
• 13:54 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P87355 and previous config saved to /var/cache/conftool/dbconfig/20260112-135413-marostegui.json
• 13:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T413525)', diff saved to https://phabricator.wikimedia.org/P87354 and previous config saved to /var/cache/conftool/dbconfig/20260112-134404-marostegui.json
• 13:40 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1195 (T413525)', diff saved to https://phabricator.wikimedia.org/P87353 and previous config saved to /var/cache/conftool/dbconfig/20260112-134040-marostegui.json
• 13:40 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1195.eqiad.wmnet with reason: Maintenance
• 13:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T413525)', diff saved to https://phabricator.wikimedia.org/P87352 and previous config saved to /var/cache/conftool/dbconfig/20260112-134016-marostegui.json
• 13:39 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1029.eqiad.wmnet with OS trixie
• 13:39 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-test-druid1001.eqiad.wmnet with OS bookworm
• 13:38 btullis@cumin1003: START - Cookbook sre.hosts.reimage for host an-test-druid1001.eqiad.wmnet with OS bookworm
• 13:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P87351 and previous config saved to /var/cache/conftool/dbconfig/20260112-133008-marostegui.json
• 13:20 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P87350 and previous config saved to /var/cache/conftool/dbconfig/20260112-132000-marostegui.json
• 13:12 aikochou@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 13:10 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2153 (T413525)', diff saved to https://phabricator.wikimedia.org/P87349 and previous config saved to /var/cache/conftool/dbconfig/20260112-131003-marostegui.json
• 13:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T413525)', diff saved to https://phabricator.wikimedia.org/P87348 and previous config saved to /var/cache/conftool/dbconfig/20260112-130952-marostegui.json
• 13:09 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2153.codfw.wmnet with reason: Maintenance
• 13:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T413525)', diff saved to https://phabricator.wikimedia.org/P87347 and previous config saved to /var/cache/conftool/dbconfig/20260112-130943-marostegui.json
• 13:07 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2240 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87346 and previous config saved to /var/cache/conftool/dbconfig/20260112-130754-marostegui.json
• 13:07 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2240.codfw.wmnet with reason: Maintenance
• 13:03 aikochou@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 12:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P87345 and previous config saved to /var/cache/conftool/dbconfig/20260112-125934-marostegui.json
• 12:58 gkyziridis@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' .
• 12:54 aikochou@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 12:49 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P87344 and previous config saved to /var/cache/conftool/dbconfig/20260112-124926-marostegui.json
• 12:41 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 12:39 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T413525)', diff saved to https://phabricator.wikimedia.org/P87343 and previous config saved to /var/cache/conftool/dbconfig/20260112-123918-marostegui.json
• 12:35 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1186 (T413525)', diff saved to https://phabricator.wikimedia.org/P87342 and previous config saved to /var/cache/conftool/dbconfig/20260112-123546-marostegui.json
• 12:35 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1186.eqiad.wmnet with reason: Maintenance
• 12:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T413525)', diff saved to https://phabricator.wikimedia.org/P87341 and previous config saved to /var/cache/conftool/dbconfig/20260112-123533-marostegui.json
• 12:26 moritzm: revoked legacy linkrecommendation discovery certificate T365798
• 12:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P87340 and previous config saved to /var/cache/conftool/dbconfig/20260112-122525-marostegui.json
• 12:19 moritzm: revoked legacy ganeti02.svc.esams certificate T365798
• 12:17 moritzm: revoked legacy default-staging-certificate certificate T365798
• 12:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P87339 and previous config saved to /var/cache/conftool/dbconfig/20260112-121517-marostegui.json
• 12:08 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 12:06 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2146 (T413525)', diff saved to https://phabricator.wikimedia.org/P87338 and previous config saved to /var/cache/conftool/dbconfig/20260112-120631-marostegui.json
• 12:06 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2146.codfw.wmnet with reason: Maintenance
• 12:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T413525)', diff saved to https://phabricator.wikimedia.org/P87337 and previous config saved to /var/cache/conftool/dbconfig/20260112-120606-marostegui.json
• 12:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T413525)', diff saved to https://phabricator.wikimedia.org/P87336 and previous config saved to /var/cache/conftool/dbconfig/20260112-120509-marostegui.json
• 12:03 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2249 slowly with 10 steps - repooling
• 11:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P87334 and previous config saved to /var/cache/conftool/dbconfig/20260112-115557-marostegui.json
• 11:55 cgoubert@dns1004: END - running authdns-update
• 11:54 cgoubert@dns1004: START - running authdns-update
• 11:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P87332 and previous config saved to /var/cache/conftool/dbconfig/20260112-114549-marostegui.json
• 11:36 ladsgroup@deploy2002: Finished scap sync-world: Backport for extension-list: Add Test Kitchen (T407806) (duration: 38m 06s)
• 11:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T413525)', diff saved to https://phabricator.wikimedia.org/P87331 and previous config saved to /var/cache/conftool/dbconfig/20260112-113541-marostegui.json
• 11:33 btullis@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-test-druid1001.eqiad.wmnet with reason: Testing druid upgrade
• 11:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1184 (T413525)', diff saved to https://phabricator.wikimedia.org/P87329 and previous config saved to /var/cache/conftool/dbconfig/20260112-112405-marostegui.json
• 11:23 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
• 11:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T413525)', diff saved to https://phabricator.wikimedia.org/P87328 and previous config saved to /var/cache/conftool/dbconfig/20260112-112341-marostegui.json
• 11:23 ladsgroup@deploy2002: cjming, ladsgroup: Continuing with sync
• 11:22 ladsgroup@deploy2002: cjming, ladsgroup: Backport for extension-list: Add Test Kitchen (T407806) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 11:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P87326 and previous config saved to /var/cache/conftool/dbconfig/20260112-111332-marostegui.json
• 11:09 moritzm: uploaded dnsmasq 2.92-rc3 to bookworm-wikimedia/main T396864
• 11:08 cgoubert@deploy2002: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
• 11:08 cgoubert@deploy2002: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
• 11:07 cgoubert@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
• 11:06 cgoubert@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
• 11:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P87325 and previous config saved to /var/cache/conftool/dbconfig/20260112-110324-marostegui.json
• 11:01 Dreamy_Jazz: Deployed security patch for T414011
• 10:58 ladsgroup@deploy2002: Started scap sync-world: Backport for extension-list: Add Test Kitchen (T407806)
• 10:57 urbanecm@deploy2002: mwscript-k8s job started: CentralAuth:emptyGlobalUserGroup.php --wiki=metawiki oathauth-tester # T411360
• 10:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T413525)', diff saved to https://phabricator.wikimedia.org/P87323 and previous config saved to /var/cache/conftool/dbconfig/20260112-105316-marostegui.json
• 10:49 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2166 gradually with 4 steps - repooling
• 10:49 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2145 (T413525)', diff saved to https://phabricator.wikimedia.org/P87321 and previous config saved to /var/cache/conftool/dbconfig/20260112-104941-marostegui.json
• 10:49 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2145.codfw.wmnet with reason: Maintenance
• 10:41 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db1203 gradually with 4 steps - repooling
• 10:34 vgutierrez@cumin1003: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[7008,7016].*} and A:cp - test haproxy 2.8.18 upgrade (T414318)
• 10:22 vgutierrez@cumin1003: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[7008,7016].*} and A:cp - test haproxy 2.8.18 upgrade (T414318)
• 10:20 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1169 (T413525)', diff saved to https://phabricator.wikimedia.org/P87313 and previous config saved to /var/cache/conftool/dbconfig/20260112-102044-marostegui.json
• 10:20 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
• 10:20 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
• 10:19 marostegui@cumin1003: START - Cookbook sre.mysql.pool db2166 gradually with 4 steps - repooling
• 10:18 vgutierrez: fetch HAProxy 2.8.18 on thirdparty/haproxy28-bullseye (apt.wm.o) - T414318
• 10:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T413525)', diff saved to https://phabricator.wikimedia.org/P87310 and previous config saved to /var/cache/conftool/dbconfig/20260112-101520-marostegui.json
• 10:11 marostegui@cumin1003: START - Cookbook sre.mysql.pool db1203 gradually with 4 steps - repooling
• 10:08 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2166 (T413525)', diff saved to https://phabricator.wikimedia.org/P87308 and previous config saved to /var/cache/conftool/dbconfig/20260112-100821-marostegui.json
• 10:08 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2166.codfw.wmnet with reason: Maintenance
• 10:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T413525)', diff saved to https://phabricator.wikimedia.org/P87307 and previous config saved to /var/cache/conftool/dbconfig/20260112-100756-marostegui.json
• 10:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T413525)', diff saved to https://phabricator.wikimedia.org/P87305 and previous config saved to /var/cache/conftool/dbconfig/20260112-100112-marostegui.json
• 09:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P87304 and previous config saved to /var/cache/conftool/dbconfig/20260112-095748-marostegui.json
• 09:56 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1203 (T413525)', diff saved to https://phabricator.wikimedia.org/P87303 and previous config saved to /var/cache/conftool/dbconfig/20260112-095610-marostegui.json
• 09:56 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1203.eqiad.wmnet with reason: Maintenance
• 09:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T413525)', diff saved to https://phabricator.wikimedia.org/P87302 and previous config saved to /var/cache/conftool/dbconfig/20260112-095545-marostegui.json
• 09:47 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P87301 and previous config saved to /var/cache/conftool/dbconfig/20260112-094740-marostegui.json
• 09:47 marostegui@cumin1003: START - Cookbook sre.mysql.pool db2249 slowly with 10 steps - repooling
• 09:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P87299 and previous config saved to /var/cache/conftool/dbconfig/20260112-094537-marostegui.json
• 09:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T413525)', diff saved to https://phabricator.wikimedia.org/P87298 and previous config saved to /var/cache/conftool/dbconfig/20260112-093731-marostegui.json
• 09:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P87297 and previous config saved to /var/cache/conftool/dbconfig/20260112-093528-marostegui.json
• 09:32 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2165 (T413525)', diff saved to https://phabricator.wikimedia.org/P87296 and previous config saved to /var/cache/conftool/dbconfig/20260112-093237-marostegui.json
• 09:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2165.codfw.wmnet with reason: Maintenance
• 09:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T413525)', diff saved to https://phabricator.wikimedia.org/P87295 and previous config saved to /var/cache/conftool/dbconfig/20260112-093223-marostegui.json
• 09:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T413525)', diff saved to https://phabricator.wikimedia.org/P87294 and previous config saved to /var/cache/conftool/dbconfig/20260112-092520-marostegui.json
• 09:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P87293 and previous config saved to /var/cache/conftool/dbconfig/20260112-092215-marostegui.json
• 09:20 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1193 (T413525)', diff saved to https://phabricator.wikimedia.org/P87292 and previous config saved to /var/cache/conftool/dbconfig/20260112-092016-marostegui.json
• 09:20 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 09:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T413525)', diff saved to https://phabricator.wikimedia.org/P87291 and previous config saved to /var/cache/conftool/dbconfig/20260112-091950-marostegui.json
• 09:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P87290 and previous config saved to /var/cache/conftool/dbconfig/20260112-091206-marostegui.json
• 09:10 dzahn@dns1004: END - running authdns-update
• 09:10 mutante: DNS - adding new language code 'kai' - https://en.wikipedia.org/wiki/Karai-karai
• 09:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P87289 and previous config saved to /var/cache/conftool/dbconfig/20260112-090942-marostegui.json
• 09:09 dzahn@dns1004: START - running authdns-update
• 09:04 bwojtowicz@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 09:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T413525)', diff saved to https://phabricator.wikimedia.org/P87288 and previous config saved to /var/cache/conftool/dbconfig/20260112-090158-marostegui.json
• 09:01 bwojtowicz@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 08:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P87287 and previous config saved to /var/cache/conftool/dbconfig/20260112-085934-marostegui.json
• 08:59 XioNoX: eqiad pfw - remove old LVS BGP config (replaced by bird) - T414015
• 08:55 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2164 (T413525)', diff saved to https://phabricator.wikimedia.org/P87286 and previous config saved to /var/cache/conftool/dbconfig/20260112-085459-marostegui.json
• 08:54 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2164.codfw.wmnet with reason: Maintenance
• 08:54 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T413525)', diff saved to https://phabricator.wikimedia.org/P87285 and previous config saved to /var/cache/conftool/dbconfig/20260112-085434-marostegui.json
• 08:49 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T413525)', diff saved to https://phabricator.wikimedia.org/P87284 and previous config saved to /var/cache/conftool/dbconfig/20260112-084926-marostegui.json
• 08:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P87283 and previous config saved to /var/cache/conftool/dbconfig/20260112-084426-marostegui.json
• 08:44 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1192 (T413525)', diff saved to https://phabricator.wikimedia.org/P87282 and previous config saved to /var/cache/conftool/dbconfig/20260112-084425-marostegui.json
• 08:44 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 08:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T413525)', diff saved to https://phabricator.wikimedia.org/P87281 and previous config saved to /var/cache/conftool/dbconfig/20260112-084401-marostegui.json
• 08:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P87280 and previous config saved to /var/cache/conftool/dbconfig/20260112-083418-marostegui.json
• 08:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P87279 and previous config saved to /var/cache/conftool/dbconfig/20260112-083353-marostegui.json
• 08:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T413525)', diff saved to https://phabricator.wikimedia.org/P87278 and previous config saved to /var/cache/conftool/dbconfig/20260112-082409-marostegui.json
• 08:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P87277 and previous config saved to /var/cache/conftool/dbconfig/20260112-082344-marostegui.json
• 08:19 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2163 (T413525)', diff saved to https://phabricator.wikimedia.org/P87276 and previous config saved to /var/cache/conftool/dbconfig/20260112-081912-marostegui.json
• 08:19 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2163.codfw.wmnet with reason: Maintenance
• 08:18 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T413525)', diff saved to https://phabricator.wikimedia.org/P87275 and previous config saved to /var/cache/conftool/dbconfig/20260112-081847-marostegui.json
• 08:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T413525)', diff saved to https://phabricator.wikimedia.org/P87274 and previous config saved to /var/cache/conftool/dbconfig/20260112-081336-marostegui.json
• 08:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P87273 and previous config saved to /var/cache/conftool/dbconfig/20260112-080839-marostegui.json
• 08:08 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1178 (T413525)', diff saved to https://phabricator.wikimedia.org/P87272 and previous config saved to /var/cache/conftool/dbconfig/20260112-080827-marostegui.json
• 08:08 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 08:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T413525)', diff saved to https://phabricator.wikimedia.org/P87271 and previous config saved to /var/cache/conftool/dbconfig/20260112-080813-marostegui.json
• 07:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P87270 and previous config saved to /var/cache/conftool/dbconfig/20260112-075830-marostegui.json
• 07:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P87269 and previous config saved to /var/cache/conftool/dbconfig/20260112-075805-marostegui.json
• 07:48 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T413525)', diff saved to https://phabricator.wikimedia.org/P87268 and previous config saved to /var/cache/conftool/dbconfig/20260112-074822-marostegui.json
• 07:47 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P87267 and previous config saved to /var/cache/conftool/dbconfig/20260112-074756-marostegui.json
• 07:43 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2154 (T413525)', diff saved to https://phabricator.wikimedia.org/P87266 and previous config saved to /var/cache/conftool/dbconfig/20260112-074325-marostegui.json
• 07:43 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2154.codfw.wmnet with reason: Maintenance
• 07:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T413525)', diff saved to https://phabricator.wikimedia.org/P87265 and previous config saved to /var/cache/conftool/dbconfig/20260112-074300-marostegui.json
• 07:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T413525)', diff saved to https://phabricator.wikimedia.org/P87264 and previous config saved to /var/cache/conftool/dbconfig/20260112-073748-marostegui.json
• 07:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P87263 and previous config saved to /var/cache/conftool/dbconfig/20260112-073251-marostegui.json
• 07:32 moritzm: updated trixie installer image to 13.3 T414179
• 07:32 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1177 (T413525)', diff saved to https://phabricator.wikimedia.org/P87262 and previous config saved to /var/cache/conftool/dbconfig/20260112-073242-marostegui.json
• 07:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 07:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T413525)', diff saved to https://phabricator.wikimedia.org/P87261 and previous config saved to /var/cache/conftool/dbconfig/20260112-073218-marostegui.json
• 07:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P87260 and previous config saved to /var/cache/conftool/dbconfig/20260112-072243-marostegui.json
• 07:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P87259 and previous config saved to /var/cache/conftool/dbconfig/20260112-072209-marostegui.json
• 07:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T413525)', diff saved to https://phabricator.wikimedia.org/P87258 and previous config saved to /var/cache/conftool/dbconfig/20260112-071235-marostegui.json
• 07:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P87257 and previous config saved to /var/cache/conftool/dbconfig/20260112-071201-marostegui.json
• 07:07 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2152 (T413525)', diff saved to https://phabricator.wikimedia.org/P87256 and previous config saved to /var/cache/conftool/dbconfig/20260112-070724-marostegui.json
• 07:07 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2152.codfw.wmnet with reason: Maintenance
• 07:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T413525)', diff saved to https://phabricator.wikimedia.org/P87255 and previous config saved to /var/cache/conftool/dbconfig/20260112-070153-marostegui.json
• 06:58 XioNoX: push pfw policy - T414116
• 06:56 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1172 (T413525)', diff saved to https://phabricator.wikimedia.org/P87254 and previous config saved to /var/cache/conftool/dbconfig/20260112-065646-marostegui.json
• 06:56 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 06:53 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 06:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T413525)', diff saved to https://phabricator.wikimedia.org/P87253 and previous config saved to /var/cache/conftool/dbconfig/20260112-065325-marostegui.json
• 06:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P87252 and previous config saved to /var/cache/conftool/dbconfig/20260112-064317-marostegui.json
• 06:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P87251 and previous config saved to /var/cache/conftool/dbconfig/20260112-063309-marostegui.json
• 06:24 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.newpool (exit_code=97) pool db2249.codfw.wmnet: Pooling for the first time in x1 T407941
• 06:24 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db2249.codfw.wmnet: Pooling for the first time in x1 T407941
• 06:23 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.newpool (exit_code=97) pool db2249: Pooling for the first time in x1 T407941
• 06:23 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db2249: Pooling for the first time in x1 T407941
• 06:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T413525)', diff saved to https://phabricator.wikimedia.org/P87249 and previous config saved to /var/cache/conftool/dbconfig/20260112-062300-marostegui.json
• 06:20 marostegui@cumin1003: END (ERROR) - Cookbook sre.mysql.newpool (exit_code=97) pool db2249: Pooling for the first time in x1 T407941
• 06:19 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db2249: Pooling for the first time in x1 T407941
• 06:18 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1167 (T413525)', diff saved to https://phabricator.wikimedia.org/P87248 and previous config saved to /var/cache/conftool/dbconfig/20260112-061800-marostegui.json
• 06:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 06:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 06:14 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2205.codfw.wmnet with reason: Maintenance
• 06:03 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2204.codfw.wmnet with reason: Maintenance
• 06:01 marostegui@cumin1003: dbctl commit (dc=all): 'Add db2249 to dbctl T407941', diff saved to https://phabricator.wikimedia.org/P87247 and previous config saved to /var/cache/conftool/dbconfig/20260112-060128-marostegui.json
• 05:55 marostegui: Disable GTID on db1195 for testing T315642
• 05:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2218.codfw.wmnet with reason: Maintenance
• 05:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1236.eqiad.wmnet with reason: Maintenance
• 05:39 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 05:38 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87246 and previous config saved to /var/cache/conftool/dbconfig/20260112-053846-marostegui.json
• 05:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P87245 and previous config saved to /var/cache/conftool/dbconfig/20260112-052838-marostegui.json
• 05:18 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P87244 and previous config saved to /var/cache/conftool/dbconfig/20260112-051829-marostegui.json
• 05:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87243 and previous config saved to /var/cache/conftool/dbconfig/20260112-050821-marostegui.json
• 04:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2239.codfw.wmnet with reason: Maintenance
• 04:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87242 and previous config saved to /var/cache/conftool/dbconfig/20260112-042700-marostegui.json
• 04:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P87241 and previous config saved to /var/cache/conftool/dbconfig/20260112-041652-marostegui.json
• 04:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237', diff saved to https://phabricator.wikimedia.org/P87240 and previous config saved to /var/cache/conftool/dbconfig/20260112-040643-marostegui.json
• 03:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2237 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87239 and previous config saved to /var/cache/conftool/dbconfig/20260112-035635-marostegui.json
• 00:50 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 00:50 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 00:50 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 00:49 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 00:49 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 00:49 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply

2026-01-11

• 19:12 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1244 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87238 and previous config saved to /var/cache/conftool/dbconfig/20260111-191238-marostegui.json
• 19:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 19:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87237 and previous config saved to /var/cache/conftool/dbconfig/20260111-191214-marostegui.json
• 19:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P87235 and previous config saved to /var/cache/conftool/dbconfig/20260111-190206-marostegui.json
• 18:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P87234 and previous config saved to /var/cache/conftool/dbconfig/20260111-185157-marostegui.json
• 18:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87233 and previous config saved to /var/cache/conftool/dbconfig/20260111-184150-marostegui.json
• 18:05 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2237 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87232 and previous config saved to /var/cache/conftool/dbconfig/20260111-180532-marostegui.json
• 18:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2237.codfw.wmnet with reason: Maintenance
• 18:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87231 and previous config saved to /var/cache/conftool/dbconfig/20260111-180507-marostegui.json
• 17:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P87230 and previous config saved to /var/cache/conftool/dbconfig/20260111-175459-marostegui.json
• 17:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236', diff saved to https://phabricator.wikimedia.org/P87229 and previous config saved to /var/cache/conftool/dbconfig/20260111-174451-marostegui.json
• 17:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2236 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87228 and previous config saved to /var/cache/conftool/dbconfig/20260111-173442-marostegui.json
• 15:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 (T413525)', diff saved to https://phabricator.wikimedia.org/P87227 and previous config saved to /var/cache/conftool/dbconfig/20260111-152746-marostegui.json
• 15:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P87226 and previous config saved to /var/cache/conftool/dbconfig/20260111-151738-marostegui.json
• 15:12 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 15:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 (T413525)', diff saved to https://phabricator.wikimedia.org/P87225 and previous config saved to /var/cache/conftool/dbconfig/20260111-151238-marostegui.json
• 15:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P87224 and previous config saved to /var/cache/conftool/dbconfig/20260111-150729-marostegui.json
• 15:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P87223 and previous config saved to /var/cache/conftool/dbconfig/20260111-150230-marostegui.json
• 14:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2222 (T413525)', diff saved to https://phabricator.wikimedia.org/P87222 and previous config saved to /var/cache/conftool/dbconfig/20260111-145721-marostegui.json
• 14:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P87221 and previous config saved to /var/cache/conftool/dbconfig/20260111-145221-marostegui.json
• 14:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1253 (T413525)', diff saved to https://phabricator.wikimedia.org/P87220 and previous config saved to /var/cache/conftool/dbconfig/20260111-144213-marostegui.json
• 14:29 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2222 (T413525)', diff saved to https://phabricator.wikimedia.org/P87219 and previous config saved to /var/cache/conftool/dbconfig/20260111-142919-marostegui.json
• 14:29 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2222.codfw.wmnet with reason: Maintenance
• 14:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T413525)', diff saved to https://phabricator.wikimedia.org/P87218 and previous config saved to /var/cache/conftool/dbconfig/20260111-142854-marostegui.json
• 14:28 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1253 (T413525)', diff saved to https://phabricator.wikimedia.org/P87217 and previous config saved to /var/cache/conftool/dbconfig/20260111-142759-marostegui.json
• 14:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1253.eqiad.wmnet with reason: Maintenance
• 14:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T413525)', diff saved to https://phabricator.wikimedia.org/P87216 and previous config saved to /var/cache/conftool/dbconfig/20260111-142735-marostegui.json
• 14:18 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P87215 and previous config saved to /var/cache/conftool/dbconfig/20260111-141846-marostegui.json
• 14:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P87214 and previous config saved to /var/cache/conftool/dbconfig/20260111-141726-marostegui.json
• 14:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P87213 and previous config saved to /var/cache/conftool/dbconfig/20260111-140837-marostegui.json
• 14:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P87212 and previous config saved to /var/cache/conftool/dbconfig/20260111-140718-marostegui.json
• 13:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T413525)', diff saved to https://phabricator.wikimedia.org/P87211 and previous config saved to /var/cache/conftool/dbconfig/20260111-135829-marostegui.json
• 13:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T413525)', diff saved to https://phabricator.wikimedia.org/P87210 and previous config saved to /var/cache/conftool/dbconfig/20260111-135710-marostegui.json
• 13:43 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1231 (T413525)', diff saved to https://phabricator.wikimedia.org/P87209 and previous config saved to /var/cache/conftool/dbconfig/20260111-134355-marostegui.json
• 13:43 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1231.eqiad.wmnet with reason: Maintenance
• 13:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T413525)', diff saved to https://phabricator.wikimedia.org/P87208 and previous config saved to /var/cache/conftool/dbconfig/20260111-134330-marostegui.json
• 13:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P87207 and previous config saved to /var/cache/conftool/dbconfig/20260111-133322-marostegui.json
• 13:30 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2221 (T413525)', diff saved to https://phabricator.wikimedia.org/P87206 and previous config saved to /var/cache/conftool/dbconfig/20260111-133019-marostegui.json
• 13:30 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2221.codfw.wmnet with reason: Maintenance
• 13:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T413525)', diff saved to https://phabricator.wikimedia.org/P87205 and previous config saved to /var/cache/conftool/dbconfig/20260111-132955-marostegui.json
• 13:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P87204 and previous config saved to /var/cache/conftool/dbconfig/20260111-132314-marostegui.json
• 13:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P87203 and previous config saved to /var/cache/conftool/dbconfig/20260111-131946-marostegui.json
• 13:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T413525)', diff saved to https://phabricator.wikimedia.org/P87202 and previous config saved to /var/cache/conftool/dbconfig/20260111-131305-marostegui.json
• 13:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P87201 and previous config saved to /var/cache/conftool/dbconfig/20260111-130938-marostegui.json
• 12:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T413525)', diff saved to https://phabricator.wikimedia.org/P87200 and previous config saved to /var/cache/conftool/dbconfig/20260111-125930-marostegui.json
• 12:45 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1227 (T413525)', diff saved to https://phabricator.wikimedia.org/P87199 and previous config saved to /var/cache/conftool/dbconfig/20260111-124525-marostegui.json
• 12:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1227.eqiad.wmnet with reason: Maintenance
• 12:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T413525)', diff saved to https://phabricator.wikimedia.org/P87198 and previous config saved to /var/cache/conftool/dbconfig/20260111-124501-marostegui.json
• 12:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P87197 and previous config saved to /var/cache/conftool/dbconfig/20260111-123452-marostegui.json
• 12:32 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2220 (T413525)', diff saved to https://phabricator.wikimedia.org/P87196 and previous config saved to /var/cache/conftool/dbconfig/20260111-123205-marostegui.json
• 12:31 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2220.codfw.wmnet with reason: Maintenance
• 12:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T413525)', diff saved to https://phabricator.wikimedia.org/P87195 and previous config saved to /var/cache/conftool/dbconfig/20260111-123140-marostegui.json
• 12:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P87194 and previous config saved to /var/cache/conftool/dbconfig/20260111-122444-marostegui.json
• 12:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P87193 and previous config saved to /var/cache/conftool/dbconfig/20260111-122132-marostegui.json
• 12:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T413525)', diff saved to https://phabricator.wikimedia.org/P87192 and previous config saved to /var/cache/conftool/dbconfig/20260111-121436-marostegui.json
• 12:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P87191 and previous config saved to /var/cache/conftool/dbconfig/20260111-121124-marostegui.json
• 12:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1202 (T413525)', diff saved to https://phabricator.wikimedia.org/P87190 and previous config saved to /var/cache/conftool/dbconfig/20260111-120139-marostegui.json
• 12:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1202.eqiad.wmnet with reason: Maintenance
• 12:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T413525)', diff saved to https://phabricator.wikimedia.org/P87189 and previous config saved to /var/cache/conftool/dbconfig/20260111-120116-marostegui.json
• 12:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T413525)', diff saved to https://phabricator.wikimedia.org/P87188 and previous config saved to /var/cache/conftool/dbconfig/20260111-120115-marostegui.json
• 11:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P87187 and previous config saved to /var/cache/conftool/dbconfig/20260111-115107-marostegui.json
• 11:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P87186 and previous config saved to /var/cache/conftool/dbconfig/20260111-114059-marostegui.json
• 11:33 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2208 (T413525)', diff saved to https://phabricator.wikimedia.org/P87185 and previous config saved to /var/cache/conftool/dbconfig/20260111-113331-marostegui.json
• 11:33 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2208.codfw.wmnet with reason: Maintenance
• 11:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T413525)', diff saved to https://phabricator.wikimedia.org/P87184 and previous config saved to /var/cache/conftool/dbconfig/20260111-113051-marostegui.json
• 11:18 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1194 (T413525)', diff saved to https://phabricator.wikimedia.org/P87183 and previous config saved to /var/cache/conftool/dbconfig/20260111-111805-marostegui.json
• 11:17 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1194.eqiad.wmnet with reason: Maintenance
• 11:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T413525)', diff saved to https://phabricator.wikimedia.org/P87182 and previous config saved to /var/cache/conftool/dbconfig/20260111-111751-marostegui.json
• 11:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P87181 and previous config saved to /var/cache/conftool/dbconfig/20260111-110743-marostegui.json
• 11:07 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2200.codfw.wmnet with reason: Maintenance
• 10:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P87180 and previous config saved to /var/cache/conftool/dbconfig/20260111-105735-marostegui.json
• 10:47 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T413525)', diff saved to https://phabricator.wikimedia.org/P87179 and previous config saved to /var/cache/conftool/dbconfig/20260111-104726-marostegui.json
• 10:42 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2198.codfw.wmnet with reason: Maintenance
• 10:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T413525)', diff saved to https://phabricator.wikimedia.org/P87178 and previous config saved to /var/cache/conftool/dbconfig/20260111-104219-marostegui.json
• 10:34 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1191 (T413525)', diff saved to https://phabricator.wikimedia.org/P87177 and previous config saved to /var/cache/conftool/dbconfig/20260111-103435-marostegui.json
• 10:34 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1191.eqiad.wmnet with reason: Maintenance
• 10:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T413525)', diff saved to https://phabricator.wikimedia.org/P87176 and previous config saved to /var/cache/conftool/dbconfig/20260111-103409-marostegui.json
• 10:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P87175 and previous config saved to /var/cache/conftool/dbconfig/20260111-103211-marostegui.json
• 10:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P87174 and previous config saved to /var/cache/conftool/dbconfig/20260111-102401-marostegui.json
• 10:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P87173 and previous config saved to /var/cache/conftool/dbconfig/20260111-102203-marostegui.json
• 10:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P87172 and previous config saved to /var/cache/conftool/dbconfig/20260111-101352-marostegui.json
• 10:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T413525)', diff saved to https://phabricator.wikimedia.org/P87171 and previous config saved to /var/cache/conftool/dbconfig/20260111-101155-marostegui.json
• 10:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T413525)', diff saved to https://phabricator.wikimedia.org/P87170 and previous config saved to /var/cache/conftool/dbconfig/20260111-100344-marostegui.json
• 09:49 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1181 (T413525)', diff saved to https://phabricator.wikimedia.org/P87169 and previous config saved to /var/cache/conftool/dbconfig/20260111-094928-marostegui.json
• 09:49 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance
• 09:49 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T413525)', diff saved to https://phabricator.wikimedia.org/P87168 and previous config saved to /var/cache/conftool/dbconfig/20260111-094903-marostegui.json
• 09:39 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2182 (T413525)', diff saved to https://phabricator.wikimedia.org/P87167 and previous config saved to /var/cache/conftool/dbconfig/20260111-093907-marostegui.json
• 09:38 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P87166 and previous config saved to /var/cache/conftool/dbconfig/20260111-093855-marostegui.json
• 09:38 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2182.codfw.wmnet with reason: Maintenance
• 09:38 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T413525)', diff saved to https://phabricator.wikimedia.org/P87165 and previous config saved to /var/cache/conftool/dbconfig/20260111-093836-marostegui.json
• 09:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P87164 and previous config saved to /var/cache/conftool/dbconfig/20260111-092846-marostegui.json
• 09:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P87163 and previous config saved to /var/cache/conftool/dbconfig/20260111-092828-marostegui.json
• 09:18 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T413525)', diff saved to https://phabricator.wikimedia.org/P87162 and previous config saved to /var/cache/conftool/dbconfig/20260111-091838-marostegui.json
• 09:18 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P87161 and previous config saved to /var/cache/conftool/dbconfig/20260111-091819-marostegui.json
• 09:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T413525)', diff saved to https://phabricator.wikimedia.org/P87160 and previous config saved to /var/cache/conftool/dbconfig/20260111-090811-marostegui.json
• 09:04 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1174 (T413525)', diff saved to https://phabricator.wikimedia.org/P87159 and previous config saved to /var/cache/conftool/dbconfig/20260111-090455-marostegui.json
• 09:04 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
• 08:37 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1243 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87158 and previous config saved to /var/cache/conftool/dbconfig/20260111-083753-marostegui.json
• 08:37 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1243.eqiad.wmnet with reason: Maintenance
• 08:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87157 and previous config saved to /var/cache/conftool/dbconfig/20260111-083729-marostegui.json
• 08:37 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2168 (T413525)', diff saved to https://phabricator.wikimedia.org/P87156 and previous config saved to /var/cache/conftool/dbconfig/20260111-083659-marostegui.json
• 08:36 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2168.codfw.wmnet with reason: Maintenance
• 08:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T413525)', diff saved to https://phabricator.wikimedia.org/P87155 and previous config saved to /var/cache/conftool/dbconfig/20260111-083634-marostegui.json
• 08:35 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 08:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T413525)', diff saved to https://phabricator.wikimedia.org/P87154 and previous config saved to /var/cache/conftool/dbconfig/20260111-083519-marostegui.json
• 08:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P87153 and previous config saved to /var/cache/conftool/dbconfig/20260111-082721-marostegui.json
• 08:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P87152 and previous config saved to /var/cache/conftool/dbconfig/20260111-082626-marostegui.json
• 08:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P87151 and previous config saved to /var/cache/conftool/dbconfig/20260111-082511-marostegui.json
• 08:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P87150 and previous config saved to /var/cache/conftool/dbconfig/20260111-081712-marostegui.json
• 08:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P87149 and previous config saved to /var/cache/conftool/dbconfig/20260111-081617-marostegui.json
• 08:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P87148 and previous config saved to /var/cache/conftool/dbconfig/20260111-081502-marostegui.json
• 08:07 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2236 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87147 and previous config saved to /var/cache/conftool/dbconfig/20260111-080744-marostegui.json
• 08:07 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2236.codfw.wmnet with reason: Maintenance
• 08:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87146 and previous config saved to /var/cache/conftool/dbconfig/20260111-080720-marostegui.json
• 08:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87145 and previous config saved to /var/cache/conftool/dbconfig/20260111-080704-marostegui.json
• 08:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T413525)', diff saved to https://phabricator.wikimedia.org/P87144 and previous config saved to /var/cache/conftool/dbconfig/20260111-080609-marostegui.json
• 08:04 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T413525)', diff saved to https://phabricator.wikimedia.org/P87143 and previous config saved to /var/cache/conftool/dbconfig/20260111-080454-marostegui.json
• 07:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P87142 and previous config saved to /var/cache/conftool/dbconfig/20260111-075712-marostegui.json
• 07:47 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P87141 and previous config saved to /var/cache/conftool/dbconfig/20260111-074703-marostegui.json
• 07:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87140 and previous config saved to /var/cache/conftool/dbconfig/20260111-073655-marostegui.json
• 07:34 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2159 (T413525)', diff saved to https://phabricator.wikimedia.org/P87139 and previous config saved to /var/cache/conftool/dbconfig/20260111-073453-marostegui.json
• 07:34 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2159.codfw.wmnet with reason: Maintenance
• 07:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T413525)', diff saved to https://phabricator.wikimedia.org/P87138 and previous config saved to /var/cache/conftool/dbconfig/20260111-073429-marostegui.json
• 07:33 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1170 (T413525)', diff saved to https://phabricator.wikimedia.org/P87137 and previous config saved to /var/cache/conftool/dbconfig/20260111-073354-marostegui.json
• 07:33 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
• 07:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T413525)', diff saved to https://phabricator.wikimedia.org/P87136 and previous config saved to /var/cache/conftool/dbconfig/20260111-073330-marostegui.json
• 07:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P87135 and previous config saved to /var/cache/conftool/dbconfig/20260111-072421-marostegui.json
• 07:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P87134 and previous config saved to /var/cache/conftool/dbconfig/20260111-072322-marostegui.json
• 07:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P87133 and previous config saved to /var/cache/conftool/dbconfig/20260111-071412-marostegui.json
• 07:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P87132 and previous config saved to /var/cache/conftool/dbconfig/20260111-071314-marostegui.json
• 07:04 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T413525)', diff saved to https://phabricator.wikimedia.org/P87131 and previous config saved to /var/cache/conftool/dbconfig/20260111-070404-marostegui.json
• 07:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T413525)', diff saved to https://phabricator.wikimedia.org/P87130 and previous config saved to /var/cache/conftool/dbconfig/20260111-070306-marostegui.json
• 06:50 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1158 (T413525)', diff saved to https://phabricator.wikimedia.org/P87129 and previous config saved to /var/cache/conftool/dbconfig/20260111-065018-marostegui.json
• 06:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 06:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1158.eqiad.wmnet with reason: Maintenance
• 06:32 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2150 (T413525)', diff saved to https://phabricator.wikimedia.org/P87128 and previous config saved to /var/cache/conftool/dbconfig/20260111-063232-marostegui.json
• 06:32 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance
• 06:30 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1223.eqiad.wmnet with reason: Maintenance
• 04:36 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2239.codfw.wmnet with reason: Maintenance
• 04:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 (T413525)', diff saved to https://phabricator.wikimedia.org/P87127 and previous config saved to /var/cache/conftool/dbconfig/20260111-043614-marostegui.json
• 04:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P87126 and previous config saved to /var/cache/conftool/dbconfig/20260111-042606-marostegui.json
• 04:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P87125 and previous config saved to /var/cache/conftool/dbconfig/20260111-041558-marostegui.json
• 04:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2227 (T413525)', diff saved to https://phabricator.wikimedia.org/P87124 and previous config saved to /var/cache/conftool/dbconfig/20260111-040549-marostegui.json
• 03:20 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2227 (T413525)', diff saved to https://phabricator.wikimedia.org/P87123 and previous config saved to /var/cache/conftool/dbconfig/20260111-032015-marostegui.json
• 03:20 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2227.codfw.wmnet with reason: Maintenance
• 03:20 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T413525)', diff saved to https://phabricator.wikimedia.org/P87122 and previous config saved to /var/cache/conftool/dbconfig/20260111-032001-marostegui.json
• 03:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P87121 and previous config saved to /var/cache/conftool/dbconfig/20260111-030953-marostegui.json
• 02:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P87120 and previous config saved to /var/cache/conftool/dbconfig/20260111-025945-marostegui.json
• 02:49 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T413525)', diff saved to https://phabricator.wikimedia.org/P87119 and previous config saved to /var/cache/conftool/dbconfig/20260111-024936-marostegui.json
• 02:05 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2209 (T413525)', diff saved to https://phabricator.wikimedia.org/P87118 and previous config saved to /var/cache/conftool/dbconfig/20260111-020520-marostegui.json
• 02:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2209.codfw.wmnet with reason: Maintenance
• 02:04 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T413525)', diff saved to https://phabricator.wikimedia.org/P87117 and previous config saved to /var/cache/conftool/dbconfig/20260111-020454-marostegui.json
• 01:54 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P87116 and previous config saved to /var/cache/conftool/dbconfig/20260111-015446-marostegui.json
• 01:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P87115 and previous config saved to /var/cache/conftool/dbconfig/20260111-014438-marostegui.json
• 01:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T413525)', diff saved to https://phabricator.wikimedia.org/P87114 and previous config saved to /var/cache/conftool/dbconfig/20260111-013429-marostegui.json
• 00:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2194 (T413525)', diff saved to https://phabricator.wikimedia.org/P87113 and previous config saved to /var/cache/conftool/dbconfig/20260111-004632-marostegui.json
• 00:46 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2194.codfw.wmnet with reason: Maintenance
• 00:46 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T413525)', diff saved to https://phabricator.wikimedia.org/P87112 and previous config saved to /var/cache/conftool/dbconfig/20260111-004608-marostegui.json
• 00:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P87111 and previous config saved to /var/cache/conftool/dbconfig/20260111-003559-marostegui.json
• 00:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 00:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P87110 and previous config saved to /var/cache/conftool/dbconfig/20260111-002551-marostegui.json
• 00:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T413525)', diff saved to https://phabricator.wikimedia.org/P87109 and previous config saved to /var/cache/conftool/dbconfig/20260111-001543-marostegui.json

2026-01-10

• 23:55 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 23:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T413525)', diff saved to https://phabricator.wikimedia.org/P87108 and previous config saved to /var/cache/conftool/dbconfig/20260110-235506-marostegui.json
• 23:44 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P87107 and previous config saved to /var/cache/conftool/dbconfig/20260110-234458-marostegui.json
• 23:34 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P87106 and previous config saved to /var/cache/conftool/dbconfig/20260110-233450-marostegui.json
• 23:28 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2190 (T413525)', diff saved to https://phabricator.wikimedia.org/P87105 and previous config saved to /var/cache/conftool/dbconfig/20260110-232856-marostegui.json
• 23:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2190.codfw.wmnet with reason: Maintenance
• 23:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T413525)', diff saved to https://phabricator.wikimedia.org/P87104 and previous config saved to /var/cache/conftool/dbconfig/20260110-232832-marostegui.json
• 23:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T413525)', diff saved to https://phabricator.wikimedia.org/P87103 and previous config saved to /var/cache/conftool/dbconfig/20260110-232441-marostegui.json
• 23:18 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P87102 and previous config saved to /var/cache/conftool/dbconfig/20260110-231824-marostegui.json
• 23:09 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1212 (T413525)', diff saved to https://phabricator.wikimedia.org/P87101 and previous config saved to /var/cache/conftool/dbconfig/20260110-230930-marostegui.json
• 23:09 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance
• 23:09 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1212.eqiad.wmnet with reason: Maintenance
• 23:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T413525)', diff saved to https://phabricator.wikimedia.org/P87100 and previous config saved to /var/cache/conftool/dbconfig/20260110-230843-marostegui.json
• 23:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P87099 and previous config saved to /var/cache/conftool/dbconfig/20260110-230816-marostegui.json
• 22:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P87098 and previous config saved to /var/cache/conftool/dbconfig/20260110-225835-marostegui.json
• 22:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T413525)', diff saved to https://phabricator.wikimedia.org/P87097 and previous config saved to /var/cache/conftool/dbconfig/20260110-225807-marostegui.json
• 22:48 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P87096 and previous config saved to /var/cache/conftool/dbconfig/20260110-224826-marostegui.json
• 22:38 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T413525)', diff saved to https://phabricator.wikimedia.org/P87095 and previous config saved to /var/cache/conftool/dbconfig/20260110-223818-marostegui.json
• 22:23 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1198 (T413525)', diff saved to https://phabricator.wikimedia.org/P87094 and previous config saved to /var/cache/conftool/dbconfig/20260110-222354-marostegui.json
• 22:23 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
• 22:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T413525)', diff saved to https://phabricator.wikimedia.org/P87093 and previous config saved to /var/cache/conftool/dbconfig/20260110-222330-marostegui.json
• 22:22 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1242 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87092 and previous config saved to /var/cache/conftool/dbconfig/20260110-222231-marostegui.json
• 22:22 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance
• 22:22 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87091 and previous config saved to /var/cache/conftool/dbconfig/20260110-222207-marostegui.json
• 22:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P87090 and previous config saved to /var/cache/conftool/dbconfig/20260110-221322-marostegui.json
• 22:11 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P87089 and previous config saved to /var/cache/conftool/dbconfig/20260110-221158-marostegui.json
• 22:07 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2177 (T413525)', diff saved to https://phabricator.wikimedia.org/P87088 and previous config saved to /var/cache/conftool/dbconfig/20260110-220750-marostegui.json
• 22:07 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
• 22:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T413525)', diff saved to https://phabricator.wikimedia.org/P87087 and previous config saved to /var/cache/conftool/dbconfig/20260110-220725-marostegui.json
• 22:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P87086 and previous config saved to /var/cache/conftool/dbconfig/20260110-220314-marostegui.json
• 22:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P87085 and previous config saved to /var/cache/conftool/dbconfig/20260110-220150-marostegui.json
• 21:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P87084 and previous config saved to /var/cache/conftool/dbconfig/20260110-215716-marostegui.json
• 21:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T413525)', diff saved to https://phabricator.wikimedia.org/P87083 and previous config saved to /var/cache/conftool/dbconfig/20260110-215305-marostegui.json
• 21:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87082 and previous config saved to /var/cache/conftool/dbconfig/20260110-215142-marostegui.json
• 21:51 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2219 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87081 and previous config saved to /var/cache/conftool/dbconfig/20260110-215104-marostegui.json
• 21:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance
• 21:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87080 and previous config saved to /var/cache/conftool/dbconfig/20260110-215037-marostegui.json
• 21:47 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P87079 and previous config saved to /var/cache/conftool/dbconfig/20260110-214708-marostegui.json
• 21:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P87078 and previous config saved to /var/cache/conftool/dbconfig/20260110-214023-marostegui.json
• 21:38 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1189 (T413525)', diff saved to https://phabricator.wikimedia.org/P87077 and previous config saved to /var/cache/conftool/dbconfig/20260110-213801-marostegui.json
• 21:37 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
• 21:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T413525)', diff saved to https://phabricator.wikimedia.org/P87076 and previous config saved to /var/cache/conftool/dbconfig/20260110-213736-marostegui.json
• 21:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T413525)', diff saved to https://phabricator.wikimedia.org/P87075 and previous config saved to /var/cache/conftool/dbconfig/20260110-213700-marostegui.json
• 21:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P87074 and previous config saved to /var/cache/conftool/dbconfig/20260110-213015-marostegui.json
• 21:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P87073 and previous config saved to /var/cache/conftool/dbconfig/20260110-212728-marostegui.json
• 21:20 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87072 and previous config saved to /var/cache/conftool/dbconfig/20260110-212006-marostegui.json
• 21:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P87071 and previous config saved to /var/cache/conftool/dbconfig/20260110-211720-marostegui.json
• 21:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T413525)', diff saved to https://phabricator.wikimedia.org/P87070 and previous config saved to /var/cache/conftool/dbconfig/20260110-210711-marostegui.json
• 20:52 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1175 (T413525)', diff saved to https://phabricator.wikimedia.org/P87069 and previous config saved to /var/cache/conftool/dbconfig/20260110-205226-marostegui.json
• 20:52 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
• 20:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T413525)', diff saved to https://phabricator.wikimedia.org/P87068 and previous config saved to /var/cache/conftool/dbconfig/20260110-205201-marostegui.json
• 20:48 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2156 (T413525)', diff saved to https://phabricator.wikimedia.org/P87067 and previous config saved to /var/cache/conftool/dbconfig/20260110-204804-marostegui.json
• 20:47 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
• 20:47 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T413525)', diff saved to https://phabricator.wikimedia.org/P87066 and previous config saved to /var/cache/conftool/dbconfig/20260110-204739-marostegui.json
• 20:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P87065 and previous config saved to /var/cache/conftool/dbconfig/20260110-204154-marostegui.json
• 20:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P87064 and previous config saved to /var/cache/conftool/dbconfig/20260110-203731-marostegui.json
• 20:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P87063 and previous config saved to /var/cache/conftool/dbconfig/20260110-203146-marostegui.json
• 20:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P87062 and previous config saved to /var/cache/conftool/dbconfig/20260110-202722-marostegui.json
• 20:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T413525)', diff saved to https://phabricator.wikimedia.org/P87061 and previous config saved to /var/cache/conftool/dbconfig/20260110-202138-marostegui.json
• 20:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T413525)', diff saved to https://phabricator.wikimedia.org/P87060 and previous config saved to /var/cache/conftool/dbconfig/20260110-201714-marostegui.json
• 20:06 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1166 (T413525)', diff saved to https://phabricator.wikimedia.org/P87059 and previous config saved to /var/cache/conftool/dbconfig/20260110-200615-marostegui.json
• 20:06 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
• 20:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T413525)', diff saved to https://phabricator.wikimedia.org/P87058 and previous config saved to /var/cache/conftool/dbconfig/20260110-200601-marostegui.json
• 19:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P87057 and previous config saved to /var/cache/conftool/dbconfig/20260110-195553-marostegui.json
• 19:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P87056 and previous config saved to /var/cache/conftool/dbconfig/20260110-194544-marostegui.json
• 19:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T413525)', diff saved to https://phabricator.wikimedia.org/P87055 and previous config saved to /var/cache/conftool/dbconfig/20260110-193536-marostegui.json
• 19:27 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2149 (T413525)', diff saved to https://phabricator.wikimedia.org/P87054 and previous config saved to /var/cache/conftool/dbconfig/20260110-192742-marostegui.json
• 19:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
• 19:21 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1157 (T413525)', diff saved to https://phabricator.wikimedia.org/P87053 and previous config saved to /var/cache/conftool/dbconfig/20260110-192058-marostegui.json
• 19:20 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
• 18:42 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 18:13 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1222.eqiad.wmnet with reason: Maintenance
• 11:31 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2210 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87052 and previous config saved to /var/cache/conftool/dbconfig/20260110-113128-marostegui.json
• 11:31 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance
• 11:31 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1241 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87051 and previous config saved to /var/cache/conftool/dbconfig/20260110-113110-marostegui.json
• 11:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87050 and previous config saved to /var/cache/conftool/dbconfig/20260110-113104-marostegui.json
• 11:31 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance
• 11:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87049 and previous config saved to /var/cache/conftool/dbconfig/20260110-113045-marostegui.json
• 11:20 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P87048 and previous config saved to /var/cache/conftool/dbconfig/20260110-112055-marostegui.json
• 11:20 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P87047 and previous config saved to /var/cache/conftool/dbconfig/20260110-112037-marostegui.json
• 11:10 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P87046 and previous config saved to /var/cache/conftool/dbconfig/20260110-111047-marostegui.json
• 11:10 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P87045 and previous config saved to /var/cache/conftool/dbconfig/20260110-111028-marostegui.json
• 11:00 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87044 and previous config saved to /var/cache/conftool/dbconfig/20260110-110039-marostegui.json
• 11:00 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87043 and previous config saved to /var/cache/conftool/dbconfig/20260110-110020-marostegui.json
• 01:55 zabe@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
• 01:54 zabe@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
• 01:16 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1238 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87042 and previous config saved to /var/cache/conftool/dbconfig/20260110-011558-marostegui.json
• 01:15 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance
• 01:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87041 and previous config saved to /var/cache/conftool/dbconfig/20260110-011534-marostegui.json
• 01:13 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 13m 00s)
• 01:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P87040 and previous config saved to /var/cache/conftool/dbconfig/20260110-010525-marostegui.json
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image
• 00:59 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2206 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87039 and previous config saved to /var/cache/conftool/dbconfig/20260110-005937-marostegui.json
• 00:59 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance
• 00:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P87038 and previous config saved to /var/cache/conftool/dbconfig/20260110-005517-marostegui.json
• 00:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P87037 and previous config saved to /var/cache/conftool/dbconfig/20260110-004509-marostegui.json

2026-01-09

• 21:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 21:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 (T413525)', diff saved to https://phabricator.wikimedia.org/P87035 and previous config saved to /var/cache/conftool/dbconfig/20260109-212830-marostegui.json
• 21:18 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P87034 and previous config saved to /var/cache/conftool/dbconfig/20260109-211822-marostegui.json
• 21:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P87033 and previous config saved to /var/cache/conftool/dbconfig/20260109-210813-marostegui.json
• 20:58 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1259 (T413525)', diff saved to https://phabricator.wikimedia.org/P87032 and previous config saved to /var/cache/conftool/dbconfig/20260109-205805-marostegui.json
• 20:28 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1259 (T413525)', diff saved to https://phabricator.wikimedia.org/P87031 and previous config saved to /var/cache/conftool/dbconfig/20260109-202821-marostegui.json
• 20:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1259.eqiad.wmnet with reason: Maintenance
• 20:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 (T413525)', diff saved to https://phabricator.wikimedia.org/P87030 and previous config saved to /var/cache/conftool/dbconfig/20260109-202756-marostegui.json
• 20:23 jhathaway@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on wdqs1029.eqiad.wmnet with reason: T412451
• 20:21 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol1008-dev.eqiad.wmnet with OS trixie
• 20:19 jhathaway@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1029.eqiad.wmnet with OS trixie
• 20:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P87029 and previous config saved to /var/cache/conftool/dbconfig/20260109-201748-marostegui.json
• 20:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P87028 and previous config saved to /var/cache/conftool/dbconfig/20260109-200739-marostegui.json
• 20:02 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol1008-dev.eqiad.wmnet with reason: host reimage
• 20:00 jhathaway@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1029.eqiad.wmnet with reason: host reimage
• 19:59 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol1008-dev.eqiad.wmnet with reason: host reimage
• 19:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1254 (T413525)', diff saved to https://phabricator.wikimedia.org/P87027 and previous config saved to /var/cache/conftool/dbconfig/20260109-195731-marostegui.json
• 19:54 jhathaway@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1029.eqiad.wmnet with reason: host reimage
• 19:49 cstone: payments-wiki upgraded from 02dbb1e2 to 14d8501a
• 19:43 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcontrol1008-dev.eqiad.wmnet with OS trixie
• 19:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T413525)', diff saved to https://phabricator.wikimedia.org/P87026 and previous config saved to /var/cache/conftool/dbconfig/20260109-194001-marostegui.json
• 19:36 jhathaway@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1029.eqiad.wmnet with OS trixie
• 19:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P87025 and previous config saved to /var/cache/conftool/dbconfig/20260109-192953-marostegui.json
• 19:28 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1254 (T413525)', diff saved to https://phabricator.wikimedia.org/P87024 and previous config saved to /var/cache/conftool/dbconfig/20260109-192844-marostegui.json
• 19:28 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1254.eqiad.wmnet with reason: Maintenance
• 19:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P87023 and previous config saved to /var/cache/conftool/dbconfig/20260109-191944-marostegui.json
• 19:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T413525)', diff saved to https://phabricator.wikimedia.org/P87022 and previous config saved to /var/cache/conftool/dbconfig/20260109-190936-marostegui.json
• 19:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 19:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T413525)', diff saved to https://phabricator.wikimedia.org/P87021 and previous config saved to /var/cache/conftool/dbconfig/20260109-190128-marostegui.json
• 18:57 jhathaway@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1029.eqiad.wmnet with OS trixie
• 18:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P87020 and previous config saved to /var/cache/conftool/dbconfig/20260109-185120-marostegui.json
• 18:49 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:47 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 18:46 cmooney@dns2005: END - running authdns-update
• 18:46 cmooney@dns2005: START - running authdns-update
• 18:44 cmooney@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 18:44 cmooney@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add DNS for codfw lsw1-a4 to mr1-codfw IPv6 IPs - cmooney@cumin1003"
• 18:44 cmooney@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add DNS for codfw lsw1-a4 to mr1-codfw IPv6 IPs - cmooney@cumin1003"
• 18:41 cmooney@cumin1003: START - Cookbook sre.dns.netbox
• 18:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P87019 and previous config saved to /var/cache/conftool/dbconfig/20260109-184111-marostegui.json
• 18:39 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2238 (T413525)', diff saved to https://phabricator.wikimedia.org/P87018 and previous config saved to /var/cache/conftool/dbconfig/20260109-183939-marostegui.json
• 18:39 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2238.codfw.wmnet with reason: Maintenance
• 18:39 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T413525)', diff saved to https://phabricator.wikimedia.org/P87017 and previous config saved to /var/cache/conftool/dbconfig/20260109-183926-marostegui.json
• 18:39 jhathaway@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1029.eqiad.wmnet with reason: host reimage
• 18:31 jhathaway@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1029.eqiad.wmnet with reason: host reimage
• 18:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T413525)', diff saved to https://phabricator.wikimedia.org/P87016 and previous config saved to /var/cache/conftool/dbconfig/20260109-183103-marostegui.json
• 18:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P87015 and previous config saved to /var/cache/conftool/dbconfig/20260109-182917-marostegui.json
• 18:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P87014 and previous config saved to /var/cache/conftool/dbconfig/20260109-181908-marostegui.json
• 18:13 jhathaway@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1029.eqiad.wmnet with OS trixie
• 18:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T413525)', diff saved to https://phabricator.wikimedia.org/P87013 and previous config saved to /var/cache/conftool/dbconfig/20260109-180900-marostegui.json
• 18:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1233 (T413525)', diff saved to https://phabricator.wikimedia.org/P87012 and previous config saved to /var/cache/conftool/dbconfig/20260109-180112-marostegui.json
• 18:01 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1233.eqiad.wmnet with reason: Maintenance
• 18:00 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T413525)', diff saved to https://phabricator.wikimedia.org/P87011 and previous config saved to /var/cache/conftool/dbconfig/20260109-180047-marostegui.json
• 17:56 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2226 (T413525)', diff saved to https://phabricator.wikimedia.org/P87010 and previous config saved to /var/cache/conftool/dbconfig/20260109-175609-marostegui.json
• 17:56 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2226.codfw.wmnet with reason: Maintenance
• 17:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T413525)', diff saved to https://phabricator.wikimedia.org/P87009 and previous config saved to /var/cache/conftool/dbconfig/20260109-175544-marostegui.json
• 17:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P87008 and previous config saved to /var/cache/conftool/dbconfig/20260109-175039-marostegui.json
• 17:48 jhathaway@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1029.eqiad.wmnet with OS trixie
• 17:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P87007 and previous config saved to /var/cache/conftool/dbconfig/20260109-174536-marostegui.json
• 17:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P87006 and previous config saved to /var/cache/conftool/dbconfig/20260109-174031-marostegui.json
• 17:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P87005 and previous config saved to /var/cache/conftool/dbconfig/20260109-173527-marostegui.json
• 17:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T413525)', diff saved to https://phabricator.wikimedia.org/P87004 and previous config saved to /var/cache/conftool/dbconfig/20260109-173023-marostegui.json
• 17:29 jhathaway@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1029.eqiad.wmnet with reason: host reimage
• 17:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T413525)', diff saved to https://phabricator.wikimedia.org/P87003 and previous config saved to /var/cache/conftool/dbconfig/20260109-172519-marostegui.json
• 17:23 jhathaway@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1029.eqiad.wmnet with reason: host reimage
• 17:13 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:13 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add missing DNS for IPV6 - pt1979@cumin2002"
• 17:13 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add missing DNS for IPV6 - pt1979@cumin2002"
• 17:10 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 17:07 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 17:07 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add missing DNS for IPV6 - pt1979@cumin2002"
• 17:07 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add missing DNS for IPV6 - pt1979@cumin2002"
• 17:04 jhathaway@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1029.eqiad.wmnet with OS trixie
• 17:00 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1229 (T413525)', diff saved to https://phabricator.wikimedia.org/P87002 and previous config saved to /var/cache/conftool/dbconfig/20260109-170033-marostegui.json
• 17:00 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1229.eqiad.wmnet with reason: Maintenance
• 16:57 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 16:56 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 16:56 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2225 (T413525)', diff saved to https://phabricator.wikimedia.org/P87001 and previous config saved to /var/cache/conftool/dbconfig/20260109-165619-marostegui.json
• 16:56 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 16:56 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2225.codfw.wmnet with reason: Maintenance
• 16:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T413525)', diff saved to https://phabricator.wikimedia.org/P87000 and previous config saved to /var/cache/conftool/dbconfig/20260109-165554-marostegui.json
• 16:54 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 16:54 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add missing DNS for IPV6 - pt1979@cumin2002"
• 16:54 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add missing DNS for IPV6 - pt1979@cumin2002"
• 16:49 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 16:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P86999 and previous config saved to /var/cache/conftool/dbconfig/20260109-164546-marostegui.json
• 16:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P86998 and previous config saved to /var/cache/conftool/dbconfig/20260109-163538-marostegui.json
• 16:33 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 16:33 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T413525)', diff saved to https://phabricator.wikimedia.org/P86997 and previous config saved to /var/cache/conftool/dbconfig/20260109-163320-marostegui.json
• 16:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T413525)', diff saved to https://phabricator.wikimedia.org/P86996 and previous config saved to /var/cache/conftool/dbconfig/20260109-162529-marostegui.json
• 16:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P86995 and previous config saved to /var/cache/conftool/dbconfig/20260109-162312-marostegui.json
• 16:13 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2199.codfw.wmnet with reason: Maintenance
• 16:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86994 and previous config saved to /var/cache/conftool/dbconfig/20260109-161326-marostegui.json
• 16:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P86993 and previous config saved to /var/cache/conftool/dbconfig/20260109-161304-marostegui.json
• 16:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P86992 and previous config saved to /var/cache/conftool/dbconfig/20260109-160318-marostegui.json
• 16:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T413525)', diff saved to https://phabricator.wikimedia.org/P86991 and previous config saved to /var/cache/conftool/dbconfig/20260109-160255-marostegui.json
• 15:58 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 15:57 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2207 (T413525)', diff saved to https://phabricator.wikimedia.org/P86990 and previous config saved to /var/cache/conftool/dbconfig/20260109-155743-marostegui.json
• 15:57 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2207.codfw.wmnet with reason: Maintenance
• 15:55 pt1979@cumin2002: START - Cookbook sre.dns.netbox
• 15:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P86989 and previous config saved to /var/cache/conftool/dbconfig/20260109-155309-marostegui.json
• 15:52 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1197 (T413525)', diff saved to https://phabricator.wikimedia.org/P86988 and previous config saved to /var/cache/conftool/dbconfig/20260109-155207-marostegui.json
• 15:52 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
• 15:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T413525)', diff saved to https://phabricator.wikimedia.org/P86987 and previous config saved to /var/cache/conftool/dbconfig/20260109-155143-marostegui.json
• 15:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86986 and previous config saved to /var/cache/conftool/dbconfig/20260109-154301-marostegui.json
• 15:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P86985 and previous config saved to /var/cache/conftool/dbconfig/20260109-154136-marostegui.json
• 15:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P86984 and previous config saved to /var/cache/conftool/dbconfig/20260109-153128-marostegui.json
• 15:29 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 15:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T413525)', diff saved to https://phabricator.wikimedia.org/P86983 and previous config saved to /var/cache/conftool/dbconfig/20260109-152935-marostegui.json
• 15:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T413525)', diff saved to https://phabricator.wikimedia.org/P86982 and previous config saved to /var/cache/conftool/dbconfig/20260109-152120-marostegui.json
• 15:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P86981 and previous config saved to /var/cache/conftool/dbconfig/20260109-151927-marostegui.json
• 15:17 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1003.eqiad.wmnet with OS trixie
• 15:10 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1188 (T413525)', diff saved to https://phabricator.wikimedia.org/P86980 and previous config saved to /var/cache/conftool/dbconfig/20260109-151029-marostegui.json
• 15:10 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
• 15:10 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T413525)', diff saved to https://phabricator.wikimedia.org/P86979 and previous config saved to /var/cache/conftool/dbconfig/20260109-151005-marostegui.json
• 15:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P86978 and previous config saved to /var/cache/conftool/dbconfig/20260109-150918-marostegui.json
• 15:03 gkyziridis@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 15:02 gkyziridis@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 14:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P86977 and previous config saved to /var/cache/conftool/dbconfig/20260109-145956-marostegui.json
• 14:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T413525)', diff saved to https://phabricator.wikimedia.org/P86976 and previous config saved to /var/cache/conftool/dbconfig/20260109-145910-marostegui.json
• 14:54 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1003.eqiad.wmnet with reason: host reimage
• 14:49 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P86975 and previous config saved to /var/cache/conftool/dbconfig/20260109-144948-marostegui.json
• 14:49 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1003.eqiad.wmnet with reason: host reimage
• 14:46 gkyziridis@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 14:45 gkyziridis@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
• 14:41 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1221 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86974 and previous config saved to /var/cache/conftool/dbconfig/20260109-144151-marostegui.json
• 14:41 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on 6 hosts with reason: Maintenance
• 14:41 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance
• 14:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86973 and previous config saved to /var/cache/conftool/dbconfig/20260109-144116-marostegui.json
• 14:39 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T413525)', diff saved to https://phabricator.wikimedia.org/P86972 and previous config saved to /var/cache/conftool/dbconfig/20260109-143940-marostegui.json
• 14:35 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS trixie
• 14:35 andrew@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudbackup1003']
• 14:34 andrew@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudbackup1003']
• 14:31 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2189 (T413525)', diff saved to https://phabricator.wikimedia.org/P86971 and previous config saved to /var/cache/conftool/dbconfig/20260109-143105-marostegui.json
• 14:31 andrew@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cloudbackup1003']
• 14:30 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2189.codfw.wmnet with reason: Maintenance
• 14:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T413525)', diff saved to https://phabricator.wikimedia.org/P86970 and previous config saved to /var/cache/conftool/dbconfig/20260109-143040-marostegui.json
• 14:29 andrew@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudbackup1003']
• 14:28 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudbackup1003.eqiad.wmnet with OS trixie
• 14:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P86968 and previous config saved to /var/cache/conftool/dbconfig/20260109-142100-marostegui.json
• 14:20 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P86967 and previous config saved to /var/cache/conftool/dbconfig/20260109-142033-marostegui.json
• 14:10 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86966 and previous config saved to /var/cache/conftool/dbconfig/20260109-141052-marostegui.json
• 14:10 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P86965 and previous config saved to /var/cache/conftool/dbconfig/20260109-141024-marostegui.json
• 14:06 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1182 (T413525)', diff saved to https://phabricator.wikimedia.org/P86964 and previous config saved to /var/cache/conftool/dbconfig/20260109-140604-marostegui.json
• 14:05 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
• 14:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T413525)', diff saved to https://phabricator.wikimedia.org/P86963 and previous config saved to /var/cache/conftool/dbconfig/20260109-140539-marostegui.json
• 14:00 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T413525)', diff saved to https://phabricator.wikimedia.org/P86962 and previous config saved to /var/cache/conftool/dbconfig/20260109-140016-marostegui.json
• 13:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P86961 and previous config saved to /var/cache/conftool/dbconfig/20260109-135531-marostegui.json
• 13:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P86960 and previous config saved to /var/cache/conftool/dbconfig/20260109-134522-marostegui.json
• 13:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T413525)', diff saved to https://phabricator.wikimedia.org/P86959 and previous config saved to /var/cache/conftool/dbconfig/20260109-133514-marostegui.json
• 13:26 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2175 (T413525)', diff saved to https://phabricator.wikimedia.org/P86958 and previous config saved to /var/cache/conftool/dbconfig/20260109-132651-marostegui.json
• 13:26 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
• 13:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T413525)', diff saved to https://phabricator.wikimedia.org/P86957 and previous config saved to /var/cache/conftool/dbconfig/20260109-132636-marostegui.json
• 13:23 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1162 (T413525)', diff saved to https://phabricator.wikimedia.org/P86956 and previous config saved to /var/cache/conftool/dbconfig/20260109-132340-marostegui.json
• 13:23 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
• 13:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T413525)', diff saved to https://phabricator.wikimedia.org/P86955 and previous config saved to /var/cache/conftool/dbconfig/20260109-132316-marostegui.json
• 13:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P86954 and previous config saved to /var/cache/conftool/dbconfig/20260109-131628-marostegui.json
• 13:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P86953 and previous config saved to /var/cache/conftool/dbconfig/20260109-131306-marostegui.json
• 13:10 marostegui: Deploy schema change on s3 primary master (this will take a few hours) T414183
• 13:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P86952 and previous config saved to /var/cache/conftool/dbconfig/20260109-130619-marostegui.json
• 13:04 marostegui: Deploy schema change on s4 primary master T414183
• 13:03 marostegui: Deploy schema change on s8 primary master T414183
• 13:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P86951 and previous config saved to /var/cache/conftool/dbconfig/20260109-130258-marostegui.json
• 13:02 marostegui: Deploy schema change on s1 primary master T414183
• 12:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T413525)', diff saved to https://phabricator.wikimedia.org/P86950 and previous config saved to /var/cache/conftool/dbconfig/20260109-125611-marostegui.json
• 12:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T413525)', diff saved to https://phabricator.wikimedia.org/P86949 and previous config saved to /var/cache/conftool/dbconfig/20260109-125250-marostegui.json
• 12:37 marostegui: Deploy schema change on s5 primary master T414183
• 12:32 marostegui: Deploy schema change on s7 primary master T414183
• 12:23 marostegui: Deploy schema change on s2 primary master T414183
• 12:21 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2148 (T413525)', diff saved to https://phabricator.wikimedia.org/P86947 and previous config saved to /var/cache/conftool/dbconfig/20260109-122157-marostegui.json
• 12:21 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1156 (T413525)', diff saved to https://phabricator.wikimedia.org/P86946 and previous config saved to /var/cache/conftool/dbconfig/20260109-122145-marostegui.json
• 12:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
• 12:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 12:21 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
• 12:19 marostegui: Deploy schema change on s6 primary master T414183
• 12:16 aikochou@deploy2002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 12:16 marostegui: Deploy schema change on s7 primary master T414178
• 12:14 aikochou@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 12:11 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2192.codfw.wmnet with reason: Maintenance
• 11:23 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T413525)', diff saved to https://phabricator.wikimedia.org/P86944 and previous config saved to /var/cache/conftool/dbconfig/20260109-112302-marostegui.json
• 11:15 jelto@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 11:14 jelto@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 11:14 jelto@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 11:14 jelto@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 11:14 jelto@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 11:13 jelto@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
• 11:12 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P86943 and previous config saved to /var/cache/conftool/dbconfig/20260109-111254-marostegui.json
• 11:09 moritzm: revoked legacy config-master discovery cert T365798
• 11:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P86942 and previous config saved to /var/cache/conftool/dbconfig/20260109-110245-marostegui.json
• 10:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T413525)', diff saved to https://phabricator.wikimedia.org/P86941 and previous config saved to /var/cache/conftool/dbconfig/20260109-105237-marostegui.json
• 10:50 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2228 (T413525)', diff saved to https://phabricator.wikimedia.org/P86940 and previous config saved to /var/cache/conftool/dbconfig/20260109-105008-marostegui.json
• 10:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2228.codfw.wmnet with reason: Maintenance
• 10:49 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T413525)', diff saved to https://phabricator.wikimedia.org/P86939 and previous config saved to /var/cache/conftool/dbconfig/20260109-104942-marostegui.json
• 10:39 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P86938 and previous config saved to /var/cache/conftool/dbconfig/20260109-103934-marostegui.json
• 10:38 gehel: depooling / repooling wdqs-main@eqiad servers one by one to allow time to recover and catch up on updates.
• 10:36 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool es1049: test
• 10:29 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P86936 and previous config saved to /var/cache/conftool/dbconfig/20260109-102925-marostegui.json
• 10:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2223 (T413525)', diff saved to https://phabricator.wikimedia.org/P86934 and previous config saved to /var/cache/conftool/dbconfig/20260109-101917-marostegui.json
• 10:16 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2223 (T413525)', diff saved to https://phabricator.wikimedia.org/P86933 and previous config saved to /var/cache/conftool/dbconfig/20260109-101648-marostegui.json
• 10:16 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2223.codfw.wmnet with reason: Maintenance
• 10:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T413525)', diff saved to https://phabricator.wikimedia.org/P86932 and previous config saved to /var/cache/conftool/dbconfig/20260109-101622-marostegui.json
• 10:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P86930 and previous config saved to /var/cache/conftool/dbconfig/20260109-100614-marostegui.json
• 09:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P86929 and previous config saved to /var/cache/conftool/dbconfig/20260109-095606-marostegui.json
• 09:51 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool es1049: test
• 09:50 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newdepool (exit_code=0) depool es1049: test
• 09:50 marostegui@cumin1003: START - Cookbook sre.mysql.newdepool depool es1049: test
• 09:46 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T413525)', diff saved to https://phabricator.wikimedia.org/P86926 and previous config saved to /var/cache/conftool/dbconfig/20260109-094558-marostegui.json
• 09:31 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2213 (T413525)', diff saved to https://phabricator.wikimedia.org/P86925 and previous config saved to /var/cache/conftool/dbconfig/20260109-093138-marostegui.json
• 09:31 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2213.codfw.wmnet with reason: Maintenance
• 09:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T413525)', diff saved to https://phabricator.wikimedia.org/P86924 and previous config saved to /var/cache/conftool/dbconfig/20260109-093114-marostegui.json
• 09:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P86923 and previous config saved to /var/cache/conftool/dbconfig/20260109-092105-marostegui.json
• 09:10 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P86922 and previous config saved to /var/cache/conftool/dbconfig/20260109-091058-marostegui.json
• 09:07 urbanecm: [urbanecm@deploy2002 ~]$ kubectl delete job/growthexperiments-updatementeedata-s1-29460615 # T414167
• 09:00 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T413525)', diff saved to https://phabricator.wikimedia.org/P86921 and previous config saved to /var/cache/conftool/dbconfig/20260109-090050-marostegui.json
• 08:48 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2211 (T413525)', diff saved to https://phabricator.wikimedia.org/P86920 and previous config saved to /var/cache/conftool/dbconfig/20260109-084834-marostegui.json
• 08:48 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2211.codfw.wmnet with reason: Maintenance
• 08:47 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2201.codfw.wmnet with reason: Maintenance
• 08:47 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T413525)', diff saved to https://phabricator.wikimedia.org/P86919 and previous config saved to /var/cache/conftool/dbconfig/20260109-084733-marostegui.json
• 08:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P86918 and previous config saved to /var/cache/conftool/dbconfig/20260109-083725-marostegui.json
• 08:30 gehel: restarting blazegraph on wdqs-main@eqiad - high thread count
• 08:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P86917 and previous config saved to /var/cache/conftool/dbconfig/20260109-082717-marostegui.json
• 08:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T413525)', diff saved to https://phabricator.wikimedia.org/P86916 and previous config saved to /var/cache/conftool/dbconfig/20260109-081708-marostegui.json
• 08:01 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2178 (T413525)', diff saved to https://phabricator.wikimedia.org/P86915 and previous config saved to /var/cache/conftool/dbconfig/20260109-080058-marostegui.json
• 08:00 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2178.codfw.wmnet with reason: Maintenance
• 08:00 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T413525)', diff saved to https://phabricator.wikimedia.org/P86914 and previous config saved to /var/cache/conftool/dbconfig/20260109-080033-marostegui.json
• 07:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P86913 and previous config saved to /var/cache/conftool/dbconfig/20260109-075025-marostegui.json
• 07:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P86912 and previous config saved to /var/cache/conftool/dbconfig/20260109-074017-marostegui.json
• 07:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T413525)', diff saved to https://phabricator.wikimedia.org/P86911 and previous config saved to /var/cache/conftool/dbconfig/20260109-073008-marostegui.json
• 07:16 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2171 (T413525)', diff saved to https://phabricator.wikimedia.org/P86910 and previous config saved to /var/cache/conftool/dbconfig/20260109-071621-marostegui.json
• 07:16 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2171.codfw.wmnet with reason: Maintenance
• 07:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T413525)', diff saved to https://phabricator.wikimedia.org/P86909 and previous config saved to /var/cache/conftool/dbconfig/20260109-071606-marostegui.json
• 07:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P86908 and previous config saved to /var/cache/conftool/dbconfig/20260109-070558-marostegui.json
• 06:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P86907 and previous config saved to /var/cache/conftool/dbconfig/20260109-065549-marostegui.json
• 06:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T413525)', diff saved to https://phabricator.wikimedia.org/P86906 and previous config saved to /var/cache/conftool/dbconfig/20260109-064541-marostegui.json
• 06:31 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2157 (T413525)', diff saved to https://phabricator.wikimedia.org/P86905 and previous config saved to /var/cache/conftool/dbconfig/20260109-063154-marostegui.json
• 06:31 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2157.codfw.wmnet with reason: Maintenance
• 06:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1210.eqiad.wmnet with reason: Maintenance
• 05:32 eileen: civicrm upgraded from 9978d78e to 197629c6
• 04:59 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2172 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86904 and previous config saved to /var/cache/conftool/dbconfig/20260109-045935-marostegui.json
• 04:59 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
• 04:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86903 and previous config saved to /var/cache/conftool/dbconfig/20260109-045910-marostegui.json
• 04:49 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P86902 and previous config saved to /var/cache/conftool/dbconfig/20260109-044902-marostegui.json
• 04:38 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P86901 and previous config saved to /var/cache/conftool/dbconfig/20260109-043854-marostegui.json
• 04:28 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86900 and previous config saved to /var/cache/conftool/dbconfig/20260109-042845-marostegui.json
• 03:45 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup2003.codfw.wmnet with OS trixie
• 03:38 eileen: config revision changed from fa2242aa to a31e117c
• 03:28 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup2003.codfw.wmnet with reason: host reimage
• 03:21 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup2003.codfw.wmnet with reason: host reimage
• 03:03 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup2003.codfw.wmnet with OS trixie
• 03:03 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1199 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86899 and previous config saved to /var/cache/conftool/dbconfig/20260109-030336-marostegui.json
• 03:03 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance
• 03:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86898 and previous config saved to /var/cache/conftool/dbconfig/20260109-030311-marostegui.json
• 03:03 andrew@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudbackup2003.codfw.wmnet with OS trixie
• 02:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P86897 and previous config saved to /var/cache/conftool/dbconfig/20260109-025303-marostegui.json
• 02:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P86896 and previous config saved to /var/cache/conftool/dbconfig/20260109-024254-marostegui.json
• 02:32 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86895 and previous config saved to /var/cache/conftool/dbconfig/20260109-023246-marostegui.json
• 02:27 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS trixie
• 02:25 andrew@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudbackup1003']
• 01:18 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 18m 17s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image
• 00:24 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 00:24 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 00:24 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 00:24 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 00:24 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 00:23 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
• 00:22 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 00:22 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 00:22 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 00:22 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 00:22 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 00:22 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
• 00:22 eileen: civicrm upgraded from b1e76341 to 9978d78e
• 00:18 dani@deploy2002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
• 00:18 dani@deploy2002: helmfile [codfw] START helmfile.d/services/miscweb: apply
• 00:18 dani@deploy2002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
• 00:18 dani@deploy2002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
• 00:18 dani@deploy2002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
• 00:18 dani@deploy2002: helmfile [staging] START helmfile.d/services/miscweb: apply
• 00:17 eileen: config revision changed from d9484233 to fa2242aa

2026-01-08

• 22:41 cjming@deploy2002: mwscript-k8s job started: namespaceDupes igwiki --fix # T406178
• 22:34 arlolra@deploy2002: Finished scap sync-world: Backport for Igwiki: add draft namespace (T406178) (duration: 07m 04s)
• 22:30 arlolra@deploy2002: pppery, arlolra: Continuing with sync
• 22:29 arlolra@deploy2002: pppery, arlolra: Backport for Igwiki: add draft namespace (T406178) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:27 arlolra@deploy2002: Started scap sync-world: Backport for Igwiki: add draft namespace (T406178)
• 22:27 eileen: civicrm upgraded from 2689332e to b1e76341
• 22:21 arlolra@deploy2002: Finished scap sync-world: Backport for Deploy PRV to 27 wikis (T413108) (duration: 18m 32s)
• 22:15 arlolra@deploy2002: arlolra: Continuing with sync
• 22:09 arlolra@deploy2002: arlolra: Backport for Deploy PRV to 27 wikis (T413108) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:03 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup2003.codfw.wmnet with reason: host reimage
• 22:02 arlolra@deploy2002: Started scap sync-world: Backport for Deploy PRV to 27 wikis (T413108)
• 21:58 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup2003.codfw.wmnet with reason: host reimage
• 21:57 jsn@deploy2002: Finished scap sync-world: Backport for extension-list: Add PersonalDashboard (T412528) (duration: 37m 41s)
• 21:45 jsn@deploy2002: jsn: Continuing with sync
• 21:44 jsn@deploy2002: jsn: Backport for extension-list: Add PersonalDashboard (T412528) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:42 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup2003.codfw.wmnet with OS trixie
• 21:35 andrew@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudbackup2003']
• 21:28 andrew@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudbackup1003']
• 21:27 andrew@cumin2002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['cloudbackup2003']
• 21:27 andrew@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudbackup2003']
• 21:25 andrew@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudbackup2003']
• 21:25 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudbackup2003.codfw.wmnet with OS trixie
• 21:21 andrew@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudbackup1003.eqiad.wmnet with OS trixie
• 21:20 jsn@deploy2002: Started scap sync-world: Backport for extension-list: Add PersonalDashboard (T412528)
• 21:14 sbassett@deploy2002: Finished scap sync-world: Backport for Set CSP Report Only mode for all wikis (T291867) (duration: 08m 20s)
• 21:10 sbassett@deploy2002: sbassett: Continuing with sync
• 21:08 sbassett@deploy2002: sbassett: Backport for Set CSP Report Only mode for all wikis (T291867) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:06 sbassett@deploy2002: Started scap sync-world: Backport for Set CSP Report Only mode for all wikis (T291867)
• 21:00 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup2003.codfw.wmnet with OS trixie
• 21:00 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS trixie
• 21:00 andrew@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudbackup1003.eqiad.wmnet with OS trixie
• 20:53 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS trixie
• 20:52 andrew@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup2003.codfw.wmnet with OS trixie
• 20:52 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudbackup1003.eqiad.wmnet with OS trixie
• 20:48 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1003.eqiad.wmnet with reason: host reimage
• 20:42 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1003.eqiad.wmnet with reason: host reimage
• 20:35 zabe: zabe@deploy2002:~$ foreachwiki refreshImageMetadata.php --mediatype AUDIO --mime unknown/wav --force --oldimage # T414077
• 20:33 zabe: zabe@deploy2002:~$ foreachwiki refreshImageMetadata.php --mediatype AUDIO --mime unknown/wav --force # T414077
• 20:27 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup2003.codfw.wmnet with OS trixie
• 20:27 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS trixie
• 20:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 20:27 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 20:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T413525)', diff saved to https://phabricator.wikimedia.org/P86894 and previous config saved to /var/cache/conftool/dbconfig/20260108-202652-marostegui.json
• 20:26 zabe: zabe@deploy2002:~$ mwscript refreshImageMetadata.php commonswiki --mediatype AUDIO --mime unknown/wav --force # T414077
• 20:25 andrew@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudbackup1003.eqiad.wmnet with OS trixie
• 20:24 andrew@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host cloudbackup2003.codfw.wmnet with OS trixie
• 20:17 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS trixie
• 20:17 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup2003.codfw.wmnet with OS trixie
• 20:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P86893 and previous config saved to /var/cache/conftool/dbconfig/20260108-201643-marostegui.json
• 20:08 zabe@deploy2002: Finished scap sync-world: Backport for MimeAnalyzer: Fix syntax error in MAJOR_MIME_TYPES array (T414077) (duration: 15m 31s)
• 20:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P86892 and previous config saved to /var/cache/conftool/dbconfig/20260108-200635-marostegui.json
• 20:04 zabe@deploy2002: zabe: Continuing with sync
• 19:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T413525)', diff saved to https://phabricator.wikimedia.org/P86891 and previous config saved to /var/cache/conftool/dbconfig/20260108-195627-marostegui.json
• 19:55 zabe@deploy2002: zabe: Backport for MimeAnalyzer: Fix syntax error in MAJOR_MIME_TYPES array (T414077) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 19:53 zabe@deploy2002: Started scap sync-world: Backport for MimeAnalyzer: Fix syntax error in MAJOR_MIME_TYPES array (T414077)
• 19:50 zabe@deploy2002: Finished scap sync-world: Backport for fixed typo of word initial in the WmfConfig.php file (T201491), Enable phan on more php files (duration: 08m 09s)
• 19:47 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1230 (T413525)', diff saved to https://phabricator.wikimedia.org/P86889 and previous config saved to /var/cache/conftool/dbconfig/20260108-194731-marostegui.json
• 19:47 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1230.eqiad.wmnet with reason: Maintenance
• 19:47 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 19:46 zabe@deploy2002: zabe, ebenezerrao: Continuing with sync
• 19:46 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T413525)', diff saved to https://phabricator.wikimedia.org/P86888 and previous config saved to /var/cache/conftool/dbconfig/20260108-194649-marostegui.json
• 19:44 zabe@deploy2002: zabe, ebenezerrao: Backport for fixed typo of word initial in the WmfConfig.php file (T201491), Enable phan on more php files synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 19:42 zabe@deploy2002: Started scap sync-world: Backport for fixed typo of word initial in the WmfConfig.php file (T201491), Enable phan on more php files
• 19:42 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudbackup1003.eqiad.wmnet with OS trixie
• 19:42 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudbackup2003.codfw.wmnet with OS trixie
• 19:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P86887 and previous config saved to /var/cache/conftool/dbconfig/20260108-193641-marostegui.json
• 19:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P86886 and previous config saved to /var/cache/conftool/dbconfig/20260108-192633-marostegui.json
• 19:24 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group2 to 1.46.0-wmf.10 refs T408280
• 19:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T413525)', diff saved to https://phabricator.wikimedia.org/P86885 and previous config saved to /var/cache/conftool/dbconfig/20260108-191624-marostegui.json
• 19:07 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1207 (T413525)', diff saved to https://phabricator.wikimedia.org/P86884 and previous config saved to /var/cache/conftool/dbconfig/20260108-190727-marostegui.json
• 19:07 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 19:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T413525)', diff saved to https://phabricator.wikimedia.org/P86883 and previous config saved to /var/cache/conftool/dbconfig/20260108-190702-marostegui.json
• 18:57 eileen: civicrm upgraded from b454e4ba to 2689332e
• 18:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P86882 and previous config saved to /var/cache/conftool/dbconfig/20260108-185654-marostegui.json
• 18:46 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P86881 and previous config saved to /var/cache/conftool/dbconfig/20260108-184645-marostegui.json
• 18:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T413525)', diff saved to https://phabricator.wikimedia.org/P86880 and previous config saved to /var/cache/conftool/dbconfig/20260108-183637-marostegui.json
• 18:26 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1200 (T413525)', diff saved to https://phabricator.wikimedia.org/P86879 and previous config saved to /var/cache/conftool/dbconfig/20260108-182641-marostegui.json
• 18:26 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1200.eqiad.wmnet with reason: Maintenance
• 18:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T413525)', diff saved to https://phabricator.wikimedia.org/P86878 and previous config saved to /var/cache/conftool/dbconfig/20260108-182627-marostegui.json
• 18:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P86877 and previous config saved to /var/cache/conftool/dbconfig/20260108-181619-marostegui.json
• 18:15 bd808@deploy2002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
• 18:14 bd808@deploy2002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
• 18:12 bd808@deploy2002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
• 18:12 bd808@deploy2002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
• 18:11 bd808@deploy2002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
• 18:11 bd808@deploy2002: helmfile [staging] START helmfile.d/services/developer-portal: apply
• 18:06 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P86876 and previous config saved to /var/cache/conftool/dbconfig/20260108-180611-marostegui.json
• 17:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T413525)', diff saved to https://phabricator.wikimedia.org/P86875 and previous config saved to /var/cache/conftool/dbconfig/20260108-175602-marostegui.json
• 17:50 ladsgroup@deploy2002: Finished scap sync-world: Backport for Only generate 120,250 thumbs (temporary) (T408062 T412971) (duration: 17m 34s)
• 17:46 ladsgroup@deploy2002: mvernon, ladsgroup: Continuing with sync
• 17:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1185 (T413525)', diff saved to https://phabricator.wikimedia.org/P86874 and previous config saved to /var/cache/conftool/dbconfig/20260108-174606-marostegui.json
• 17:45 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1185.eqiad.wmnet with reason: Maintenance
• 17:45 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T413525)', diff saved to https://phabricator.wikimedia.org/P86873 and previous config saved to /var/cache/conftool/dbconfig/20260108-174542-marostegui.json
• 17:35 ladsgroup@deploy2002: mvernon, ladsgroup: Backport for Only generate 120,250 thumbs (temporary) (T408062 T412971) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 17:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P86872 and previous config saved to /var/cache/conftool/dbconfig/20260108-173534-marostegui.json
• 17:33 ladsgroup@deploy2002: Started scap sync-world: Backport for Only generate 120,250 thumbs (temporary) (T408062 T412971)
• 17:32 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup2003.codfw.wmnet with OS trixie
• 17:31 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudbackup2003.codfw.wmnet with OS trixie
• 17:31 jynus: restarted wmf_auto_restart_prometheus-mysqld-exporter.service @ db2231
• 17:25 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2155 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86871 and previous config saved to /var/cache/conftool/dbconfig/20260108-172551-marostegui.json
• 17:25 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
• 17:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86870 and previous config saved to /var/cache/conftool/dbconfig/20260108-172526-marostegui.json
• 17:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P86869 and previous config saved to /var/cache/conftool/dbconfig/20260108-172526-marostegui.json
• 17:21 oblivian@deploy2002: helmfile [eqiad] DONE helmfile.d/services/thumbor: sync
• 17:21 oblivian@deploy2002: helmfile [eqiad] START helmfile.d/services/thumbor: sync
• 17:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P86867 and previous config saved to /var/cache/conftool/dbconfig/20260108-171517-marostegui.json
• 17:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T413525)', diff saved to https://phabricator.wikimedia.org/P86868 and previous config saved to /var/cache/conftool/dbconfig/20260108-171517-marostegui.json
• 17:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P86866 and previous config saved to /var/cache/conftool/dbconfig/20260108-170509-marostegui.json
• 17:02 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1161 (T413525)', diff saved to https://phabricator.wikimedia.org/P86865 and previous config saved to /var/cache/conftool/dbconfig/20260108-170241-marostegui.json
• 17:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 17:02 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1161.eqiad.wmnet with reason: Maintenance
• 17:01 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T413525)', diff saved to https://phabricator.wikimedia.org/P86864 and previous config saved to /var/cache/conftool/dbconfig/20260108-170156-marostegui.json
• 16:55 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86863 and previous config saved to /var/cache/conftool/dbconfig/20260108-165501-marostegui.json
• 16:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P86862 and previous config saved to /var/cache/conftool/dbconfig/20260108-165148-marostegui.json
• 16:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P86861 and previous config saved to /var/cache/conftool/dbconfig/20260108-164140-marostegui.json
• 16:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T413525)', diff saved to https://phabricator.wikimedia.org/P86860 and previous config saved to /var/cache/conftool/dbconfig/20260108-163131-marostegui.json
• 16:18 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1159 (T413525)', diff saved to https://phabricator.wikimedia.org/P86858 and previous config saved to /var/cache/conftool/dbconfig/20260108-161848-marostegui.json
• 16:18 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1159.eqiad.wmnet with reason: Maintenance
• 16:14 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1029.eqiad.wmnet with OS trixie
• 16:10 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 (T413525)', diff saved to https://phabricator.wikimedia.org/P86857 and previous config saved to /var/cache/conftool/dbconfig/20260108-161038-marostegui.json
• 16:00 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P86856 and previous config saved to /var/cache/conftool/dbconfig/20260108-160029-marostegui.json
• 15:54 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1029.eqiad.wmnet with reason: host reimage
• 15:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229', diff saved to https://phabricator.wikimedia.org/P86855 and previous config saved to /var/cache/conftool/dbconfig/20260108-155021-marostegui.json
• 15:48 elukey@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1029.eqiad.wmnet with reason: host reimage
• 15:44 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup2003.codfw.wmnet with OS trixie
• 15:44 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS trixie
• 15:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2229 (T413525)', diff saved to https://phabricator.wikimedia.org/P86854 and previous config saved to /var/cache/conftool/dbconfig/20260108-154013-marostegui.json
• 15:29 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1029.eqiad.wmnet with OS trixie
• 15:24 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2229 (T413525)', diff saved to https://phabricator.wikimedia.org/P86853 and previous config saved to /var/cache/conftool/dbconfig/20260108-152432-marostegui.json
• 15:24 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2229.codfw.wmnet with reason: Maintenance
• 15:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T413525)', diff saved to https://phabricator.wikimedia.org/P86852 and previous config saved to /var/cache/conftool/dbconfig/20260108-152407-marostegui.json
• 15:21 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1190 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86851 and previous config saved to /var/cache/conftool/dbconfig/20260108-152103-marostegui.json
• 15:20 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
• 15:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P86850 and previous config saved to /var/cache/conftool/dbconfig/20260108-151400-marostegui.json
• 15:10 Lucas_WMDE: UTC afternoon backport+config window done
• 15:09 mszwarc@deploy2002: Finished scap sync-world: Backport for Revert^2 "Silence TransactionProfiler warnings in CheckUserPrivateEventsHandler", Update API call in edit.js with rvslots (T412762) (duration: 09m 11s)
• 15:05 mszwarc@deploy2002: ladsgroup, mszwarc: Continuing with sync
• 15:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P86849 and previous config saved to /var/cache/conftool/dbconfig/20260108-150351-marostegui.json
• 15:02 mszwarc@deploy2002: ladsgroup, mszwarc: Backport for Revert^2 "Silence TransactionProfiler warnings in CheckUserPrivateEventsHandler", Update API call in edit.js with rvslots (T412762) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 15:00 mszwarc@deploy2002: Started scap sync-world: Backport for Revert^2 "Silence TransactionProfiler warnings in CheckUserPrivateEventsHandler", Update API call in edit.js with rvslots (T412762)
• 14:58 elukey@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1029.eqiad.wmnet with OS trixie
• 14:55 lucaswerkmeister-wmde@deploy2002: Finished scap sync-world: Backport for Enable the MEX / wbui2025 beta feature on wikidata (T403015) (duration: 08m 49s)
• 14:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T413525)', diff saved to https://phabricator.wikimedia.org/P86848 and previous config saved to /var/cache/conftool/dbconfig/20260108-145343-marostegui.json
• 14:51 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, arthurtaylor: Continuing with sync
• 14:51 urbanecm: Add `mszwarc` to `wmf-deployment` on Gerrit (existing deployer, T404697)
• 14:48 lucaswerkmeister-wmde@deploy2002: lucaswerkmeister-wmde, arthurtaylor: Backport for Enable the MEX / wbui2025 beta feature on wikidata (T403015) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:47 jgreen@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
• 14:46 lucaswerkmeister-wmde@deploy2002: Started scap sync-world: Backport for Enable the MEX / wbui2025 beta feature on wikidata (T403015)
• 14:45 jgreen@cumin1003: START - Cookbook sre.dns.netbox
• 14:43 mszwarc@deploy2002: Finished scap sync-world: Backport for [enwikiquote] Enable block feature for AbuseFilter (T413530), [ruwiki] Disable setting a cookie for blocked anonymous users (T413737), [enwikiquote] Add new autopatrolled and patroller usergroups (T413848) (duration: 08m 36s)
• 14:39 mszwarc@deploy2002: mszwarc, superpes: Continuing with sync
• 14:37 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2224 (T413525)', diff saved to https://phabricator.wikimedia.org/P86847 and previous config saved to /var/cache/conftool/dbconfig/20260108-143755-marostegui.json
• 14:37 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2224.codfw.wmnet with reason: Maintenance
• 14:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T413525)', diff saved to https://phabricator.wikimedia.org/P86846 and previous config saved to /var/cache/conftool/dbconfig/20260108-143730-marostegui.json
• 14:36 mszwarc@deploy2002: mszwarc, superpes: Backport for [enwikiquote] Enable block feature for AbuseFilter (T413530), [ruwiki] Disable setting a cookie for blocked anonymous users (T413737), [enwikiquote] Add new autopatrolled and patroller usergroups (T413848) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified t
• 14:36 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1029.eqiad.wmnet with OS trixie
• 14:34 jgreen@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 14:34 mszwarc@deploy2002: Started scap sync-world: Backport for [enwikiquote] Enable block feature for AbuseFilter (T413530), [ruwiki] Disable setting a cookie for blocked anonymous users (T413737), [enwikiquote] Add new autopatrolled and patroller usergroups (T413848)
• 14:33 moritzm: installing systemd bugfix updates from Bookworm point release
• 14:31 mszwarc@deploy2002: Finished scap sync-world: Backport for Revert "Silence TransactionProfiler warnings in CheckUserPrivateEventsHandler" (duration: 07m 34s)
• 14:27 mszwarc@deploy2002: mszwarc, trainbranchbot: Continuing with sync
• 14:27 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P86845 and previous config saved to /var/cache/conftool/dbconfig/20260108-142722-marostegui.json
• 14:26 jgreen@cumin1003: START - Cookbook sre.dns.netbox
• 14:26 mszwarc@deploy2002: mszwarc, trainbranchbot: Backport for Revert "Silence TransactionProfiler warnings in CheckUserPrivateEventsHandler" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:25 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2231.codfw.wmnet onto db2249.codfw.wmnet
• 14:23 mszwarc@deploy2002: Started scap sync-world: Backport for Revert "Silence TransactionProfiler warnings in CheckUserPrivateEventsHandler"
• 14:19 btullis@dns1004: END - running authdns-update
• 14:18 btullis@dns1004: START - running authdns-update
• 14:17 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P86844 and previous config saved to /var/cache/conftool/dbconfig/20260108-141714-marostegui.json
• 14:15 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 14:15 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 14:08 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2231 gradually with 4 steps - Pool db2231.codfw.wmnet in after cloning
• 14:08 mszwarc@deploy2002: Sync cancelled.
• 14:07 jmm@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest2006.codfw.wmnet with OS bookworm
• 14:07 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T413525)', diff saved to https://phabricator.wikimedia.org/P86842 and previous config saved to /var/cache/conftool/dbconfig/20260108-140705-marostegui.json
• 14:06 mszwarc@deploy2002: mszwarc: Backport for Silence TransactionProfiler warnings in CheckUserPrivateEventsHandler (T413929) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:04 mszwarc@deploy2002: Started scap sync-world: Backport for Silence TransactionProfiler warnings in CheckUserPrivateEventsHandler (T413929)
• 14:03 jgreen@cumin1003: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
• 13:58 jgreen@cumin1003: START - Cookbook sre.dns.netbox
• 13:57 dpogorzelski@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 13:51 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2217 (T413525)', diff saved to https://phabricator.wikimedia.org/P86839 and previous config saved to /var/cache/conftool/dbconfig/20260108-135111-marostegui.json
• 13:51 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2217.codfw.wmnet with reason: Maintenance
• 13:50 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance
• 13:50 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T413525)', diff saved to https://phabricator.wikimedia.org/P86838 and previous config saved to /var/cache/conftool/dbconfig/20260108-135028-marostegui.json
• 13:47 jgreen@dns1004: END - running authdns-update
• 13:46 jgreen@dns1004: START - running authdns-update
• 13:40 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P86837 and previous config saved to /var/cache/conftool/dbconfig/20260108-134020-marostegui.json
• 13:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P86835 and previous config saved to /var/cache/conftool/dbconfig/20260108-133011-marostegui.json
• 13:23 marostegui@cumin1003: START - Cookbook sre.mysql.pool db2231 gradually with 4 steps - Pool db2231.codfw.wmnet in after cloning
• 13:20 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T413525)', diff saved to https://phabricator.wikimedia.org/P86833 and previous config saved to /var/cache/conftool/dbconfig/20260108-132003-marostegui.json
• 13:19 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 13:18 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 13:17 moritzm: installing imagemagick security updates
• 13:16 aikochou@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revise-tone-task-generator' for release 'main' .
• 13:13 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2193 (T413525)', diff saved to https://phabricator.wikimedia.org/P86832 and previous config saved to /var/cache/conftool/dbconfig/20260108-131327-marostegui.json
• 13:13 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2193.codfw.wmnet with reason: Maintenance
• 13:13 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T413525)', diff saved to https://phabricator.wikimedia.org/P86831 and previous config saved to /var/cache/conftool/dbconfig/20260108-131303-marostegui.json
• 13:02 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P86828 and previous config saved to /var/cache/conftool/dbconfig/20260108-130255-marostegui.json
• 12:53 moritzm: installing libsodium security updates on bullseye
• 12:52 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P86827 and previous config saved to /var/cache/conftool/dbconfig/20260108-125247-marostegui.json
• 12:42 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T413525)', diff saved to https://phabricator.wikimedia.org/P86826 and previous config saved to /var/cache/conftool/dbconfig/20260108-124238-marostegui.json
• 12:40 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 12:39 ozge@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 12:35 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 12:35 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2180 (T413525)', diff saved to https://phabricator.wikimedia.org/P86825 and previous config saved to /var/cache/conftool/dbconfig/20260108-123526-marostegui.json
• 12:35 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2180.codfw.wmnet with reason: Maintenance
• 12:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T413525)', diff saved to https://phabricator.wikimedia.org/P86824 and previous config saved to /var/cache/conftool/dbconfig/20260108-123501-marostegui.json
• 12:33 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host sretest2006.codfw.wmnet with OS bookworm
• 12:24 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P86823 and previous config saved to /var/cache/conftool/dbconfig/20260108-122453-marostegui.json
• 12:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P86822 and previous config saved to /var/cache/conftool/dbconfig/20260108-121445-marostegui.json
• 12:04 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T413525)', diff saved to https://phabricator.wikimedia.org/P86821 and previous config saved to /var/cache/conftool/dbconfig/20260108-120436-marostegui.json
• 11:53 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 11:52 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 11:50 ozge@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 11:46 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2169 (T413525)', diff saved to https://phabricator.wikimedia.org/P86820 and previous config saved to /var/cache/conftool/dbconfig/20260108-114627-marostegui.json
• 11:46 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2169.codfw.wmnet with reason: Maintenance
• 11:46 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T413525)', diff saved to https://phabricator.wikimedia.org/P86819 and previous config saved to /var/cache/conftool/dbconfig/20260108-114602-marostegui.json
• 11:35 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P86818 and previous config saved to /var/cache/conftool/dbconfig/20260108-113554-marostegui.json
• 11:25 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P86817 and previous config saved to /var/cache/conftool/dbconfig/20260108-112546-marostegui.json
• 11:16 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
• 11:16 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
• 11:15 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T413525)', diff saved to https://phabricator.wikimedia.org/P86816 and previous config saved to /var/cache/conftool/dbconfig/20260108-111537-marostegui.json
• 11:10 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
• 11:10 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
• 10:57 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2158 (T413525)', diff saved to https://phabricator.wikimedia.org/P86815 and previous config saved to /var/cache/conftool/dbconfig/20260108-105703-marostegui.json
• 10:56 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2158.codfw.wmnet with reason: Maintenance
• 10:56 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T413525)', diff saved to https://phabricator.wikimedia.org/P86814 and previous config saved to /var/cache/conftool/dbconfig/20260108-105639-marostegui.json
• 10:46 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P86812 and previous config saved to /var/cache/conftool/dbconfig/20260108-104630-marostegui.json
• 10:39 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 10:39 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 10:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P86811 and previous config saved to /var/cache/conftool/dbconfig/20260108-103622-marostegui.json
• 10:35 mutante: wikitech wiki - made lsobanski an admin - T414065
• 10:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T413525)', diff saved to https://phabricator.wikimedia.org/P86810 and previous config saved to /var/cache/conftool/dbconfig/20260108-102613-marostegui.json
• 10:23 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 10:13 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 10:07 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2151 (T413525)', diff saved to https://phabricator.wikimedia.org/P86809 and previous config saved to /var/cache/conftool/dbconfig/20260108-100757-marostegui.json
• 10:07 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2151.codfw.wmnet with reason: Maintenance
• 09:59 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1201.eqiad.wmnet with reason: Maintenance
• 09:46 claime: Rebuilding ratelimit image - T414002
• 09:38 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab
• 09:31 XioNoX: remove pybal BGP group on pfw1-codfw (replaced with Bird) - T414015
• 09:08 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 09:08 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T413525)', diff saved to https://phabricator.wikimedia.org/P86808 and previous config saved to /var/cache/conftool/dbconfig/20260108-090759-marostegui.json
• 08:57 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P86807 and previous config saved to /var/cache/conftool/dbconfig/20260108-085751-marostegui.json
• 08:47 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P86806 and previous config saved to /var/cache/conftool/dbconfig/20260108-084742-marostegui.json
• 08:37 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T413525)', diff saved to https://phabricator.wikimedia.org/P86805 and previous config saved to /var/cache/conftool/dbconfig/20260108-083734-marostegui.json
• 08:30 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1187 (T413525)', diff saved to https://phabricator.wikimedia.org/P86804 and previous config saved to /var/cache/conftool/dbconfig/20260108-083026-marostegui.json
• 08:30 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1187.eqiad.wmnet with reason: Maintenance
• 08:30 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T413525)', diff saved to https://phabricator.wikimedia.org/P86803 and previous config saved to /var/cache/conftool/dbconfig/20260108-083001-marostegui.json
• 08:20 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab
• 08:19 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P86802 and previous config saved to /var/cache/conftool/dbconfig/20260108-081953-marostegui.json
• 08:16 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica
• 08:09 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P86801 and previous config saved to /var/cache/conftool/dbconfig/20260108-080945-marostegui.json
• 08:07 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab replica
• 08:05 jelto@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica
• 07:59 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T413525)', diff saved to https://phabricator.wikimedia.org/P86800 and previous config saved to /var/cache/conftool/dbconfig/20260108-075937-marostegui.json
• 07:55 jelto@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica
• 07:52 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1180 (T413525)', diff saved to https://phabricator.wikimedia.org/P86799 and previous config saved to /var/cache/conftool/dbconfig/20260108-075220-marostegui.json
• 07:52 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1180.eqiad.wmnet with reason: Maintenance
• 07:51 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 (T413525)', diff saved to https://phabricator.wikimedia.org/P86798 and previous config saved to /var/cache/conftool/dbconfig/20260108-075155-marostegui.json
• 07:41 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P86797 and previous config saved to /var/cache/conftool/dbconfig/20260108-074147-marostegui.json
• 07:31 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173', diff saved to https://phabricator.wikimedia.org/P86796 and previous config saved to /var/cache/conftool/dbconfig/20260108-073139-marostegui.json
• 07:21 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1173 (T413525)', diff saved to https://phabricator.wikimedia.org/P86795 and previous config saved to /var/cache/conftool/dbconfig/20260108-072130-marostegui.json
• 07:14 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1173 (T413525)', diff saved to https://phabricator.wikimedia.org/P86794 and previous config saved to /var/cache/conftool/dbconfig/20260108-071413-marostegui.json
• 07:14 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1173.eqiad.wmnet with reason: Maintenance
• 07:14 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T413525)', diff saved to https://phabricator.wikimedia.org/P86793 and previous config saved to /var/cache/conftool/dbconfig/20260108-071359-marostegui.json
• 07:03 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P86792 and previous config saved to /var/cache/conftool/dbconfig/20260108-070351-marostegui.json
• 06:53 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P86791 and previous config saved to /var/cache/conftool/dbconfig/20260108-065342-marostegui.json
• 06:43 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T413525)', diff saved to https://phabricator.wikimedia.org/P86790 and previous config saved to /var/cache/conftool/dbconfig/20260108-064333-marostegui.json
• 06:36 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1168 (T413525)', diff saved to https://phabricator.wikimedia.org/P86789 and previous config saved to /var/cache/conftool/dbconfig/20260108-063642-marostegui.json
• 06:36 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1168.eqiad.wmnet with reason: Maintenance
• 06:36 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T413525)', diff saved to https://phabricator.wikimedia.org/P86788 and previous config saved to /var/cache/conftool/dbconfig/20260108-063616-marostegui.json
• 06:26 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P86787 and previous config saved to /var/cache/conftool/dbconfig/20260108-062608-marostegui.json
• 06:16 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P86786 and previous config saved to /var/cache/conftool/dbconfig/20260108-061600-marostegui.json
• 06:05 marostegui@cumin1003: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T413525)', diff saved to https://phabricator.wikimedia.org/P86785 and previous config saved to /var/cache/conftool/dbconfig/20260108-060551-marostegui.json
• 06:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool db2144: test
• 06:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 06:05 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
• 06:05 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool db2144: test
• 06:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newdepool (exit_code=0) depool db2144: test
• 06:05 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 06:05 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
• 06:05 marostegui@cumin1003: START - Cookbook sre.mysql.newdepool depool db2144: test
• 06:04 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newpool (exit_code=0) pool pc1013: test
• 06:04 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 06:03 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
• 06:03 marostegui@cumin1003: START - Cookbook sre.mysql.newpool pool pc1013: test
• 06:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.newdepool (exit_code=0) depool pc1013: test
• 06:02 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0)
• 06:02 marostegui@cumin1003: START - Cookbook sre.mysql.parsercache
• 06:02 marostegui@cumin1003: START - Cookbook sre.mysql.newdepool depool pc1013: test
• 05:59 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db1165 (T413525)', diff saved to https://phabricator.wikimedia.org/P86780 and previous config saved to /var/cache/conftool/dbconfig/20260108-055901-marostegui.json
• 05:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 05:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 05:51 marostegui@cumin1003: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) db2231 - Depool db2231.codfw.wmnet to then clone it to db2249.codfw.wmnet - marostegui@cumin1003
• 05:51 marostegui@cumin1003: START - Cookbook sre.mysql.depool db2231 - Depool db2231.codfw.wmnet to then clone it to db2249.codfw.wmnet - marostegui@cumin1003
• 05:51 marostegui@cumin1003: START - Cookbook sre.mysql.clone of db2231.codfw.wmnet onto db2249.codfw.wmnet
• 05:34 marostegui@cumin1003: dbctl commit (dc=all): 'Depooling db2147 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86778 and previous config saved to /var/cache/conftool/dbconfig/20260108-053449-marostegui.json
• 05:34 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
• 05:34 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 04:36 eileen: civicrm upgraded from e81dd402 to b454e4ba
• 02:12 eileen: civicrm upgraded from 3dc5b0c3 to e81dd402
• 01:14 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 12m 59s)
• 01:01 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

2026-01-07

• 23:14 eileen: config revision changed from 310a8242 to d9484233
• 23:14 eileen: civicrm upgraded from ee3e61ce to 3dc5b0c3
• 22:57 dduvall@deploy2002: Finished scap sync-world: Backport for Specials: Add space before page tools in changes lists (T414031) (duration: 06m 30s)
• 22:53 dduvall@deploy2002: dduvall: Continuing with sync
• 22:53 dduvall@deploy2002: dduvall: Backport for Specials: Add space before page tools in changes lists (T414031) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:51 dduvall@deploy2002: Started scap sync-world: Backport for Specials: Add space before page tools in changes lists (T414031)
• 22:15 dduvall@deploy2002: Finished scap sync-world: Backport for Add Flags::$suppressed (T414016) (duration: 06m 45s)
• 22:11 dduvall@deploy2002: reedy, dduvall: Continuing with sync
• 22:11 dduvall@deploy2002: reedy, dduvall: Backport for Add Flags::$suppressed (T414016) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:09 dduvall@deploy2002: Started scap sync-world: Backport for Add Flags::$suppressed (T414016)
• 21:54 inflatador: bking@clouddumps100{1,2} created /srv/dumps/xmldatadumps/public/other/cirrussearch/DEPRECATED.txt T366248
• 21:35 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.10 refs T408280
• 21:23 reedy@deploy2002: Finished scap sync-world: Backport for Bump cache key version of FilterLookup::getAllActiveFiltersInGroup (T414016), MWScript: Remove curl_close() (T413538) (duration: 06m 30s)
• 21:19 reedy@deploy2002: reedy: Continuing with sync
• 21:19 reedy@deploy2002: reedy: Backport for Bump cache key version of FilterLookup::getAllActiveFiltersInGroup (T414016), MWScript: Remove curl_close() (T413538) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:17 reedy@deploy2002: Started scap sync-world: Backport for Bump cache key version of FilterLookup::getAllActiveFiltersInGroup (T414016), MWScript: Remove curl_close() (T413538)
• 19:41 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.10 refs T408280
• 19:34 dduvall: rolling back 1.46.0-wmf.10 from group1 due to a large number (630,829) of new deprecation warnings (T408280)
• 19:32 jhathaway: upload corto 1.0.22 to apt repo
• 19:22 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.10 refs T408280
• 17:33 urandom: restarting restbase1035-{a,b,c}/Cassandra to re-add /dev/sdc — T413678
• 16:37 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
• 16:37 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
• 16:34 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
• 16:33 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
• 16:33 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 16:33 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 16:25 sukhe@dns1004: END - running authdns-update
• 16:24 sukhe@dns1004: START - running authdns-update
• 16:22 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart (exit_code=0) rolling restart_daemons on A:dnsbox
• 16:18 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1032.eqiad.wmnet with OS trixie
• 16:02 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1032.eqiad.wmnet with OS trixie
• 15:48 cgoubert@deploy2002: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
• 15:48 cgoubert@deploy2002: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
• 15:42 cgoubert@deploy2002: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
• 15:42 cgoubert@deploy2002: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
• 15:40 moritzm: uploaded Bird 2.18-1~wmf12u2 to component/bird-routed-ganeti for bookworm-wikimedia T413740
• 15:37 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 15:37 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 15:32 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1032.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:29 elukey@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1032.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 15:25 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: sync
• 15:25 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: sync
• 15:23 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart rolling restart_daemons on A:dnsbox
• 15:15 ecarg@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 15:15 cgoubert@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 15:15 cgoubert@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 15:15 ecarg@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 15:15 ecarg@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 15:14 ecarg@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 15:13 ecarg@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 15:13 ecarg@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 15:09 ecarg@deploy2002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
• 15:08 ecarg@deploy2002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
• 15:08 ecarg@deploy2002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
• 15:08 ecarg@deploy2002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
• 15:07 ecarg@deploy2002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
• 15:06 ecarg@deploy2002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
• 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough
• 14:50 bearloga@deploy2002: Finished scap sync-world: Backport for EventStreamConfig: enrich stream with more headers (duration: 07m 37s)
• 14:46 bearloga@deploy2002: bearloga: Continuing with sync
• 14:46 sukhe@cumin1003: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
• 14:45 bearloga@deploy2002: bearloga: Backport for EventStreamConfig: enrich stream with more headers synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:42 bearloga@deploy2002: Started scap sync-world: Backport for EventStreamConfig: enrich stream with more headers
• 14:32 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wdqs1032.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:24 urbanecm@deploy2002: Finished scap sync-world: Backport for fix(ConfirmEmail): Save settings after generating tokens (T413435), Stop setting $wgCampaignEventsEnableContributionTracking (T410939) (duration: 08m 05s)
• 14:24 elukey@cumin1003: START - Cookbook sre.hosts.provision for host wdqs1032.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
• 14:23 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1032.eqiad.wmnet with OS trixie
• 14:22 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Improve performance - oblivian@cumin1003"
• 14:22 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Improve performance - oblivian@cumin1003
• 14:21 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Improve performance - oblivian@cumin1003
• 14:21 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Improve performance - oblivian@cumin1003"
• 14:20 urbanecm@deploy2002: urbanecm, daimona: Continuing with sync
• 14:18 urbanecm@deploy2002: urbanecm, daimona: Backport for fix(ConfirmEmail): Save settings after generating tokens (T413435), Stop setting $wgCampaignEventsEnableContributionTracking (T410939) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:17 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1032.eqiad.wmnet with OS trixie
• 14:17 elukey@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1032.eqiad.wmnet with OS trixie
• 14:16 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Improve performance - oblivian@cumin1003"
• 14:16 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Improve performance - oblivian@cumin1003
• 14:16 urbanecm@deploy2002: Started scap sync-world: Backport for fix(ConfirmEmail): Save settings after generating tokens (T413435), Stop setting $wgCampaignEventsEnableContributionTracking (T410939)
• 14:15 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Improve performance - oblivian@cumin1003
• 14:15 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Improve performance - oblivian@cumin1003"
• 14:14 elukey@cumin1003: START - Cookbook sre.hosts.reimage for host wdqs1032.eqiad.wmnet with OS trixie
• 14:12 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Improve performance - oblivian@cumin1003"
• 14:12 oblivian@cumin1003: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Improve performance - oblivian@cumin1003
• 14:11 oblivian@cumin1003: START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Improve performance - oblivian@cumin1003
• 14:11 oblivian@cumin1003: START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Improve performance - oblivian@cumin1003"
• 13:49 cmooney@deploy2002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
• 13:48 cmooney@deploy2002: helmfile [eqiad] START helmfile.d/services/zotero: apply
• 13:40 cmooney@deploy2002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
• 13:40 cmooney@deploy2002: helmfile [codfw] START helmfile.d/services/zotero: apply
• 13:37 ladsgroup@deploy2002: Finished scap sync-world: Backport for Update config to reflect new standard set of thumbnail sizes (WE 5.4.7) (T408062 T412971) (duration: 07m 02s)
• 13:36 cmooney@deploy2002: helmfile [staging] DONE helmfile.d/services/zotero: apply
• 13:36 cmooney@deploy2002: helmfile [staging] START helmfile.d/services/zotero: apply
• 13:33 ladsgroup@deploy2002: mvernon, ladsgroup: Continuing with sync
• 13:32 ladsgroup@deploy2002: mvernon, ladsgroup: Backport for Update config to reflect new standard set of thumbnail sizes (WE 5.4.7) (T408062 T412971) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 13:30 ladsgroup@deploy2002: Started scap sync-world: Backport for Update config to reflect new standard set of thumbnail sizes (WE 5.4.7) (T408062 T412971)
• 13:20 dzahn@dns1004: END - running authdns-update
• 13:19 dzahn@dns1004: START - running authdns-update
• 13:15 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 11 echo-subscriptions-email-mention (T406724)
• 13:05 btullis@cumin1003: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host an-worker1198.eqiad.wmnet
• 12:26 mvolz@deploy2002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
• 12:26 mvolz@deploy2002: helmfile [eqiad] START helmfile.d/services/zotero: apply
• 12:26 mvolz@deploy2002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
• 12:25 mvolz@deploy2002: helmfile [codfw] START helmfile.d/services/zotero: apply
• 12:18 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/zotero: apply
• 12:17 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/zotero: apply
• 12:11 mvolz@deploy2002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
• 12:11 mvolz@deploy2002: helmfile [eqiad] START helmfile.d/services/citoid: apply
• 12:10 mvolz@deploy2002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
• 12:09 mvolz@deploy2002: helmfile [codfw] START helmfile.d/services/citoid: apply
• 12:08 mvolz@deploy2002: helmfile [staging] DONE helmfile.d/services/citoid: apply
• 12:08 mvolz@deploy2002: helmfile [staging] START helmfile.d/services/citoid: apply
• 11:47 moritzm: restart dovecot on mx-out to pick up sodium security updates
• 11:45 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-worker1198.eqiad.wmnet
• 11:44 btullis@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1168.eqiad.wmnet
• 11:34 btullis@cumin1003: START - Cookbook sre.hosts.reboot-single for host an-worker1168.eqiad.wmnet
• 11:08 javiermonton@deploy2002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: sync
• 11:07 javiermonton@deploy2002: helmfile [codfw] START helmfile.d/services/eventgate-analytics: sync
• 11:05 javiermonton@deploy2002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: sync
• 11:05 javiermonton@deploy2002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: sync
• 09:58 javiermonton@deploy2002: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
• 09:58 javiermonton@deploy2002: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
• 09:38 moritzm: uploaded Bird 2.18-1~wmf12u1 to component/bird-routed-ganeti for bookworm-wikimedia T413740
• 09:14 kevinbazira@deploy2002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
• 09:10 jynus: restarting bacula-sd daemon on backup1012 T413853
• 08:57 awight@deploy2002: Finished scap sync-world: Backport for Configure new stream mediawiki.wmde_page_summary (T413891) (duration: 07m 28s)
• 08:53 awight@deploy2002: awight: Continuing with sync
• 08:52 awight@deploy2002: awight: Backport for Configure new stream mediawiki.wmde_page_summary (T413891) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 08:49 awight@deploy2002: Started scap sync-world: Backport for Configure new stream mediawiki.wmde_page_summary (T413891)
• 08:45 jdlrobson@deploy2002: Sync cancelled.
• 08:18 moritzm: installing libsodium security updates
• 07:55 krinkle@deploy2002: helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
• 07:54 krinkle@deploy2002: helmfile [eqiad] START helmfile.d/services/mw-experimental: apply

2026-01-06

• 22:37 jdlrobson@deploy2002: jdlrobson, aude: Backport for Add wordmark logo to beta cluster for Minerva (mobile) (T413217) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:35 jdlrobson@deploy2002: Started scap sync-world: Backport for Add wordmark logo to beta cluster for Minerva (mobile) (T413217)
• 22:23 jdlrobson@deploy2002: Finished scap sync-world: Backport for Update VE core submodule to master (24389ad60), Update VE core submodule to master (f53cf0905) (T413356) (duration: 10m 17s)
• 22:19 jdlrobson@deploy2002: jdlrobson: Continuing with sync
• 22:15 jdlrobson@deploy2002: jdlrobson: Backport for Update VE core submodule to master (24389ad60), Update VE core submodule to master (f53cf0905) (T413356) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:13 jdlrobson@deploy2002: Started scap sync-world: Backport for Update VE core submodule to master (24389ad60), Update VE core submodule to master (f53cf0905) (T413356)
• 21:06 eileen: civicrm upgraded from dc2511ee to ee3e61ce
• 19:13 dduvall@deploy2002: rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.10 refs T408280
• 18:22 cdanis: 💙cdanis@cumin1003.eqiad.wmnet ~ 🕜☕ sudo cumin 'A:cp' 'enable-puppet "cdanis deploy I9b2f6edc6b7"'
• 17:56 cdanis: 💙cdanis@cumin1003.eqiad.wmnet ~ 🕐☕ sudo cumin 'A:cp' 'disable-puppet "cdanis deploy I9b2f6edc6b7"'
• 17:07 urbanecm@deploy2002: Finished scap sync-world: Backport for growthexperiments: Add pcmwiki (T409480) (duration: 10m 27s)
• 17:03 urbanecm@deploy2002: urbanecm: Continuing with sync
• 16:59 urbanecm@deploy2002: urbanecm: Backport for growthexperiments: Add pcmwiki (T409480) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 16:57 urbanecm@deploy2002: Started scap sync-world: Backport for growthexperiments: Add pcmwiki (T409480)
• 16:50 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet1006.eqiad.wmnet with OS trixie
• 16:33 sukhe: sukhe@cp1100:~$ sudo ats-backend-restart
• 16:28 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet1006.eqiad.wmnet with reason: host reimage
• 16:28 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet1006.eqiad.wmnet with reason: host reimage
• 16:13 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudnet1006.eqiad.wmnet with OS trixie
• 16:08 brennen@deploy2002: Finished deploy [phabricator/deployment@f31c251]: deploy phab1004 for T413832 (duration: 00m 54s)
• 16:07 brennen@deploy2002: Started deploy [phabricator/deployment@f31c251]: deploy phab1004 for T413832
• 16:07 brennen@deploy2002: Finished deploy [phabricator/deployment@f31c251]: deploy phab2002 for T413832 (duration: 00m 32s)
• 16:06 brennen@deploy2002: Started deploy [phabricator/deployment@f31c251]: deploy phab2002 for T413832
• 15:28 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudnet1005.eqiad.wmnet with OS trixie
• 15:06 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet1005.eqiad.wmnet with reason: host reimage
• 15:05 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudnet1005.eqiad.wmnet with reason: host reimage
• 14:50 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudnet1005.eqiad.wmnet with OS trixie
• 14:46 dcausse: closing the UTC afternoon backport window
• 14:44 dcausse@deploy2002: Finished scap sync-world: Backport for cirrus: allow title natural sort (T40403) (duration: 22m 10s)
• 14:40 dcausse@deploy2002: dcausse: Continuing with sync
• 14:38 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirtlocal1003.eqiad.wmnet with OS trixie
• 14:34 jgleeson: payments-wiki upgraded from 4dbf3b57 to 53c258af
• 14:30 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 14:30 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 14:26 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 14:24 dcausse@deploy2002: dcausse: Backport for cirrus: allow title natural sort (T40403) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:22 dcausse@deploy2002: Started scap sync-world: Backport for cirrus: allow title natural sort (T40403)
• 14:19 stran@deploy2002: Finished scap sync-world: Backport for Deploy IRS to pilot wikis (T413773 T413774 T413775 T413776 T413777) (duration: 14m 01s)
• 14:16 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 14:15 stran@deploy2002: stran: Continuing with sync
• 14:13 andrew@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudvirtlocal1003.eqiad.wmnet with reason: host reimage
• 14:12 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirtlocal1003.eqiad.wmnet with reason: host reimage
• 14:07 stran@deploy2002: stran: Backport for Deploy IRS to pilot wikis (T413773 T413774 T413775 T413776 T413777) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:05 stran@deploy2002: Started scap sync-world: Backport for Deploy IRS to pilot wikis (T413773 T413774 T413775 T413776 T413777)
• 13:56 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 13:55 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirtlocal1003.eqiad.wmnet with OS trixie
• 13:53 daniel@deploy2002: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
• 13:52 daniel@deploy2002: helmfile [staging] START helmfile.d/services/rest-gateway: apply
• 13:46 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 12:58 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 12:58 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 12:52 moritzm: uploaded Bird 2.18-1~wmf13u1 to component/bird-routed-ganeti for trixie-wikimedia T413740
• 12:29 andrew@dns1004: END - running authdns-update
• 12:28 andrew@dns1004: START - running authdns-update
• 12:18 ladsgroup@deploy2002: Finished scap sync-world: Backport for Reduce VP9 transcode resolution steps (T413031) (duration: 08m 02s)
• 12:17 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 20940
• 12:14 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 20940
• 12:14 ladsgroup@deploy2002: ladsgroup: Continuing with sync
• 12:12 ladsgroup@deploy2002: ladsgroup: Backport for Reduce VP9 transcode resolution steps (T413031) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:10 ladsgroup@deploy2002: Started scap sync-world: Backport for Reduce VP9 transcode resolution steps (T413031)
• 11:59 cgoubert@deploy2002: Finished scap sync-world: 1223643: Revert "apache: Don't redirect RestSandbox on wikimedia.org" (duration: 03m 34s)
• 11:58 cgoubert@deploy2002: cgoubert: Continuing with sync
• 11:58 cgoubert@deploy2002: cgoubert: 1223643: Revert "apache: Don't redirect RestSandbox on wikimedia.org" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 11:58 Amir1: ladsgroup@deploy2002:~$ mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 11 echo-subscriptions-email-page-review (T406724)
• 11:57 cgoubert@deploy2002: Started scap sync-world: 1223643: Revert "apache: Don't redirect RestSandbox on wikimedia.org"
• 11:32 cgoubert@deploy2002: cgoubert: 1223188: apache: Don't redirect RestSandbox on wikimedia.org synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 11:31 cgoubert@deploy2002: Started scap sync-world: 1223188: apache: Don't redirect RestSandbox on wikimedia.org
• 11:21 claime: Deploying apache config change T396807 - 1223188
• 10:58 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'clear' for AS: 400566
• 10:58 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'clear' for AS: 400566
• 10:47 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 3223
• 10:46 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'email' for AS: 3223
• 10:43 kharlan@deploy2002: Finished scap sync-world: Backport for QuickSurveys: Enable coverage percentages for safety survey (T413022) (duration: 09m 27s)
• 10:40 moritzm: installing aom bugfix updates from Bookworm point release
• 10:36 kharlan@deploy2002: kharlan: Continuing with sync
• 10:35 kharlan@deploy2002: kharlan: Backport for QuickSurveys: Enable coverage percentages for safety survey (T413022) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 10:33 kharlan@deploy2002: Started scap sync-world: Backport for QuickSurveys: Enable coverage percentages for safety survey (T413022)
• 10:32 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'clear' for AS: 55818
• 10:31 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'clear' for AS: 55818
• 10:30 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 18734
• 10:30 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 18734
• 10:28 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 400566
• 10:28 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 400566
• 10:27 ayounsi@cumin1003: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 208915
• 10:27 ayounsi@cumin1003: START - Cookbook sre.network.peering with action 'configure' for AS: 208915
• 10:22 mvernon@cumin2002: END (FAIL) - Cookbook sre.swift.remove-ghost-objects (exit_code=99) from container wikipedia-commons-local-public.f7 in codfw
• 10:19 mvernon@cumin2002: START - Cookbook sre.swift.remove-ghost-objects from container wikipedia-commons-local-public.f7 in codfw
• 09:40 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 09:14 XioNoX: push pfw policies - T413833
• 09:07 moritzm: installing e2fsprogs updates from Bookworm point release
• 08:12 moritzm: installing distro-info-data updates
• 08:08 hashar@deploy2002: Finished deploy [gerrit/gerrit@c323101]: Disable banner for the 2025 developer survey (duration: 00m 11s)
• 08:08 hashar@deploy2002: Started deploy [gerrit/gerrit@c323101]: Disable banner for the 2025 developer survey
• 07:54 moritzm: installing bash updates from Bookworm point release
• 04:55 eileen: civicrm upgraded from 68ea4eab to dc2511ee
• 04:47 mwpresync@deploy2002: Finished scap sync-world: testwikis to 1.46.0-wmf.10 refs T408280 (duration: 44m 16s)
• 04:03 mwpresync@deploy2002: Started scap sync-world: testwikis to 1.46.0-wmf.10 refs T408280
• 03:04 eileen: config revision changed from 76dff80b to 596df184
• 02:44 eileen: revision changed from ddb2f36b to 76dff80b
• 02:26 eileen: civicrm upgraded from 6bd3c988 to 8751ed14
• 01:13 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 12m 32s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image
• 00:36 eileen: civicrm upgraded from 9d26c426 to 6bd3c988

2026-01-05

• 23:38 ejegg: payments-wiki upgraded from 857e80f2 to 4dbf3b57
• 22:58 reedy@deploy2002: Finished scap sync-world: Backport for Remove WebAuthn keys that no longer need Wikimedia overrides (T413287) (duration: 06m 24s)
• 22:54 reedy@deploy2002: reedy: Continuing with sync
• 22:53 reedy@deploy2002: reedy: Backport for Remove WebAuthn keys that no longer need Wikimedia overrides (T413287) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:51 reedy@deploy2002: Started scap sync-world: Backport for Remove WebAuthn keys that no longer need Wikimedia overrides (T413287)
• 22:36 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirtlocal1002.eqiad.wmnet with OS trixie
• 22:31 cjming: end of UTC late backport window
• 22:29 cjming@deploy2002: Finished scap sync-world: Backport for Fetch user from primary DB when saving settings (T411804) (duration: 06m 06s)
• 22:25 cjming@deploy2002: d3r1ck01, cjming: Continuing with sync
• 22:25 cjming@deploy2002: d3r1ck01, cjming: Backport for Fetch user from primary DB when saving settings (T411804) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 22:23 cjming@deploy2002: Started scap sync-world: Backport for Fetch user from primary DB when saving settings (T411804)
• 21:55 cjming@deploy2002: Finished scap sync-world: Backport for [config] Set Category Collation for arwiktionary (T413338) (duration: 09m 23s)
• 21:51 cjming@deploy2002: hubaishan, cjming: Continuing with sync
• 21:48 cjming@deploy2002: hubaishan, cjming: Backport for [config] Set Category Collation for arwiktionary (T413338) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:46 cjming@deploy2002: Started scap sync-world: Backport for [config] Set Category Collation for arwiktionary (T413338)
• 21:44 andrew@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirtlocal1002.eqiad.wmnet with reason: host reimage
• 21:44 cjming@deploy2002: Finished scap sync-world: Backport for arbcom_zhwiki: Logo Changes (T413649) (duration: 11m 12s)
• 21:38 cjming@deploy2002: zhaofjx, cjming: Continuing with sync
• 21:38 jhancock@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2249.codfw.wmnet with OS bookworm
• 21:38 jhancock@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:37 andrew@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirtlocal1002.eqiad.wmnet with reason: host reimage
• 21:37 cjming@deploy2002: zhaofjx, cjming: Backport for arbcom_zhwiki: Logo Changes (T413649) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 21:33 cjming@deploy2002: Started scap sync-world: Backport for arbcom_zhwiki: Logo Changes (T413649)
• 21:30 jhancock@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1003"
• 21:20 andrew@cumin2002: START - Cookbook sre.hosts.reimage for host cloudvirtlocal1002.eqiad.wmnet with OS trixie
• 21:15 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on db2249.codfw.wmnet with reason: host reimage
• 21:07 jhancock@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on db2249.codfw.wmnet with reason: host reimage
• 20:58 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host db2249.codfw.wmnet with OS bookworm
• 20:50 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2249.codfw.wmnet with OS bookworm
• 20:07 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host db2249.codfw.wmnet with OS bookworm
• 19:13 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply
• 19:12 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-video: apply
• 19:12 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: apply
• 19:11 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-timeline: apply
• 19:11 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 19:10 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 19:10 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-media: apply
• 19:10 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-media: apply
• 19:09 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox-constraints: apply
• 19:09 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox-constraints: apply
• 19:08 swfrench@deploy2002: helmfile [codfw] DONE helmfile.d/services/shellbox: apply
• 19:07 swfrench@deploy2002: helmfile [codfw] START helmfile.d/services/shellbox: apply
• 18:59 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
• 18:58 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
• 18:58 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
• 18:57 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
• 18:57 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 18:56 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 18:56 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
• 18:56 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
• 18:55 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
• 18:54 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
• 18:54 swfrench@deploy2002: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
• 18:53 swfrench@deploy2002: helmfile [eqiad] START helmfile.d/services/shellbox: apply
• 18:51 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
• 18:51 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-video: apply
• 18:50 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
• 18:50 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
• 18:49 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
• 18:49 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply
• 18:48 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
• 18:48 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-media: apply
• 18:48 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
• 18:47 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
• 18:47 swfrench@deploy2002: helmfile [staging] DONE helmfile.d/services/shellbox: apply
• 18:46 swfrench@deploy2002: helmfile [staging] START helmfile.d/services/shellbox: apply
• 18:36 swfrench@deploy2002: Finished scap sync-world: Rebuild deployment to pick up new production image (duration: 36m 10s)
• 18:33 andrew@dns1004: END - running authdns-update
• 18:32 andrew@dns1004: START - running authdns-update
• 18:15 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 18:15 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 18:01 swfrench@deploy2002: Started scap sync-world: Rebuild deployment to pick up new production image
• 17:57 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 17:57 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 17:55 bking@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 17:55 bking@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-ipoid: apply
• 17:47 swfrench-wmf: reprepro include php8.3_8.3.29-1+wmf11u2 in component/php83
• 17:47 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2249.codfw.wmnet with OS bookworm
• 17:08 dancy@deploy2002: Installation of scap version "4.231.0" completed for 2 hosts
• 17:07 dancy@deploy2002: Installing scap version "4.231.0" for 2 host(s)
• 16:48 jdrewniak@deploy2002: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 56s)
• 16:46 jdrewniak@deploy2002: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 06m 08s)
• 16:30 btullis@deploy2002: Finished scap build-images: Building new mediawiki-cli version to pick up https://gerrit.wikimedia.org/r/c/operations/dumps/+/1223213 (duration: 00m 51s)
• 16:29 btullis@deploy2002: Started scap build-images: Building new mediawiki-cli version to pick up https://gerrit.wikimedia.org/r/c/operations/dumps/+/1223213
• 16:26 jhancock@cumin1003: START - Cookbook sre.hosts.reimage for host db2249.codfw.wmnet with OS bookworm
• 16:24 jhancock@cumin1003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2249.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:23 jhancock@cumin1003: START - Cookbook sre.hosts.provision for host db2249.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
• 16:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 16:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1263.eqiad.wmnet with reason: Maintenance
• 16:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1262.eqiad.wmnet with reason: Maintenance
• 16:11 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1261.eqiad.wmnet with reason: Maintenance
• 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1260.eqiad.wmnet with reason: Maintenance
• 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1252.eqiad.wmnet with reason: Maintenance
• 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1249.eqiad.wmnet with reason: Maintenance
• 16:10 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1248.eqiad.wmnet with reason: Maintenance
• 16:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1247.eqiad.wmnet with reason: Maintenance
• 16:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 16:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1244.eqiad.wmnet with reason: Maintenance
• 16:09 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1243.eqiad.wmnet with reason: Maintenance
• 16:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1242.eqiad.wmnet with reason: Maintenance
• 16:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1241.eqiad.wmnet with reason: Maintenance
• 16:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1238.eqiad.wmnet with reason: Maintenance
• 16:08 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 6 hosts with reason: Maintenance
• 16:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1221.eqiad.wmnet with reason: Maintenance
• 16:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1199.eqiad.wmnet with reason: Maintenance
• 16:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1190.eqiad.wmnet with reason: Maintenance
• 16:07 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 15:54 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 15:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1226.eqiad.wmnet with reason: Maintenance
• 15:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1214.eqiad.wmnet with reason: Maintenance
• 15:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1203.eqiad.wmnet with reason: Maintenance
• 15:53 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1193.eqiad.wmnet with reason: Maintenance
• 15:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1192.eqiad.wmnet with reason: Maintenance
• 15:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1178.eqiad.wmnet with reason: Maintenance
• 15:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1177.eqiad.wmnet with reason: Maintenance
• 15:52 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1172.eqiad.wmnet with reason: Maintenance
• 15:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 15:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 15:51 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1167.eqiad.wmnet with reason: Maintenance
• 15:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 15:48 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1253.eqiad.wmnet with reason: Maintenance
• 15:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1231.eqiad.wmnet with reason: Maintenance
• 15:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1227.eqiad.wmnet with reason: Maintenance
• 15:47 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1202.eqiad.wmnet with reason: Maintenance
• 15:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1194.eqiad.wmnet with reason: Maintenance
• 15:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1191.eqiad.wmnet with reason: Maintenance
• 15:46 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance
• 15:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1174.eqiad.wmnet with reason: Maintenance
• 15:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance
• 15:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance
• 15:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 15:45 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1158.eqiad.wmnet with reason: Maintenance
• 14:46 hashar@deploy2002: Finished deploy [gerrit/gerrit@8c906b0]: Reword schedule deployment link - T412992 (duration: 00m 12s)
• 14:46 hashar@deploy2002: Started deploy [gerrit/gerrit@8c906b0]: Reword schedule deployment link - T412992
• 14:43 stran@deploy2002: Finished scap sync-world: Backport for trwikisource: Create rollbacker user group (T410931), Enable protection indicators for ruwiki, CommonSettings: Remove EOL REL1_39 from ExtensionDistributor (T403199) (duration: 20m 31s)
• 14:42 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 14:40 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1240.eqiad.wmnet with reason: Maintenance
• 14:39 stran@deploy2002: reedy, stran, neriah: Continuing with sync
• 14:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on 6 hosts with reason: Maintenance
• 14:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1212.eqiad.wmnet with reason: Maintenance
• 14:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1198.eqiad.wmnet with reason: Maintenance
• 14:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1189.eqiad.wmnet with reason: Maintenance
• 14:31 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1175.eqiad.wmnet with reason: Maintenance
• 14:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1166.eqiad.wmnet with reason: Maintenance
• 14:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1157.eqiad.wmnet with reason: Maintenance
• 14:25 stran@deploy2002: reedy, stran, neriah: Backport for trwikisource: Create rollbacker user group (T410931), Enable protection indicators for ruwiki, CommonSettings: Remove EOL REL1_39 from ExtensionDistributor (T403199) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:23 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1150.eqiad.wmnet with reason: Maintenance
• 14:23 stran@deploy2002: Started scap sync-world: Backport for trwikisource: Create rollbacker user group (T410931), Enable protection indicators for ruwiki, CommonSettings: Remove EOL REL1_39 from ExtensionDistributor (T403199)
• 14:16 stran@deploy2002: Finished scap sync-world: Backport for wgEnableProtectionIndicators true for frwiktionary (T413724) (duration: 06m 41s)
• 14:12 stran@deploy2002: stran, bunnypranav: Continuing with sync
• 14:11 stran@deploy2002: stran, bunnypranav: Backport for wgEnableProtectionIndicators true for frwiktionary (T413724) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 14:09 stran@deploy2002: Started scap sync-world: Backport for wgEnableProtectionIndicators true for frwiktionary (T413724)
• 14:05 moritzm: installing jinja2 security updates
• 13:37 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
• 13:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1259.eqiad.wmnet with reason: Maintenance
• 13:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1254.eqiad.wmnet with reason: Maintenance
• 13:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1239.eqiad.wmnet with reason: Maintenance
• 13:36 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1233.eqiad.wmnet with reason: Maintenance
• 13:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1229.eqiad.wmnet with reason: Maintenance
• 13:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 13:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1197.eqiad.wmnet with reason: Maintenance
• 13:35 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1188.eqiad.wmnet with reason: Maintenance
• 13:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1182.eqiad.wmnet with reason: Maintenance
• 13:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1162.eqiad.wmnet with reason: Maintenance
• 13:34 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 13:33 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1156.eqiad.wmnet with reason: Maintenance
• 13:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
• 13:28 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1245.eqiad.wmnet with reason: Maintenance
• 13:27 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1230.eqiad.wmnet with reason: Maintenance
• 13:27 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1216.eqiad.wmnet with reason: Maintenance
• 13:26 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1207.eqiad.wmnet with reason: Maintenance
• 13:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1200.eqiad.wmnet with reason: Maintenance
• 13:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1185.eqiad.wmnet with reason: Maintenance
• 13:25 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
• 13:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1161.eqiad.wmnet with reason: Maintenance
• 13:24 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1159.eqiad.wmnet with reason: Maintenance
• 13:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
• 13:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1225.eqiad.wmnet with reason: Maintenance
• 13:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance
• 13:22 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1180.eqiad.wmnet with reason: Maintenance
• 13:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1173.eqiad.wmnet with reason: Maintenance
• 13:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1168.eqiad.wmnet with reason: Maintenance
• 13:21 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1015,1019].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
• 13:20 fceratto@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1165.eqiad.wmnet with reason: Maintenance
• 13:00 kharlan@deploy2002: Finished scap sync-world: Backport for QuickSurveys: Enable safety survey at 0% coverage (T413022) (duration: 09m 14s)
• 12:55 kharlan@deploy2002: kharlan: Continuing with sync
• 12:52 kharlan@deploy2002: kharlan: Backport for QuickSurveys: Enable safety survey at 0% coverage (T413022) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 12:50 kharlan@deploy2002: Started scap sync-world: Backport for QuickSurveys: Enable safety survey at 0% coverage (T413022)
• 12:39 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
• 12:39 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply
• 12:28 btullis@deploy2002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
• 12:26 kevinbazira@deploy2002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
• 12:26 btullis@deploy2002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
• 11:26 cgoubert@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on 34 hosts with reason: up for decom
• 11:25 cgoubert@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on 16 hosts with reason: up for decom
• 11:24 cgoubert@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on 24 hosts with reason: up for decom
• 10:57 moritzm: installing Postgresql 13 security updates
• 09:22 zabe@deploy2002: Finished scap sync-world: Backport for Pin imagelinks migration to old schema (T299953) (duration: 06m 55s)
• 09:18 zabe@deploy2002: zabe: Continuing with sync
• 09:17 zabe@deploy2002: zabe: Backport for Pin imagelinks migration to old schema (T299953) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 09:15 zabe@deploy2002: Started scap sync-world: Backport for Pin imagelinks migration to old schema (T299953)
• 09:14 moritzm: installing Django security updates
• 09:12 zabe@deploy2002: Finished scap sync-world: Backport for SpecialPageLanguage: Use OOUI infuse if language selector is present (T413313) (duration: 48m 25s)
• 08:59 zabe@deploy2002: abi, zabe: Continuing with sync
• 08:59 zabe@deploy2002: abi, zabe: Backport for SpecialPageLanguage: Use OOUI infuse if language selector is present (T413313) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
• 08:59 Emperor: depool / restart swift / repool ms-fe1014 T360913
• 08:58 Emperor: depool / restart swift / repool ms-fe1010 T360913
• 08:24 zabe@deploy2002: Started scap sync-world: Backport for SpecialPageLanguage: Use OOUI infuse if language selector is present (T413313)
• 07:20 moritzm: installing net-snmp security updates
• 04:07 Amir1: mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 11 echo-subscriptions-web-article-linked (T406724)

2026-01-04

• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

2026-01-03

• 01:19 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 18m 24s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

2026-01-02

• 14:32 inflatador: bking@wmf3062 restarting blazegraph on wdqs1014.eqiad.wmnet,wdqs1013.eqiad.wmnet,wdqs1019.eqiad.wmnet
• 01:13 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 13m 05s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

2026-01-01

• 12:31 Emperor: restart corto.service on alert1002
• 01:36 mwpresync@deploy2002: Finished scap build-images: Publishing wmf/next image (duration: 35m 31s)
• 01:00 mwpresync@deploy2002: Started scap build-images: Publishing wmf/next image

Other archives

See Server Admin Log/Archives.