Nova Resource:Admin/SAL

2024-12-04

• 17:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
• 17:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet'
• 17:16 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1035.eqiad.wmnet' (T380893)
• 17:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' (T380893)
• 15:09 andrewbogott: rebooted cloudinfra-cloudvps-puppetserver-1, it's so busy that it's unresponsive
• 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1035.eqiad.wmnet}' (T380731)
• 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1035.eqiad.wmnet}' (T380731)
• 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' (T380893)
• 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' (T380893)

2024-11-27

• 13:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet'
• 13:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'

2024-11-26

• 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1062.eqiad.wmnet}' (T380731)
• 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1062.eqiad.wmnet}' (T380731)
• 15:16 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 15:15 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:15 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 15:15 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 15:14 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/145
• 15:13 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/145
• 15:12 rook@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 15:12 rook@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 13:31 dcaro: added cloudcephmon1004 to the ceph mon pool
• 12:41 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
• 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 12:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 12:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 05:40 andrewbogott: rebooting tools-sgebastion-10.tools.eqiad1.wikimedia.cloud to get NFS things remounted
• 04:56 andrewbogott: 'systemctl restart nfs-server' on tools-nfs-2.tools.eqiad1.wikimedia.cloud

2024-11-25

• 14:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 14:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 14:40 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 13:55 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 13:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 13:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:28 arturo: create IPv6 networks in eqiad1 (T380174), then reverted because network outage
• 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 10:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-11-23

• 14:58 taavi: removed broken records for re-created VMs in .eqiad.wmflabs zone

2024-11-22

• 14:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 14:35 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:48 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 13:48 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-11-20

• 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:26 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

2024-11-19

• 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 13:43 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:43 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
• 13:37 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:35 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
• 11:39 arturo: [codfw1dev] performing rabbit full reset T380208
• 10:26 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 10:24 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
• 10:24 arturo: [codfw1dev] restart rabbitmq and nova/neutron services for T380208

2024-11-16

• 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 08:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 07:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 07:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-11-15

• 19:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 17:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 17:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 17:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 17:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 17:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 17:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 17:10 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 17:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 17:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 17:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 17:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 17:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 16:30 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:29 arturo: [codfw1dev] restart rabbitmq and designate
• 16:29 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack

2024-11-14

• 09:52 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 09:48 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack

2024-11-13

• 13:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120
• 13:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/120

2024-11-11

• 14:49 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console

2024-11-08

• 16:47 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:45 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
• 16:42 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:42 arturo: [codfw1dev] restart all nova services and rabbitmq out of despair
• 16:42 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
• 13:45 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:44 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
• 13:33 aborrero@cloudcumin2001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 13:32 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
• 13:28 arturo: [codfw1dev] restart rabbitmq, openstack services logs show connection errors

2024-11-07

• 17:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 17:42 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 13:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 13:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-11-06

• 13:09 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
• 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
• 13:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
• 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/117
• 11:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 11:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/116
• 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/116
• 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 09:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-11-05

• 10:24 arturo: [codfw1dev] disable puppet and make changes for testing T378192

2024-11-04

• 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 16:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 15:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 14:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:09 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:08 aborrero@cloudcumin2001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 14:08 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
• 14:08 aborrero@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 14:08 aborrero@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
• 13:52 arturo: [codfw1dev] live-hack cloudlb2001-dev and cloudcontrol2004-dev for T378192
• 13:52 arturo: [codfw1dev] restart rabbitmq
• 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-10-29

• 15:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 13:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-10-28

• 17:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:00 arturo: [codfw1dev] restarting rabbitmq, misbehaving
• 17:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 16:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 16:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:22 dhinus: apt full-upgrade and reboot for cloudcumin*
• 16:21 dhinus: upgrade spicerack from 8.8.0 to 8.15.1 on cloudcumin*

2024-10-24

• 15:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-10-22

• 15:02 taavi: recover access to User:Labslogbot T376220

2024-10-21

• 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 12:06 arturo: [codfw1dev] restart rabbitmq, tofu shows error talking to the designate API

2024-10-16

• 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 14:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-10-15

• 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 15:38 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
• 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 15:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
• 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 15:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
• 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
• 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
• 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
• 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
• 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 11:33 arturo: cloudgw maintenance, firewall change for T374714
• 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-10-11

• 15:31 arturo: cloudgw maintenance firewall change T374716
• 09:51 arturo: cloudgw network maintenance related to T376879
• 08:33 arturo: [codfw1dev] reboot cloudgw2002-dev/2003-dev because network connectivity issues

2024-10-10

• 12:04 arturo: manual network failover in cloudgw because maintenance related to T376879
• 10:40 dhinus: cumin 'cloudrabbit*' 'systemctl restart rabbitmq-server' T376802
• 09:19 arturo: [codfw1dev] enable IPv6 on cloudgw (T374716)

2024-10-09

• 14:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 14:20 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 14:19 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 14:19 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 14:16 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 14:16 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 10:58 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 10:58 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 10:55 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 10:55 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 10:54 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 10:53 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 10:53 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 10:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93
• 10:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95
• 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/95
• 10:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 10:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch

2024-10-08

• 11:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-10-07

• 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-10-04

• 15:12 arturo: cloudservice1005/1006: enable puppet and restore /etc/powerdns/recursor.conf to non-debug mode (T374830)
• 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 08:32 arturo: cloudservice1005/1006: disable puppet and set `quiet=no` in /etc/powerdns/recursor.conf (T374830)

2024-10-03

• 14:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 14:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 14:54 arturo: [codfw1dev] delete default security group rule list, now tracking them via tofu-infra
• 14:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 14:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 14:51 arturo: delete default security group rule list, now tracking them via tofu-infra
• 14:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-10-02

• 15:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:23 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan+apply for main branch
• 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:55 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch (T376211)
• 10:21 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch (T376211)
• 10:16 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch (T376211)
• 10:15 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch (T376211)
• 10:05 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch (T376211)
• 10:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch (T376211)
• 09:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (T376211)
• 09:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (T376211)
• 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (T376211)
• 09:10 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/77 (T376211)

2024-10-01

• 15:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 09:53 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 08:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 08:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 08:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)

2024-09-30

• 20:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T372814)
• 16:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
• 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
• 16:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node (T372814)
• 13:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
• 13:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T372814)
• 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
• 13:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
• 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
• 11:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
• 11:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
• 11:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
• 11:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
• 10:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
• 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
• 10:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reset_weights (exit_code=0)
• 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
• 10:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.reset_weights (exit_code=99)
• 10:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reset_weights
• 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
• 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
• 10:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=99)
• 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
• 09:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
• 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
• 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
• 09:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
• 09:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
• 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
• 09:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
• 09:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
• 09:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
• 09:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
• 09:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
• 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=0)
• 09:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
• 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.reset_weights (exit_code=99)
• 09:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.reset_weights
• 09:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 09:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)

2024-09-27

• 15:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 13:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 13:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 13:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 10:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 10:56 arturo: [codfw1dev] restart rabbitmq again
• 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 10:15 arturo: [codfw1dev] restart rabbitmq
• 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 10:04 arturo: [codfw1dev] enable IPv6 on the neutron virtual router T375847

2024-09-26

• 14:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 14:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node (T372814)
• 14:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T372814)
• 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node (T372814)
• 10:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T372814)
• 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 08:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 08:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 08:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 08:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 08:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 08:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 08:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 08:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 08:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 08:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 07:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 07:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 07:47 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 07:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 07:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 07:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 07:45 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 07:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)
• 07:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T372814)
• 07:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T372814)

2024-09-25

• 14:11 arturo: [codfw1dev] start proxy-02 vm on proxy-codfw1dev project, it was in shutoff mode for unknown reasons
• 10:33 arturo: [codfw1dev] cleanup unused security groups (T375604)
• 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 09:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:48 arturo: [codfw1dev] restart rabbitmq on all cloudcontrol servers
• 09:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 09:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 09:35 arturo: [codfw1dev] deletre a bunch of tests and seemingly unused projects

2024-09-24

• 19:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643)
• 16:00 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T348643)
• 14:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T348643)
• 14:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node (T348643)
• 14:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 13:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 12:56 arturo: [codfw1dev] restart rabbitmq-server on all 3 nodes
• 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack
• 09:10 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_rack (exit_code=97)
• 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack

2024-09-23

• 19:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_rack (exit_code=99)
• 15:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 15:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 15:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 15:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 15:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 15:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 14:42 arturo: put cloudvirt1048 in the network-ovs aggregate T364457
• 14:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_rack
• 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:37 arturo: [codfw1dev] restart rabbitmq
• 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 12:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 12:23 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch

2024-09-21

• 10:17 dhinus: nova host-evacuate cloudvirt1063 (T375223)
• 09:59 dhinus: openstack aggregate remove host ceph cloudvirt1063 (T375223)
• 09:59 dhinus: openstack aggregate add host maintenance cloudvirt1063 (T375223)
• 09:42 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1063.eqiad.wmnet'
• 09:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063.eqiad.wmnet'

2024-09-20

• 11:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 10:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 08:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T373740)
• 08:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T373740)

2024-09-19

• 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' (T373740)
• 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' (T373740)
• 15:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T373740)
• 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T373740)
• 10:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51
• 10:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51
• 09:56 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) (T374043)
• 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node (T374043)
• 09:51 arturo: [codfw1dev] play with neutron default security group rules (delete, create them, etc) T375111
• 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 09:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 09:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 09:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 09:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50
• 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50

2024-09-18

• 15:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
• 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
• 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
• 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
• 15:21 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
• 15:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/49
• 15:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
• 15:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
• 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
• 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/48
• 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/47
• 11:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/47
• 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 08:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 08:39 dcaro: restarted rabbitmq-server on all cloudrabbits
• 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 08:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 08:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.restart_openstack

2024-09-17

• 21:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T374043)
• 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node (T374043)
• 16:11 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) (T374043)
• 16:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node (T374043)
• 15:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 15:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 15:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 15:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 14:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 14:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 14:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46
• 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/46

2024-09-16

• 14:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 14:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 14:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
• 14:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
• 14:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
• 14:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/45
• 11:28 arturo: [codfw1dev] created VM bastion-codfw1dev-04 to replace current bastion -03 (T374828)

2024-09-12

• 10:51 arturo: merging change to keystone wmf hooks https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071230 (T374020)

2024-09-11

• 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 16:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 16:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 16:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
• 15:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
• 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
• 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
• 15:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 15:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:32 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
• 15:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
• 15:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
• 15:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
• 15:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 15:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 15:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
• 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/44
• 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
• 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/43
• 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
• 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
• 12:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
• 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/42
• 11:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 11:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/41
• 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/41
• 11:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 10:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 10:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 07:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet' (T374467)
• 07:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2004-dev.codfw.wmnet' (T374467)

2024-09-10

• 12:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 11:37 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 10:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 10:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:42 dcaro: hard-rebooting cloudvirt2004-dev (codfw1dev) having io/hardware issues
• 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 09:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 08:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 08:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/40
• 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T373986)
• 08:45 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T373986)

2024-09-09

• 21:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T373986)
• 16:36 dcaro: cleaned up dns leaks
• 15:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 15:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/39
• 15:23 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/39
• 15:05 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T373986)
• 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T373986)
• 14:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:06 arturo: merged change to cloudgw NAT setting https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071189
• 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 12:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 11:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 10:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 10:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 10:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 10:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:42 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:38 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/38
• 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T373986)

2024-09-06

• 23:18 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T373986)
• 18:17 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T373986)
• 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T373986)
• 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T373986)
• 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T373986)
• 07:27 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T373986)

2024-09-05

• 21:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T373986)
• 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' (T374043)
• 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' (T374043)
• 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' (T374043)
• 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' (T374043)
• 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T373986)
• 17:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' (T374043)
• 17:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' (T374043)
• 17:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1032.eqiad.wmnet' (T374043)
• 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T373986)
• 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet' (T374043)
• 16:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' (T374043)
• 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' (T374043)
• 14:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 14:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 14:19 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
• 14:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
• 13:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
• 13:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
• 13:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
• 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/37
• 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T373986)
• 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T373986)
• 12:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 12:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/36
• 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/36
• 10:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 10:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/35
• 10:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/35
• 09:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 09:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/34
• 09:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/34
• 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/33
• 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/33
• 09:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 09:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
• 09:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
• 09:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
• 09:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/32
• 09:22 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/31
• 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/31
• 09:16 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 09:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 09:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 08:46 arturo: [codfw1dev] restart rabbitmq @ codfw1dev T374002

2024-09-04

• 21:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T373986)
• 19:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 17:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 17:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T373986)
• 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 15:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 15:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 15:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/30
• 12:18 arturo: [codfw1dev] restart rabbitmq-server.service on all 3 cloudcontrols, all nova-compute agents are down complaining about rabbitmq being unreachable

2024-09-02

• 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:40 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 11:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29
• 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29
• 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 11:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 11:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
• 11:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 10:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch
• 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 10:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 10:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 10:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 10:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 10:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28

2024-08-31

• 13:55 andrewbogott: moving tools-redis-7 off of cloudvirt1048 just in case T373740
• 13:39 andrewbogott: rebooting cloudvirt1048 from mgmt, it seems to have crashed

2024-08-30

• 12:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 11:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-08-29

• 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 18:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

2024-08-25

• 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-08-23

• 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' (T369044)
• 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' (T369044)
• 00:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 00:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 00:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 00:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-08-22

• 23:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 23:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) (T369044)
• 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T369044)
• 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 20:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1047.eqiad.wmnet' (T369044)
• 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' (T369044)
• 20:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' (T369044)
• 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' (T369044)
• 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' (T369044)
• 20:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' (T369044)
• 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' (T369044)
• 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' (T369044)
• 20:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' (T369044)
• 20:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' (T369044)
• 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' (T369044)
• 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' (T369044)
• 20:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' (T369044)
• 20:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' (T369044)
• 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' (T369044)
• 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' (T369044)
• 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' (T369044)
• 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' (T369044)
• 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' (T369044)
• 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' (T369044)
• 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' (T369044)
• 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' (T369044)
• 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' (T369044)
• 19:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' (T369044)
• 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' (T369044)
• 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' (T369044)
• 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' (T369044)
• 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.unset_maintenance (exit_code=0) (T369044)
• 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.unset_maintenance (T369044)
• 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' (T369044)
• 19:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=97) on host 'cloudvirt1039' (T369044)
• 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039' (T369044)
• 19:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1037' (T369044)
• 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037' (T369044)
• 19:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1035' (T369044)
• 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035' (T369044)
• 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' (T369044)
• 19:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' (T369044)
• 19:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1033' (T369044)
• 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033' (T369044)
• 19:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1052' (T369044)
• 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052' (T369044)
• 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' (T369044)
• 18:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' (T369044)
• 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' (T369044)
• 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' (T369044)
• 18:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' (T369044)
• 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' (T369044)
• 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' (T369044)
• 17:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' (T369044)
• 17:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' (T369044)
• 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' (T369044)
• 17:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol1005.eqiad.wmnet' (T369044)
• 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' (T369044)
• 17:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' (T369044)
• 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' (T369044)
• 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices1006.eqiad.wmnet' (T369044)
• 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' (T369044)
• 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' (T369044)
• 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' (T369044)
• 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' (T369044)
• 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' (T369044)
• 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) (T369044)
• 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T369044)

2024-08-20

• 03:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)

2024-08-19

• 23:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 23:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 23:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 23:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 23:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 23:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 23:16 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
• 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 18:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

2024-08-17

• 03:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 03:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

2024-08-16

• 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 15:20 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97)
• 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 15:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
• 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node
• 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T363344)
• 05:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T363344)
• 04:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T363344)
• 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T363344)
• 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T363344)
• 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T363344)
• 04:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T363344)
• 04:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T363344)
• 04:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T363344)
• 04:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T363344)
• 03:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T363344)
• 03:57 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T363344)
• 03:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T363344)
• 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T363344)
• 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T363344)
• 03:50 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T363344)
• 03:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T363344)
• 01:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:12 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:08 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

2024-08-15

• 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 19:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 19:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
• 16:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:27 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
• 16:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
• 08:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 04:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 04:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 04:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 04:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 04:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 04:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 04:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 04:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 04:19 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 04:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 04:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 03:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 03:03 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 03:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 03:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 03:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 03:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 02:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:39 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 01:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 01:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 00:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 00:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 00:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 00:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 00:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 00:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

2024-08-14

• 23:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 23:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 23:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 23:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 23:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 23:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 23:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 23:22 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 15:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
• 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 15:05 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
• 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 04:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-08-12

• 15:44 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) (T363344)
• 11:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
• 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (T363344)
• 11:51 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) (T363344)
• 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (T363344)
• 11:51 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) (T363344)
• 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (T363344)
• 08:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=0)
• 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
• 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
• 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
• 08:37 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97) (T371878)
• 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack (T371878)
• 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
• 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack
• 08:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) (T371878)
• 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack (T371878)
• 08:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99) (T371878)
• 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_rack (T371878)
• 08:17 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=97)
• 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_rack

2024-08-09

• 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T371878)
• 18:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T371878)
• 13:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)
• 13:38 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (T371878)
• 13:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)
• 13:35 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (T371878)
• 13:34 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)
• 11:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T371878)
• 05:36 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)
• 05:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T371878)
• 00:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)
• 00:47 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (T371878)
• 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)

2024-08-08

• 23:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T371878)
• 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet'
• 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet'
• 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet'
• 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet'
• 18:39 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
• 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet'
• 18:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)
• 18:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T371878)
• 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet'
• 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)
• 18:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T371878)
• 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)
• 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
• 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T371878)
• 18:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet'
• 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
• 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
• 17:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt2004-dev.codfw.wmnet'
• 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet'
• 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet'
• 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet'
• 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet'
• 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet'
• 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet'
• 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1001-dev.eqiad.wmnet'
• 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1001-dev.eqiad.wmnet'
• 16:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudbackup1002-dev.eqiad.wmnet'
• 16:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
• 16:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
• 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.eqiad.wmnet'
• 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudbackup1002-dev.codfw.wmnet'
• 16:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudbackup1002-dev.codfw.wmnet'
• 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet'
• 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
• 16:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
• 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet'
• 16:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2006-dev.codfw.wmnet'
• 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2006-dev.codfw.wmnet'
• 16:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet'
• 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
• 16:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
• 16:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
• 16:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
• 16:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
• 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
• 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet'
• 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2005-dev.codfw.wmnet'
• 15:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet'
• 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet'
• 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.wikimedia.org'
• 15:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.wikimedia.org'
• 15:51 andrewbogott: upgrading codfw1dev to openstack version caracal https://phabricator.wikimedia.org/T369044
• 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) (T369044)
• 15:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T369044)
• 15:49 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) (T369044)
• 15:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T369044)
• 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
• 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 14:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
• 14:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
• 13:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.wait_for_rebalance
• 13:00 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97)
• 12:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 12:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
• 12:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 11:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T371878)
• 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node (T371878)
• 09:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 09:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
• 07:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 07:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=99)
• 06:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 05:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 05:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 00:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 00:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 00:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)

2024-08-07

• 20:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 20:11 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=97)
• 20:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 20:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 20:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 18:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 17:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
• 17:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 17:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 17:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 17:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 17:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 17:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 17:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 17:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 16:56 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=97)
• 16:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node
• 16:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node
• 16:40 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 15:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T371878)
• 08:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T371878)
• 06:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T371878)
• 03:08 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T371878)
• 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)

2024-08-06

• 21:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node (T371878)
• 19:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T371878)
• 18:36 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet'
• 18:21 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1042.eqiad.wmnet'
• 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet'
• 18:17 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet'
• 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet'
• 18:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet'
• 17:59 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet'
• 17:58 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet'
• 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
• 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet'
• 17:45 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet'
• 17:43 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet'
• 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 17:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet'
• 17:32 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet'
• 17:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet'
• 17:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet'
• 17:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet'
• 17:21 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet'
• 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet'
• 17:14 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet'
• 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet'
• 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 17:01 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet'
• 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 17:00 wmbot~andrew@bullseye: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
• 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:48 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
• 16:47 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
• 16:47 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
• 16:37 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
• 16:37 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
• 16:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:04 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 16:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 15:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 15:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T371878)
• 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (T371878)
• 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 15:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 15:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 15:25 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 15:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 15:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1036.eqiad.wmnet'
• 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
• 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 15:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 15:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 15:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance

2024-08-01

• 13:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 13:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 03:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 03:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 03:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 03:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 02:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 02:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 01:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 01:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 01:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 01:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 01:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-07-31

• 23:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 23:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 23:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 23:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 16:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 15:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 15:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 15:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:07 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28
• 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:24 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:22 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 08:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 08:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27

2024-07-30

• 11:29 arturo: installing nova security updates (T371240)

2024-07-29

• 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 14:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 11:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 11:28 arturo: [codfw1dev] restarting rabbitmq-server on all cloudcontrols, nova-compute cannot contact it
• 11:00 arturo: [codfw1dev] installing nova security updates (T371240)
• 10:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 09:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 08:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27

2024-07-25

• 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 14:46 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 14:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 14:45 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 14:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:16 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:15 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:09 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 12:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 12:55 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/27
• 11:40 arturo: manually restart maintain-dbusers in cloudcontrol1005 to see if that makes any difference
• 11:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 11:03 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 11:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 10:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/26
• 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 09:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 08:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 08:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 08:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
• 08:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
• 08:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
• 08:27 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
• 08:25 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
• 08:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/25
• 08:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 08:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 07:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 07:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23

2024-07-24

• 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 16:01 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 15:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 15:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 15:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:46 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:43 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:38 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:37 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:37 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:34 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:28 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 15:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/23
• 11:59 arturo: restarted maintain-dbusers.service in cloudcontrol1005, it was stuck doing nothing
• 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 10:44 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 10:43 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 10:42 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24
• 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/24

2024-07-23

• 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 13:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 10:59 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:52 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:51 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:48 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/22
• 10:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 10:08 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 10:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
• 10:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
• 09:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 08:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 08:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 05:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 05:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 05:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 05:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 03:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 03:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 02:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 02:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 00:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 00:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)

2024-07-22

• 23:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 23:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 21:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 21:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 20:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 20:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 18:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
• 16:07 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
• 16:05 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
• 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/21
• 15:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 15:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 15:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 15:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 15:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 15:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 15:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 14:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 14:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 13:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 13:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 13:28 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:14 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:10 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:06 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:06 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:04 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 13:04 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:56 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:47 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:35 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:31 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 12:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 12:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch
• 11:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch
• 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 11:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 11:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/17
• 11:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 11:35 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 11:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 11:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch
• 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 11:29 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch
• 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.tofu running tofu plan for main branch
• 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 08:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy

2024-07-21

• 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 09:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 09:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 06:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 06:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 03:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 03:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 03:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 03:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 00:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 00:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 00:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 00:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

2024-07-20

• 21:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 21:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 21:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 21:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 18:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 18:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 18:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 18:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 16:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 15:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 13:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 12:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 12:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 09:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 07:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 07:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 06:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 06:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 04:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 04:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 03:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 03:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 01:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 01:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 00:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 00:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

2024-07-19

• 22:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 22:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 21:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 21:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 19:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 19:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 18:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 18:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 16:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 13:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 07:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 07:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 06:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 06:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 04:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 04:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 04:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 04:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 01:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 01:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 01:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 01:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

2024-07-18

• 22:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 22:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 22:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 22:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 22:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 22:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 19:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 19:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 19:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 19:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 16:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 13:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 13:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 13:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 13:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 12:31 dhinus: upgrade spicerack from 8.5 to 8.8 on cloudcumin*
• 10:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 07:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 07:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 07:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 07:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 04:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 04:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 04:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 04:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 03:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 01:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 01:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 01:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 01:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

2024-07-17

• 22:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 22:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 22:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 22:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 20:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 20:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 19:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 19:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 17:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 17:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 14:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 13:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 13:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 11:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 10:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 10:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 08:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 08:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 07:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 05:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 05:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 04:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 04:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 02:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 02:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 01:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 01:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)

2024-07-16

• 23:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 23:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 22:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 22:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 20:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 19:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 19:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 17:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 17:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 16:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 16:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 13:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 13:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
• 11:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 11:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 11:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 11:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 10:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.wait_for_rebalance (exit_code=0)
• 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.wait_for_rebalance
• 09:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 09:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 09:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 09:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 09:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 09:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 09:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 09:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 08:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 08:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 08:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 08:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 08:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 08:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 08:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 08:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 08:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 08:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

2024-07-15

• 22:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 22:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 20:40 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 20:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 20:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 20:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 18:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 18:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 17:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0)
• 16:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 14:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 14:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 14:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy

2024-07-11

• 13:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 13:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add
• 12:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 11:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 11:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 11:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 11:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 02:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 02:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 02:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 02:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 01:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 01:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 01:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 01:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 01:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 01:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 01:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 01:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 01:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 01:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 01:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 01:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 01:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'
• 01:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1061.eqiad.wmnet'
• 01:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1061.eqiad.wmnet'

2024-07-08

• 17:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T309789)
• 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)
• 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) (T309789)
• 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789)

2024-07-06

• 14:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 14:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-07-05

• 10:33 arturo: aborrero@cloudcephmon1001:~$ sudo ceph osd unset norebalance
• 10:31 arturo: aborrero@cloudcephmon1001:~$ sudo ceph osd unset noin
• 08:56 arturo: installing nova/glance/cinder security updates T369138

2024-07-04

• 20:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)
• 20:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 14:52 dcaro: rebooting cloudcontrol1007 due to systemd-journal service failing to start
• 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)

2024-07-03

• 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) (T309789)
• 16:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789)
• 12:28 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 12:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)

2024-07-02

• 19:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 14:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)

2024-07-01

• 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) (T309789)
• 14:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789)
• 12:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T309789)
• 12:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789)

2024-06-27

• 22:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet'
• 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1059.eqiad.wmnet'
• 19:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1059.eqiad.wmnet'
• 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1059.eqiad.wmnet'
• 18:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet'
• 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1064.eqiad.wmnet'
• 17:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet'
• 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1067.eqiad.wmnet'
• 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet'
• 17:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1058.eqiad.wmnet'
• 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet'
• 17:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1066.eqiad.wmnet'
• 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet'
• 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1065.eqiad.wmnet'
• 17:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet'
• 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1057.eqiad.wmnet'
• 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)
• 15:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)
• 15:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 15:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)
• 15:21 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) (T309789)
• 15:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)
• 13:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) (T309789)
• 12:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet'
• 12:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirtlocal1001.eqiad.wmnet'
• 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789)
• 12:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
• 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.unset_cluster_maintenance
• 12:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
• 12:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
• 10:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
• 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.unset_cluster_maintenance
• 10:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
• 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
• 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
• 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
• 10:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
• 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
• 10:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
• 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
• 10:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
• 10:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
• 10:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
• 10:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
• 08:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
• 08:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance
• 07:55 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=99)
• 07:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.set_cluster_in_maintenance

2024-06-26

• 22:32 andrewbogott: disabled all g3.* flavors in eqiad1
• 19:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 17:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)
• 15:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)
• 14:46 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 14:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)
• 14:45 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T309789)
• 14:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T309789)
• 11:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0) (T309789)
• 10:03 dcaro: taking cloudcephosd1006 out of the pool (T348643)
• 10:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789)
• 10:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99) (T309789)
• 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy (T309789)

2024-06-25

• 22:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet'
• 22:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet'
• 22:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet'
• 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet'
• 22:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
• 22:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet'
• 22:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt2001-dev.codw.wmnet'
• 22:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codw.wmnet'
• 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet'
• 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2005-dev.codfw.wmnet'
• 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet'
• 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2004-dev.codfw.wmnet'
• 16:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt2006-dev.codfw.wmnet'
• 16:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2006-dev.codfw.wmnet'
• 03:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 03:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 03:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 03:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 03:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-06-24

• 19:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet'
• 18:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056.eqiad.wmnet'
• 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet'
• 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet'
• 16:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet'
• 16:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet'

2024-06-21

• 10:36 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 10:36 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 10:35 wmbot~arturo@nostromo: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97)
• 10:35 wmbot~arturo@nostromo: START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 10:35 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 10:35 wmbot~arturo@nostromo: START - Cookbook wmcs.openstack.cloudvirt.vm_console
• 09:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 09:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 08:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' (T368129)
• 08:28 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' (T368129)
• 04:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 4e612eb8-04e1-4541-941d-a05519eed60a
• 04:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 4e612eb8-04e1-4541-941d-a05519eed60a

2024-06-20

• 21:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 14789ac1-bc06-4677-9bb0-66c16c887427
• 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 14789ac1-bc06-4677-9bb0-66c16c887427
• 17:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet'
• 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet'
• 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 14:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet'
• 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet'
• 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet'
• 13:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 614f9c99-86f1-410f-8ef5-e33d23215ff5
• 13:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 614f9c99-86f1-410f-8ef5-e33d23215ff5
• 13:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051.eqiad.wmnet'
• 13:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet'
• 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet'
• 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet'
• 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet'
• 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet'
• 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet'
• 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet'
• 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet'
• 10:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 09:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet'
• 09:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet'
• 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet'
• 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1045.eqiad.wmnet'
• 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 04:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 77140f83-1a12-43b7-9e47-0e779503a525
• 04:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 77140f83-1a12-43b7-9e47-0e779503a525
• 04:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server d0b1d9d5-1aec-4a05-a2d7-6d8522a365dc
• 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server d0b1d9d5-1aec-4a05-a2d7-6d8522a365dc
• 04:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 2d2e8925-b50f-483a-82ee-e6a1c588e5be
• 04:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 2d2e8925-b50f-483a-82ee-e6a1c588e5be
• 04:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 138be95d-93ad-4a85-9245-a0a508711555
• 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 270b6533-dc99-4e5d-a642-c61138b11891
• 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 270b6533-dc99-4e5d-a642-c61138b11891
• 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 138be95d-93ad-4a85-9245-a0a508711555
• 04:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server a3e945dc-3548-47fa-8ce3-bf1426ff3b15
• 04:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 0258b810-29af-448a-af5e-ed39e19286df
• 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server a3e945dc-3548-47fa-8ce3-bf1426ff3b15
• 04:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 37e23659-4516-4bd8-a9be-4cc55def5560
• 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 0258b810-29af-448a-af5e-ed39e19286df
• 04:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server e94378df-7a48-4b11-b44a-bf69aaf132bd
• 04:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server e94378df-7a48-4b11-b44a-bf69aaf132bd
• 04:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 7da824f3-c9f3-4460-b9b2-2a894268a7ca
• 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 37e23659-4516-4bd8-a9be-4cc55def5560
• 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 63b82d38-3026-408f-8dcc-0ecbd1a3c870
• 04:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 7da824f3-c9f3-4460-b9b2-2a894268a7ca
• 04:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 7cb371bb-a53a-4e65-a1cf-f1a8264a9166
• 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 63b82d38-3026-408f-8dcc-0ecbd1a3c870
• 04:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 6616fcbf-a49e-4e03-b735-84d31b535c08
• 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 6616fcbf-a49e-4e03-b735-84d31b535c08
• 04:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 7cb371bb-a53a-4e65-a1cf-f1a8264a9166
• 04:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 6f1171db-9d7d-466f-aaa7-cb14e1a6af41
• 04:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 77140f83-1a12-43b7-9e47-0e779503a525
• 04:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 6f1171db-9d7d-466f-aaa7-cb14e1a6af41
• 04:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 1dc3fcee-cf27-4351-ad60-384478624ea3
• 04:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 77140f83-1a12-43b7-9e47-0e779503a525
• 04:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 1dc3fcee-cf27-4351-ad60-384478624ea3
• 04:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 3df62375-e75d-4068-9c71-13519b6bf927
• 04:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 3af9b40f-d29d-4216-9b64-0ebb10f94c7c
• 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 3df62375-e75d-4068-9c71-13519b6bf927
• 04:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 270b6533-dc99-4e5d-a642-c61138b11891
• 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 270b6533-dc99-4e5d-a642-c61138b11891
• 04:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server ae9fa949-e4ce-4ffb-ad9f-4e5a3812d031
• 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server ae9fa949-e4ce-4ffb-ad9f-4e5a3812d031
• 04:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 452dc8d3-6ee0-412c-90ba-66f21f9d09c1
• 04:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 3af9b40f-d29d-4216-9b64-0ebb10f94c7c
• 04:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 452dc8d3-6ee0-412c-90ba-66f21f9d09c1
• 04:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server e94378df-7a48-4b11-b44a-bf69aaf132bd
• 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server e94378df-7a48-4b11-b44a-bf69aaf132bd
• 04:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 47b0da1d-e50a-42c1-8cd9-dc255cc2f1a3
• 04:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 47b0da1d-e50a-42c1-8cd9-dc255cc2f1a3
• 02:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' (T368007)
• 02:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1063.eqiad.wmnet' (T368007)
• 02:05 andrewbogott: cloudvirt1063 is unresponsive, cycling power from racadm

2024-06-19

• 15:43 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1044.eqiad.wmnet'
• 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
• 15:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet'
• 12:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1044.eqiad.wmnet'
• 12:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet'
• 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1043.eqiad.wmnet'
• 11:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet'
• 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1042.eqiad.wmnet'
• 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server f667d3c2-379a-48d7-ad44-4f3933bdb871
• 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server f667d3c2-379a-48d7-ad44-4f3933bdb871
• 01:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 71e4296d-039d-4452-9d92-69b9f8eb3aba
• 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 71e4296d-039d-4452-9d92-69b9f8eb3aba
• 01:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server a7725cd2-6162-41a3-8add-4dc0668b233b
• 01:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server a7725cd2-6162-41a3-8add-4dc0668b233b
• 01:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server 02bb9b5a-cadf-4bee-9b63-519b1e9b485b
• 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 02bb9b5a-cadf-4bee-9b63-519b1e9b485b
• 01:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server 02bb9b5a-cadf-4bee-9b63-519b1e9b485b
• 01:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server 02bb9b5a-cadf-4bee-9b63-519b1e9b485b

2024-06-18

• 21:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.reboot_node (exit_code=0) on hosts matched by 'D{cloudcontrol1006.eqiad.wmnet}'
• 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.reboot_node on hosts matched by 'D{cloudcontrol1006.eqiad.wmnet}'

2024-06-17

• 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' (T364457)
• 20:03 andrewbogott: repaced ovs hosts in the 'ceph' aggregate
• 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' (T364457)
• 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' (T364457)
• 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' (T364457)
• 19:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' (T364457)
• 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' (T364457)
• 18:16 andrewbogott: temporarily removing all ovs hosts from the 'ceph' aggregate so the scheduler will stop putting linuxbridge hosts on ovs hosts and breaking them
• 17:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' (T364457)
• 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' (T364457)
• 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' (T364457)
• 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' (T364457)
• 13:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 13:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
• 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet'
• 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 10:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
• 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet'

2024-06-14

• 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet'
• 13:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet'

2024-06-13

• 16:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 16:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 13:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet'
• 13:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet'
• 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2003-dev.codfw.wmnet'
• 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2003-dev.codfw.wmnet'

2024-06-12

• 11:49 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1031.eqiad.wmnet'
• 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet'
• 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet'
• 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet'
• 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet'
• 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet'

2024-06-11

• 15:55 dcaro: restarting mon service on cloudcephmon1002 to try to release the 2 stuck ops left
• 13:30 taavi: pin all existing eqiad1 flavors to linuxbridge hypervisors T364458
• 12:28 taavi: add all existing eqiad1 cloudvirts to new network-linuxbridge aggregate T364458
• 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 10:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 09:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-06-07

• 19:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-06-05

• 11:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)
• 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.bootstrap_and_add

2024-06-04

• 15:49 taavi: drop hopefully-unused 68.10.in-addr.arpa. from designate T361220

2024-05-30

• 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 02:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-05-29

• 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
• 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) (T364984)
• 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T364984)
• 18:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) (T364984)
• 18:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T364984)

2024-05-28

• 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 19:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 19:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 14:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-05-21

• 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
• 08:18 taavi: stop neutron services on cloudnet1005 T364459

2024-05-20

• 14:46 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
• 14:46 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
• 14:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
• 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
• 14:20 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=0)
• 14:19 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
• 14:18 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
• 14:18 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
• 14:17 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
• 14:17 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
• 14:16 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
• 14:16 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs
• 14:15 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs (exit_code=99)
• 14:15 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.migrate_to_ovs

2024-05-16

• 08:55 taavi: delete 'monitoring' project https://phabricator.wikimedia.org/T365105

2024-05-15

• 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' (T319184)
• 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1041.eqiad.wmnet' (T319184)

2024-05-14

• 19:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 00:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 00:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 00:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 00:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-05-10

• 13:23 andrewbogott: deploying updated 2024.1 Horizon

2024-05-09

• 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-04-25

• 21:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' (T356287)
• 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' (T356287)
• 21:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' (T356287)
• 20:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' (T356287)
• 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' (T356287)
• 20:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' (T356287)
• 20:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' (T356287)
• 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' (T356287)
• 20:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' (T356287)
• 20:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' (T356287)
• 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' (T356287)
• 20:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' (T356287)
• 20:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' (T356287)
• 20:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' (T356287)
• 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' (T356287)
• 20:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' (T356287)
• 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' (T356287)
• 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' (T356287)
• 20:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' (T356287)
• 20:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' (T356287)
• 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' (T356287)
• 20:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' (T356287)
• 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' (T356287)
• 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' (T356287)
• 20:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' (T356287)
• 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' (T356287)
• 20:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' (T356287)
• 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' (T356287)
• 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' (T356287)
• 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' (T356287)
• 19:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' (T356287)
• 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' (T356287)
• 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' (T356287)
• 19:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' (T356287)
• 19:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' (T356287)
• 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' (T356287)
• 19:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' (T356287)
• 19:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' (T356287)
• 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' (T356287)
• 19:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' (T356287)
• 19:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' (T356287)
• 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' (T356287)
• 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' (T356287)
• 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' (T356287)
• 19:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' (T356287)
• 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' (T356287)
• 19:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' (T356287)
• 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' (T356287)
• 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' (T356287)
• 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' (T356287)
• 19:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' (T356287)
• 18:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' (T356287)
• 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' (T356287)
• 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' (T356287)
• 18:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' (T356287)
• 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' (T356287)
• 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' (T356287)
• 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' (T356287)
• 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' (T356287)
• 18:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' (T356287)
• 18:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' (T356287)
• 18:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' (T356287)
• 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' (T356287)
• 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' (T356287)
• 18:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' (T356287)
• 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' (T356287)
• 18:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' (T356287)
• 18:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' (T356287)
• 18:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' (T356287)
• 18:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' (T356287)
• 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' (T356287)
• 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' (T356287)
• 18:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' (T356287)
• 18:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' (T356287)
• 18:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' (T356287)
• 17:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' (T356287)
• 17:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' (T356287)
• 17:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' (T356287)
• 17:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' (T356287)
• 17:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' (T356287)
• 17:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' (T356287)
• 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' (T356287)
• 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' (T356287)
• 17:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' (T356287)
• 17:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' (T356287)
• 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 17:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1006.eqiad.wmnet' (T356287)
• 17:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' (T356287)
• 17:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) on host 'cloudvirt1033' (T356287)
• 17:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033' (T356287)
• 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' (T356287)
• 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' (T356287)
• 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' (T356287)
• 17:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' (T356287)
• 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' (T356287)
• 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' (T356287)
• 17:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1005.eqiad.wmnet' (T356287)
• 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1005.eqiad.wmnet' (T356287)
• 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' (T356287)
• 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' (T356287)
• 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' (T356287)
• 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' (T356287)
• 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' (T356287)
• 16:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) (T356287)
• 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T356287)

2024-04-23

• 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2002-dev.codfw.wmnet'
• 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2002-dev.codfw.wmnet'

2024-04-18

• 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
• 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2008-dev.codfw.wmnet'
• 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudvirt2001-dev.codfw.wmnet'
• 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2008-dev.codfw.wmnet'

2024-04-17

• 14:12 dcaro: deleting dns leaks
• 01:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 01:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 01:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' (T356287)
• 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' (T356287)
• 01:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' (T356287)
• 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' (T356287)
• 01:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' (T356287)
• 01:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' (T356287)

2024-04-16

• 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' (T356287)
• 21:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' (T356287)
• 21:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2005-dev.codfw.wmnet' (T356287)
• 21:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2005-dev.codfw.wmnet' (T356287)
• 21:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' (T356287)
• 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' (T356287)
• 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2004-dev.codfw.wmnet' (T356287)
• 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' (T356287)
• 20:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2001-dev.codfw.wmnet' (T356287)
• 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2001-dev.codfw.wmnet' (T356287)
• 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2005-dev.codfw.wmnet' (T356287[A)
• 19:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2005-dev.codfw.wmnet' (T356287[A)
• 19:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices2004-dev.codfw.wmnet' (T356287[A)
• 19:20 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' (T356287[A)
• 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004-dev.codfw.wmnet' (T356287[A)
• 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004-dev.codfw.wmnet' (T356287[A)
• 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.codfw.wmnet' (T356287)
• 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.codfw.wmnet' (T356287)
• 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudservices2004.eqiad.wmnet' (T356287)
• 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices2004.eqiad.wmnet' (T356287)
• 19:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99) (T356287)
• 19:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T356287)

2024-04-15

• 11:19 taavi: update spicerack to 8.5.0 on cloudcumin2001

2024-04-10

• 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-04-09

• 18:13 andrewbogott: rebooting cloudinfra-cloudvps-puppetserver-1; unresponsive
• 13:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-04-05

• 14:37 taavi: run maintain-replica-indexes on all web replicas T361945
• 14:27 taavi: run maintain-replica-indexes on remaining analytics replicas T361945
• 14:17 taavi: run maintain-replica-indexes on clouddb1017 T361945

2024-04-04

• 18:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-04-03

• 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 15:01 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' (T319184)
• 14:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1040.eqiad.wmnet' (T319184)
• 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' (T319184)
• 11:34 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1039.eqiad.wmnet' (T319184)
• 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' (T319184)
• 10:25 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1038.eqiad.wmnet' (T319184)
• 10:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 10:21 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' (T319184)
• 09:17 taavi: manually delete prometheus-node-textfile-wmcs-dnsleaks.service and related files from cloudservices1005/6, leftovers of the designate api to cloudcontrol migration
• 09:09 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1037.eqiad.wmnet' (T319184)
• 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

2024-04-02

• 15:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=0)
• 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' (T319184)
• 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1036.eqiad.wmnet' (T319184)
• 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 12:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 12:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 12:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.depool_and_destroy (exit_code=99)
• 12:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.depool_and_destroy
• 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' (T319184)
• 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1035.eqiad.wmnet' (T319184)

2024-03-27

• 11:47 taavi: deleting about 2k stale puppet certs by running wmcs-puppetcertleaks in delete mode

2024-03-22

• 10:25 dcaro: adding back cloudcephosd1034 to the pool after doing the performance tests (T348643)

2024-03-21

• 22:07 andrewbogott: doing dist-upgrade on cloudcontrol nodes to get mariadb upgraded for T357133
• 22:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2004-dev.codfw.wmnet' (T357133)
• 22:04 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2004-dev.codfw.wmnet' (T357133)
• 11:08 dcaro: restarting nova-api on cloudcontrol1007

2024-03-20

• 17:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643)
• 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node (T348643)
• 15:27 dcaro: turning off cloudcephosd1030 to swap some disks (T348643)
• 15:25 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643)
• 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node (T348643)
• 15:00 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T348643)
• 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node (T348643)

2024-03-19

• 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643)
• 15:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node (T348643)
• 15:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T348643)
• 13:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.osd.drain_node (T348643)

2024-03-18

• 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-03-09

• 16:43 andrewbogott: restarted nova-api on cloudcontrol1006

2024-03-08

• 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 11:21 arturo: restarted nova-api in cloudcontrol1007, it was complaining about mysql broken pipe
• 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 10:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 10:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 10:35 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97)
• 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 10:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 10:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance

2024-03-07

• 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt2001-dev.codfw.wmnet'
• 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt2001-dev.codfw.wmnet'
• 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0)
• 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance

2024-03-06

• 17:46 dhinus: running "wmcs-dnsleaks --delete" to clean up 2 leaked records (tools-sgeweblight-10-32)
• 15:36 dcaro: renewing puppet ca cert for cloud-puppetmaster-03
• 15:24 dcaro: renewing puppet ca cert for cloudinfra-internal puppetmaster

2024-03-04

• 14:56 dhinus: delete project "loggerdiscordbot" in favor of new project "discordbots" T358337,T358427
• 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.ceph.reboot_node (exit_code=0) (T359049)
• 12:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.ceph.reboot_node (T359049)

2024-03-01

• 14:59 taavi: removing wmf-auto-restart-cron from all VMs without cron via cumin - https://gerrit.wikimedia.org/r/c/operations/puppet/+/1007328/ T358343
• 12:07 dcaro: restarted nova-api on cloudcontrol100* as it was very slow
• 12:04 dcaro: restarted nova-api on cloudcontrol1005 as it was very slow

2024-02-27

• 18:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-02-26

• 10:02 arturo: deleting nskaggs account from gerrit's wmcs-trusted group

2024-02-22

• 13:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 13:58 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' (T319184)
• 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' (T319184)
• 12:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1034.eqiad.wmnet' (T319184)
• 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' (T319184)
• 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-02-21

• 13:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T319184)
• 13:43 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T319184)
• 13:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T319184)
• 13:19 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T319184)
• 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' (T319184)
• 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' (T319184)
• 12:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' (T319184)
• 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' (T319184)

2024-02-20

• 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 11:45 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
• 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=0) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)
• 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)

2024-02-19

• 12:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)
• 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)
• 12:02 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)
• 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)
• 12:00 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)
• 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184)
• 10:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=0) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184)
• 09:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184)
• 09:49 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=99) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184)
• 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184)

2024-02-15

• 17:34 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.roll_reboot_cloudnets (T356975)
• 15:34 taavi: restart radosgw in eqiad as I am seeing 500 errors
• 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
• 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' (T319184)
• 11:58 dhinus: restore correct aggregate "localdisk" for cloudvirtlocal1001 and remove "maintenance"
• 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' (T319184)
• 05:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1044.eqiad.wmnet}'
• 05:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1045.eqiad.wmnet}'
• 05:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1044.eqiad.wmnet}'
• 05:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1047.eqiad.wmnet}'
• 05:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1046.eqiad.wmnet}'
• 04:52 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1047.eqiad.wmnet}'
• 04:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1046.eqiad.wmnet[B}'
• 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1046.eqiad.wmnet[B}'
• 04:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}'
• 04:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1048.eqiad.wmnet}'
• 04:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}'
• 04:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}'
• 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}'
• 04:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1048.eqiad.wmnet}'
• 04:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1051.eqiad.wmnet}'
• 04:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1050.eqiad.wmnet}'
• 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1051.eqiad.wmnet}'
• 04:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1050.eqiad.wmnet}'
• 04:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1052.eqiad.wmnet}'
• 04:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1053.eqiad.wmnet}'
• 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1052.eqiad.wmnet}'
• 03:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}'
• 03:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}'
• 03:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1053.eqiad.wmnet}'
• 03:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}'
• 03:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1055.eqiad.wmnet}'
• 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}'
• 03:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}'
• 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}'
• 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1055.eqiad.wmnet}'

2024-02-14

• 22:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1057.eqiad.wmnet}'
• 21:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1056.eqiad.wmnet}'
• 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1057.eqiad.wmnet}'
• 21:38 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1056.eqiad.wmnet}'
• 21:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1058.eqiad.wmnet}'
• 21:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1059.eqiad.wmnet}'
• 21:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1058.eqiad.wmnet}'
• 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1060.eqiad.wmnet}'
• 20:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1059.eqiad.wmnet}'
• 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1061.eqiad.wmnet}'
• 20:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1060.eqiad.wmnet}'
• 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1060.eqiad.wmnet}'
• 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1060.eqiad.wmnet}'
• 20:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1060.eqiad.wmnet}'
• 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1061.eqiad.wmnet}'
• 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1060.eqiad.wmnet}'
• 20:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1062.eqiad.wmnet}'
• 20:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1064.eqiad.wmnet}'
• 20:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1062.eqiad.wmnet}'
• 20:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1065.eqiad.wmnet}'
• 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1064.eqiad.wmnet}'
• 20:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1065.eqiad.wmnet}'
• 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1066.eqiad.wmnet}'
• 19:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1067.eqiad.wmnet}'
• 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1066.eqiad.wmnet}'
• 19:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1067.eqiad.wmnet}'
• 19:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1063.eqiad.wmnet}'
• 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1063.eqiad.wmnet}'
• 19:51 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'D{cloudvirtXXXX.eqiad.wmnet}'
• 19:51 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirtXXXX.eqiad.wmnet}'
• 19:49 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 19:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 19:49 wmbot~andrew@bullseye: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97)
• 19:49 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 19:48 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 19:45 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 19:45 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 19:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1063.eqiad.wmnet}'
• 19:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1063.eqiad.wmnet}'
• 19:38 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 19:38 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 19:36 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 19:36 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 19:34 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 19:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 19:34 wmbot~andrew@bullseye: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 19:34 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 19:32 wmbot~andrew@bullseye: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 17:13 wmbot~fran@wmf3169: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirtlocal1001.eqiad.wmnet}' (T356975)
• 17:12 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirtlocal1001.eqiad.wmnet}' (T356975)
• 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1043.eqiad.wmnet}'
• 16:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1043.eqiad.wmnet}'
• 16:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1043.eqiad.wmnet}'
• 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1043.eqiad.wmnet}'
• 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1042.eqiad.wmnet}'
• 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1042.eqiad.wmnet}'
• 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1041.eqiad.wmnet}'
• 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1040.eqiad.wmnet}'
• 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1040.eqiad.wmnet}'
• 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1039.eqiad.wmnet}'
• 14:09 taavi: creating some missing $PROJECT.wmcloud.org. DNS zones
• 14:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1039.eqiad.wmnet}'
• 14:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1038.eqiad.wmnet}'
• 13:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1038.eqiad.wmnet}'
• 13:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1037.eqiad.wmnet}'
• 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1037.eqiad.wmnet}'
• 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1036.eqiad.wmnet}'
• 13:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1036.eqiad.wmnet}'
• 13:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1035.eqiad.wmnet}'
• 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1035.eqiad.wmnet}'
• 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1034.eqiad.wmnet}'
• 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1034.eqiad.wmnet}'
• 12:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1033.eqiad.wmnet}'
• 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1033.eqiad.wmnet}'
• 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1032.eqiad.wmnet}'
• 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1032.eqiad.wmnet}'
• 11:21 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 10:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 10:15 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::codfw1dev::virt_ceph}'
• 09:31 taavi: failover all dumps traffic to clouddumps1001
• 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::codfw1dev::virt_ceph}'
• 08:16 taavi: reboot clouddumps1001 for kernel updates

2024-02-13

• 17:07 wmbot~taavi@runko: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=97) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 17:07 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 16:06 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 16:06 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 16:04 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 16:04 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 16:04 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 16:04 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::virt_ceph}'
• 16:02 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P{P:openstack::eqiad1::nova::compute::service}'
• 16:02 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P{P:openstack::eqiad1::nova::compute::service}'
• 16:02 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'P:openstack::eqiad1::nova::compute::service'
• 16:02 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'P:openstack::eqiad1::nova::compute::service'
• 16:01 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1001.eqiad.wmnet}'
• 16:01 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1001.eqiad.wmnet}'
• 15:10 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) (T356975)
• 14:55 wmbot~fran@wmf3169: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot (T356975)

2024-02-12

• 17:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0)
• 17:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
• 17:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0)
• 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
• 17:23 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99)
• 17:21 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudnet.reboot_node
• 17:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
• 17:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 17:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
• 16:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 16:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 16:36 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 16:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 16:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
• 16:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 16:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
• 16:14 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
• 15:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0)
• 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 15:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99)
• 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node
• 01:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 01:34 andrewbogott: resetting eqiad1 rabbitmq in hopes of resolving neutron double message warnings
• 01:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 00:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 00:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-02-11

• 21:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 21:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 20:57 andrewbogott: running wmcs.openstack.restart_openstack for all eqiad1 services
• 20:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 20:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 11:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-02-08

• 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 13:43 taavi: deploy change to exclude cloud-private networks from general egress NAT https://phabricator.wikimedia.org/T356850

2024-02-06

• 20:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 20:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 02:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 02:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 02:09 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
• 02:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-02-05

• 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 20:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 20:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 20:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:55 andrewbogott: rebuilt bookworm base image in eqiad1 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/992677

2024-02-02

• 13:54 arturo: [codfw1dev] cleanup /etc/network/interfaces on cloudlb2003-dev from puppet leftovers

2024-02-01

• 10:55 taavi: invite aborrero to /repos/cloud, /toolforge-repos, /cloudvps-repos on gitlab

2024-01-29

• 12:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-01-26

• 09:01 taavi: joining cloudrabbit1001/2 to the cluster on 1003 T345610

2024-01-25

• 16:48 andrewbogott: taavi just moved all rabbitmq traffic to cloudrabbit1003 as part of T345610
• 16:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 13:27 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
• 13:27 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-2.tools.eqiad1.wikimedia.cloud to the cluster
• 13:15 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
• 13:15 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-1.tools.eqiad1.wikimedia.cloud to the cluster
• 12:48 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
• 12:48 wmbot~taavi@runko: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-1.toolsbeta.eqiad1.wikimedia.cloud to the cluster

2024-01-24

• 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
• 11:37 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster

2024-01-22

• 16:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 15:56 andrewbogott: restarting openstack sevices on eqiad1 to clean up from the mariadb restarts
• 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:07 andrewbogott: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/992192 and resetting galera cluster in eqiad1

2024-01-21

• 05:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 05:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-01-19

• 23:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 23:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-01-18

• 20:10 taavi: mysql:labsdbaccounts@m5-master.eqiad.wmnet [labsdbaccounts]> update account_host set status = 'absent' where id = 137613;
• 12:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 12:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-101.tools.eqiad1.wikimedia.cloud to the cluster

2024-01-17

• 15:17 andrewbogott: "systemctl restart mariadb@s4.service mariadb@s6.service" on clouddb1015. System is in danger of oom and there are no obvious long queries running

2024-01-16

• 14:25 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 14:25 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 14:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
• 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
• 14:23 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
• 14:23 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
• 13:55 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 13:55 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 13:54 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 13:54 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 13:53 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99)
• 13:53 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 13:52 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0)
• 13:52 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance
• 11:35 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) for cloudvirt1060.eqiad.wmnet
• 11:34 wmbot~taavi@runko: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot for cloudvirt1060.eqiad.wmnet
• 11:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) (T355061)
• 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot (T355061)
• 09:41 taavi: drop dbproxy1018/9 grants from all clouddb hosts T346947
• 09:24 taavi: move cloudvirt2004-dev from 'failed' to 'active' in netbox - seems like that was for T348531 which is now resolved

2024-01-15

• 14:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) (T355061)
• 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot (T355061)
• 14:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) (T355061)
• 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot (T355061)
• 12:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2024-01-11

• 10:51 dcaro: restarting striker.service on cloudweb1003 as it seems non-responsive

2024-01-10

• 17:43 bd808: Blocking Developer accounts connected to invalid/legacy wikimedia.org email addresses (T218239)
• 16:48 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0) (T346631)
• 16:48 wmbot~fran@wmf3169: test message3 from local cookbook (T346631)
• 16:48 wmbot~fran@wmf3169: START - Cookbook wmcs.do_log_msg (T346631)

2024-01-09

• 17:50 wmbot~fran@wmf3169: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0) (T346631)
• 17:49 wmbot~fran@wmf3169: test message2 from local cookbook (T346631)
• 17:49 wmbot~fran@wmf3169: START - Cookbook wmcs.do_log_msg (T346631)
• 17:43 wmbot~fran@wmf3169: %(message)s (T346631)
• 17:43 wmbot~fran@wmf3169: %(message)s (T346631)
• 17:42 wmbot~fran@wmf3169: %(message)s (T346631)

2024-01-08

• 15:52 taavi: verify wmcloud.org, wmflabs.org and toolforge.org in gmail postmaster console to figure out how much google likes us (T354112)

2024-01-07

• 19:34 andrewbogott: removed cloudvirt1063 from 'ceph' aggregate, added to 'maintenance' aggregate T353408
• 19:34 andrewbogott: evacuating all VMs from cloudvirt1063. T353408

2024-01-02

• 16:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) (T353408)
• 16:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T353408)
• 10:22 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
• 10:22 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.vm_console

2023-12-31

• 21:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:35 andrewbogott: running openstack service restart cookbook in eqiad1 in response to a bunch of service down alerts
• 21:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-12-21

• 16:52 dhinus: puppet node deactivate cloudvirt1063.eqiad.wmnet T353406
• 03:01 andrewbogott: restarting mariadb on cloudcontrol1005, hoping to get Galera back in sync

2023-12-20

• 19:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-12-18

• 17:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 17:23 andrewbogott: restarting all eqiad1 openstack services after a rabbitmq upgrade/rebuild for T353646
• 15:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 15:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-12-15

• 13:13 dcaro: restarted nova-fullstack on codfw as it was stuck (and alerting through stale prometheus file)

2023-12-14

• 00:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99)
• 00:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.set_maintenance
• 00:25 andrewbogott: evacuating hosts from cloudvirt1063 and depooling. T353406

2023-12-13

• 16:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=99)

2023-12-12

• 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 20:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 17:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 17:45 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-100.tools.eqiad1.wikimedia.cloud to the cluster
• 16:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 16:11 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-99.tools.eqiad1.wikimedia.cloud to the cluster
• 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 15:49 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-98.tools.eqiad1.wikimedia.cloud to the cluster

2023-12-10

• 18:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:27 andrewbogott: restarting all openstack API servers, hoping to make things a bit more responsive
• 18:25 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-12-08

• 12:00 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud (T353055)
• 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud (T353055)
• 11:58 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97) on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
• 11:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud
• 09:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) (T345084)
• 09:32 dcaro: restarting nova and keystone as they are getting too slow (T345084)
• 09:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack (T345084)
• 09:32 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) (T345084)
• 09:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack (T345084)

2023-12-07

• 13:12 dcaro: rebooting cloudcephosd1001 to make sure puppet7 migration went ok

2023-12-04

• 00:08 andrewbogott: rebooting cloudcontrol1006 to recover from full disk error

2023-12-03

• 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-12-02

• 12:27 taavi: powercycle cloudvirt1063 T352595
• 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 11:28 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-97.tools.eqiad1.wikimedia.cloud to the cluster
• 11:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 11:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-96.tools.eqiad1.wikimedia.cloud to the cluster
• 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 10:50 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-95.tools.eqiad1.wikimedia.cloud to the cluster
• 00:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 00:21 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-94.tools.eqiad1.wikimedia.cloud to the cluster
• 00:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 00:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-93.tools.eqiad1.wikimedia.cloud to the cluster
• 00:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 00:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-92.tools.eqiad1.wikimedia.cloud to the cluster
• 00:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 00:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-91.tools.eqiad1.wikimedia.cloud to the cluster
• 00:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 00:05 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-90.tools.eqiad1.wikimedia.cloud to the cluster
• 00:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
• 00:01 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-89.tools.eqiad1.wikimedia.cloud to the cluster

2023-12-01

• 17:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 17:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:19 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T351171)
• 16:19 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T351171)
• 15:49 andrewbogott: reimaging cloudcontrol1005 due to widespread misbehavior
• 14:24 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T351171)
• 14:20 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T351171)
• 14:19 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97) (T351171)
• 14:18 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T351171)
• 13:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 11:55 taavi: restart neutron-rpc-server.service on eqiad1 cloudcontrols

2023-11-30

• 20:44 andrewbogott: generating to application credentials for the tests that run on tf-infra-test
• 19:54 andrewbogott: reimaged cloudrabbit100[23] after https://gerrit.wikimedia.org/r/c/operations/puppet/+/979127. I didn't reimage 1001 because that will require rebuilding the whole cluster but I did remove the related packages.
• 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet'
• 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet'
• 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet'
• 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet'
• 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet'
• 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet'
• 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet'
• 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet'
• 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet'
• 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet'
• 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet'
• 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet'
• 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet'
• 17:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' (T348843)
• 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet'
• 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet'
• 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet'
• 17:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet'
• 17:47 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' (T348843)
• 17:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' (T348843)
• 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet'
• 17:42 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' (T348843)
• 17:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' (T348843)
• 17:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' (T348843)
• 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' (T348843)
• 17:30 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' (T348843)
• 17:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' (T348843)
• 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' (T348843)
• 17:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' (T348843)
• 17:19 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' (T348843)
• 17:19 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' (T348843)
• 17:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' (T348843)
• 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' (T348843)
• 17:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' (T348843)
• 17:09 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' (T348843)
• 17:04 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' (T348843)
• 17:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' (T348843)
• 16:59 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' (T348843)
• 16:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' (T348843)
• 16:54 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' (T348843)
• 16:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' (T348843)
• 16:49 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' (T348843)
• 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1049.eqiad.wmnet' (T348843)
• 16:39 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1049.eqiad.wmnet' (T348843)
• 16:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' (T348843)
• 16:34 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' (T348843)
• 16:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' (T348843)
• 16:29 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' (T348843)
• 16:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' (T348843)
• 16:15 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' (T348843)
• 16:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' (T348843)
• 16:10 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' (T348843)
• 16:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' (T348843)
• 16:05 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' (T348843)
• 16:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' (T348843)
• 16:00 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' (T348843)
• 15:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' (T348843)
• 15:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' (T348843)
• 15:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' (T348843)
• 15:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' (T348843)
• 15:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1053.eqiad.wmnet' (T348843)
• 15:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' (T348843)
• 15:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' (T348843)
• 15:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' (T348843)
• 15:37 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' (T348843)
• 15:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' (T348843)
• 15:31 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1050.eqiad.wmnet' (T348843)
• 15:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' (T348843)
• 15:22 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' (T348843)
• 15:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' (T348843)
• 15:17 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' (T348843)
• 15:17 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' (T348843)
• 15:13 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' (T348843)
• 15:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' (T348843)
• 15:07 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' (T348843)
• 15:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' (T348843)
• 15:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' (T348843)
• 15:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' (T348843)
• 14:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' (T348843)
• 14:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' (T348843)
• 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' (T348843)
• 14:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' (T348843)
• 14:38 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' (T348843)
• 14:16 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1006.eqiad.wmnet' (T348843)
• 14:03 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet1005.eqiad.wmnet' (T348843)
• 13:55 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet1005.eqiad.wmnet' (T348843)
• 13:43 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=0) (T348843)
• 13:43 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudweb.set_maintenance (T348843)
• 12:18 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1005.eqiad.wmnet' (T348843)
• 12:04 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1005.eqiad.wmnet' (T348843)
• 11:59 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' (T348843)
• 11:45 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' (T348843)
• 11:44 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' (T348843)
• 11:28 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' (T348843)

2023-11-29

• 15:27 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudservices1006.eqiad.wmnet' (T348843)
• 15:15 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudservices1006.eqiad.wmnet' (T348843)
• 14:59 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) (T348843)
• 14:50 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T348843)

2023-11-28

• 14:18 taavi: moving wiki replica DNS to use cloudlbs instead of the old proxy VMs T346947

2023-11-27

• 19:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 19:35 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-11-24

• 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 12:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 12:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 11:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-11-22

• 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 13:00 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-11-21

• 10:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-11-20

• 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-11-17

• 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-11-16

• 12:09 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 12:05 taavi@cloudcumin2001: START - Cookbook wmcs.openstack.restart_openstack
• 11:23 dhinus: upgraded spicerack from 8.0.2 to 8.0.3 on cloudcumins
• 05:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 05:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-11-15

• 09:50 taavi: move cloudlb hosts to use the nftables firewall backend

2023-11-14

• 21:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' (T345811)
• 20:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' (T345811)
• 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056.eqiad.wmnet' (T345811)
• 20:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' (T345811)
• 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055.eqiad.wmnet' (T345811)
• 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 20:09 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 20:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 20:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 19:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054.eqiad.wmnet' (T345811)
• 19:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 19:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 19:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 19:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1053.eqiad.wmnet' (T345811)
• 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' (T345811)
• 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' (T345811)
• 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1052.eqiad.wmnet' (T345811)
• 18:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1050.eqiad.wmnet' (T345811)
• 18:31 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1049.eqiad.wmnet' (T345811)
• 18:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' (T345811)
• 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049.eqiad.wmnet' (T345811)
• 18:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' (T345811)
• 18:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' (T345811)
• 18:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047.eqiad.wmnet' (T345811)
• 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048.eqiad.wmnet' (T345811)
• 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' (T345811)
• 17:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047.eqiad.wmnet' (T345811)
• 17:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' (T345811)
• 12:03 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' (T345811)
• 11:44 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1046.eqiad.wmnet' (T345811)
• 10:10 taavi: restart kiwix-mirror-update on clouddumps1001
• 05:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) (T345811)
• 05:02 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 04:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 04:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) (T345811)
• 04:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 04:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) (T345811)
• 03:50 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 03:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) (T345811)
• 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 03:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
• 03:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) (T345811)
• 03:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) (T345811)
• 02:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
• 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 02:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 02:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) (T345811)
• 02:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 02:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 02:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 02:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 02:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 02:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) (T345811)
• 02:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 02:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)

2023-11-13

• 22:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) (T345811)
• 22:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 22:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 22:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) (T345811)
• 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) (T345811)
• 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 21:30 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 21:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) (T345811)
• 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
• 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 21:00 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 19:31 andrewbogott: rebooting cloudcontrol2005-dev, trying to fix general misbehavior
• 19:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
• 19:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) (T345811)
• 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 19:01 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)
• 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
• 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 18:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 18:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 18:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 18:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1033.eqiad.wmnet'
• 18:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1032.eqiad.wmnet'
• 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet'
• 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1032.eqiad.wmnet'
• 17:09 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T345811)
• 17:09 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T345811)
• 16:57 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) (T345811)
• 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T345811)
• 15:37 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' (T345811)
• 15:19 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' (T345811)
• 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 09:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-11-11

• 02:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 02:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 02:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 02:14 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 02:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
• 01:58 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 01:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
• 01:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 01:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
• 01:05 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 01:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0)
• 00:48 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 00:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 00:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 00:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 00:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 00:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 00:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 00:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 00:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 00:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1058'
• 00:39 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1058'
• 00:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 00:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.safe_reboot

2023-11-09

• 21:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:54 wm-bot2: fran@wmf3169 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1025.eqiad.wmnet' (T345811)
• 14:52 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1025.eqiad.wmnet' (T345811)
• 14:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) (T345811)
• 14:50 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345811)

2023-11-08

• 16:43 andrewbogott: created foundationmemory project for T350760

2023-11-06

• 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 20:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:27 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 18:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 18:19 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
• 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 15:43 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
• 15:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)

2023-11-05

• 00:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

2023-11-04

• 23:05 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 22:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 20:12 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 16:00 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 13:13 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 04:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 01:56 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)

2023-11-03

• 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
• 16:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 16:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 13:13 dhinus: triggering neutron failover from cloudnet1005 to cloudnet1006 (T345811)
• 06:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 01:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 01:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
• 01:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 01:45 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)

2023-11-02

• 20:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 17:32 taavi: merged cloudcontrol firewall cleanup patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/971211
• 16:46 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
• 16:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
• 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 16:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 13:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 13:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 11:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 11:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 07:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 07:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 05:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 03:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 03:47 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
• 03:44 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)

2023-11-01

• 23:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
• 21:29 taavi: re-enable puppet on cloudcontrol2006-dev.codfw.wmnet which has fallen off of puppetdb
• 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 21:15 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 20:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
• 19:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
• 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 14:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
• 14:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 14:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
• 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
• 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
• 09:04 taavi: reset local cookbook changes on cloudcumin1001 which were causing issues with puppet runs
• 09:02 taavi: restart nova-fullstack which had had some issues after yesterday's cloudcontrol1007 reimage
• 02:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 00:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

2023-10-31

• 23:46 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 18:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 14:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 14:11 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
• 14:10 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 10:16 dhinus: upgrading mariadb-server in cloudcontrol1005 (T345811)
• 10:04 dhinus: upgrading mariadb-server in cloudcontrol1006 (T345811)
• 09:51 dhinus: upgrading mariadb-server in cloudcontrol1007, second attempt (T345811)
• 06:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
• 05:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 02:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0) (348643)
• 02:27 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (348643)
• 02:27 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (348643)
• 02:26 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 01:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 01:59 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)
• 01:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (348643)

2023-10-30

• 17:40 andrewbogott: rebooting tools-db-1.tools.eqiad1.wikimedia.cloud for yet another oom death
• 17:09 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
• 16:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (348643)
• 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 16:55 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.drain_node (348643)
• 16:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:53 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:39 dhinus: upgrading mariadb-server in cloudcontrol1007 (T345811)
• 16:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)
• 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (T348643)
• 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)
• 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (T348643)
• 16:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)
• 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (T348643)
• 16:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)
• 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (T348643)
• 16:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)
• 16:30 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (T348643)
• 16:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) (T348643)
• 16:29 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node (T348643)
• 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node
• 16:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 16:17 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.osd.undrain_node

2023-10-28

• 07:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)

2023-10-27

• 15:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 15:38 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 15:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 15:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 15:29 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 15:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 15:27 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 15:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 13:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 13:09 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
• 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 12:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 12:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 09:05 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 09:04 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

2023-10-26

• 14:02 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
• 09:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

2023-10-25

• 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster
• 11:00 taavi: update cloudcumins to spicerack 8.x
• 10:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster

2023-10-24

• 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-10-23

• 16:48 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 16:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 15:27 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 15:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:06 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project catalyst in eqiad1
• 15:06 wm-bot2: fran@wmf3169 START - Cookbook wmcs.vps.create_project for project catalyst in eqiad1
• 10:36 taavi: merged change https://gerrit.wikimedia.org/r/c/operations/puppet/+/966494 which touches the pdns web server config

2023-10-20

• 15:17 dcaro: upgraded cloudcephosd1004 to v15 (T349363)
• 14:25 dcaro: upgraded cloudcephosd1003 to v15 (T349363)
• 13:20 dcaro: upgraded cloudcephosd1002 to v15 (T349363)
• 10:33 dcaro: upgraded cloudcephosd1001 to v15 (T349363)
• 08:26 dcaro: codfw ceph enabled diskprediction_local module, will take a bit to populate/start getting predictions (T348716)

2023-10-17

• 15:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) (T349109)
• 15:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T349109)

2023-10-16

• 03:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 03:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-10-13

• 23:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 23:10 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 19:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 19:24 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 17:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:42 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 08:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) (T341285)
• 08:31 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)
• 08:30 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) (T341285)
• 08:20 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)

2023-10-12

• 17:16 wm-bot2: fran@wmf3169 END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97) (T341285)
• 17:16 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)

2023-10-11

• 10:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) (T341285)
• 10:36 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (T341285)
• 10:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 07:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 06:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node

2023-10-10

• 17:25 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 14:50 dcaro: removing ~100 dangling backup snapshots from eqiad1-compute ceph pool
• 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) (T341285)
• 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (T341285)
• 12:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 11:38 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) (T341285)
• 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (T341285)
• 11:33 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99)
• 11:32 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.safe_reboot
• 11:00 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=99) (T341285)
• 11:00 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (T341285)
• 10:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) (T341285)
• 10:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (T341285)
• 10:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) (T341285)
• 09:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (T341285)
• 09:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) (T341285)
• 09:43 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (T341285)
• 08:19 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 08:18 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 08:18 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 08:17 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 08:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 08:17 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 08:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 08:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 08:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node

2023-10-09

• 17:16 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 16:26 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) (T341285)
• 16:18 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)
• 15:49 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) (T341285)
• 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)
• 15:26 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) (T341285)
• 13:55 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) (T341285)
• 13:45 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)
• 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 13:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 13:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 13:04 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 09:10 dcaro: undrained cephosd1011
• 09:09 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 09:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 09:06 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 09:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 09:05 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 09:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 07:35 taavi: restart postgresql on cloudbackup2001 T348431

2023-10-05

• 16:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) (T341285)
• 16:29 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)
• 16:24 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) (T341285)
• 16:11 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)
• 15:30 arturo: operating on cloudgw @ eqiad1 (T347469)
• 14:07 wm-bot2: dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.drain_rack (exit_code=99)
• 12:55 arturo: doing cloudgw maintenance operations T347469
• 11:54 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_rack
• 10:57 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 09:54 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) (T341285)
• 09:52 arturo: [codfw1dev] aborrero@cloudcontrol2001-dev:~ $ sudo keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
• 09:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)
• 09:40 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) (T341285)
• 09:33 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)
• 08:50 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 07:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 07:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 00:04 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)

2023-10-04

• 20:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 15:55 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 14:54 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) (T341285)
• 14:44 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)
• 14:41 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) (T341285)
• 14:40 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (T341285)
• 13:50 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) (T341285)
• 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 12:08 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 07:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 07:21 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 07:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 07:12 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 07:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node

2023-10-03

• 19:38 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 12:35 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 12:32 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 12:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 12:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 11:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
• 08:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 08:43 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0)
• 08:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 08:39 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 08:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 08:23 dcaro: set .rgw.root pool on eqiad as rgw app (`ceph osd pool application enable .rgw.root rgw`)

2023-10-02

• 17:39 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 16:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 16:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 14:30 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 13:58 taavi: cloudcontrol1005,7: `sudo systemctl reset-failed keystone_sync_keys_from_cloudcontrol1006.eqiad.wmnet.service`
• 13:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 13:40 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99)
• 13:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node
• 13:37 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T316544)
• 12:26 arturo: [codfw1dev] run `update domains set master = '185.15.57.25:5354 185.15.57.26:5354 172.20.5.9:5354 172.20.5.8:5354';` in cloudservies2005-dev
• 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node (T316544)
• 11:55 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T316544)
• 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node (T316544)
• 11:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T316544)
• 08:44 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.drain_node (T316544)

2023-09-29

• 12:43 taavi: taavi@cloudcontrol1005 ~ $ os subnet set a69bdfad-d7d2-4cfa-8231-3d6d3e0074c9 --no-dns-nameservers --dns-nameserver 172.20.255.1
• 08:36 taavi: start script to fix networking on broken bullseye instances T347665

2023-09-28

• 20:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 20:26 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 18:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 18:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 12:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=0)
• 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 12:01 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99)
• 12:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.osd.undrain_node
• 09:48 arturo: rebooting cloudgw1001/1002 for sysctl and kernel upgrades

2023-09-27

• 12:07 arturo: merging cloudgw firewall changes https://gerrit.wikimedia.org/r/c/operations/puppet/+/961360
• 09:36 taavi: move maintain-dbusers to cloudcontrol1005
• 01:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 01:29 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-09-26

• 17:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 17:06 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-09-24

• 15:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:35 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-09-23

• 08:40 taavi: restart keystone

2023-09-22

• 14:03 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
• 14:02 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 14:02 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
• 14:01 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 14:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.undrain_node (exit_code=0)
• 14:00 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
• 14:00 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
• 13:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 13:56 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
• 13:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 13:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.undrain_node (exit_code=0)
• 13:52 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
• 13:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.undrain_node (exit_code=99)
• 13:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
• 13:48 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.undrain_node (exit_code=99)
• 13:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.undrain_node
• 13:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
• 13:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 13:46 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
• 13:44 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 13:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
• 13:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 13:41 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
• 13:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.drain_node (exit_code=0)
• 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 12:37 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
• 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 12:33 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
• 12:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 12:33 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
• 12:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node
• 11:43 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.ceph.drain_node (exit_code=99)
• 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.drain_node

2023-09-21

• 16:35 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-3.tools.eqiad1.wikimedia.cloud
• 16:34 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-3.tools.eqiad1.wikimedia.cloud
• 02:12 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.restart_openstack (exit_code=97)
• 02:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-09-20

• 21:49 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:45 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 21:38 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 21:35 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 21:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 21:31 root@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 11:12 arturo: moving openstack API endpoint to cloudlb (T346439)
• 10:28 arturo: running SQL command `update domains set master="172.20.1.5:5354 172.20.2.4:5354 185.15.56.162:5354 185.15.56.163:5354";` on cloudservices1005/1006 (T346042)
• 10:06 arturo: running SQL command `update domains set master="185.15.56.162:5354 185.15.56.163:5354"` on cloudservices1005/1005 (T346042)

2023-09-19

• 18:54 andrewbogott: depooling clouddb1019 to let it recover from high memory use. T346826
• 15:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-09-18

• 11:53 taavi: update designate urls in Keystone to point to openstack-next, until cloudlb is serving the main openstack address T346042
• 11:40 arturo: decomission cloudservices1005 T346042 in preparation for re-racking
• 08:45 arturo: hardcode `185.15.56.161 openstack.eqiad1.wikimediacloud.org` in /etc/hosts in cloudcontrol1005 for T346441

2023-09-15

• 11:43 arturo: merging NAT change for T346426 in cloudgw
• 10:33 arturo: faiolver cloudgw1001 into cloudgw1002, investigating a nftables syntax error (T346432)

2023-09-14

• 21:25 wm-bot2: andrew@bullseye END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 21:25 wm-bot2: andrew@bullseye START - Cookbook wmcs.openstack.cloudvirt.drain
• 21:23 wm-bot2: andrew@bullseye END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 21:23 wm-bot2: andrew@bullseye START - Cookbook wmcs.openstack.cloudvirt.drain
• 21:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99)
• 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain
• 17:13 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) (T345810)
• 17:06 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain (T345810)
• 17:01 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) (T345810)
• 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain (T345810)
• 16:51 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) (T345810)
• 16:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain (T345810)
• 16:29 wm-bot2: fran@wmf3169 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) (T345810)
• 16:29 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.drain (T345810)
• 16:25 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) (T345810)
• 16:25 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.drain (T345810)
• 16:12 topranks: DNS operation: remove old DNS entry for ns0.openstack.eqiad1.wikimediacloud.org. on wikimedia authdns (was pointing to 208.80.154.148)
• 16:10 topranks: DNS operation: add new DNS entry for ns0.openstack.eqiad1.wikimediacloud.org. on wikimedia authdns pointing to 185.15.56.162
• 14:42 arturo: DNS operation: route 208.80.154.148 to cloudservices1006 in anticipation of cloudservices1005 decom (T346042)
• 12:11 arturo: enable puppet on cloudservices1006 to drop local NAT hacks and enable new DNS auth IP address (T346042)

2023-09-13

• 17:11 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=0) (T345811)
• 17:08 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node (T345811)
• 17:04 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) (T345811)
• 17:04 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node (T345811)
• 16:57 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) (T345811)
• 16:57 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudnet.reboot_node (T345811)
• 16:56 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) (T345811)
• 16:56 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node (T345811)
• 16:53 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) (T345811)
• 16:53 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node (T345811)
• 16:49 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) (T345811)
• 16:49 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudnet.reboot_node (T345811)
• 16:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudnet.reboot_node (exit_code=99) (T345811)
• 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudnet.reboot_node (T345811)
• 01:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 01:57 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 01:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 01:55 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 01:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 01:54 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-09-12

• 18:06 andrewbogott: update domains set master='185.15.56.163:5354 208.80.154.11:5354 10.64.151.4:5354'; on cloudservices1005 + cloudservices1006
• 17:59 andrewbogott: mysql:root@localhost [pdns]> update domains set master='185.15.56.163:5354 208.80.154.11:5354';
• 17:59 andrewbogott: "designate-manage pool update' on cloudservices1005 to remove cloudservices1004 from the pool
• 17:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 15:12 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 15:07 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-09-11

• 12:36 arturo: update DNS resolver cloud-wide to use 172.20.255.1 (T342621)

2023-09-10

• 02:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 02:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 02:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99)
• 02:41 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-09-07

• 10:09 dhinus: reimaging cloudcontrol2001-dev to bookworm (T345810)

2023-09-06

• 16:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97)
• 16:56 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
• 16:53 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
• 16:52 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
• 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
• 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
• 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
• 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
• 16:51 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
• 16:51 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
• 16:37 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
• 16:37 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
• 16:37 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
• 16:35 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
• 16:35 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
• 16:34 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99)
• 16:34 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
• 16:23 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=97)
• 16:23 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node
• 16:13 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.cloudweb.set_maintenance (exit_code=99)
• 16:12 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.set_maintenance

2023-09-05

• 12:34 arturo: synced pdns database from cloudservices1004 to cloudservices1006 (T345240)
• 12:18 arturo: updating pools.yaml in all cloudservices designate nodes (T345240)
• 10:54 arturo: running SQL command `update domains set master="208.80.154.11:5354 208.80.154.148:5354 10.64.151.4:5354";` on all 3 cloudservices nodes (T345240)

2023-09-04

• 15:58 arturo: stop and mask designate-sink.service @ cloudservices1006
• 14:19 arturo: started all designate services on cloudservices1006 T345240
• 10:46 arturo: added designate galera DB grants for cloudlb T345240
• 08:40 arturo: stopped all designate services on cloudservices1006 T345240

2023-09-01

• 10:28 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.network.tests (exit_code=1) (T345282)
• 10:25 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.network.tests (T345282)

2023-08-31

• 15:14 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=99)
• 15:14 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status
• 12:49 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudnet.show (exit_code=0)
• 12:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudnet.show
• 12:48 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.unset_cluster_maintenance (exit_code=0)
• 12:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.unset_cluster_maintenance
• 12:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
• 12:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.ceph.set_cluster_in_maintenance
• 12:46 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.ceph.set_cluster_in_maintenance (exit_code=0)
• 12:46 wm-bot2: fran@wmf3169 START - Cookbook wmcs.ceph.set_cluster_in_maintenance

2023-08-30

• 13:07 wm-bot2: dcaro testing stuff

2023-08-28

• 15:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 15:05 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:05 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:04 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:04 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:04 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:04 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:04 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:04 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:04 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:04 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:04 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:03 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:02 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:01 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:01 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:01 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] (T345084) - cookbook ran by root@cloudcumin1001
• 15:01 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-08-23

• 16:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 16:10 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
• 16:10 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
• 16:10 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 16:10 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 16:09 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
• 16:09 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
• 16:09 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 16:09 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 16:09 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 16:09 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 16:08 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 16:08 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-08-15

• 19:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 19:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 19:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 19:32 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 19:32 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 19:32 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 16:01 andrewbogott: rebooting cloudvirt2001-dev in an attempt to figure out what's happening with bastions
• 15:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 15:42 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
• 15:42 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
• 15:42 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 15:42 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 15:42 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
• 15:42 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
• 15:41 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 15:41 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 15:41 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 15:40 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 15:40 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 15:40 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 15:40 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack
• 12:39 dcaro: removed some logs from the cloudmetrics1003:/var/log/carbon/ directory and stopped the carbon processes (they were crashing and filling up the disk with logs)

2023-08-14

• 22:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0)
• 22:11 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by root@cloudcumin1001
• 22:11 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by root@cloudcumin1001
• 22:11 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 22:11 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 22:10 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
• 22:10 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by root@cloudcumin1001
• 22:10 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 22:09 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 22:09 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 22:09 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by root@cloudcumin1001
• 22:09 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 22:08 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by root@cloudcumin1001
• 22:08 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.restart_openstack

2023-08-07

• 09:39 taavi: cloud vps graphite service was disabled: https://wikitech.wikimedia.org/wiki/News/2023_Cloud_VPS_metrics_changes T326266

2023-08-03

• 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0)
• 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.do_log_msg
• 13:07 wm-bot2: Test SAL log (T341793) - cookbook ran by root@cloudcumin1001

2023-07-31

• 16:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.do_log_msg (exit_code=0)
• 16:16 wm-bot2: Test SAL log (T325756) - cookbook ran by root@cloudcumin1001
• 16:15 fnegri@cloudcumin1001: START - Cookbook wmcs.do_log_msg
• 15:33 wm-bot2: Test SAL log (T325756) - cookbook ran by root@cloudcumin1001
• 14:42 wm-bot2: Test SAL log (T325756) - cookbook ran by root@cloudcumin1001
• 14:35 wm-bot2: Test SAL log (T325756) - cookbook ran by root@cloudcumin1001
• 13:55 andrewbogott: recreating the codfw1dev galera cluster according to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Galera -- mariadb is stopped (and won't start) on all three cloudcontrol nodes
• 13:39 wm-bot2: Test SAL log (T325756) - cookbook ran by root@cloudcumin1001

2023-07-27

• 10:52 arturo: adding cloud-private subnet to cloudnet1005/1006 hosts in eqiad1 (T342619)

2023-07-19

• 14:02 wm-bot2: Draining cloudvirt2002-dev.codfw.wmnet (T335840) - cookbook ran by raymond@ubuntu
• 14:02 wm-bot2: Safe rebooting cloudvirt2002-dev.codfw.wmnet (T335840) - cookbook ran by raymond@ubuntu

2023-07-17

• 15:55 arturo: cloudcontrol1005 was shutdown earlier today (T341495)
• 15:35 arturo: [codfw1dev] cloudweb2002-dev up and running after reracking (T327919)
• 15:11 arturo: [codfw1dev] powered off cloudweb2002-dev for reracking (T327919)
• 12:45 taavi: removing diamond from remaining buster instances T317032

2023-07-13

• 10:48 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by dcaro@urcuchillay
• 10:48 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by dcaro@urcuchillay
• 10:48 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by dcaro@urcuchillay
• 10:48 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by dcaro@urcuchillay
• 10:48 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by dcaro@urcuchillay
• 10:47 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by dcaro@urcuchillay
• 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:47 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:47 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:47 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:47 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:47 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:47 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:46 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:46 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:46 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:46 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:46 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:46 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:46 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:46 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:46 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:45 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
• 10:45 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
• 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:45 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:45 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:44 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:44 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:44 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:44 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:44 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:44 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:44 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:44 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:44 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:44 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:43 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:43 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:43 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:43 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:43 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:43 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:42 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by dcaro@urcuchillay
• 10:42 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:42 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:42 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:42 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:42 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:42 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:38 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay
• 10:33 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by dcaro@urcuchillay

2023-07-07

• 10:28 taavi: backfilling {project}.wmcloud.org and other currently-named DNS zones to projects that don't have them

2023-07-06

• 17:02 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 17:01 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 17:01 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 17:01 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:01 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:01 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:00 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 17:00 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

2023-07-04

• 14:08 wm-bot2: Test SAL log (T325756) - cookbook ran by root@cloudcumin1001
• 14:07 wm-bot2: Test SAL log (T325756) - cookbook ran by root@cloudcumin1001

2023-06-30

• 21:37 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 21:35 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 21:35 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:35 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:35 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:35 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 21:34 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

2023-06-29

• 19:49 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 19:49 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 19:49 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:49 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:49 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 19:49 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 19:48 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 19:48 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:48 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:48 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:48 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 19:47 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 19:34 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 19:34 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 19:34 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:34 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:34 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 19:33 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
• 19:33 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 19:33 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
• 19:33 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
• 19:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:30 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 19:30 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

2023-06-26

• 08:46 arturo: [codfw1dev] manually start mariadb service @ cloudinfra-db-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud

2023-06-24

• 19:21 wm-bot2: Created new flavor: g3.cores16.ram34.disk20 (id:7dd33202-32c3-4bc7-b2d4-10c2ebe7e5c5) - cookbook ran by andrew@bullseye
• 19:20 wm-bot2: Created new flavor: g3.cores16.ram34816.disk20.admin (id:d76925a8-b58b-489e-a3d9-c68c7744b551) - cookbook ran by andrew@bullseye

2023-06-23

• 16:45 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 16:45 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
• 16:45 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
• 16:45 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
• 16:45 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 16:45 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:44 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:43 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:43 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:43 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:43 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:43 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:43 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:41 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:40 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:40 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:40 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:40 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:40 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:40 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:40 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 13:59 andrewbogott: rebooting every VM in codfw1dev

2023-06-13

• 15:35 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 15:35 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
• 15:35 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
• 15:35 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
• 15:35 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 15:35 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:35 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:35 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:34 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:33 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 15:33 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:33 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:33 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:32 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:31 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:31 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:31 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:31 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 15:31 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:31 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:31 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:31 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:31 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:31 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:22 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
• 15:11 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
• 14:58 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye

2023-06-12

• 19:37 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 19:37 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 19:37 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:37 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:37 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 19:37 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 19:37 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 19:36 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:36 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:36 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:36 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 19:36 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 19:35 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 19:35 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 11:57 arturo: [codfw1dev] refresh various occurrences of old FQDNs in instance puppet via horizon (T324992)

2023-06-08

• 16:56 andrewbogott: deleting all Stretch base images from glance
• 16:45 andrewbogott: updated the bullseye image with https://cloud.debian.org/images/cloud/bullseye/20230601-1398/debian-11-genericcloud-amd64-20230601-1398.tar.xz
• 12:17 wm-bot2: Drained cloudvirt1047.eqiad.wmnet (T334644) - cookbook ran by dcaro@vulcanus
• 12:17 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 02920314-1efe-4934-ad81-d2a6cf2e17ab, use this to unset) (T334644) - cookbook ran by dcaro@vulcanus
• 12:16 wm-bot2: Draining cloudvirt1047.eqiad.wmnet (T334644) - cookbook ran by dcaro@vulcanus
• 12:06 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 769349bf-465f-4f0c-a8f3-f2423631ba7e, use this to unset) (T334644) - cookbook ran by dcaro@vulcanus
• 12:05 wm-bot2: Draining cloudvirt1047.eqiad.wmnet (T334644) - cookbook ran by dcaro@vulcanus

2023-06-07

• 10:06 dcaro: upgraded ruby2.5 to latest fixed version on all buster VMs
• 08:10 dcaro: downgrading ruby2.5 to previous backport on all buster VMs

2023-06-06

• 19:09 andrewbogott: also increased RAM and secgroup-rule quota for Trove T337882
• 19:06 andrewbogott: increased trove secgroups, instances, volumes quotas from 40 to 100. Trove is too popular! T337882
• 18:31 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 18:31 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:30 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:29 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:28 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:27 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:27 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:27 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
• 18:27 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:27 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:27 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:27 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:27 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:27 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:26 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
• 18:26 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
• 17:55 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 17:55 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
• 17:55 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
• 17:54 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
• 17:54 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 17:54 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 17:42 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 17:42 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
• 17:41 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 17:41 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye
• 17:41 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['cinder-scheduler', 'cinder-volume'] - cookbook ran by andrew@bullseye

2023-06-05

• 09:41 arturo: [codfw1dev] rebooting bastion-codfw1dev-02 (no IP address in the main interface) T336963

2023-06-02

• 14:40 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by arturo@nostromo
• 12:59 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 12:58 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 12:58 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 12:44 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 12:44 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 12:43 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

2023-05-26

• 16:03 andrewbogott: "maintain-views --all-databases --replace-all" on clouddb1021 for T337446

2023-05-22

• 19:54 andrewbogott: deleting project 'citelearn' as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#SHUTDOWN_citelearn

2023-05-21

• 23:29 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 23:29 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 23:29 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 23:29 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 23:29 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 23:29 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 23:28 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 23:28 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 23:28 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 23:28 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 23:28 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 23:27 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 21:45 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 21:45 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 21:45 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:45 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:45 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 21:45 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 21:45 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 21:45 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:45 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:45 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:44 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 21:44 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

2023-05-19

• 14:53 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 14:53 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 14:53 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 14:53 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 14:53 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 14:53 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 14:53 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 14:52 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 14:52 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 14:52 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 14:52 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 14:52 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

2023-05-18

• 21:39 andrewbogott: deleting obsolete roles '4d8cad783d6342efa8414d7d36fbc034 | projectadmin_renamed_for_T330759' and 'f473273fac7146b3bdbf22e5d4504f95 | user_renamed_for_T330759' on eqiad1. State pre-deletion is dumped to /root/allassignmentspredeletion.txt on cloudcontrol1007.
• 21:35 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 21:35 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 21:35 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:35 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:35 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 21:35 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 21:34 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 21:34 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:34 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:34 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 21:34 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 21:33 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 19:20 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 19:20 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 19:20 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:20 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:20 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 19:20 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 19:19 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 19:19 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:19 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:19 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 19:19 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 19:18 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:12 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 16:12 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 16:12 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:12 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:12 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 16:12 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 16:11 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:11 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:11 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:11 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:11 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:10 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 15:22 wm-bot2: Restarting openstack services on cloudcontrol2001-dev@local1: ['cinder-volume'] - cookbook ran by andrew@bullseye
• 15:21 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 15:21 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 15:21 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 15:21 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:21 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:21 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:21 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 15:20 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

2023-05-15

• 14:28 wm-bot2: Drained cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
• 14:28 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 96ce2ed0-3aff-4d04-be0b-e16513070617, use this to unset) - cookbook ran by andrew@bullseye
• 14:27 wm-bot2: Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
• 14:23 wm-bot2: Drained cloudvirt1027.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 14:17 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 110176f8-04d5-4110-bb7d-1ab272bd8be2, use this to unset) (T316544) - cookbook ran by dcaro@vulcanus
• 14:16 wm-bot2: Draining cloudvirt1027.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 14:13 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: c6e92e13-49f4-4db3-8a13-8692ccfd3bc9, use this to unset) - cookbook ran by andrew@bullseye
• 14:12 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
• 14:12 wm-bot2: Drained cloudvirt1035.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 14:11 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: fa730dec-848f-45fb-9eda-e74bd874c5c9, use this to unset) - cookbook ran by andrew@bullseye
• 14:10 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
• 14:06 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 14:06 wm-bot2: Restarting openstack services on cloudcontrol1006: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
• 14:06 wm-bot2: Restarting openstack services on cloudcontrol1007: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
• 14:06 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 14:06 wm-bot2: Restarting openstack services on cloudcontrol1005: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye
• 14:01 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: eb1cfac0-d481-4baa-b9cd-15e5fbcef495, use this to unset) - cookbook ran by andrew@bullseye
• 14:00 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
• 13:58 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 3e6c3ff3-7d55-4777-9032-b867a257eced, use this to unset) - cookbook ran by andrew@bullseye
• 13:57 wm-bot2: Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye
• 13:53 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: 0693664a-df78-417e-ba34-590e5a0a9981, use this to unset) - cookbook ran by andrew@bullseye
• 13:52 wm-bot2: Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye
• 13:49 wm-bot2: Set cloudvirt cloudvirt1035.eqiad.wmnet maintenance (downtime id: e6929ab8-4bc3-4186-817b-9b53dbd597c6, use this to unset) (T316544) - cookbook ran by dcaro@vulcanus
• 13:48 wm-bot2: Draining cloudvirt1035.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:40 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:40 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:40 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:40 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:39 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 13:38 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:38 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:38 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:38 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:38 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:38 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:38 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:38 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:37 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:36 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 13:36 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:36 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:36 andrewbogott: restarting nova services in eqiad1, trying to free up db connections
• 13:36 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:36 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:36 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:36 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
• 13:33 wm-bot2: Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: fef43d73-6fd4-4dde-a0ac-95fd69a9b0c1, use this to unset) (T316544) - cookbook ran by dcaro@vulcanus
• 13:32 wm-bot2: Draining cloudvirt1034.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 13:29 wm-bot2: Draining cloudvirt1027.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 13:24 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 5f867662-e824-498c-a715-2e2ad50f0bb5, use this to unset) (T316544) - cookbook ran by dcaro@vulcanus
• 13:23 wm-bot2: Draining cloudvirt1027.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 13:08 wm-bot2: Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: 0f515150-1313-41d4-a5f6-9bc00ce9b245, use this to unset) (T316544) - cookbook ran by dcaro@vulcanus
• 13:07 wm-bot2: Draining cloudvirt1033.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 13:07 wm-bot2: Drained cloudvirt1032.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 12:50 wm-bot2: Set cloudvirt cloudvirt1032.eqiad.wmnet maintenance (downtime id: e826adc3-addd-44d8-b39e-ae7bd2df1e60, use this to unset) (T316544) - cookbook ran by dcaro@vulcanus
• 12:49 wm-bot2: Draining cloudvirt1032.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 12:49 wm-bot2: Drained cloudvirt1031.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 12:14 wm-bot2: Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 4154c818-744c-4d84-9883-cae7a5826ed5, use this to unset) (T316544) - cookbook ran by dcaro@vulcanus
• 12:13 wm-bot2: Draining cloudvirt1027.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 12:13 wm-bot2: Drained cloudvirt1026.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 12:04 wm-bot2: Set cloudvirt cloudvirt1026.eqiad.wmnet maintenance (downtime id: cdfc3d01-ec1e-483c-9a38-834193e487ff, use this to unset) (T316544) - cookbook ran by dcaro@vulcanus
• 12:03 wm-bot2: Draining cloudvirt1026.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 12:03 wm-bot2: Drained cloudvirt1025.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus
• 11:51 wm-bot2: Set cloudvirt cloudvirt1025.eqiad.wmnet maintenance (downtime id: 6a56757d-35de-499e-8209-728bcf62a22a, use this to unset) (T316544) - cookbook ran by dcaro@vulcanus
• 11:50 wm-bot2: Draining cloudvirt1025.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus

2023-05-12

• 17:52 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

2023-05-10

• 16:09 wm-bot2: Restarting openstack services on cloudservices1005: ['designate-producer', 'designate-sink', 'designate-worker', 'designate-central', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 16:09 wm-bot2: Restarting openstack services on cloudservices1004: ['designate-worker', 'designate-api', 'designate-mdns', 'designate-producer', 'designate-central', 'designate-sink', 'designate-agent'] - cookbook ran by andrew@bullseye
• 16:09 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
• 16:09 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
• 16:09 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 16:09 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:08 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:07 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:07 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:07 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:07 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:07 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:07 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:06 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:05 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:04 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:04 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:04 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:00 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:00 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:00 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:00 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:00 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:00 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:00 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 16:00 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:00 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 16:00 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:59 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:58 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:58 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:58 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:58 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:58 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:58 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:58 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:58 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 15:58 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:57 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:57 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:57 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:57 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:57 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:57 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:57 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudbackup2002: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:56 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:55 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:55 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:55 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:55 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:55 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:55 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:55 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:54 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:53 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:53 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:53 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:53 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:53 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:53 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:53 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:53 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:53 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:53 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:52 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 15:52 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:52 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:52 andrewbogott: running "cookbook -c ~/.config/spicerack/cookbook_config.yaml wmcs.openstack.restart_openstack --cluster-name eqiad1 --all" to pick up changes for testing T336379
• 15:52 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:52 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:52 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 15:52 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 03:05 andrewbogott: "systemctl restart puppet-enc" on enc-1.cloudinfra.eqiad1.wikimedia.cloud. Seems to have crashed.

2023-05-05

• 16:07 wm-bot2: Drained cloudvirt1024.eqiad.wmnet (T336064) - cookbook ran by andrew@bullseye
• 16:03 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 95995009-09d6-496e-8cd2-0cfac93d3cf7, use this to unset) (T336064) - cookbook ran by andrew@bullseye
• 16:02 wm-bot2: Draining cloudvirt1024.eqiad.wmnet (T336064) - cookbook ran by andrew@bullseye
• 16:01 wm-bot2: Drained cloudvirt1023.eqiad.wmnet (T336064) - cookbook ran by andrew@bullseye
• 15:51 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 53c46cae-00af-4664-97ff-266b393335bb, use this to unset) (T336064) - cookbook ran by andrew@bullseye
• 15:50 wm-bot2: Draining cloudvirt1023.eqiad.wmnet (T336064) - cookbook ran by andrew@bullseye
• 15:49 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 528ea4f6-8088-475e-937f-098ffba861b6, use this to unset) - cookbook ran by andrew@bullseye
• 15:47 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 3ef85b5e-d9d9-4b24-901b-a3058a7d0615, use this to unset) - cookbook ran by andrew@bullseye
• 15:44 andrewbogott: moved cloudvirt1023 and cloudvirt1024 from 'ceph' aggregate to 'maintenance' aggregate, prep for decom T336064
• 15:44 andrewbogott: moved cloudvirt1028 from 'localdisk' aggregate to 'maintenance' aggregate. Nothing new should be scheduled here, local storage should now move to cloudvirtlocal100x
• 15:41 andrewbogott: moved cloudvirt1055 and cloudvirt1056 from 'spare' to 'ceph' aggregate. Prep for removing two obsolete cloudvirts, 1023 and 1024. T336064

2023-05-04

• 22:49 andrewbogott: removed fullstack-* puppet reports on puppetmaster-02.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud and cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs to free up disk space

2023-05-02

• 13:01 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
• 13:01 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
• 12:31 wm-bot2: Destroying OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
• 12:30 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
• 11:53 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
• 11:53 wm-bot2: Added 1 new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] - cookbook ran by dcaro@vulcanus
• 11:53 wm-bot2: Added OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
• 11:53 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
• 11:52 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus

2023-05-01

• 17:09 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
• 17:09 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
• 17:08 wm-bot2: Adding OSD cloudcephosd2001-dev.codfw.wmnet... (1/1) - cookbook ran by dcaro@vulcanus
• 17:08 wm-bot2: Adding new OSDs ['cloudcephosd2001-dev.codfw.wmnet'] to the cluster - cookbook ran by dcaro@vulcanus
• 15:22 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus
• 13:53 taavi: running wmcs-novastats-puppetleaks in real mode T334127
• 13:51 wm-bot2: Depooling OSDs with ids in [0] on cloudcephosd2001-dev from codfw1 - cookbook ran by dcaro@vulcanus

2023-04-18

• 22:52 wm-bot2: Restarting openstack services on cloudcontrol1007: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
• 22:52 wm-bot2: Restarting openstack services on cloudcontrol1006: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
• 22:52 wm-bot2: Restarting openstack services on cloudcontrol1005: ['neutron-api', 'neutron-rpc-server'] - cookbook ran by andrew@bullseye
• 22:52 wm-bot2: Restarting openstack services on cloudvirt1056: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1019: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1060: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1023: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1038: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1036: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1039: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1050: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1061: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1028: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1020: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1052: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1040: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1043: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1026: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1030: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1048: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1025: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Restarting openstack services on cloudvirt1035: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1055: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1042: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1059: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1057: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1032: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1024: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1029: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1044: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1047: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1034: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1058: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1045: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1041: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1031: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1046: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1027: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudnet1006: ['neutron-linuxbridge-agent', 'neutron-metadata-agent', 'neutron-dhcp-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudnet1005: ['neutron-linuxbridge-agent', 'neutron-dhcp-agent', 'neutron-metadata-agent'] - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Restarting openstack services on cloudvirt1054: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:49 wm-bot2: Restarting openstack services on cloudvirt1033: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:49 wm-bot2: Restarting openstack services on cloudvirt1037: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:49 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:49 wm-bot2: Restarting openstack services on cloudvirt1051: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:49 wm-bot2: Restarting openstack services on cloudvirt1053: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:49 wm-bot2: Restarting openstack services on cloudvirt1049: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1003: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1002: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:48 wm-bot2: Restarting openstack services on cloudvirtlocal1001: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:48 wm-bot2: Restarting openstack services on cloudvirt1054: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:48 wm-bot2: Restarting openstack services on cloudvirt1055: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:48 wm-bot2: Restarting openstack services on cloudvirt1060: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1058: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1059: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1061: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1057: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1056: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1051: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1050: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1049: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 22:47 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1019: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1020: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:45 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:44 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:44 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:44 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:44 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:44 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:41 andrewbogott: resetting rabbitmq on cloudrabbit1003 due to splitbrain
• 22:40 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:40 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:40 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:39 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:39 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:39 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:39 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:36 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:36 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:36 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:34 wm-bot2: Restarting openstack services on cloudvirt1053: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:34 wm-bot2: Restarting openstack services on cloudvirt1052: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:34 wm-bot2: Restarting openstack services on cloudvirt1048: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:33 wm-bot2: Restarting openstack services on cloudcontrol1007: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 22:33 wm-bot2: Restarting openstack services on cloudcontrol1006: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1003: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1002: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:33 wm-bot2: Restarting openstack services on cloudvirt-wdqs1001: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:33 wm-bot2: Restarting openstack services on cloudvirt1047: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:33 wm-bot2: Restarting openstack services on cloudvirt1038: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:32 wm-bot2: Restarting openstack services on cloudvirt1042: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:32 wm-bot2: Restarting openstack services on cloudvirt1044: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:32 wm-bot2: Restarting openstack services on cloudvirt1041: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:31 wm-bot2: Restarting openstack services on cloudvirt1046: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:31 wm-bot2: Restarting openstack services on cloudvirt1043: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:31 wm-bot2: Restarting openstack services on cloudvirt1045: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:31 wm-bot2: Restarting openstack services on cloudvirt1040: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:30 wm-bot2: Restarting openstack services on cloudvirt1036: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:30 wm-bot2: Restarting openstack services on cloudvirt1034: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:30 wm-bot2: Restarting openstack services on cloudvirt1039: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:30 wm-bot2: Restarting openstack services on cloudvirt1037: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:30 wm-bot2: Restarting openstack services on cloudvirt1035: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:30 wm-bot2: Restarting openstack services on cloudvirt1033: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:29 wm-bot2: Restarting openstack services on cloudvirt1031: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:29 wm-bot2: Restarting openstack services on cloudvirt1032: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:29 wm-bot2: Restarting openstack services on cloudcontrol1005: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 22:29 wm-bot2: Restarting openstack services on cloudvirt1028: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:29 wm-bot2: Restarting openstack services on cloudvirt1019: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:29 wm-bot2: Restarting openstack services on cloudvirt1020: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:29 wm-bot2: Restarting openstack services on cloudvirt1030: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:29 wm-bot2: Restarting openstack services on cloudvirt1027: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:29 wm-bot2: Restarting openstack services on cloudvirt1023: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:28 wm-bot2: Restarting openstack services on cloudvirt1026: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:28 wm-bot2: Restarting openstack services on cloudvirt1029: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:28 wm-bot2: Restarting openstack services on cloudvirt1025: ['nova-compute'] - cookbook ran by andrew@bullseye
• 22:28 wm-bot2: Restarting openstack services on cloudvirt1024: ['nova-compute'] - cookbook ran by andrew@bullseye

2023-04-17

• 08:49 wm-bot2: Increased quotas by 9 cores, 1 instances, 16 ram (T334695) - cookbook ran by dcaro@vulcanus

2023-04-06

• 17:03 andrewbogott: running wmcs-wikireplica-dns on cloudcontrol1005 to update tools-db dns entries

2023-04-04

• 17:23 andrewbogott: resetting all three rabbitmq nodes and restarting all openstack services as per https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Rabbitmq#Resetting_the_HA_setup

2023-03-28

• 13:56 andrewbogott: depooling cloudweb1003 before switch upgrade
• 10:58 dhinus: disabled tool "wb" by clicking the disable button at https://toolsadmin.wikimedia.org/tools/id/wb T328693
• 08:34 arturo: cleanup neutron agents for cloudvirt1021/1022 (decom)
• 08:32 arturo: cleanup neutron agents for cloudvirt1017 (decom)

2023-03-27

• 14:30 wm-bot2: Drained cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye
• 14:19 wm-bot2: Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 3f43d3ca-696c-4d3c-8d5b-e57984f0eb86, use this to unset) - cookbook ran by andrew@bullseye
• 14:18 wm-bot2: Draining cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye
• 14:17 wm-bot2: Drained cloudvirt1023.eqiad.wmnet - cookbook ran by andrew@bullseye
• 14:08 wm-bot2: Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: f4767781-ea26-453b-9521-847a8340a249, use this to unset) - cookbook ran by andrew@bullseye
• 14:07 wm-bot2: Draining cloudvirt1023.eqiad.wmnet - cookbook ran by andrew@bullseye
• 14:04 wm-bot2: Drained cloudvirt1022.eqiad.wmnet - cookbook ran by andrew@bullseye
• 13:55 wm-bot2: Set cloudvirt cloudvirt1022.eqiad.wmnet maintenance (downtime id: 0e794cfe-5896-46c1-842d-c34719140d4f, use this to unset) - cookbook ran by andrew@bullseye
• 13:54 wm-bot2: Draining cloudvirt1022.eqiad.wmnet - cookbook ran by andrew@bullseye
• 13:54 wm-bot2: Drained cloudvirt1021.eqiad.wmnet - cookbook ran by andrew@bullseye
• 13:48 wm-bot2: Set cloudvirt cloudvirt1021.eqiad.wmnet maintenance (downtime id: e7a904ca-003d-450a-ad85-886fb80dfc41, use this to unset) - cookbook ran by andrew@bullseye
• 13:47 wm-bot2: Draining cloudvirt1021.eqiad.wmnet - cookbook ran by andrew@bullseye
• 13:46 wm-bot2: Drained cloudvirt1017.eqiad.wmnet - cookbook ran by andrew@bullseye
• 13:36 wm-bot2: Set cloudvirt cloudvirt1017.eqiad.wmnet maintenance (downtime id: 0ff0090f-26c3-4278-a9c9-cce518559408, use this to unset) - cookbook ran by andrew@bullseye
• 13:35 wm-bot2: Draining cloudvirt1017.eqiad.wmnet - cookbook ran by andrew@bullseye

2023-03-22

• 12:41 taavi: delete wmde-templates-alpha project T332773

2023-03-08

• 21:45 bd808: maintain-kubeusers container in CrashLoopBackoff, investigating
• 13:49 dcaro: stopping puppet on labostre1004 to debug maintain-dbusers

2023-03-07

• 16:06 andrewbogott: updated application credential roles, replacing 'user' with 'reader' and 'projectadmin' with 'member': update application_credential_role set role_id='f75a3c410bca4e96a1cf6ac103b0ccaf' where role_id='f473273fac7146b3bdbf22e5d4504f95' and update application_credential_role set role_id='38676f30eaeb44518bf7e144a73c8da6' where role_id='4d8cad783d6342efa8414d7d36fbc034'
• 10:11 dcaro: there was a little unavailability for some VMs while ceph was starting to rebalance things, but it seems stable and moving data around (T331141)
• 09:37 dcaro: Changing ceph crush map to allow rack HA on eqiad1 cluster (T331141)

2023-03-03

• 12:12 arturo: installing haproxy updates (T331119)

2023-03-02

• 16:17 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1010.eqiad.wmnet'] (T329504) - cookbook ran by dcaro@vulcanus
• 14:21 wm-bot2: Added 1 new OSDs ['cloudcephosd1010.eqiad.wmnet'] (T329504) - cookbook ran by dcaro@vulcanus
• 14:21 wm-bot2: Added OSD cloudcephosd1010.eqiad.wmnet... (1/1) (T329504) - cookbook ran by dcaro@vulcanus
• 14:13 wm-bot2: Finished rebooting node cloudcephosd1010.eqiad.wmnet (T329504) - cookbook ran by dcaro@vulcanus
• 14:10 wm-bot2: Rebooting node cloudcephosd1010.eqiad.wmnet (T329504) - cookbook ran by dcaro@vulcanus
• 14:09 wm-bot2: Adding OSD cloudcephosd1010.eqiad.wmnet... (1/1) (T329504) - cookbook ran by dcaro@vulcanus
• 14:09 wm-bot2: Adding new OSDs ['cloudcephosd1010.eqiad.wmnet'] to the cluster (T329504) - cookbook ran by dcaro@vulcanus
• 10:43 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1005.eqiad.wmnet'] (T329504) - cookbook ran by dcaro@vulcanus
• 08:44 wm-bot2: Added 1 new OSDs ['cloudcephosd1005.eqiad.wmnet'] (T329504) - cookbook ran by dcaro@vulcanus
• 08:44 wm-bot2: Added OSD cloudcephosd1005.eqiad.wmnet... (1/1) (T329504) - cookbook ran by dcaro@vulcanus
• 08:36 wm-bot2: Finished rebooting node cloudcephosd1005.eqiad.wmnet (T329504) - cookbook ran by dcaro@vulcanus
• 08:33 wm-bot2: Rebooting node cloudcephosd1005.eqiad.wmnet (T329504) - cookbook ran by dcaro@vulcanus
• 08:32 wm-bot2: Adding OSD cloudcephosd1005.eqiad.wmnet... (1/1) (T329504) - cookbook ran by dcaro@vulcanus
• 08:32 wm-bot2: Adding new OSDs ['cloudcephosd1005.eqiad.wmnet'] to the cluster (T329504) - cookbook ran by dcaro@vulcanus

2023-02-28

• 22:47 andrewbogott: adding new 'member' role assignment to every user/project pair that currently has the 'user' assignment. T330759
• 09:47 wm-bot2: Depooled and destroyed OSD daemons [79, 78, 77, 76, 75, 74, 73, 72] and removed the OSD host cloudcephosd1010 from the CRUSH map. (T329504) - cookbook ran by dcaro@vulcanus
• 09:46 wm-bot2: Destroying OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 (T329504) - cookbook ran by dcaro@vulcanus
• 09:28 wm-bot2: Depooling OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 (T329504) - cookbook ran by dcaro@vulcanus
• 09:13 wm-bot2: Depooling OSDs with ids in [79, 78, 77, 76, 75, 74, 73, 72] on cloudcephosd1010 from eqiad1 (T329504) - cookbook ran by dcaro@vulcanus
• 09:12 wm-bot2: Depooled and destroyed OSD daemons [39, 38, 37, 36, 35, 34, 33, 32] and removed the OSD host cloudcephosd1005 from the CRUSH map. (T329504) - cookbook ran by dcaro@vulcanus
• 09:11 wm-bot2: Destroying OSDs with ids in [39, 38, 37, 36, 35, 34, 33, 32] on cloudcephosd1005 from eqiad1 (T329504) - cookbook ran by dcaro@vulcanus
• 08:55 wm-bot2: Depooling OSDs with ids in [39, 38, 37, 36, 35, 34, 33, 32] on cloudcephosd1005 from eqiad1 (T329504) - cookbook ran by dcaro@vulcanus

2023-02-27

• 21:01 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1004.eqiad.wmnet'] (T329502) - cookbook ran by dcaro@vulcanus
• 18:53 wm-bot2: Added 1 new OSDs ['cloudcephosd1004.eqiad.wmnet'] (T329502) - cookbook ran by dcaro@vulcanus
• 18:53 wm-bot2: Added OSD cloudcephosd1004.eqiad.wmnet... (1/1) (T329502) - cookbook ran by dcaro@vulcanus
• 18:45 wm-bot2: Finished rebooting node cloudcephosd1004.eqiad.wmnet (T329502) - cookbook ran by dcaro@vulcanus
• 18:42 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet (T329502) - cookbook ran by dcaro@vulcanus
• 18:41 wm-bot2: Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) (T329502) - cookbook ran by dcaro@vulcanus
• 18:41 wm-bot2: Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster (T329502) - cookbook ran by dcaro@vulcanus
• 18:38 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet (T329502) - cookbook ran by dcaro@vulcanus
• 18:38 wm-bot2: Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) (T329502) - cookbook ran by dcaro@vulcanus
• 18:38 wm-bot2: Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster (T329502) - cookbook ran by dcaro@vulcanus
• 18:14 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1003.eqiad.wmnet'] (T329502) - cookbook ran by dcaro@vulcanus
• 16:13 wm-bot2: Added 1 new OSDs ['cloudcephosd1003.eqiad.wmnet'] (T329502) - cookbook ran by dcaro@vulcanus
• 16:13 wm-bot2: Added OSD cloudcephosd1003.eqiad.wmnet... (1/1) (T329502) - cookbook ran by dcaro@vulcanus
• 16:05 wm-bot2: Finished rebooting node cloudcephosd1003.eqiad.wmnet (T329502) - cookbook ran by dcaro@vulcanus
• 16:02 wm-bot2: Rebooting node cloudcephosd1003.eqiad.wmnet (T329502) - cookbook ran by dcaro@vulcanus
• 16:01 wm-bot2: Adding OSD cloudcephosd1003.eqiad.wmnet... (1/1) (T329502) - cookbook ran by dcaro@vulcanus
• 16:01 wm-bot2: Adding new OSDs ['cloudcephosd1003.eqiad.wmnet'] to the cluster (T329502) - cookbook ran by dcaro@vulcanus
• 15:59 wm-bot2: Adding OSD coludcephosd1003.eqiad.wmnet... (1/1) (T329502) - cookbook ran by dcaro@vulcanus
• 15:59 wm-bot2: Adding new OSDs ['coludcephosd1003.eqiad.wmnet'] to the cluster (T329502) - cookbook ran by dcaro@vulcanus
• 15:58 wm-bot2: Adding OSD coludcephosd1003... (1/1) (T329502) - cookbook ran by dcaro@vulcanus
• 15:58 wm-bot2: Adding new OSDs ['coludcephosd1003'] to the cluster (T329502) - cookbook ran by dcaro@vulcanus

2023-02-22

• 08:25 wm-bot2: Depooled and destroyed OSD daemons [31, 30, 29, 28, 27, 26, 25, 24] and removed the OSD host cloudcephosd1004 from the CRUSH map. (T329502) - cookbook ran by dcaro@vulcanus
• 08:25 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus
• 08:22 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus
• 08:15 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus
• 08:13 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus
• 07:23 wm-bot2: Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus
• 07:02 wm-bot2: Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus

2023-02-21

• 20:27 andrewbogott: deleted 200 more orphaned VM images with wmcs-novastats-cephleaks
• 17:18 andrewbogott: shutting down postgres on clouddb1004/1003, then shutting down the vms
• 15:50 wm-bot2: Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus
• 15:48 wm-bot2: Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus
• 14:21 wm-bot2: Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus
• 14:00 wm-bot2: Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus

2023-02-16

• 19:14 wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1002.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus
• 17:55 dcaro: Manually zapped /dev/sdc on cloudcephosd1002, probably a leftover drive since the beginning (or during the reimage the drives changed names, and this one had leftovers from the previous OS) (T329498)
• 17:47 wm-bot2: Added 1 new OSDs ['cloudcephosd1002.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus
• 17:47 wm-bot2: Added OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus
• 17:42 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus
• 17:42 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus
• 16:50 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus
• 16:50 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus
• 16:01 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus
• 16:01 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus
• 16:00 wm-bot2: Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus
• 16:00 wm-bot2: Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus
• 14:05 wm-bot2: Added 1 new OSDs ['cloudcephosd1001.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus
• 13:29 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus
• 13:29 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus
• 13:24 wm-bot2: Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus
• 13:24 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus
• 13:23 wm-bot2: Destroying OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus
• 13:21 wm-bot2: Depooling OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus
• 13:14 wm-bot2: Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus
• 13:14 wm-bot2: Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus
• 11:20 wm-bot2: Destroying OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus
• 11:19 wm-bot2: Depooling OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus
• 11:03 wm-bot2: Destroying OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus
• 11:01 wm-bot2: Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus
• 10:59 wm-bot2: Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus
• 10:15 dcaro: purges osd daemons 48 and 40 from eqiad ceph cluster (T329709)

2023-02-15

• 14:53 andrewbogott: deleting another 100 leaked VM images with wmcs-novastats-cephleaks
• 13:39 wm-bot2: Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
• 13:14 wm-bot2: Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
• 13:13 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
• 13:11 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
• 13:11 wm-bot2: Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
• 13:10 wm-bot2: Destroying OSDs with id 12345 on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus
• 13:09 wm-bot2: Destroying OSDs with id ['12345'] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus

2023-02-14

• 13:17 andrewbogott: restarting all eqiad1 openstack services because that seems to sometimes help things *shrug*

2023-02-13

• 14:06 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 8fbf6bfd-eec1-4d81-8e0d-ea431d8411ee (T329498) - cookbook ran by dcaro@vulcanus
• 13:32 taavi: re-enable puppet on labstore1004 T329377

2023-02-09

• 21:17 andrewbogott: deleted 10% of leaked VM ceph images using wmcs-novastats-cephleaks (only 10% out of an abundance of caution)

2023-02-08

• 17:08 arturo: changing to cloudgw network setup, make VIPs /32 (T295774)

2023-02-07

• 11:26 arturo: [codfw1dev] testing network changes in cloudgw, expect unrealiable network (T295774)

2023-02-04

• 13:44 taavi: drop old columns from oathauth_users table on labtestwiki T328131

2023-02-03

• 15:00 andrewbogott: restarted nova services in eqiad1 in an attempt to eke out another day or two of stability
• 14:13 taavi: attached GrapheSuppression developer account to wikitech

2023-02-02

• 13:14 dcaro_away: draining osd.48 from node cloudcephosd1001 (T316544)
• 12:57 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 7ac2b25a-d1bb-4789-8aa6-b9435b505349 (T316544) - cookbook ran by dcaro@vulcanus

2023-01-30

• 22:34 wm-bot2: Upgraded and rebooted host cloudrabbit1002.wikimedia.org - cookbook ran by andrew@bullseye
• 21:34 andrewbogott: merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/884922 and upgrading rabbitmq nodes for T328155

2023-01-27

• 20:08 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 19:22 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 19:10 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 15:25 andrewbogott: restarting openstack services in eqiad1, another attempt to address instability

2023-01-26

• 20:34 andrewbogott: shutting down mariadb on cloudbackup2001-dev, testing the waters for T328079

2023-01-22

• 03:42 andrewbogott: reset eqiad1 rabbitmq in an attempt to resolve some mild instability

2023-01-20

• 15:26 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
• 15:26 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['f47a3d91-b270-4c90-acc8-d85075a6bf8e'], use this to unset) - cookbook ran by andrew@bullseye
• 13:15 arturo: reinstall python3-neutron (to reset manual patching) on all cloudnet nodes and patch it via puppet, then restart neutron-l3-agent by hand (T327463)
• 10:12 arturo: [codfw1dev] failover neutron-l3-agent between cloudnet2005-dev/cloudnet2006-dev a couple of times T327463
• 02:17 andrewbogott: stopping neutron-l3-agent on cloudnet1005 because it's logging at a furious rate and about to fill the drive

2023-01-19

• 18:06 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
• 18:06 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['36d5af6a-7d8e-4d0c-831e-1bf05c255984'], use this to unset) - cookbook ran by andrew@bullseye
• 17:35 wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye
• 17:22 wm-bot2: Put cloudweb hosts (cloudweb2002-dev.wikimedia.org) into maintenance mode (downtime id: ['66ec4f04-d25b-4067-be9e-2fe12cb1d3ff'], use this to unset) - cookbook ran by andrew@bullseye

2023-01-18

• 22:32 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 347cb75e-215e-4b85-ae14-4ce1934c70c7, use this to unset) - cookbook ran by andrew@bullseye
• 22:01 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 9bf6212f-7fdb-4869-8190-b07387b2bc7e, use this to unset) - cookbook ran by andrew@bullseye
• 21:40 wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 11aec8ea-f443-41ae-b79a-5e1e3aa94546, use this to unset) - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 20:19 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 20:19 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 20:19 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 20:19 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 20:19 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 20:18 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 20:18 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 20:18 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 17:03 wm-bot2: Restarting openstack services on cloudservices2004-dev: ['designate-mdns', 'designate-sink', 'designate-central', 'designate-producer', 'designate-worker', 'designate-agent'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudservices2005-dev: ['designate-central', 'designate-sink', 'designate-worker', 'designate-producer', 'designate-mdns', 'designate-agent'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudbackup1002-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudbackup1001-dev: ['cinder-backup'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-volume', 'cinder-scheduler', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 17:02 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:01 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:01 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:01 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 17:01 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 14:38 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 14:38 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 14:37 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye
• 14:37 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata', 'cinder-scheduler', 'cinder-volume', 'neutron-api', 'neutron-rpc-server', 'trove-api', 'trove-conductor', 'trove-taskmanager', 'keystone', 'keystone-admin', 'glance-api', 'magnum-api', 'magnum-conductor', 'heat-api', 'heat-api-cfn', 'heat-engine'] - cookbook ran by andrew@bullseye

2023-01-17

• 19:32 wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye
• 18:04 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye
• 17:52 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
• 17:36 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
• 17:23 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye

2023-01-13

• 17:36 wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 17:36 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
• 17:36 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
• 17:35 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
• 17:35 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 17:35 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 17:33 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:33 wm-bot2: Restarting openstack services on cloudvirt2002-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:33 wm-bot2: Restarting openstack services on cloudnet2005-dev: ['neutron-metadata-agent', 'neutron-dhcp-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:33 wm-bot2: Restarting openstack services on cloudvirt2003-dev: ['neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:33 wm-bot2: Restarting openstack services on cloudnet2006-dev: ['neutron-dhcp-agent', 'neutron-metadata-agent', 'neutron-linuxbridge-agent'] - cookbook ran by andrew@bullseye
• 17:26 wm-bot2: Restarting openstack services on cloudvirt2001-dev: ['nova-compute'] - cookbook ran by andrew@bullseye
• 17:26 wm-bot2: Restarting openstack services on cloudcontrol2004-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 17:25 wm-bot2: Restarting openstack services on cloudcontrol2001-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye
• 17:21 wm-bot2: Restarting openstack services: {'cloudcontrol2001-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'], 'cloudcontrol2004-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'], 'cloudvirt2001-dev': ['nova-compute'], 'cloudvirt2002-dev': ['nova-compute'], 'cloudvirt2003-dev': ['nova-compute'], 'cloudcontrol2005-dev': ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata
• 10:41 arturo: restart backup_vm.service on cloudbackup1003/1004 to recover from a nova Unknown Error (HTTP 503)

2023-01-12

• 22:34 andrewbogott: updated the Bullseye base image with the upstream 20221219 build
• 14:12 wm-bot2: Created project checkuser-beta-wiki with default quotas. (T326740) - cookbook ran by arturo@nostromo

2023-01-06

• 18:42 wm-bot2: Safe reboot of cloudvirt2003-dev.codfw.wmnet finished successfully - cookbook ran by andrew@bullseye
• 18:42 wm-bot2: unset cloudvirt2003-dev.codfw.wmnet maintenance (aggregates: ceph) - cookbook ran by andrew@bullseye
• 18:39 wm-bot2: Drained cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
• 18:35 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: b50d9d3a-4f1d-4522-b86f-722fc9c55c87, use this to unset) - cookbook ran by andrew@bullseye
• 18:35 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
• 18:35 wm-bot2: Safe rebooting cloudvirt2003-dev.codfw.wmnet - cookbook ran by andrew@bullseye
• 18:35 wm-bot2: Draining cloudvirt2003-dev.cdofw.wmnet - cookbook ran by andrew@bullseye
• 18:35 wm-bot2: Safe rebooting cloudvirt2003-dev.cdofw.wmnet - cookbook ran by andrew@bullseye
• 00:14 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
• 00:08 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
• 00:01 wm-bot2: Upgraded and rebooted host cloudnet2006-dev.codfw.wmnet - cookbook ran by andrew@bullseye

2023-01-05

• 23:54 wm-bot2: Upgraded and rebooted host cloudnet2005-dev.codfw.wmnet - cookbook ran by andrew@bullseye
• 23:38 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 23:25 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 23:13 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 22:18 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 22:10 andrewbogott: upgrading codfw1dev openstack to version 'zed'

2023-01-04

• 21:18 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye
• 21:11 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org - cookbook ran by andrew@bullseye
• 20:12 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye
• 20:04 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org - cookbook ran by andrew@bullseye
• 14:45 wm-bot2: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] - cookbook ran by fran@wmf3169
• 14:44 wm-bot2: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by fran@wmf3169
• 14:41 wm-bot2: Rebooting node cloudcephmon2006-dev.codfw.wmnet - cookbook ran by fran@wmf3169
• 14:41 wm-bot2: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by fran@wmf3169
• 14:38 wm-bot2: Rebooting node cloudcephmon2005-dev.codfw.wmnet - cookbook ran by fran@wmf3169
• 14:37 wm-bot2: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by fran@wmf3169
• 14:34 wm-bot2: Rebooting node cloudcephmon2004-dev.codfw.wmnet - cookbook ran by fran@wmf3169
• 14:34 wm-bot2: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev - cookbook ran by fran@wmf3169

2023-01-03

• 22:11 wm-bot2: Upgraded and rebooted host cloudservices2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 21:55 wm-bot2: Upgraded and rebooted host cloudservices2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 15:25 taavi: restart designate-sink everywhere to pick up wmf-sink changes

2022-12-25

• 14:21 taavi: register developer account 'instance-puppet-user-dev' to update the codfw1dev instance-puppet repo without access to the eqiad1 repo T318504

2022-12-22

• 15:16 dcaro: added submit rights for JenkinsBot on all cloud/* gerrit repos

2022-12-21

• 04:59 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 04:59 wm-bot2: Finished rebooting node cloudcephosd1029.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 04:35 wm-bot2: Rebooting node cloudcephosd1029.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 04:35 wm-bot2: Finished rebooting node cloudcephosd1028.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 04:12 wm-bot2: Rebooting node cloudcephosd1028.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 04:12 wm-bot2: Finished rebooting node cloudcephosd1027.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 03:48 wm-bot2: Rebooting node cloudcephosd1027.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 03:48 wm-bot2: Finished rebooting node cloudcephosd1026.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 03:24 wm-bot2: Rebooting node cloudcephosd1026.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 03:24 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 03:00 wm-bot2: Rebooting node cloudcephosd1025.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 03:00 wm-bot2: Finished rebooting node cloudcephosd1024.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 02:36 wm-bot2: Rebooting node cloudcephosd1024.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 02:36 wm-bot2: Finished rebooting node cloudcephosd1023.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 02:12 wm-bot2: Rebooting node cloudcephosd1023.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 02:12 wm-bot2: Finished rebooting node cloudcephosd1022.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 01:47 wm-bot2: Rebooting node cloudcephosd1022.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 01:47 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 01:22 wm-bot2: Rebooting node cloudcephosd1021.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 01:22 wm-bot2: Finished rebooting node cloudcephosd1020.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 00:58 wm-bot2: Rebooting node cloudcephosd1020.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 00:58 wm-bot2: Finished rebooting node cloudcephosd1019.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 00:34 wm-bot2: Rebooting node cloudcephosd1019.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 00:34 wm-bot2: Finished rebooting node cloudcephosd1018.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 00:09 wm-bot2: Rebooting node cloudcephosd1018.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 00:09 wm-bot2: Finished rebooting node cloudcephosd1017.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye

2022-12-20

• 23:45 wm-bot2: Rebooting node cloudcephosd1017.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 23:45 wm-bot2: Finished rebooting node cloudcephosd1016.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 23:21 wm-bot2: Rebooting node cloudcephosd1016.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 23:21 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 22:56 wm-bot2: Rebooting node cloudcephosd1015.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 22:55 wm-bot2: Finished rebooting node cloudcephosd1014.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 22:30 wm-bot2: Rebooting node cloudcephosd1014.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 22:30 wm-bot2: Finished rebooting node cloudcephosd1013.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 22:08 andrewbogott: restarting openstack services with wmcs.openstack.restart_openstack due to miscellaneous trove failures
• 22:06 wm-bot2: Rebooting node cloudcephosd1013.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 22:06 wm-bot2: Finished rebooting node cloudcephosd1012.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 21:40 wm-bot2: Rebooting node cloudcephosd1012.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 21:40 wm-bot2: Finished rebooting node cloudcephosd1011.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 21:15 wm-bot2: Rebooting node cloudcephosd1011.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 21:15 wm-bot2: Finished rebooting node cloudcephosd1010.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 20:50 wm-bot2: Rebooting node cloudcephosd1010.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 20:50 wm-bot2: Finished rebooting node cloudcephosd1009.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 20:26 wm-bot2: Rebooting node cloudcephosd1009.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 20:26 wm-bot2: Finished rebooting node cloudcephosd1008.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 20:01 wm-bot2: Rebooting node cloudcephosd1008.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 20:01 wm-bot2: Finished rebooting node cloudcephosd1007.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 19:35 wm-bot2: Rebooting node cloudcephosd1007.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 19:35 wm-bot2: Finished rebooting node cloudcephosd1006.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 19:10 wm-bot2: Rebooting node cloudcephosd1006.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 19:10 wm-bot2: Finished rebooting node cloudcephosd1005.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 18:45 wm-bot2: Rebooting node cloudcephosd1005.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 18:45 wm-bot2: Finished rebooting node cloudcephosd1004.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 18:20 wm-bot2: Rebooting node cloudcephosd1004.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 18:20 wm-bot2: Finished rebooting node cloudcephosd1003.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 17:56 wm-bot2: Rebooting node cloudcephosd1003.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 17:56 wm-bot2: Finished rebooting node cloudcephosd1002.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 17:33 wm-bot2: Rebooting node cloudcephosd1002.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 17:33 wm-bot2: Finished rebooting node cloudcephosd1001.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 17:07 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 17:07 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
• 16:49 wm-bot2: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T325132) - cookbook ran by andrew@bullseye
• 16:48 wm-bot2: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 16:46 wm-bot2: Rebooting node cloudcephosd2003-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 16:46 wm-bot2: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Rebooting node cloudcephosd2002-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 16:42 wm-bot2: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 16:40 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 16:40 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
• 16:39 wm-bot2: Rebooting node cloudcephosd2001-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 16:39 wm-bot2: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T325132) - cookbook ran by andrew@bullseye
• 15:41 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:41 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl
• 15:40 wm-bot2: Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T325132) - cookbook ran by andrew@bullseye
• 15:40 wm-bot2: Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:37 wm-bot2: Rebooting node cloudcephosd2003-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:37 wm-bot2: Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:33 wm-bot2: Rebooting node cloudcephosd2002-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:33 wm-bot2: Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:30 wm-bot2: Rebooting node cloudcephosd2001-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:30 wm-bot2: Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T325132) - cookbook ran by andrew@bullseye
• 15:29 wm-bot2: Finished rebooting the nodes ['cloudcephmon2004-dev', 'cloudcephmon2005-dev', 'cloudcephmon2006-dev'] (T325132) - cookbook ran by andrew@bullseye
• 15:29 wm-bot2: Finished rebooting node cloudcephmon2006-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:25 wm-bot2: Rebooting node cloudcephmon2006-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:25 wm-bot2: Finished rebooting node cloudcephmon2005-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:22 wm-bot2: Rebooting node cloudcephmon2005-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:22 wm-bot2: Finished rebooting node cloudcephmon2004-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:19 wm-bot2: Rebooting node cloudcephmon2004-dev.codfw.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:19 wm-bot2: Rebooting the nodes cloudcephmon2004-dev,cloudcephmon2005-dev,cloudcephmon2006-dev (T325132) - cookbook ran by andrew@bullseye
• 15:18 wm-bot2: Finished rebooting the nodes ['cloudcephmon1001', 'cloudcephmon1002', 'cloudcephmon1003'] (T325132) - cookbook ran by andrew@bullseye
• 15:18 wm-bot2: Finished rebooting node cloudcephmon1003.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:15 wm-bot2: Rebooting node cloudcephmon1003.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:15 wm-bot2: Finished rebooting node cloudcephmon1002.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:11 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:11 wm-bot2: Finished rebooting node cloudcephmon1001.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:08 wm-bot2: Rebooting node cloudcephmon1001.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
• 15:08 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 (T325132) - cookbook ran by andrew@bullseye

2022-12-17

• 07:50 taavi: deleted project packagist-mirror per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#packagist-mirror

2022-12-16

• 19:36 volans: restarted sshd twice on bastion-restricted-eqiad1-02 to debug SSH connections for T319401
• 08:46 dcaro: restart designate-sink on both cloudservice hosts (T322279)
• 08:45 dcaro: restart designate-sink on both cloudservice hosts

2022-12-07

• 22:07 andrewbogott: systemctl restart libvirt-guests.service on cloudvirt1019 to get ceph/rbd working on VMS on this hypervisor

2022-12-03

• 19:24 taavi: restart designate-sink on both cloudservices hosts

2022-11-30

• 20:03 andrewbogott: changing all rabbitmq queues to quorum queues. Will be noisy! T318816
• 02:54 wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye

2022-11-28

• 13:00 wm-bot2: unset cloudvirt1043.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
• 10:28 wm-bot2: Drained cloudvirt1043.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 10:19 wm-bot2: Set cloudvirt cloudvirt1043.eqiad.wmnet maintenance (downtime id: bb94dd24-fef9-4c9c-8f79-b6e15023ce69, use this to unset) (T319184) - cookbook ran by arturo@nostromo
• 10:18 wm-bot2: Draining cloudvirt1043.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo

2022-11-25

• 10:54 wm-bot2: deleted VM canary2001-dev-2 from cloudvirt2001-dev - cookbook ran by arturo@nostromo
• 10:54 wm-bot2: created VM canary2001-dev-3 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
• 10:53 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:5b2ca632-2ea0-4007-9b40-4f84f8e2428b) - cookbook ran by arturo@nostromo
• 10:46 wm-bot2: Created new flavor: g3.cores1.ram1.disk20.admin (id:fecbd56d-0969-45f3-80fd-2b463a5b6270) - cookbook ran by arturo@nostromo

2022-11-24

• 16:53 wm-bot2: deleted VM canary2001-dev-1 from cloudvirt2001-dev - cookbook ran by arturo@nostromo
• 16:53 wm-bot2: created VM canary2001-dev-2 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
• 16:42 wm-bot2: created VM canary2003-dev-1 in cloudvirt2003-dev - cookbook ran by arturo@nostromo
• 16:42 wm-bot2: created VM canary2002-dev-1 in cloudvirt2002-dev - cookbook ran by arturo@nostromo
• 16:42 wm-bot2: created VM canary2001-dev-1 in cloudvirt2001-dev - cookbook ran by arturo@nostromo
• 16:36 wm-bot2: Created new flavor: cloudvirt-canary-ceph (id:0d06701a-2845-4298-b2b4-fabf8b1ddcbb) - cookbook ran by arturo@nostromo
• 13:03 wm-bot2: unset cloudvirt1044.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
• 11:51 wm-bot2: Drained cloudvirt1044.eqiad.wmnet - cookbook ran by arturo@nostromo
• 11:37 wm-bot2: Set cloudvirt cloudvirt1044.eqiad.wmnet maintenance (downtime id: 10076ecc-0f94-4d56-9bbd-bebb48bdc126, use this to unset) - cookbook ran by arturo@nostromo
• 11:35 wm-bot2: Draining cloudvirt1044.eqiad.wmnet - cookbook ran by arturo@nostromo
• 10:16 dcaro: removed ip6 dns name entry from nb for coluddb* (T323550)
• 09:53 dcaro: removed ip6 dns entry from nb for coluddb1013 (T323550)

2022-11-23

• 15:00 wm-bot2: unset cloudvirt1045.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
• 13:46 wm-bot2: Drained cloudvirt1045.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 13:35 wm-bot2: Set cloudvirt cloudvirt1045.eqiad.wmnet maintenance (downtime id: 2386b468-0f21-4ecb-91e2-e19ace66881d, use this to unset) (T319184) - cookbook ran by arturo@nostromo
• 13:34 wm-bot2: Draining cloudvirt1045.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 13:20 wm-bot2: unset cloudvirt1046.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
• 12:17 wm-bot2: Drained cloudvirt1046.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 12:04 wm-bot2: Set cloudvirt cloudvirt1046.eqiad.wmnet maintenance (downtime id: 6291d38e-c04c-4aa0-88da-2b329874a9b9, use this to unset) (T319184) - cookbook ran by arturo@nostromo
• 12:03 wm-bot2: Draining cloudvirt1046.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 12:02 wm-bot2: unset cloudvirt1047.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
• 11:32 arturo: [codfw1dev] created project cloudvirt-canary
• 10:13 wm-bot2: Drained cloudvirt1047.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 10:01 wm-bot2: Set cloudvirt cloudvirt1047.eqiad.wmnet maintenance (downtime id: 2c7ccc17-2be3-427d-aaec-57fadca0de5b, use this to unset) (T319184) - cookbook ran by arturo@nostromo
• 10:00 wm-bot2: Draining cloudvirt1047.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo

2022-11-22

• 13:26 wm-bot2: unset cloudvirt1048.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
• 12:15 wm-bot2: unset cloudvirt1049.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo
• 10:58 wm-bot2: Drained cloudvirt1049.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 10:37 wm-bot2: Set cloudvirt cloudvirt1049.eqiad.wmnet maintenance (downtime id: 3ae0a20d-3cf0-4eba-bea6-45aa61d8ad00, use this to unset) (T319184) - cookbook ran by arturo@nostromo
• 10:36 wm-bot2: Draining cloudvirt1049.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 10:21 wm-bot2: Unset cloudvirt cloudvirt1050.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo

2022-11-21

• 16:19 wm-bot2: Drained cloudvirt1050.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 16:11 wm-bot2: Set cloudvirt cloudvirt1050.eqiad.wmnet maintenance (downtime id: 0b154a93-a9d3-4ac7-bcfc-b67c49abe97b, use this to unset) (T319184) - cookbook ran by arturo@nostromo
• 16:09 wm-bot2: Draining cloudvirt1050.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 16:07 wm-bot2: Unset cloudvirt cloudvirt1051.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
• 15:18 wm-bot2: Drained cloudvirt1051.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 15:02 wm-bot2: Set cloudvirt cloudvirt1051.eqiad.wmnet maintenance (downtime id: 1de1174b-ec46-47a3-911c-b5808ce37028, use this to unset) (T319184) - cookbook ran by arturo@nostromo
• 15:00 wm-bot2: Draining cloudvirt1051.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 14:57 wm-bot2: Unset cloudvirt cloudvirt1052.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
• 12:53 wm-bot2: Drained cloudvirt1052.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 12:30 wm-bot2: Set cloudvirt cloudvirt1052.eqiad.wmnet maintenance (downtime id: 30db8a9b-08db-456f-8106-53188ff5f989, use this to unset) (T319184) - cookbook ran by arturo@nostromo
• 12:29 wm-bot2: Draining cloudvirt1052.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo
• 12:15 wm-bot2: Unset cloudvirt cloudvirt1053.eqiad.wmnet maintenance - cookbook ran by arturo@nostromo
• 10:13 arturo: drained cloudvirt1053 in preparation for reimage (was spare anyway)

2022-11-18

• 13:37 arturo: [codfw1dev] reimaged cloudvirt2001-dev and cloudvirt2002-dev
• 11:36 wm-bot2: Set cloudvirt cloudvirt2001-dev.codfw.wmnet maintenance (downtime id: 2fb9df34-fc2d-45b9-b21f-c0ec09008b92, use this to unset) - cookbook ran by arturo@nostromo
• 11:36 wm-bot2: Draining cloudvirt2001-dev.codfw.wmnet - cookbook ran by arturo@nostromo

2022-11-16

• 20:07 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 19:56 wm-bot2: Upgraded and rebooted host cloudservices2004-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 19:16 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
• 18:59 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
• 18:51 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 12:52 arturo: failovered cloudgw1002 into cloudgw1001 for reimage, IRC bots were briefly disconnected

2022-11-14

• 20:22 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye
• 20:07 wm-bot2: Upgraded and rebooted host cloudcontrol1007.wikimedia.org - cookbook ran by andrew@bullseye
• 19:55 wm-bot2: Upgraded and rebooted host cloudcontrol1006.wikimedia.org - cookbook ran by andrew@bullseye
• 19:42 wm-bot2: Upgraded and rebooted host cloudcontrol1005.wikimedia.org - cookbook ran by andrew@bullseye
• 19:28 andrewbogott: beginning OpenStack upgrade in eqiad1 -- T305828
• 19:22 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
• 19:14 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet - cookbook ran by andrew@bullseye
• 19:08 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 12:52 arturo: cleanup old network vlan interface names from /etc/network/interfaces in cloudnet1005/1006

2022-11-11

• 13:24 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: edad3915-b7c6-4b23-bb9c-ab13b04a41c5, use this to unset) (T319184) - cookbook ran by arturo@nostromo
• 13:23 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet (T319184) - cookbook ran by arturo@nostromo
• 13:17 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: a09a9868-8aae-4dc0-8510-a3923b703060, use this to unset) (T319184) - cookbook ran by arturo@nostromo
• 13:16 wm-bot2: Draining cloudvirt2003-dev.codfw.wmnet (T319184) - cookbook ran by arturo@nostromo

2022-11-10

• 16:19 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: 346013ec-ce4e-497e-ad65-2d215b14998c, use this to unset). (T319184) - cookbook ran by arturo@nostromo
• 16:18 wm-bot2: Draining 'cloudvirt2002-dev.codfw.wmnet'. (T319184) - cookbook ran by arturo@nostromo
• 16:13 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: a40a8487-22ee-4bc6-bbe4-694a615e3bf5, use this to unset). (T319184) - cookbook ran by arturo@nostromo

2022-11-08

• 11:17 taavi: backfilling security groups for metricsinfra access on all projects T288108

2022-11-07

• 21:01 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org (T305828) - cookbook ran by andrew@bullseye
• 20:50 wm-bot2: Upgraded and rebooted host cloudservices1005.wikimedia.org (T305828) - cookbook ran by andrew@bullseye
• 20:40 andrewbogott: upgrading eqiad1 designate to version 'yoga'

2022-11-04

• 17:57 andrewbogott: removing cinderv2 API endpoints from keystone catalog; this is deprecated and removed in Yoga. prep for T305828

2022-11-03

• 19:24 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet (T305828) - cookbook ran by andrew@bullseye
• 19:19 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet (T305828) - cookbook ran by andrew@bullseye
• 00:08 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org (T305828) - cookbook ran by andrew@bullseye

2022-11-02

• 23:39 wm-bot2: Upgraded and rebooted host cloudbackup1001-dev.eqiad.wmnet (T305828) - cookbook ran by andrew@bullseye
• 23:38 wm-bot2: Upgraded and rebooted host cloudnet2006-dev.codfw.wmnet (T305828) - cookbook ran by andrew@bullseye
• 23:29 wm-bot2: Upgraded and rebooted host cloudnet2005-dev.codfw.wmnet (T305828) - cookbook ran by andrew@bullseye
• 23:13 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org (T305828) - cookbook ran by andrew@bullseye
• 23:00 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org (T305828) - cookbook ran by andrew@bullseye
• 22:54 wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye
• 20:29 wm-bot2: Upgraded and rebooted host cloudcontrol2001-dev.wikimedia.org - cookbook ran by andrew@bullseye

2022-10-31

• 13:09 arturo: restart keepalived on all 4 cloudgw servers to run them with `-D` in /etc/default/keepalived to further debug T320975

2022-10-26

• 16:18 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:bf48880d-0c1b-4c2a-8e8b-778d28b16561) (T319446) - cookbook ran by dcaro@vulcanus
• 09:34 taavi: running wmcs-puppetcertleaks in delete mode
• 09:09 taavi: running wmcs-novastats-dnsleaks in delete mode

2022-10-25

• 16:03 arturo: [codfw1dev] T321220 root@cloudcontrol2001-dev:~# openstack subnet create magnum --no-dhcp --network 57017d7c-3817-429a-8aa3-b028de82cdcc --ip-version 4 --gateway auto --subnet-range 192.168.0.0/24
• 14:38 arturo: [codfw1dev] restart neutron-l3-agent in cloudnet2005-dev, it was dead after rabbit connectivity problems

2022-10-24

• 18:50 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by andrew@bullseye
• 18:50 wm-bot2: Finished rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by andrew@bullseye
• 18:47 wm-bot2: Rebooting node cloudcephmon1001.eqiad.wmnet - cookbook ran by andrew@bullseye
• 18:47 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by andrew@bullseye
• 18:38 wm-bot2: Rebooting the nodes cloudcephmon1001,cloudcephmon1002,cloudcephmon1003 - cookbook ran by andrew@bullseye
• 18:21 wm-bot2: Rebooting node cloudcephosd1001.eqiad.wmnet - cookbook ran by andrew@bullseye
• 18:21 wm-bot2: Rebooting the nodes cloudcephosd1001,cloudcephosd1002,cloudcephosd1003,cloudcephosd1004,cloudcephosd1005,cloudcephosd1006,cloudcephosd1007,cloudcephosd1008,cloudcephosd1009,cloudcephosd1010,cloudcephosd1011,cloudcephosd1012,cloudcephosd1013,cloudcephosd1014,cloudcephosd1015,cloudcephosd1016,cloudcephosd1017,cloudcephosd1018,cloudcephosd1019,cloudcephosd1020,cloudcephosd1021,cloudcephosd1022,cloudcephosd1023,cloudcephosd1024,cl

2022-10-20

• 23:23 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 23:23 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 23:20 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 23:20 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 23:19 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 23:16 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 23:07 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 23:07 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 23:06 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 23:06 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 23:03 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 23:03 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 101c41d1-d65d-4088-b6d2-eac859e45ef8, use this to unset). - cookbook ran by andrew@bullseye
• 23:03 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 23:03 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 23:03 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 23:03 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 23:01 wm-bot2: Unset cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 22:57 wm-bot2: Drained 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:51 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 7197ce34-cc57-4677-a1a9-05e20cd0dd80, use this to unset). - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 22:50 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 22:46 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:38 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 68e4cc1d-9def-444e-84a7-21b0b5adfa72, use this to unset). - cookbook ran by andrew@bullseye
• 22:37 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 73388c1e-369b-40af-a2c1-96936128c324, use this to unset). - cookbook ran by andrew@bullseye
• 22:37 wm-bot2: Set cloudvirt 'cloudvirt1026.eqiad.wmnet' maintenance (downtime id: 12feeca2-ea59-48e9-9235-1cecf5b384cd, use this to unset). - cookbook ran by andrew@bullseye
• 22:37 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:37 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:37 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:37 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 22:37 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:36 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 22:36 wm-bot2: Draining 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:36 wm-bot2: Safe rebooting 'cloudvirt1026.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:35 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 22:35 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 22:35 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 22:35 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 22:34 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 1b34587f-2770-4d2e-bb31-c8ad14632d39, use this to unset). - cookbook ran by andrew@bullseye
• 22:34 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:34 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:33 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:33 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:32 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:28 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 22:28 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 22:24 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:13 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: e75b00eb-7f58-4821-8139-3dfc6e97a92a, use this to unset). - cookbook ran by andrew@bullseye
• 22:12 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 9151511e-bcc0-4367-a976-7cff060308aa, use this to unset). - cookbook ran by andrew@bullseye
• 22:12 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 241c0bd9-c3cf-40e4-95dc-9b51d9823fe9, use this to unset). - cookbook ran by andrew@bullseye
• 22:12 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: 7a53c274-1d5f-4c94-9a0d-721c3e8f7239, use this to unset). - cookbook ran by andrew@bullseye
• 22:12 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:12 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:11 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:11 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:11 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:11 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:11 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:11 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 22:10 wm-bot2: Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 22:10 wm-bot2: Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 22:06 wm-bot2: Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:58 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 21:58 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 21:58 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 21:58 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 21:55 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 21:55 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 21:54 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:54 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:51 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:38 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 0b75b44c-1efe-4edf-8f5e-42a67e8d3b13, use this to unset). - cookbook ran by andrew@bullseye
• 21:38 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: aeb9c3a6-961a-4873-85d4-929248aebb8b, use this to unset). - cookbook ran by andrew@bullseye
• 21:38 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: eb14cc04-3293-4cf8-a46f-1f87d8b0bcc4, use this to unset). - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 84a3cd99-2643-495a-86b6-7ae80b11a30d, use this to unset). - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:37 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:36 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:32 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 21:32 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 21:28 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:28 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 01a0b69c-2e27-4331-9635-403a668dac29, use this to unset). - cookbook ran by andrew@bullseye
• 21:27 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:27 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:23 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: ea29f65f-6d86-4568-8db9-a85faa827447, use this to unset). - cookbook ran by andrew@bullseye
• 21:22 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:22 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:19 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 21:19 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 21:19 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 21:19 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 21:18 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 21:18 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 21:16 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:15 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:14 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:01 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: d6b6bb4d-c7c0-4bd2-9a02-dee9a5ec6a3e, use this to unset). - cookbook ran by andrew@bullseye
• 21:01 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 4e109149-7d67-403f-8b21-f829235ea491, use this to unset). - cookbook ran by andrew@bullseye
• 21:01 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: 341f4b8c-6d6e-4190-9f2d-a1a1483abadd, use this to unset). - cookbook ran by andrew@bullseye
• 21:01 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: f76e8f14-920e-4d51-a44e-321a55dcb0c5, use this to unset). - cookbook ran by andrew@bullseye
• 21:00 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:00 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:00 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:00 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:00 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:00 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:00 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 21:00 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:56 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 20:56 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 20:54 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 20:35 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:34 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:32 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 0b0d8090-4d31-4d43-8545-c25209e2ef58, use this to unset). - cookbook ran by andrew@bullseye
• 20:31 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:31 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:21 wm-bot2: Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance (downtime id: 3d928554-a79c-419a-bcc4-c8d63791d8e7, use this to unset). - cookbook ran by andrew@bullseye
• 20:21 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 3d3c8aa5-45f7-429e-a9d7-b5181b29dc13, use this to unset). - cookbook ran by andrew@bullseye
• 20:21 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: b9027020-8a90-4c40-b0e6-6c8767f87917, use this to unset). - cookbook ran by andrew@bullseye
• 20:21 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: f0d60fab-64e9-4da2-826c-229d31d0fbc2, use this to unset). - cookbook ran by andrew@bullseye
• 20:21 wm-bot2: Draining 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:21 wm-bot2: Safe rebooting 'cloudvirt1043.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Safe rebooting 'cloudvirt1042.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Draining 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Safe rebooting 'cloudvirt1041.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Draining 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Safe rebooting 'cloudvirt1040.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Safe reboot of 'cloudvirt1044.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 20:19 wm-bot2: Safe reboot of 'cloudvirt1047.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 20:19 wm-bot2: Unset cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 20:18 wm-bot2: Safe reboot of 'cloudvirt1046.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 20:18 wm-bot2: Unset cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 20:16 wm-bot2: Drained 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:15 wm-bot2: Drained 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:15 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 20:15 wm-bot2: Unset cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 20:14 wm-bot2: Drained 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 20:11 wm-bot2: Drained 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:55 wm-bot2: Set cloudvirt 'cloudvirt1046.eqiad.wmnet' maintenance (downtime id: b65da7e2-9fd2-4a1e-8dd4-88ca65936ae4, use this to unset). - cookbook ran by andrew@bullseye
• 19:55 wm-bot2: Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance (downtime id: cbbf114c-9a4c-4dd2-9b58-50da8bd896ca, use this to unset). - cookbook ran by andrew@bullseye
• 19:55 wm-bot2: Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance (downtime id: 5969beaf-72bf-4af9-ae4d-8e5c3331e78e, use this to unset). - cookbook ran by andrew@bullseye
• 19:55 wm-bot2: Set cloudvirt 'cloudvirt1047.eqiad.wmnet' maintenance (downtime id: b8f7389d-79b4-411d-b3d5-ec8f92eba101, use this to unset). - cookbook ran by andrew@bullseye
• 19:54 wm-bot2: Draining 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:54 wm-bot2: Safe rebooting 'cloudvirt1047.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:54 wm-bot2: Draining 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:54 wm-bot2: Safe rebooting 'cloudvirt1046.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:54 wm-bot2: Draining 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:54 wm-bot2: Safe rebooting 'cloudvirt1045.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:54 wm-bot2: Draining 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:54 wm-bot2: Safe rebooting 'cloudvirt1044.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:53 wm-bot2: Safe reboot of 'cloudvirt1051.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 19:53 wm-bot2: Unset cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 19:51 wm-bot2: Safe reboot of 'cloudvirt1049.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 19:51 wm-bot2: Unset cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 19:50 wm-bot2: Safe reboot of 'cloudvirt1048.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 19:50 wm-bot2: Unset cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 19:49 wm-bot2: Drained 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:36 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: c5dbfa7b-72fc-4156-8257-af224a725b78, use this to unset). - cookbook ran by andrew@bullseye
• 19:36 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:36 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:33 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:33 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:30 wm-bot2: Set cloudvirt 'cloudvirt1048.eqiad.wmnet' maintenance (downtime id: c1a3e92c-cb04-46e4-ac5d-784c79601b05, use this to unset). - cookbook ran by andrew@bullseye
• 19:30 wm-bot2: Set cloudvirt 'cloudvirt1049.eqiad.wmnet' maintenance (downtime id: 62fdc4ee-c8d5-4892-aeb9-a681cdcbc84b, use this to unset). - cookbook ran by andrew@bullseye
• 19:29 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 9ec407cd-5c00-407a-a57d-794f6c68f947, use this to unset). - cookbook ran by andrew@bullseye
• 19:29 wm-bot2: Set cloudvirt 'cloudvirt1051.eqiad.wmnet' maintenance (downtime id: 712683ea-d58f-484b-8efb-89d1c21aa0e3, use this to unset). - cookbook ran by andrew@bullseye
• 19:29 wm-bot2: Draining 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:29 wm-bot2: Safe rebooting 'cloudvirt1048.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:29 wm-bot2: Draining 'cloudvirt1049.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:29 wm-bot2: Safe rebooting 'cloudvirt1049.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:28 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:28 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:28 wm-bot2: Draining 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:28 wm-bot2: Safe rebooting 'cloudvirt1051.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:25 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye
• 19:25 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye
• 19:21 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:04 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 18fae8d8-7353-4f67-90d7-8df9b3fb1ccb, use this to unset). - cookbook ran by andrew@bullseye
• 19:03 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:03 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:01 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 3d3cffa3-abc6-4901-83d9-3bcc4b02fd3c, use this to unset). - cookbook ran by andrew@bullseye
• 19:00 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 19:00 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:54 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:54 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:51 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:51 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:51 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:50 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:50 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:50 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:46 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye
• 18:46 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. - cookbook ran by andrew@bullseye

2022-10-15

• 17:38 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki Slevinski # T320867

2022-10-13

• 12:19 wm-bot2: OSDs (['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034']) upgraded successfully B-) (T309786) - cookbook ran by dcaro@vulcanus
• 11:31 wm-bot2: Upgrading OSDs and rebooting the nodes ['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034'] (T309786) - cookbook ran by dcaro@vulcanus
• 11:30 wm-bot2: OSDs (['cloudcephosd1025', 'cloudcephosd1026']) upgraded successfully B-) (T309786) - cookbook ran by dcaro@vulcanus

2022-10-10

• 14:01 dcaro: test2
• 01:22 andrewbogott: restarting designate-sink on cloudservices100[45], possible example of T316614

2022-10-09

• 12:04 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki DatGuy # T320301

2022-10-07

• 13:40 andrewbogott: dhinus is resetting rabbitmq cluster in an attempt to resolve a suspected (by Andrew) split-brain
• 11:33 arturo: rabbitmq-server.service @ cloudrabbit1002 is again up and running (T320232)
• 10:24 arturo: stopping rabbitmq-server.service @ cloudrabbit1002 (T320232)
• 10:19 arturo: restarting nova-conductor in all 3 cloudcontrols (T320232)
• 09:45 arturo: restarting rabbitmq-server.service @ cloudrabbit1002 (T320232)

2022-10-06

• 15:55 arturo: cloudnet1005 & cloudnet1006 now in service. Secom cloudnet1003 & cloudnet1004. Drop neutron agents, etc. (T316284)
• 11:54 arturo: rebooting cloudnet1005/1006 to see if they have the right network config (T316284)
• 11:50 arturo: set neutron l3 agents on cloudnet1005/1006 as down `root@cloudcontrol1005:~# neutron agent-update --admin-state-down <uuid>` (T316284)
• 11:40 arturo: [codfw1dev] rebooting both network nodes to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/839492
• 10:14 arturo: [codfw1dev] restart neutron-l3-agent on cloudnet2006-dev, it was dead

2022-10-05

• 14:40 wm-bot2: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) (T319418) - cookbook ran by fran@wmf3169
• 14:40 wm-bot2: Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster (T319418) - cookbook ran by fran@wmf3169
• 14:28 arturo: adding cloudinstances2b-gw router to l3 agents on cloudnet1005/1006 (T316284)
• 13:11 wm-bot2: Added 1 new OSDs ['cloudcephosd1034.eqiad.wmnet'] (T314870) - cookbook ran by fran@wmf3169
• 13:11 wm-bot2: Added OSD cloudcephosd1034.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169
• 13:02 wm-bot2: Finished rebooting node cloudcephosd1034.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169
• 12:58 wm-bot2: Rebooting node cloudcephosd1034.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169
• 12:58 wm-bot2: Adding OSD cloudcephosd1034.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169
• 12:58 wm-bot2: Adding new OSDs ['cloudcephosd1034.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@wmf3169

2022-10-04

• 16:40 wm-bot2: Added 1 new OSDs ['cloudcephosd1033.eqiad.wmnet'] (T314870) - cookbook ran by fran@wmf3169
• 16:40 wm-bot2: Added OSD cloudcephosd1033.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169
• 14:34 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169
• 14:30 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169
• 14:30 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169
• 14:30 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@wmf3169
• 14:20 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
• 14:17 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
• 14:16 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 14:16 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
• 10:59 wm-bot2: Finished rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
• 10:56 wm-bot2: Rebooting node cloudcephosd1033.eqiad.wmnet - cookbook ran by fran@wmf3169
• 10:55 wm-bot2: Adding OSD cloudcephosd1033.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 10:55 wm-bot2: Adding new OSDs ['cloudcephosd1033.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

2022-09-30

• 14:52 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169
• 14:52 wm-bot2: Added OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 14:48 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 14:48 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
• 14:16 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. (T319025) - cookbook ran by andrew@buster
• 14:15 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: 64eac5c6-4b1d-4269-98fd-8e5bed42ce40, use this to unset). (T319025) - cookbook ran by andrew@buster
• 14:15 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. (T319025) - cookbook ran by andrew@buster
• 14:15 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. (T319025) - cookbook ran by andrew@buster
• 14:09 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
• 14:05 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: fb99c967-b974-4314-a2fa-31ed0e883dd3, use this to unset). - cookbook ran by andrew@buster
• 14:04 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. - cookbook ran by andrew@buster
• 13:30 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169
• 13:30 wm-bot2: Added OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 13:26 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 13:26 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
• 13:16 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 13:16 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
• 13:08 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 13:08 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
• 12:45 wm-bot2: Finished rebooting node cloudcephosd1031.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169
• 12:42 wm-bot2: Rebooting node cloudcephosd1031.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169
• 12:41 wm-bot2: Adding OSD cloudcephosd1031.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169
• 12:41 wm-bot2: Adding new OSDs ['cloudcephosd1031.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@wmf3169
• 12:26 arturo: [codfw1dev] cloudnet2005/2006-dev are now on a single NIC setup (T318824)
• 11:44 arturo: sysctl change (cleanup) on cloudnet1003/1004

2022-09-27

• 10:48 wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169
• 10:35 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169
• 10:32 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169
• 10:32 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169
• 10:32 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@wmf3169
• 10:26 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
• 10:23 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
• 10:22 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 10:22 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
• 10:17 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
• 10:14 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
• 10:14 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 10:14 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
• 10:09 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
• 10:05 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
• 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
• 10:05 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 10:05 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169

2022-09-26

• 18:37 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
• 18:37 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
• 18:33 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
• 18:31 andrewbogott: rebooting cloudvirt1028 for T317391
• 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1002.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1003.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 18:28 wm-bot2: Safe reboot of 'cloudvirt-wdqs1001.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 18:28 wm-bot2: Unset cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 18:25 wm-bot2: Drained 'cloudvirt-wdqs1003.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1003.eqiad.wmnet' maintenance (downtime id: 6d1c4c53-76b9-493f-8c44-eb0413dfb9d0, use this to unset). (T317391) - cookbook ran by andrew@buster
• 18:25 wm-bot2: Drained 'cloudvirt-wdqs1002.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1002.eqiad.wmnet' maintenance (downtime id: 6bb80b65-616c-4e45-be4f-d2cdd8abb2bf, use this to unset). (T317391) - cookbook ran by andrew@buster
• 18:25 wm-bot2: Drained 'cloudvirt-wdqs1001.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 18:25 wm-bot2: Set cloudvirt 'cloudvirt-wdqs1001.eqiad.wmnet' maintenance (downtime id: a3bba7e7-bcd3-482d-a486-06b6f53fd899, use this to unset). (T317391) - cookbook ran by andrew@buster
• 18:24 wm-bot2: Draining 'cloudvirt-wdqs1003.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1003.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 18:24 wm-bot2: Draining 'cloudvirt-wdqs1002.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1002.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 18:24 wm-bot2: Draining 'cloudvirt-wdqs1001.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 18:24 wm-bot2: Safe rebooting 'cloudvirt-wdqs1001.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 18:18 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 8fb59651-fd42-4aee-a978-36bc7f79b4d5, use this to unset). - cookbook ran by andrew@buster
• 18:18 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
• 18:17 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. - cookbook ran by andrew@buster
• 18:01 wm-bot2: Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 18:01 wm-bot2: Unset cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 17:58 wm-bot2: Drained 'cloudvirt1023.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:51 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 17:51 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 17:47 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
• 17:47 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
• 17:47 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:45 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:38 wm-bot2: Set cloudvirt 'cloudvirt1023.eqiad.wmnet' maintenance (downtime id: abfc35bb-331e-4d7e-bb92-35e675abce32, use this to unset). (T317391) - cookbook ran by andrew@buster
• 17:37 wm-bot2: Draining 'cloudvirt1023.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:37 wm-bot2: Safe rebooting 'cloudvirt1023.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:37 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 17:37 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 17:34 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:33 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: d4fe6ea8-76eb-45f7-b6cf-66f54ba9801c, use this to unset). (T317391) - cookbook ran by andrew@buster
• 17:33 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: c0671bda-9aae-4960-85be-58aaa9c8e021, use this to unset). - cookbook ran by andrew@buster
• 17:32 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:32 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:32 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:32 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:31 wm-bot2: Safe reboot of 'cloudvirt1029.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
• 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
• 17:31 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 17:31 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 17:28 wm-bot2: Drained 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:28 wm-bot2: Drained 'cloudvirt1030.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:24 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: be6559f9-4d72-4492-b9b9-23c636d6554e, use this to unset). (T317391) - cookbook ran by andrew@buster
• 17:23 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:23 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:21 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 237dfb1b-5382-408d-aa35-1aa1cfe4c5af, use this to unset). (T317391) - cookbook ran by andrew@buster
• 17:20 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:20 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:20 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 17:20 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 17:17 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: 088b97cd-0cf2-4c27-a67f-077ffa1c1c5e, use this to unset). - cookbook ran by andrew@buster
• 17:16 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:16 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 91625000-aa6d-4887-bf34-ef7785a4af4a, use this to unset). (T317391) - cookbook ran by andrew@buster
• 17:16 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:16 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:16 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:15 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:15 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
• 17:15 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster
• 17:15 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 7364c4fe-9f6b-4539-b6fa-1e768b602a1b, use this to unset). (T317391) - cookbook ran by andrew@buster
• 17:14 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:14 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:12 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:12 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: cb8f8a21-73c0-4f54-9412-074d82582cb4, use this to unset). - cookbook ran by andrew@buster
• 17:11 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:11 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:09 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: d23dabd2-3bfc-4ce0-9533-cd1cf910f8e1, use this to unset). - cookbook ran by andrew@buster
• 17:08 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:08 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:05 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: 4d7ddae1-f621-4dd5-8616-29b7699b692f, use this to unset). (T317391) - cookbook ran by andrew@buster
• 17:04 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:04 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:04 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 4628dd6f-5b49-417f-b6ab-5c41992192b4, use this to unset). - cookbook ran by andrew@buster
• 17:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:02 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: c95529cf-b2a5-4553-bd6e-6ad4dcb2a105, use this to unset). (T317391) - cookbook ran by andrew@buster
• 17:01 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 36c68112-a964-4a7c-a093-8a6d481dea7d, use this to unset). - cookbook ran by andrew@buster
• 17:01 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:01 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 17:00 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
• 17:00 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. - cookbook ran by andrew@buster
• 16:49 wm-bot2: Safe reboot of 'cloudvirt1050.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 16:49 wm-bot2: Unset cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 16:45 wm-bot2: Drained 'cloudvirt1050.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 16:31 wm-bot2: Set cloudvirt 'cloudvirt1050.eqiad.wmnet' maintenance (downtime id: 1244c159-65bb-476e-a702-8f43c253e499, use this to unset). (T317391) - cookbook ran by andrew@buster
• 16:30 wm-bot2: Draining 'cloudvirt1050.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 16:30 wm-bot2: Safe rebooting 'cloudvirt1050.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:22 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
• 13:19 wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@wmf3169
• 13:18 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by fran@wmf3169
• 13:18 wm-bot2: Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster - cookbook ran by fran@wmf3169
• 12:32 dcaro: Changed the collation of labsdbaccount db to utf8mb4_bin (T318047)
• 10:14 arturo: deployed new version of maintain-dbusers (T318047)

2022-09-25

• 15:06 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 15:06 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: a050e47f-3a63-41ff-accb-3993c1e8b593, use this to unset). (T317391) - cookbook ran by andrew@buster
• 15:05 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 15:05 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 15:03 wm-bot2: Safe reboot of 'cloudvirt1052.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 15:03 wm-bot2: Unset cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 14:58 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:58 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 1a49d773-4dc9-4a6f-bd49-8663db77ce66, use this to unset). (T317391) - cookbook ran by andrew@buster
• 14:57 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:57 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:54 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: ab52dd42-68a6-4159-b345-e5625d209b57, use this to unset). (T317391) - cookbook ran by andrew@buster
• 14:53 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:53 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:37 wm-bot2: Set cloudvirt 'cloudvirt1052.eqiad.wmnet' maintenance (downtime id: 10337415-3095-4834-8538-b3a64bd6b18d, use this to unset). (T317391) - cookbook ran by andrew@buster
• 14:37 wm-bot2: Draining 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:37 wm-bot2: Safe rebooting 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:53 wm-bot2: Drained 'cloudvirt1053.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:53 wm-bot2: Set cloudvirt 'cloudvirt1053.eqiad.wmnet' maintenance (downtime id: 4cdbed3a-3775-4913-8c3b-afd57ae1ca55, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:52 wm-bot2: Draining 'cloudvirt1053.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:52 wm-bot2: Safe rebooting 'cloudvirt1053.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster

2022-09-24

• 17:37 andrewbogott: restarting neutron api on cloudcontrol1006; cause of outage unknown
• 17:35 andrewbogott: restarting neutron-linuxbridge-agent on cloudvirt1022

2022-09-22

• 15:14 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 15:14 wm-bot2: Unset cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 15:10 wm-bot2: Drained 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 15:10 wm-bot2: Set cloudvirt 'cloudvirt1025.eqiad.wmnet' maintenance (downtime id: 04849437-a304-45a1-a037-538741a2f801, use this to unset). (T317391) - cookbook ran by andrew@buster
• 15:09 wm-bot2: Draining 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 15:09 wm-bot2: Safe rebooting 'cloudvirt1025.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 15:07 wm-bot2: Safe reboot of 'cloudvirt1017.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 15:07 wm-bot2: Unset cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 15:04 wm-bot2: Drained 'cloudvirt1017.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 15:03 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 8810a919-61e7-4ba5-af7f-c41d1e1c8c8b, use this to unset). (T317391) - cookbook ran by andrew@buster
• 15:03 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 15:03 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:59 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 14:59 wm-bot2: Unset cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 14:56 wm-bot2: Drained 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:42 wm-bot2: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance (downtime id: b6f8e0b6-f35c-41fd-8035-6efde61db3a3, use this to unset). (T317391) - cookbook ran by andrew@buster
• 14:42 wm-bot2: Safe reboot of 'cloudvirt1027.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 14:42 wm-bot2: Unset cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 14:42 wm-bot2: Draining 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:42 wm-bot2: Safe rebooting 'cloudvirt1021.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:41 wm-bot2: Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 14:41 wm-bot2: Unset cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 14:41 wm-bot2: Set cloudvirt 'cloudvirt1017.eqiad.wmnet' maintenance (downtime id: 784bff2c-8160-44dc-9bcf-ff23055a0527, use this to unset). (T317391) - cookbook ran by andrew@buster
• 14:40 wm-bot2: Draining 'cloudvirt1017.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:40 wm-bot2: Safe rebooting 'cloudvirt1017.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:39 wm-bot2: Drained 'cloudvirt1027.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:38 wm-bot2: Drained 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:38 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 21152e9e-bc67-4024-b68e-fd671c398642, use this to unset). (T317391) - cookbook ran by andrew@buster
• 14:37 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:37 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:37 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: f07ddb1d-e8da-43f3-8140-bc66789d37c8, use this to unset). (T317391) - cookbook ran by andrew@buster
• 14:36 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:36 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:33 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 14:33 wm-bot2: Unset cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 14:30 wm-bot2: Drained 'cloudvirt1024.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:16 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 4e4b03fc-e3f1-440e-9097-cce5edaa1f08, use this to unset). (T317391) - cookbook ran by andrew@buster
• 14:16 wm-bot2: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance (downtime id: d4bb2160-f492-4ddc-a5b7-eeee37007d14, use this to unset). (T317391) - cookbook ran by andrew@buster
• 14:16 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance (downtime id: 9bf00a6a-3697-4201-9a6e-76d499eccfa1, use this to unset). (T317391) - cookbook ran by andrew@buster
• 14:15 wm-bot2: Draining 'cloudvirt1027.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:15 wm-bot2: Safe rebooting 'cloudvirt1027.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:15 wm-bot2: Draining 'cloudvirt1024.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:15 wm-bot2: Safe rebooting 'cloudvirt1024.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:15 wm-bot2: Draining 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:15 wm-bot2: Safe rebooting 'cloudvirt1022.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 14:13 wm-bot2: Safe reboot of 'cloudvirt1030.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 14:13 wm-bot2: Unset cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 13:47 wm-bot2: Set cloudvirt 'cloudvirt1029.eqiad.wmnet' maintenance (downtime id: f2888490-1804-4b23-b7b0-677853fb9ef7, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:46 wm-bot2: Draining 'cloudvirt1029.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:46 wm-bot2: Safe rebooting 'cloudvirt1029.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:46 wm-bot2: Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance (downtime id: 8a3e8b0a-ced3-4667-a121-66df673c7ed8, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:46 wm-bot2: Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 13:46 wm-bot2: Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 13:45 wm-bot2: Draining 'cloudvirt1030.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:45 wm-bot2: Safe rebooting 'cloudvirt1030.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:45 wm-bot2: Safe reboot of 'cloudvirt1032.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 13:45 wm-bot2: Unset cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 13:43 wm-bot2: Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 35c761b3-f0de-4b23-9b11-2ed2e474c89f, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:42 wm-bot2: Draining 'cloudvirt1031.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:42 wm-bot2: Safe rebooting 'cloudvirt1031.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:42 wm-bot2: Drained 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:42 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 6121a0e1-754a-47fa-b51b-265f6386f396, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:41 wm-bot2: Drained 'cloudvirt1032.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:41 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:41 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:40 wm-bot2: Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 13:40 wm-bot2: Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 13:39 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 189ae1bb-c69d-4eb0-aee9-90ab1f492aa8, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:38 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:38 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:36 wm-bot2: Drained 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:35 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: acc53776-68bf-47a4-bb82-c571b3df2945, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:35 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:35 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:31 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 07770ae7-3e7e-4615-8174-513767f95b1d, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:30 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:30 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:29 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 7ae20a7d-8bff-4bc2-91c5-b0005c8164d1, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:29 wm-bot2: Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance (downtime id: b596f855-8019-49d2-9657-0d513ad7acb5, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:29 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:29 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:28 wm-bot2: Draining 'cloudvirt1032.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:28 wm-bot2: Safe rebooting 'cloudvirt1032.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:28 wm-bot2: Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 13:28 wm-bot2: Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 13:25 wm-bot2: Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 9172c785-d2d6-46e5-bed3-2f49d1033f78, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:24 wm-bot2: Draining 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:24 wm-bot2: Safe rebooting 'cloudvirt1033.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:24 wm-bot2: Drained 'cloudvirt1034.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:23 wm-bot2: Safe reboot of 'cloudvirt1036.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 13:23 wm-bot2: Unset cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 13:19 wm-bot2: Drained 'cloudvirt1036.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:04 wm-bot2: Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance (downtime id: 85b4e0e2-e635-485f-9ce5-341981b3887f, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:04 wm-bot2: Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: 9494c283-02c8-42da-98c3-139b4d119ef8, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:04 wm-bot2: Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: d181f301-249f-4b5a-bd85-47d87262d0ed, use this to unset). (T317391) - cookbook ran by andrew@buster
• 13:03 wm-bot2: Draining 'cloudvirt1034.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:03 wm-bot2: Safe rebooting 'cloudvirt1034.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:03 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:03 wm-bot2: Safe rebooting 'cloudvirt1035.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:03 wm-bot2: Draining 'cloudvirt1036.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 13:03 wm-bot2: Safe rebooting 'cloudvirt1036.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster

2022-09-20

• 21:02 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 21:02 wm-bot2: Unset cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 20:58 wm-bot2: Drained 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:58 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 1c4246a4-9cee-4423-8cd1-4f52f61d503f, use this to unset). (T317391) - cookbook ran by andrew@buster
• 20:57 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:57 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:57 wm-bot2: Safe reboot of 'cloudvirt1038.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 20:57 wm-bot2: Unset cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 20:53 wm-bot2: Drained 'cloudvirt1038.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:50 wm-bot2: Safe reboot of 'cloudvirt1039.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 20:50 wm-bot2: Unset cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 20:49 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 82665ecc-4431-48fe-b255-7e9d518be217, use this to unset). (T317391) - cookbook ran by andrew@buster
• 20:48 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:48 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:47 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: cd284562-e3b0-4504-9977-2b18032bbf3c, use this to unset). (T317391) - cookbook ran by andrew@buster
• 20:46 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:46 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:46 wm-bot2: Drained 'cloudvirt1039.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:45 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 8fb2a646-16a5-4183-94a1-ee6bbbf029fa, use this to unset). (T317391) - cookbook ran by andrew@buster
• 20:44 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:44 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:32 wm-bot2: Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance (downtime id: d561fe45-1582-4cd8-bbc9-a356c39f3330, use this to unset). (T317391) - cookbook ran by andrew@buster
• 20:32 wm-bot2: Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance (downtime id: 351c365c-0228-4907-a279-01795b1e62ce, use this to unset). (T317391) - cookbook ran by andrew@buster
• 20:32 wm-bot2: Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance (downtime id: 3beec1dd-0132-4f5c-adce-5a7dc56060b6, use this to unset). (T317391) - cookbook ran by andrew@buster
• 20:31 wm-bot2: Draining 'cloudvirt1039.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:31 wm-bot2: Safe rebooting 'cloudvirt1039.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:31 wm-bot2: Draining 'cloudvirt1038.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:31 wm-bot2: Safe rebooting 'cloudvirt1038.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:31 wm-bot2: Draining 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:31 wm-bot2: Safe rebooting 'cloudvirt1037.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:30 wm-bot2: Safe reboot of 'cloudvirt1040.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 20:29 wm-bot2: Unset cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 20:28 wm-bot2: Safe reboot of 'cloudvirt1041.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 20:28 wm-bot2: Unset cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 20:26 wm-bot2: Drained 'cloudvirt1040.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:24 wm-bot2: Drained 'cloudvirt1041.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:20 wm-bot2: Safe reboot of 'cloudvirt1042.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster
• 20:20 wm-bot2: Unset cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. (T317391) - cookbook ran by andrew@buster
• 20:16 wm-bot2: Drained 'cloudvirt1042.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster
• 20:07 wm-bot2: Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance (downtime id: 353a8527-ad5e-4898-937c-303d16801e28, use this to unset). (T317391) - cookbook ran by andrew@buster
• 20:07 wm-bot2: Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance (downtime id: fc235146-f070-4723-9503-e20cbb877755, use this to unset). (T317391) - cookbook ran by andrew@buster
• 20:07 wm-bot2: Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance (downtime id: cdf712ac-c083-4a63-af83-514ca5cdc76d, use this to unset). (T317391) - cookbook ran by andrew@buster
• 20:06 wm-bot2: Draining 'cloudvirt1042.eqiad.wmnet'. (