Jump to content

Nova Resource:Tools/SAL

From Wikitech

2024-11-12

  • 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
  • 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
  • 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
  • 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice

2024-11-11

  • 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud (T352206)
  • 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud (T352206)
  • 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud (T352206)
  • 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud (T352206)
  • 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' (T352206)
  • 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' (T352206)
  • 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

2024-11-10

  • 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T362867)
  • 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 (T362867)
  • 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T362867)

2024-11-06

  • 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 (T379139)
  • 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 (T379139)
  • 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice
  • 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
  • 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
  • 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

2024-11-05

  • 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
  • 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
  • 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics
  • 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
  • 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
  • 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
  • 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico

2024-11-04

  • 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
  • 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
  • 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
  • 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
  • 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
  • 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 (T362867)
  • 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 (T362867)
  • 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 (T362867)
  • 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 (T362867)
  • 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 (T362867)
  • 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 (T362867)
  • 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 (T362867)
  • 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 (T362867)
  • 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 (T362867)
  • 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 (T362867)
  • 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 (T362867)
  • 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 (T362867)
  • 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 (T362867)
  • 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 (T362867)
  • 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 (T362867)
  • 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 (T362867)
  • 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 (T362867)
  • 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 (T362867)
  • 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 (T362867)
  • 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 (T362867)
  • 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 (T362867)
  • 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 (T362867)
  • 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 (T362867)
  • 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 (T362867)
  • 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 (T362867)
  • 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 (T362867)
  • 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 (T362867)
  • 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 (T362867)
  • 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 (T362867)
  • 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 (T362867)
  • 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 (T362867)
  • 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 (T362867)
  • 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 (T362867)
  • 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 (T362867)
  • 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 (T362867)
  • 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 (T362867)
  • 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 (T362867)
  • 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 (T362867)
  • 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 (T362867)
  • 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 (T362867)
  • 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 (T362867)
  • 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 (T362867)
  • 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 (T362867)
  • 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 (T362867)
  • 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 (T362867)
  • 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 (T362867)
  • 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 (T362867)
  • 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 (T362867)
  • 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 (T362867)
  • 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 (T362867)
  • 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 (T362867)
  • 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 (T362867)
  • 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 (T362867)
  • 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 (T362867)
  • 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 (T362867)
  • 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 (T362867)
  • 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 (T362867)
  • 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 (T362867)
  • 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 (T362867)
  • 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 (T362867)
  • 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 (T362867)
  • 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 (T362867)
  • 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 (T362867)
  • 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 (T362867)
  • 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 (T362867)
  • 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 (T362867)
  • 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 (T362867)
  • 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 (T362867)
  • 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 (T362867)
  • 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 (T362867)
  • 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 (T362867)
  • 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 (T362867)
  • 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 (T362867)
  • 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 (T362867)
  • 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 (T362867)
  • 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 (T362867)
  • 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 (T362867)
  • 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 (T362867)
  • 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 (T362867)
  • 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 (T362867)
  • 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 (T362867)
  • 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 (T362867)
  • 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 (T362867)
  • 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 (T362867)
  • 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 (T362867)
  • 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 (T362867)
  • 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 (T362867)
  • 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 (T362867)
  • 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 (T362867)
  • 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 (T362867)
  • 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 (T362867)
  • 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 (T362867)
  • 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 (T362867)
  • 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 (T362867)
  • 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 (T362867)
  • 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 (T362867)
  • 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 (T362867)
  • 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 (T362867)
  • 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 (T362867)
  • 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 (T362867)
  • 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 (T362867)
  • 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 (T362867)
  • 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 (T362867)
  • 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 (T362867)
  • 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 (T362867)
  • 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 (T362867)
  • 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 (T362867)
  • 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 (T362867)
  • 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 (T362867)
  • 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 (T362867)
  • 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 (T362867)
  • 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 (T362867)
  • 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 (T362867)
  • 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 (T362867)
  • 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 (T362867)
  • 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
  • 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 (T362867)
  • 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 (T362867)
  • 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 (T362867)
  • 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 (T362867)
  • 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 (T362867)
  • 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867)
  • 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867)
  • 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
  • 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
  • 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
  • 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867)
  • 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867)
  • 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867)
  • 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867)
  • 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 (T362867)
  • 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 (T362867)
  • 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 (T362867)
  • 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 (T362867)
  • 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 (T362867)
  • 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 (T362867)
  • 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 (T362867)
  • 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 (T362867)
  • 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 (T362867)
  • 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 (T362867)
  • 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 (T362867)
  • 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 (T362867)
  • 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 (T362867)
  • 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 (T362867)
  • 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 (T362867)
  • 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
  • 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
  • 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 (T362867)
  • 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
  • 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
  • 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 (T362867)
  • 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api
  • 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
  • 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 (T362867)
  • 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 (T362867)
  • 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 (T362867)
  • 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 (T362867)
  • 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 (T362867)
  • 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
  • 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
  • 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
  • 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
  • 10:42 dcaro: added api.svc.toolforge.org dns record entry
  • 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
  • 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
  • 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
  • 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
  • 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
  • 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
  • 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
  • 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
  • 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

2024-10-22

  • 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
  • 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23
  • 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
  • 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23
  • 09:05 arturo: restart puppetserver service for T377803

2024-10-16

  • 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

2024-10-15

  • 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
  • 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

2024-10-14

  • 09:14 dcaro: migrating pipelineruns stored versions to v1 (T376710)
  • 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
  • 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
  • 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9
  • 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

2024-10-09

  • 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
  • 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
  • 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
  • 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

2024-10-08

  • 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld (T376710)
  • 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld (T376710)
  • 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup
  • 12:27 dcaro: upgrade finished, build actions have become slower than usual (T376710), running tests and investigating
  • 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress T374908
  • 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
  • 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
  • 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
  • 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

2024-10-04

  • 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api

2024-10-02

  • 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers
  • 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

2024-10-01

  • 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
  • 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api
  • 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 10:28 dcaro: updated ci image with latest precommit versions
  • 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission
  • 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

2024-09-30

  • 18:25 taavi: run striker migrations T359428

2024-09-28

  • 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
  • 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli

2024-09-27

  • 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
  • 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

2024-09-26

  • 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
  • 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
  • 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
  • 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
  • 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission
  • 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
  • 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
  • 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
  • 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
  • 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
  • 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli
  • 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli
  • 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld
  • 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld

2024-09-25

  • 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7
  • 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7

2024-09-24

  • 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T375157)
  • 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T375157)
  • 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno (T359641)
  • 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno (T359641)

2024-09-20

  • 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T341066)
  • 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)
  • 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico (T341066)
  • 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)
  • 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico (T341066)
  • 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)
  • 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 (T359641)
  • 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 (T359641)
  • 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 (T359641)
  • 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 (T359641)
  • 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 (T359641)
  • 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 (T359641)
  • 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 (T359641)
  • 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
  • 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics (T359641)
  • 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics (T359641)

2024-09-19

  • 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) (T359641)
  • 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 (T359641)
  • 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 (T359641)
  • 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 (T359641)
  • 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) (T359641)
  • 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 (T359641)
  • 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) (T359641)
  • 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 (T359641)
  • 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli (T341066)
  • 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli (T341066)
  • 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T341066)
  • 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)
  • 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api (T341066)
  • 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)
  • 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api
  • 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)
  • 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)
  • 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network
  • 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api (T341066)
  • 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)
  • 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api (T341066)
  • 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)
  • 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api (T341066)
  • 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)
  • 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api (T341066)
  • 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)
  • 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api (T341066)
  • 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

2024-09-17

  • 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 (T359641)
  • 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 (T359641)
  • 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud (T359641)
  • 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 (T359641)
  • 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud (T359641)
  • 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 (T359641)
  • 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud (T359641)
  • 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud (T359641)
  • 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 (T359641)
  • 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 (T359641)
  • 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 (T359641)
  • 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 (T359641)
  • 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 (T359641)
  • 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 (T359641)
  • 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64
  • 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63
  • 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 (T359641)
  • 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster
  • 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 (T359641)
  • 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster
  • 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster
  • 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62
  • 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60
  • 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 (T359641)
  • 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 (T359641)
  • 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster
  • 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster
  • 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster
  • 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster (T359641)
  • 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6
  • 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56
  • 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster
  • 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 (T359641)
  • 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 (T359641)
  • 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49
  • 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31
  • 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster
  • 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 (T359641)
  • 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 (T359641)
  • 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30
  • 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 (T359641)
  • 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29
  • 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 (T359641)
  • 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 (T359641)
  • 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 (T359641)
  • 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 (T359641)
  • 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 (T359641)
  • 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28
  • 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster
  • 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 (T359641)
  • 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 (T359641)
  • 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 (T359641)
  • 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 (T359641)
  • 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 (T359641)
  • 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 (T359641)
  • 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 (T359641)
  • 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 (T359641)
  • 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster
  • 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 (T359641)
  • 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 (T359641)
  • 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 (T359641)
  • 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 (T359641)
  • 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 (T359641)
  • 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 (T359641)
  • 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster
  • 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 (T359641)
  • 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 (T359641)
  • 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 (T359641)
  • 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 (T359641)
  • 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 (T359641)
  • 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 (T359641)
  • 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 (T359641)
  • 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 (T359641)
  • 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 (T359641)
  • 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 (T359641)
  • 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T359641)
  • 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 (T359641)
  • 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 (T359641)
  • 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 (T359641)
  • 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 (T359641)
  • 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 (T359641)
  • 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 (T359641)
  • 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 (T359641)
  • 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 (T359641)
  • 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 (T359641)
  • 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 (T359641)
  • 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 (T359641)
  • 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 (T359641)
  • 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 (T359641)
  • 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 (T359641)
  • 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 (T359641)
  • 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 (T359641)
  • 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 (T359641)
  • 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 (T359641)
  • 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 (T359641)
  • 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 (T359641)
  • 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 (T359641)
  • 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 (T359641)
  • 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 (T359641)
  • 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 (T359641)
  • 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 (T359641)
  • 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 (T359641)
  • 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 (T359641)
  • 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 (T359641)
  • 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 (T359641)
  • 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 (T359641)
  • 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 (T359641)
  • 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 (T359641)
  • 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 (T359641)
  • 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 (T359641)
  • 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 (T359641)
  • 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 (T359641)
  • 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 (T359641)
  • 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 (T359641)
  • 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 (T359641)
  • 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 (T359641)
  • 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 (T359641)
  • 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 (T359641)
  • 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 (T359641)

2024-09-16

  • 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45
  • 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45
  • 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
  • 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

2024-09-13

  • 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 (T374692)
  • 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 (T374692)
  • 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 (T374692)
  • 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 (T374692)
  • 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 (T374692)
  • 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 (T374692)

2024-09-12

  • 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 (T374612)
  • 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 (T374612)
  • 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 (T374612)
  • 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 (T374612)
  • 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 (T374612)
  • 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 (T374612)

2024-09-11

  • 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
  • 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers

2024-09-09

  • 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager
  • 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager

2024-09-06

  • 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api
  • 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
  • 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6
  • 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

2024-09-05

  • 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 (T359641)
  • 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 (T359641)
  • 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) (T359641)
  • 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 (T359641)
  • 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) (T359641)
  • 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 (T359641)
  • 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 (T359641)
  • 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 (T359641)
  • 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)
  • 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) (T359641)
  • 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 (T359641)
  • 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (T359641)

2024-09-04

  • 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers
  • 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers
  • 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
  • 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
  • 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
  • 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
  • 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
  • 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
  • 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
  • 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission

2024-09-03

  • 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
  • 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer
  • 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer
  • 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
  • 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
  • 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno
  • 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
  • 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno
  • 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno
  • 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
  • 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission
  • 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission
  • 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) (T359641)
  • 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 (T359641)
  • 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 (T359641)
  • 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 (T359641)
  • 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 (T359641)
  • 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 (T359641)
  • 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 (T359641)
  • 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 (T359641)
  • 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (T359641)
  • 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
  • 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) (T359641)
  • 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 (T359641)
  • 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 (T359641)
  • 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 (T359641)
  • 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 (T359641)
  • 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 (T359641)
  • 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 (T359641)
  • 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (T359641)
  • 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
  • 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
  • 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
  • 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
  • 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
  • 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
  • 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
  • 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
  • 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15
  • 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
  • 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15
  • 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15
  • 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15

2024-09-02

  • 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 (T370249)
  • 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 (T370249)
  • 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 (T370249)
  • 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 (T370249)
  • 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 (T370249)
  • 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 (T370249)
  • 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 (T370249)
  • 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 (T370249)
  • 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 (T370249)
  • 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 (T370249)
  • 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 (T370249)
  • 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 (T370249)
  • 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 (T370249)
  • 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 (T370249)
  • 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 (T370249)
  • 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 (T370249)
  • 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
  • 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15
  • 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
  • 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15
  • 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
  • 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 (T370249)
  • 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 (T370249)
  • 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 (T370249)
  • 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15
  • 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
  • 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 (T370249)
  • 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 (T370249)
  • 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15
  • 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 (T370249)
  • 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 (T370249)
  • 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 (T370249)
  • 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 (T370249)
  • 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
  • 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 (T370249)
  • 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 (T370249)
  • 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15
  • 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 (T370249)
  • 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 (T370249)
  • 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
  • 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15
  • 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 (T370249)
  • 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 (T370249)
  • 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
  • 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 (T370249)
  • 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 (T370249)
  • 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15
  • 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 (T370249)
  • 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 (T370249)
  • 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 (T370249)
  • 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 (T370249)
  • 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 (T370249)
  • 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 (T370249)
  • 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
  • 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 (T370249)
  • 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 (T370249)
  • 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15
  • 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 (T370249)
  • 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 (T370249)
  • 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
  • 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 (T370249)
  • 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 (T370249)
  • 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15
  • 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
  • 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 (T370249)
  • 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 (T370249)
  • 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15
  • 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 (T370249)
  • 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 (T370249)
  • 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
  • 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 (T370249)
  • 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 (T370249)
  • 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15
  • 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 (T370249)
  • 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 (T370249)
  • 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
  • 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 (T370249)
  • 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 (T370249)
  • 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15
  • 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
  • 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 (T370249)
  • 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 (T370249)
  • 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15
  • 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
  • 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 (T370249)
  • 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 (T370249)
  • 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15
  • 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 (T370249)
  • 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 (T370249)
  • 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
  • 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 (T370249)
  • 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 (T370249)
  • 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15
  • 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 (T370249)
  • 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 (T370249)
  • 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
  • 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 (T370249)
  • 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 (T370249)
  • 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15
  • 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
  • 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 (T370249)
  • 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 (T370249)
  • 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15
  • 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 (T370249)
  • 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 (T370249)
  • 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
  • 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 (T370249)
  • 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 (T370249)
  • 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15
  • 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
  • 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 (T370249)
  • 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 (T370249)
  • 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15
  • 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 (T370249)
  • 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
  • 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15
  • 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
  • 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15
  • 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
  • 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15
  • 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
  • 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15
  • 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
  • 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15
  • 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
  • 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15
  • 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
  • 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15
  • 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
  • 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15
  • 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15
  • 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15
  • 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15
  • 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15
  • 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15
  • 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15
  • 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15
  • 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15
  • 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
  • 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
  • 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
  • 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
  • 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
  • 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api
  • 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api
  • 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

2024-08-29

  • 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
  • 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
  • 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx
  • 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx

2024-08-27

  • 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
  • 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2
  • 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
  • 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
  • 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico
  • 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico
  • 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
  • 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
  • 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 (T373243)
  • 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 (T373243)
  • 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 (T373243)
  • 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 (T373243)
  • 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 (T373243)
  • 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 (T373243)
  • 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 (T373243)
  • 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 (T373243)
  • 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 (T373243)
  • 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 (T373243)
  • 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 (T373243)
  • 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 (T373243)
  • 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 (T373243)
  • 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 (T373243)
  • 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
  • 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

2024-08-26

  • 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster
  • 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
  • 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster
  • 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
  • 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
  • 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster
  • 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
  • 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
  • 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
  • 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase
  • 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster
  • 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster
  • 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster
  • 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster
  • 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster
  • 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 (T373243)
  • 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 (T373243)
  • 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 (T373243)
  • 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 (T373243)
  • 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 (T373243)
  • 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 (T373243)
  • 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 (T373243)
  • 11:06 dcaro: manually deleted the coredns pods that had been around for 4d
  • 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
  • 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway
  • 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 08:18 dcaro: scale up cordens deployment to 4 replicas

2024-08-21

  • 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
  • 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api
  • 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder
  • 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder
  • 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
  • 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
  • 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
  • 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
  • 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
  • 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
  • 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission
  • 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
  • 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
  • 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
  • 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
  • 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
  • 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
  • 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
  • 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api

2024-08-19

  • 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24
  • 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24
  • 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17
  • 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17
  • 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24
  • 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24

2024-08-15

  • 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20
  • 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20

2024-08-13

  • 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6
  • 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6

2024-08-12

  • 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway
  • 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice
  • 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice
  • 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
  • 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
  • 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway
  • 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway

2024-08-08

  • 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
  • 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
  • 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
  • 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api
  • 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
  • 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api

2024-08-06

  • 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1)
  • 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console

2024-08-05

  • 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
  • 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
  • 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

2024-08-01

  • 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot
  • 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected.
  • 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55
  • 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc
  • 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
  • 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
  • 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api

2024-07-31

  • 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
  • 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
  • 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
  • 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
  • 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to T371505
  • 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api
  • 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api
  • 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43
  • 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43

2024-07-30

  • 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
  • 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
  • 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
  • 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
  • 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
  • 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
  • 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli
  • 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
  • 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
  • 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
  • 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
  • 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
  • 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli
  • 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
  • 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23
  • 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23

2024-07-29

  • 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0)
  • 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance
  • 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli
  • 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli
  • 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli
  • 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli
  • 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli
  • 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli

2024-07-25

  • 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

2024-07-24

  • 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
  • 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
  • 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
  • 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
  • 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission
  • 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

2024-07-23

  • 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
  • 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
  • 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
  • 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
  • 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2024-07-22

  • 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org
  • 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3
  • 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3
  • 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 17:00 dcaro: moving the toolforge apt repo to tools-services-06
  • 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud
  • 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud
  • 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-07-19

  • 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
  • 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2
  • 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
  • 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
  • 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6
  • 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry

2024-07-18

  • 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2024-07-17

  • 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
  • 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
  • 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

2024-07-16

  • 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
  • 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16
  • 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
  • 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16
  • 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
  • 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16
  • 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
  • 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16
  • 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
  • 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16
  • 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
  • 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16
  • 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
  • 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16
  • 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
  • 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16
  • 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
  • 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16
  • 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
  • 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16
  • 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
  • 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16
  • 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
  • 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16
  • 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
  • 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16
  • 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
  • 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
  • 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
  • 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16
  • 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
  • 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
  • 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
  • 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16
  • 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
  • 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
  • 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16
  • 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
  • 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16
  • 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
  • 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16
  • 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
  • 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16
  • 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
  • 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16
  • 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
  • 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16
  • 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
  • 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16
  • 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16
  • 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
  • 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16
  • 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
  • 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
  • 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16
  • 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
  • 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16
  • 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16
  • 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
  • 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
  • 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16
  • 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
  • 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16
  • 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16
  • 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
  • 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
  • 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16
  • 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
  • 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16
  • 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
  • 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16
  • 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
  • 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16
  • 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
  • 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16
  • 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
  • 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16
  • 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16
  • 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
  • 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
  • 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16
  • 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
  • 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16
  • 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
  • 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16
  • 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
  • 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16
  • 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16
  • 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
  • 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
  • 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
  • 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16
  • 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
  • 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16
  • 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
  • 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16
  • 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
  • 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16
  • 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
  • 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16
  • 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
  • 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16
  • 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
  • 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16
  • 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
  • 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16
  • 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
  • 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16
  • 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
  • 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
  • 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16
  • 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
  • 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16
  • 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
  • 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16
  • 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
  • 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16
  • 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
  • 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
  • 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
  • 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
  • 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
  • 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
  • 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
  • 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16
  • 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
  • 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16
  • 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
  • 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16
  • 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16
  • 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
  • 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
  • 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16
  • 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
  • 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16
  • 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
  • 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16
  • 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
  • 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16
  • 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16
  • 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
  • 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16
  • 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
  • 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16
  • 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
  • 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16
  • 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16
  • 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16
  • 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16
  • 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16
  • 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16
  • 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16
  • 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16
  • 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16
  • 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
  • 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

2024-07-15

  • 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
  • 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

2024-07-11

  • 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 (T369573)
  • 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
  • 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission

2024-07-10

  • 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
  • 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
  • 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2024-07-09

  • 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
  • 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
  • 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-07-08

  • 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37
  • 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37
  • 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
  • 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3
  • 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3
  • 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2
  • 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2
  • 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1
  • 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1
  • 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
  • 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
  • 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
  • 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
  • 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
  • 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
  • 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2024-07-05

  • 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
  • 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
  • 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
  • 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
  • 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
  • 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
  • 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
  • 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
  • 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
  • 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
  • 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
  • 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
  • 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
  • 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
  • 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
  • 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
  • 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
  • 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
  • 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
  • 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0)
  • 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0
  • 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry
  • 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
  • 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
  • 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4
  • 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-07-04

  • 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 12:57 arturo: updating kubelet flags T355881
  • 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

2024-07-03

  • 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
  • 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

2024-07-02

  • 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for T311905
  • 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings (T349197)
  • 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition (T349197)
  • 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
  • 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
  • 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
  • 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

2024-07-01

  • 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
  • 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
  • 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
  • 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
  • 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
  • 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission

2024-06-28

  • 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
  • 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
  • 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
  • 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
  • 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway

2024-06-27

  • 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23
  • 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23
  • 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1
  • 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
  • 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1
  • 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1
  • 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3
  • 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3
  • 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24
  • 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24
  • 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22
  • 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22
  • 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager
  • 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager
  • 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
  • 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
  • 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:02 arturo: drop all PSP definitions for all accounts (T368142)
  • 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap (T368142)
  • 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-06-26

  • 11:40 taavi: update pywikibot image to 9.2 T363631
  • 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:18 arturo: deploying toolforge-webservice 0.103.9 (T368463)
  • 09:18 arturo: setting kyverno policies to Enforce (T368141)
  • 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29
  • 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29

2024-06-25

  • 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-> dict[str, Any]` type annotations causing T368463
  • 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104
  • 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104
  • 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103
  • 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104
  • 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104
  • 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103
  • 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102
  • 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103
  • 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103
  • 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102
  • 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56
  • 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
  • 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
  • 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56
  • 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55
  • 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55
  • 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54
  • 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56
  • 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56
  • 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54
  • 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53
  • 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55
  • 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55
  • 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53
  • 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54
  • 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52
  • 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54
  • 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52
  • 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
  • 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
  • 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51
  • 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53
  • 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51
  • 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53
  • 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50
  • 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52
  • 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50
  • 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
  • 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
  • 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52
  • 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
  • 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50
  • 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
  • 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50
  • 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50
  • 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50
  • 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7
  • 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7
  • 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8'
  • 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8'
  • 09:44 arturo: deploy toolforge-webservice 0.103.8 (T362050)
  • 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6
  • 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6
  • 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9
  • 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9
  • 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9
  • 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9
  • 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49
  • 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49
  • 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48
  • 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49
  • 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48
  • 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49
  • 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47
  • 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48
  • 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48
  • 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47
  • 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46
  • 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46
  • 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45
  • 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47
  • 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47
  • 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45
  • 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44
  • 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46
  • 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46
  • 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44
  • 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45
  • 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45
  • 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43
  • 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
  • 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42
  • 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44
  • 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44
  • 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43
  • 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
  • 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42
  • 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
  • 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
  • 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42
  • 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41
  • 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42
  • 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41
  • 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40
  • 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40
  • 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39
  • 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41
  • 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41
  • 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39
  • 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38
  • 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40
  • 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40
  • 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38
  • 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37
  • 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39
  • 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39
  • 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37
  • 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36
  • 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
  • 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
  • 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36
  • 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35
  • 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37
  • 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37
  • 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35
  • 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34
  • 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36
  • 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36
  • 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34
  • 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35
  • 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33
  • 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35
  • 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34
  • 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34
  • 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33
  • 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33
  • 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33

2024-06-24

  • 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots.
  • 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32
  • 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32
  • 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31
  • 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32
  • 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31
  • 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32
  • 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30
  • 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31
  • 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31
  • 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30
  • 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29
  • 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30
  • 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30
  • 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29
  • 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28
  • 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29
  • 15:45 arturo: deploy toolforge-webservice 0.103.7 (T362050)
  • 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29
  • 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28
  • 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27
  • 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28
  • 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27
  • 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28
  • 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27
  • 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27
  • 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
  • 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10
  • 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10
  • 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13
  • 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13
  • 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12
  • 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12
  • 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
  • 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2
  • 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
  • 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2
  • 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2
  • 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd
  • 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd
  • 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26
  • 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26
  • 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25
  • 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25
  • 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26
  • 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24
  • 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26
  • 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
  • 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24
  • 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25
  • 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23
  • 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24
  • 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23
  • 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24
  • 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22
  • 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22
  • 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21
  • 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23
  • 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23
  • 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21
  • 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20
  • 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22
  • 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22
  • 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20
  • 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21
  • 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19
  • 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21
  • 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19
  • 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18
  • 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18
  • 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20
  • 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17
  • 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20
  • 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19
  • 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19
  • 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18
  • 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18
  • 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17
  • 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
  • 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
  • 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17
  • 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17
  • 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16
  • 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16
  • 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15
  • 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16
  • 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16
  • 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15
  • 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14
  • 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15
  • 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15
  • 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14
  • 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13
  • 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14
  • 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14
  • 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13
  • 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12
  • 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13
  • 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13
  • 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12
  • 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11
  • 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12
  • 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11
  • 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12
  • 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7
  • 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11
  • 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11
  • 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7
  • 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8
  • 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8
  • 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8
  • 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8
  • 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
  • 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
  • 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15
  • 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15
  • 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4
  • 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4
  • 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10
  • 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10
  • 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10
  • 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
  • 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10
  • 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10
  • 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
  • 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
  • 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9
  • 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9
  • 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8
  • 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9
  • 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8
  • 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9
  • 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8
  • 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
  • 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7
  • 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8
  • 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7
  • 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8
  • 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7
  • 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7
  • 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6
  • 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6
  • 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5
  • 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5
  • 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6
  • 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4
  • 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6
  • 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5
  • 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4
  • 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5
  • 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4
  • 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4
  • 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3
  • 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3
  • 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2
  • 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2
  • 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1
  • 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1
  • 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3
  • 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3
  • 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2
  • 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2
  • 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
  • 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
  • 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1
  • 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
  • 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5
  • 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5
  • 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7
  • 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7
  • 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7
  • 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43
  • 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7
  • 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43
  • 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7
  • 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7
  • 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7
  • 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43
  • 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7
  • 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6
  • 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6
  • 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43
  • 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1
  • 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1
  • 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5
  • 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5
  • 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1
  • 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1
  • 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
  • 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6
  • 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6
  • 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01
  • 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01
  • 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2
  • 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2
  • 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4
  • 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
  • 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4
  • 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2
  • 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2
  • 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2
  • 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2
  • 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8
  • 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8
  • 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05
  • 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05
  • 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04
  • 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04
  • 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8
  • 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
  • 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8
  • 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5
  • 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
  • 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5
  • 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
  • 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

2024-06-20

  • 13:09 arturo: re-deploy kyverno T368044
  • 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
  • 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
  • 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-06-19

  • 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113
  • 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2024-06-14

  • 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-06-12

  • 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for T367345.
  • 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
  • 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
  • 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
  • 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
  • 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
  • 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno
  • 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 13:45 taavi: hard reboot tools-k8s-control-7
  • 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-06-11

  • 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers
  • 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
  • 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers
  • 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers
  • 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:57 dcaro: cleaning old maintain-kubeusers configmaps
  • 10:45 dcaro: cleaning up old resourcequotas

2024-06-10

  • 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno
  • 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno

2024-06-07

  • 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2024-06-06

  • 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-06-05

  • 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 13:27 dcaro: deploying toolforge-webservice 0.103.6
  • 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13
  • 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12

2024-06-04

  • 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-06-03

  • 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0)
  • 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7
  • 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7
  • 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7
  • 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7
  • 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
  • 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
  • 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
  • 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
  • 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
  • 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
  • 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7
  • 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
  • 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
  • 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
  • 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99)
  • 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry
  • 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
  • 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission

2024-05-29

  • 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api
  • 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2024-05-28

  • 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-05-27

  • 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
  • 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

2024-05-25

  • 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-05-23

  • 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2024-05-22

  • 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
  • 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9

2024-05-15

  • 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 (T364822)
  • 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 (T364822)
  • 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 (T364822)
  • 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 (T364822)
  • 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-05-14

  • 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO
  • 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9
  • 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9

2024-05-07

  • 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-05-06

  • 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2024-05-05

  • 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
  • 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

2024-05-03

  • 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-04-30

  • 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

2024-04-26

  • 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-04-25

  • 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 09:48 taavi: update pywikibot script image to v9.1.0 T363132

2024-04-24

  • 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

2024-04-18

  • 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-04-17

  • 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50
  • 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools
  • 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50
  • 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13
  • 10:48 dcaro: rebooting tools-k8s-worker-nfs-1

2024-04-16

  • 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1
  • 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1
  • 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0'
  • 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0'

2024-04-15

  • 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2024-04-12

  • 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission
  • 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission
  • 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
  • 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
  • 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico
  • 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
  • 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico
  • 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway
  • 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission
  • 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway
  • 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission
  • 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission
  • 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-04-11

  • 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2024-04-09

  • 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
  • 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
  • 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
  • 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
  • 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
  • 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
  • 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
  • 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission
  • 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6

2024-04-08

  • 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0)
  • 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
  • 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
  • 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
  • 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99)
  • 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node
  • 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
  • 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
  • 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0)
  • 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21
  • 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21
  • 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
  • 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56
  • 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56
  • 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
  • 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
  • 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
  • 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
  • 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
  • 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
  • 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
  • 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99)
  • 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node (T349207)
  • 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5

2024-04-05

  • 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-04-03

  • 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
  • 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
  • 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
  • 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
  • 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
  • 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
  • 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
  • 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
  • 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
  • 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06
  • 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06
  • 09:45 taavi: rebuilding prebuild images for T361457

2024-04-02

  • 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 (T344717)
  • 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 (T344717)
  • 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05
  • 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05

2024-03-28

  • 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05
  • 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05
  • 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 T361223
  • 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
  • 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
  • 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy'
  • 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy'
  • 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06
  • 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06
  • 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
  • 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'

2024-03-27

  • 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01
  • 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01

2024-03-26

  • 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
  • 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
  • 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
  • 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud
  • 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry'
  • 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry'
  • 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud
  • 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud
  • 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion'
  • 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion'
  • 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11
  • 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11
  • 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 T311909

2024-03-25

  • 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector
  • 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector
  • 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
  • 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
  • 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
  • 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
  • 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud
  • 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud

2024-03-22

  • 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error)

2024-03-21

  • 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4
  • 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4
  • 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3
  • 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3
  • 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3
  • 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3
  • 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
  • 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node
  • 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
  • 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node

2024-03-20

  • 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04
  • 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04
  • 12:30 taavi: move checker service address to tools-checker-5
  • 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud
  • 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
  • 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud
  • 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud
  • 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker'
  • 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
  • 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
  • 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'
  • 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
  • 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
  • 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker'
  • 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker'

2024-03-19

  • 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images)
  • 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14
  • 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14
  • 11:19 taavi: point dev.toolforge.org to tools-bastion-12 T314665
  • 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 (T360405)

2024-03-18

  • 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 (T359638)
  • 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 (T359638)
  • 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 (T359638)
  • 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 (T359638)
  • 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 (T359638)
  • 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 (T359638)
  • 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 (T307651)
  • 13:13 taavi: restart harbor services after docker service restart
  • 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 (T307651)
  • 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 (T307651)
  • 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 (T307651)
  • 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 (T307651)
  • 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 (T307651)
  • 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 (T307651)
  • 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 (T307651)
  • 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 (T307651)
  • 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 (T307651)
  • 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 (T307651)
  • 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 (T307651)
  • 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 (T307651)
  • 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 (T307651)
  • 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 (T307651)
  • 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 (T307651)
  • 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 (T307651)
  • 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 (T307651)
  • 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 (T307651)
  • 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 (T307651)
  • 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 (T307651)
  • 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 (T307651)
  • 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 (T307651)
  • 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 (T307651)
  • 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 (T307651)
  • 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 (T307651)
  • 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 (T307651)
  • 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 (T307651)
  • 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 (T307651)
  • 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 (T307651)
  • 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 (T307651)
  • 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 (T307651)
  • 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 (T307651)
  • 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 (T307651)
  • 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 (T307651)
  • 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 (T307651)
  • 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 (T307651)
  • 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 (T307651)
  • 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 (T359638)
  • 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 (T359638)
  • 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 (T307651)
  • 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 (T307651)
  • 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 (T307651)
  • 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 (T307651)
  • 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 (T307651)
  • 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1
  • 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 (T307651)
  • 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 (T307651)
  • 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1
  • 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 (T307651)
  • 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 (T307651)
  • 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 (T307651)
  • 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 (T307651)
  • 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 (T307651)
  • 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 (T307651)
  • 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 (T307651)
  • 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 (T307651)
  • 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 (T307651)
  • 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 (T307651)
  • 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 (T307651)
  • 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 (T307651)
  • 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 (T307651)
  • 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 (T307651)
  • 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 (T307651)
  • 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 (T307651)
  • 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 (T307651)
  • 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 (T307651)
  • 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 (T307651)
  • 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 (T307651)
  • 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 (T307651)
  • 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 (T307651)
  • 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 (T307651)
  • 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 (T307651)
  • 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 (T307651)
  • 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 (T307651)
  • 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 (T307651)
  • 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 (T307651)
  • 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
  • 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 (T307651)
  • 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 (T307651)
  • 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud
  • 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 (T307651)
  • 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 (T307651)
  • 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 (T307651)
  • 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
  • 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 (T359638)
  • 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 (T359638)
  • 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 (T359638)
  • 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
  • 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
  • 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud
  • 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 (T359638)
  • 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 (T307651)
  • 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 (T307651)
  • 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 (T307651)
  • 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 (T307651)
  • 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 (T307651)
  • 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 (T307651)
  • 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 (T307651)
  • 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 (T307651)
  • 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 (T307651)
  • 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 (T307651)
  • 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 (T307651)
  • 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 (T307651)
  • 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 (T307651)
  • 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 (T307651)
  • 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 (T307651)
  • 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 (T307651)
  • 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 (T307651)
  • 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 (T307651)
  • 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 (T307651)
  • 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 (T307651)
  • 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 (T307651)
  • 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 (T307651)
  • 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 (T307651)
  • 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 (T307651)
  • 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 (T307651)
  • 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 (T307651)
  • 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 (T307651)
  • 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 (T307651)
  • 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 (T307651)
  • 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 (T307651)
  • 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 (T307651)
  • 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 (T307651)
  • 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 (T307651)
  • 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 (T307651)
  • 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 (T307651)
  • 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 (T307651)
  • 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 (T307651)
  • 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 (T307651)
  • 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 (T307651)
  • 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 (T307651)
  • 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 (T359638)
  • 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) T311913
  • 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud
  • 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
  • 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud
  • 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud
  • 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 (T359638)
  • 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 (T359638)
  • 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 (T359638)
  • 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
  • 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api
  • 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 (T359638)
  • 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 (T359638)
  • 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 (T359638)
  • 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 (T359638)
  • 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 (T307651)
  • 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 (T307651)
  • 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud
  • 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud
  • 09:27 taavi: deleted shutdown grid engine VMs T314664

2024-03-15

  • 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-03-14

  • 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48'
  • 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48'
  • 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01
  • 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
  • 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
  • 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
  • 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01
  • 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01
  • 11:02 taavi: stop grid related VMs T314664
  • 11:01 taavi: disable grid access for remaining tools still running on the grid T314664

2024-03-13

  • 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable.

2024-03-12

  • 12:38 taavi: hard reboot tools-prometheus-6
  • 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-03-11

  • 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
  • 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for T359798

2024-03-09

  • 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs

2024-03-08

  • 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-03-07

  • 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-03-06

  • 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32
  • 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32
  • 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32
  • 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 (T293552) + normal package updates
  • 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
  • 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs

2024-03-05

  • 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
  • 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
  • 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
  • 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
  • 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) (T357901)
  • 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase (T357901)
  • 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud
  • 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud

2024-03-04

  • 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state

2024-03-03

  • 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state)

2024-03-01

  • 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2024-02-29

  • 14:36 dcaro: deploy webservice 0.103.3

2024-02-28

  • 11:57 dcaro: deploy tools-webservice 0.103.2 with probes (T341919)
  • 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-02-26

  • 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) (T284656)
  • 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node (T284656)
  • 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
  • 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster (T284656)

2024-02-23

  • 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2
  • 13:32 taavi: remove toolschecker alerts for grid engine jobs T358333

2024-02-22

  • 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
  • 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api
  • 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
  • 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api
  • 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) (T284656)
  • 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node (T284656)
  • 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
  • 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
  • 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster (T284656)
  • 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51
  • 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51
  • 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38
  • 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38
  • 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25
  • 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25

2024-02-21

  • 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4
  • 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4
  • 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster
  • 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster

2024-02-20

  • 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
  • 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster
  • 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102
  • 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102
  • 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101
  • 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101
  • 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
  • 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
  • 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102
  • 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102
  • 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster
  • 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster
  • 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
  • 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud
  • 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100
  • 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100
  • 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99
  • 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99
  • 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98
  • 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98
  • 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97
  • 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97
  • 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96
  • 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96
  • 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95
  • 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95
  • 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94
  • 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94
  • 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93
  • 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93
  • 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster
  • 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92
  • 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92
  • 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6
  • 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6
  • 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
  • 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
  • 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91
  • 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91
  • 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90
  • 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90
  • 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster
  • 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89
  • 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89
  • 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster
  • 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88
  • 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88

2024-02-19

  • 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5
  • 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5
  • 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87
  • 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87
  • 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86
  • 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86
  • 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T357901)
  • 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase (T357901)
  • 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
  • 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud
  • 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85
  • 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85
  • 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84
  • 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84
  • 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83
  • 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83
  • 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82
  • 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82
  • 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81
  • 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81
  • 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-02-16

  • 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
  • 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
  • 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80
  • 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80
  • 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79
  • 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79
  • 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78
  • 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78
  • 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster
  • 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77
  • 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77

2024-02-15

  • 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4
  • 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4
  • 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76
  • 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76
  • 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75
  • 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75
  • 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster
  • 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster
  • 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7
  • 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7
  • 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster
  • 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster

2024-02-14

  • 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30
  • 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3
  • 16:30 taavi: reboot tools-sgeexec-10-23 due to high load
  • 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
  • 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud
  • 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster
  • 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74
  • 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74
  • 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster
  • 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73
  • 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73
  • 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster
  • 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72
  • 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72
  • 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster
  • 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71
  • 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71
  • 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster
  • 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70
  • 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70
  • 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster
  • 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69
  • 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69
  • 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster
  • 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68
  • 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68

2024-02-13

  • 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67
  • 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67
  • 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster
  • 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66
  • 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66
  • 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster
  • 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65
  • 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65
  • 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64
  • 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64

2024-02-12

  • 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster
  • 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62
  • 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62
  • 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster
  • 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61
  • 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61
  • 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60
  • 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60
  • 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster
  • 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59
  • 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59
  • 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58
  • 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58
  • 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster
  • 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57
  • 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57
  • 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56
  • 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56
  • 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55
  • 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55
  • 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54
  • 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54
  • 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster
  • 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15
  • 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15
  • 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53
  • 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53
  • 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52
  • 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52
  • 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2024-02-11

  • 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

2024-02-09

  • 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule
  • 13:06 taavi: reboot tools-sgecron-2 due to high load
  • 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
  • 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config
  • 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51
  • 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51
  • 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50
  • 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50
  • 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes

2024-02-08

  • 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
  • 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49
  • 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49
  • 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48
  • 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48
  • 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47
  • 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47
  • 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46
  • 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46
  • 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45
  • 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45
  • 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44
  • 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44
  • 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster
  • 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43
  • 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43
  • 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42
  • 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42

2024-02-07

  • 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
  • 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
  • 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9
  • 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9
  • 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
  • 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
  • 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
  • 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
  • 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers
  • 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
  • 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers
  • 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers

2024-02-06

  • 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes (T356507)
  • 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes (T356507)

2024-01-31

  • 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-01-30

  • 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster
  • 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9
  • 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9
  • 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster
  • 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
  • 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
  • 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8
  • 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8
  • 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster
  • 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41
  • 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41
  • 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40
  • 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40
  • 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39
  • 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39
  • 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38
  • 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38
  • 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37
  • 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37
  • 15:16 dcaro: restart harbor now that the db is clean (T356037)
  • 15:14 dcaro: restart harbor now that the db is clean (T3543)
  • 13:08 taavi: create no-op DMARC record T354112
  • 12:39 dcaro: rebuilding all the toolforge images (T354320)
  • 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data (T356037)
  • 09:33 dcaro: cleaning up old schedules on harbor (T356037)

2024-01-29

  • 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
  • 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36
  • 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36
  • 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud
  • 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud
  • 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster
  • 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35
  • 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35
  • 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34
  • 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34
  • 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33
  • 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33
  • 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32
  • 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32
  • 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31
  • 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30
  • 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster
  • 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster
  • 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster
  • 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB
  • 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB
  • 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

2024-01-26

  • 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta

2024-01-25

  • 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster
  • 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

2024-01-24

  • 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1

2024-01-23

  • 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
  • 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
  • 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
  • 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22
  • 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4
  • 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

2024-01-19

  • 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2024-01-18

  • 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17

2024-01-17

  • 18:16 dhinus: increase volume quotas for toolsdb T344717
  • 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) (T344717)
  • 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase (T344717)
  • 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 08:56 taavi: update all pre-built docker images T352886

2024-01-15

  • 09:18 taavi: reboot stuck tools-k8s-worker-84

2024-01-12

  • 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12'
  • 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12'

2024-01-11

  • 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

2024-01-10

  • 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 09:17 taavi: reboot tools-k8s-worker-98

2024-01-09

  • 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor -- T354714
  • 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it)
  • 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
  • 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder
  • 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder
  • 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 10:13 taavi: reboot tools-sgeexec-10-17 due to high load

2024-01-08

  • 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28
  • 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 10:17 taavi: reboot tools-sgeexec-10-21

2024-01-05

  • 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

2024-01-04

  • 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3

2024-01-03

  • 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs
  • 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage

2024-01-02

  • 11:06 dcaro: restart toolsdb database to flush connections (T354176)
  • 10:42 dcaro: flushed the redis db on tools-harbor-1 (T354176)
  • 10:37 dcaro: hard reboot tools-harbor-1
  • 10:13 dhinus: hard reboot tools-harbor-1

2024-01-01

  • 15:55 andrewbogott: rebooting tools-harbor-1, T354151

2023-12-30

  • 12:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

2023-12-29

  • 21:39 andrewbogott: rebooting tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud because previous reset didn't get the queue out of error state
  • 19:31 andrewbogott: restarting sge_execd on tools-sgeweblight-10-28.tools.eqiad1.wikimedia.cloud in response to error state alert

2023-12-28

  • 21:03 andrewbogott: "docker-compose restart" on tools-harbor-1
  • 19:18 andrewbogott: rebooting tools-harbor-1.tools.eqiad1.wikimedia.cloud, unresponsive

2023-12-23

  • 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 18:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-12-21

  • 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-16

2023-12-20

  • 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-14, tools-sgeexec-10-15, tools-sgeweblight-10-18, tools-sgeweblight-10-24
  • 10:01 taavi: rebooting tools-sgeweblight-10-18, -24, -25, to get rid of a large number of jobs in deleting status

2023-12-19

  • 15:39 dhinus: restarting toolsdb to apply a config change T353093
  • 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-12-18

  • 16:15 taavi: reboot tools-sgeexec-10-15, -23 due to stuck NFS processes
  • 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 14:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2023-12-16

  • 22:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 22:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
  • 20:54 bd808: Rebuilding all containers to pick up lighttpd config fix and normal package updates (T293552)
  • 08:14 dhinus: restarting toolsdb with jemalloc
  • 05:32 andrewbogott: restarting mariadb on toolsdb-1 because it's just about to go oom (or possibly just did)
  • 00:21 dhinus: restarting toolsdb again as it's again low in free mem T353093

2023-12-15

  • 20:26 andrewbogott: restarting toolsdb to avoid upcoming oom crash
  • 16:49 dhinus: restarting toolsdb before it's about to go OOM, enabling performance_schema for debugging
  • 14:40 dcaro: deploy toolforge-builds-cli 0.0.10 (T341067)
  • 13:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T341067)
  • 13:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T341067)
  • 12:19 dhinus: restarting toolsdb again to apply a config fix T353093
  • 10:48 dhinus: restarting toolsdb to apply new config T353093

2023-12-14

  • 23:02 andrewbogott: rebooting tools-db-1 yet again
  • 17:42 taavi: reboot tools-sgewebgen-10-3
  • 02:20 andrewbogott: restarting tools-db-1, oomkiller killed mariadb

2023-12-13

  • 19:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
  • 18:53 andrewbogott: rebooting tools-nfs-2 server to resolve weird file locking issues
  • 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec
  • 14:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder (T352774)
  • 14:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder (T352774)
  • 14:22 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774)
  • 14:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774)
  • 13:54 dcaro: deploy toolforge-builds-cli version 0.0.9 (with envvars support)
  • 13:32 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T338142)
  • 13:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T338142)
  • 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
  • 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-16
  • 10:48 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission (T338142)
  • 10:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission (T338142)
  • 09:49 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 09:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2023-12-12

  • 17:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 17:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-18
  • 17:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-19
  • 17:24 taavi: reboot tools-sgeexec-10-14
  • 15:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 15:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-14, tools-sgeexec-10-8
  • 15:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
  • 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 15:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 12:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774)
  • 12:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774)

2023-12-11

  • 15:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774)
  • 15:36 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774)
  • 13:43 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774)
  • 13:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774)
  • 13:29 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission (T352774)
  • 13:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission (T352774)

2023-12-09

  • 16:45 dcaro: set toolsdb back as read-write
  • 16:35 andrewbogott: rebooting tools-db-1.tools.eqiad1.wikimedia.cloud yet again
  • 07:23 dcaro: set toolsdb back as read-write
  • 00:54 taavi: set toolsdb back as read-write

2023-12-08

  • 11:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 11:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-12-07

  • 04:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-26

2023-12-05

  • 21:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 21:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
  • 19:16 andrewbogott: rebooting tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud; can't log in even with root key
  • 11:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 11:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 11:20 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 11:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 11:20 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 11:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 11:20 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 11:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 11:15 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255)
  • 11:15 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 11:01 dcaro: rebooting tools-sgeweblight-10-25 due to memory allocation issue (T352753)
  • 04:51 andrewbogott: rebooting tools-sgeweblight-10-27, tools-sgeweblight-10-17 and tools-sgeweblight-10-30; their filesystems seem locked up and I suspect NFS somehow

2023-12-04

  • 09:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 09:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-12-02

  • 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-22
  • 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-13, tools-sgeweblight-10-20
  • 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 00:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 00:08 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker role in the tools cluster
  • 00:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 00:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 00:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster
  • 00:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 00:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster

2023-12-01

  • 23:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 23:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 23:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster
  • 22:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
  • 21:22 andrewbogott: rebooting tools-sgeweblight-10-[18,21,32].tools.eqiad1.wikimedia.cloud to recover from nfs lockup
  • 21:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
  • 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors

2023-11-29

  • 23:11 bd808: Drained and hard rebooted tools-k8s-worker-40. K8s was showing inconsistent status of the node (offline per k8s-status tool, online per kubectl)
  • 22:35 bd808: Hard reboot of tools-k8s-worker-81
  • 22:33 bd808: Soft reboot of tools-k8s-worker-81
  • 22:26 bd808: Cordon, drain, and restart tools-k8s-worker-81. Instance appears to have pods from tools.cluebotng that are unresponsive to kubectl commands.

2023-11-27

  • 14:46 andrewbogott: shuffling toolforge etcd nodes all over the place in order to reimage cloudvirtlocal hosts
  • 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2023-11-23

  • 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 10:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-11-22

  • 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers (T350873)
  • 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers (T350873)
  • 10:57 taavi: deploy maintain-kubeusers patch to manage quotas from the git config T350873
  • 09:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2023-11-21

  • 10:28 taavi: restart replication on tools-db-2

2023-11-20

  • 15:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 13:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 10:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-cli' version '0.3.5'
  • 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-cli' version '0.3.5'

2023-11-17

  • 15:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.5'
  • 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.5'
  • 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api

2023-11-16

  • 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
  • 19:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
  • 13:47 taavi: reboot tools-sgecron-2 with very high load average

2023-11-14

  • 19:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 10:11 taavi: reboot unresponsive tools-sgeexec-10-22

2023-11-13

  • 22:21 taavi: reboot! tools-sgewebgen-10-3, tools-sgeweblight-10-21, tools-sgeweblight-10-32, tools-sgeexec-10-16 due to high load average and/or stuck jobs
  • 16:37 taavi: drain tools-k8s-worker-84 tools-k8s-worker-85

2023-11-09

  • 11:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 11:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers

2023-11-07

  • 11:45 taavi: reboot tools-sgeexec-10-8 which had high load average

2023-11-02

  • 13:13 taavi: wiping data directory from tools-prometheus-7 so we have least one working server T350227

2023-11-01

  • 14:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
  • 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
  • 08:47 taavi: restart puppetdb

2023-10-30

  • 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
  • 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics

2023-10-29

  • 17:46 andrewbogott: running SET GLOBAL read_only=OFF; for mariadb on tools-db-1.tools.eqiad1.wikimedia.cloud
  • 17:37 andrewbogott: rebooting tools-db-1.tools.eqiad1.wikimedia.cloud to recover from the oom-killer firing

2023-10-26

  • 08:29 taavi: root@tools-sgeweblight-10-21:~# sudo dpkg --configure -a
  • 08:18 taavi: restart sssd on tools-nfs-2

2023-10-25

  • 09:08 blancadesal: harbor up again and upgraded from 2.5 to 2.9 (T346241)
  • 08:31 blancadesal: taking harbor down for upgrade (T346241)

2023-10-24

2023-10-23

  • 15:40 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 15:39 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 14:18 dcaro: release toolforge-builds-cli 0.0.4
  • 08:22 taavi: reboot tools-sgeweblight-10-14, 24 T349425

2023-10-19

  • 12:48 taavi: flush queued webgrid jobs that had been waiting in the queue since the nfs issues last week

2023-10-18

  • 12:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 12:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-70 from 1.22.17 to 1.23.17
  • 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-88 from 1.22.17 to 1.23.17
  • 12:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-70 from 1.22.17 to 1.23.17
  • 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-69 from 1.22.17 to 1.23.17
  • 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-50 from 1.22.17 to 1.23.17
  • 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-88 from 1.22.17 to 1.23.17
  • 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-87 from 1.22.17 to 1.23.17
  • 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-69 from 1.22.17 to 1.23.17
  • 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-68 from 1.22.17 to 1.23.17
  • 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-50 from 1.22.17 to 1.23.17
  • 12:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-49 from 1.22.17 to 1.23.17
  • 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-87 from 1.22.17 to 1.23.17
  • 12:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-86 from 1.22.17 to 1.23.17
  • 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-68 from 1.22.17 to 1.23.17
  • 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-67 from 1.22.17 to 1.23.17
  • 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-49 from 1.22.17 to 1.23.17
  • 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-48 from 1.22.17 to 1.23.17
  • 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-86 from 1.22.17 to 1.23.17
  • 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-85 from 1.22.17 to 1.23.17
  • 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-67 from 1.22.17 to 1.23.17
  • 11:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-66 from 1.22.17 to 1.23.17
  • 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-48 from 1.22.17 to 1.23.17
  • 11:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-47 from 1.22.17 to 1.23.17
  • 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-85 from 1.22.17 to 1.23.17
  • 11:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-84 from 1.22.17 to 1.23.17
  • 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-66 from 1.22.17 to 1.23.17
  • 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-65 from 1.22.17 to 1.23.17
  • 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-47 from 1.22.17 to 1.23.17
  • 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-46 from 1.22.17 to 1.23.17
  • 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-84 from 1.22.17 to 1.23.17
  • 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-83 from 1.22.17 to 1.23.17
  • 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-6 from 1.22.17 to 1.23.17
  • 11:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-65 from 1.22.17 to 1.23.17
  • 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-64 from 1.22.17 to 1.23.17
  • 11:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-83 from 1.22.17 to 1.23.17
  • 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-82 from 1.22.17 to 1.23.17
  • 11:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-46 from 1.22.17 to 1.23.17
  • 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-45 from 1.22.17 to 1.23.17
  • 11:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-6 from 1.22.17 to 1.23.17
  • 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-64 from 1.22.17 to 1.23.17
  • 11:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-62 from 1.22.17 to 1.23.17
  • 11:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-5 from 1.22.17 to 1.23.17
  • 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-82 from 1.22.17 to 1.23.17
  • 11:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-81 from 1.22.17 to 1.23.17
  • 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-45 from 1.22.17 to 1.23.17
  • 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-44 from 1.22.17 to 1.23.17
  • 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-5 from 1.22.17 to 1.23.17
  • 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-81 from 1.22.17 to 1.23.17
  • 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-80 from 1.22.17 to 1.23.17
  • 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-62 from 1.22.17 to 1.23.17
  • 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-61 from 1.22.17 to 1.23.17
  • 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-44 from 1.22.17 to 1.23.17
  • 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-43 from 1.22.17 to 1.23.17
  • 11:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-4 from 1.22.17 to 1.23.17
  • 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-80 from 1.22.17 to 1.23.17
  • 11:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-79 from 1.22.17 to 1.23.17
  • 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-61 from 1.22.17 to 1.23.17
  • 11:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-60 from 1.22.17 to 1.23.17
  • 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-43 from 1.22.17 to 1.23.17
  • 11:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-42 from 1.22.17 to 1.23.17
  • 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-4 from 1.22.17 to 1.23.17
  • 11:49 dcaro: deploy toolforge-builds-cli 0.3.0 (T348866)
  • 11:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-42 from 1.22.17 to 1.23.17
  • 11:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-41 from 1.22.17 to 1.23.17
  • 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-59 from 1.22.17 to 1.23.17
  • 11:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-58 from 1.22.17 to 1.23.17
  • 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-78 from 1.22.17 to 1.23.17
  • 11:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-77 from 1.22.17 to 1.23.17
  • 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-41 from 1.22.17 to 1.23.17
  • 11:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-40 from 1.22.17 to 1.23.17
  • 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-58 from 1.22.17 to 1.23.17
  • 11:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-57 from 1.22.17 to 1.23.17
  • 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-77 from 1.22.17 to 1.23.17
  • 11:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-76 from 1.22.17 to 1.23.17
  • 11:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-40 from 1.22.17 to 1.23.17
  • 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-39 from 1.22.17 to 1.23.17
  • 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-57 from 1.22.17 to 1.23.17
  • 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-56 from 1.22.17 to 1.23.17
  • 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-76 from 1.22.17 to 1.23.17
  • 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-75 from 1.22.17 to 1.23.17
  • 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-39 from 1.22.17 to 1.23.17
  • 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-38 from 1.22.17 to 1.23.17
  • 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-56 from 1.22.17 to 1.23.17
  • 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-55 from 1.22.17 to 1.23.17
  • 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-38 from 1.22.17 to 1.23.17
  • 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-37 from 1.22.17 to 1.23.17
  • 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-75 from 1.22.17 to 1.23.17
  • 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-74 from 1.22.17 to 1.23.17
  • 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-55 from 1.22.17 to 1.23.17
  • 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-54 from 1.22.17 to 1.23.17
  • 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-37 from 1.22.17 to 1.23.17
  • 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-36 from 1.22.17 to 1.23.17
  • 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-74 from 1.22.17 to 1.23.17
  • 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-73 from 1.22.17 to 1.23.17
  • 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-54 from 1.22.17 to 1.23.17
  • 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-53 from 1.22.17 to 1.23.17
  • 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-36 from 1.22.17 to 1.23.17
  • 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-35 from 1.22.17 to 1.23.17
  • 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-73 from 1.22.17 to 1.23.17
  • 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-72 from 1.22.17 to 1.23.17
  • 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-53 from 1.22.17 to 1.23.17
  • 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-52 from 1.22.17 to 1.23.17
  • 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-35 from 1.22.17 to 1.23.17
  • 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-34 from 1.22.17 to 1.23.17
  • 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-72 from 1.22.17 to 1.23.17
  • 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-71 from 1.22.17 to 1.23.17
  • 11:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-52 from 1.22.17 to 1.23.17
  • 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-51 from 1.22.17 to 1.23.17
  • 11:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-34 from 1.22.17 to 1.23.17
  • 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-33 from 1.22.17 to 1.23.17
  • 11:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-71 from 1.22.17 to 1.23.17
  • 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-51 from 1.22.17 to 1.23.17
  • 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-33 from 1.22.17 to 1.23.17
  • 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-32 from 1.22.17 to 1.23.17
  • 11:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-32 from 1.22.17 to 1.23.17
  • 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-31 from 1.22.17 to 1.23.17
  • 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-31 from 1.22.17 to 1.23.17
  • 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-30 from 1.22.17 to 1.23.17
  • 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-30 from 1.22.17 to 1.23.17
  • 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-6 from 1.22.17 to 1.23.17
  • 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-6 from 1.22.17 to 1.23.17
  • 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-5 from 1.22.17 to 1.23.17
  • 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-5 from 1.22.17 to 1.23.17
  • 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-4 from 1.22.17 to 1.23.17
  • 11:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-4 from 1.22.17 to 1.23.17
  • 11:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.22.17 to 1.23.17 (T298005)
  • 11:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.22.17 to 1.23.17 (T298005)

2023-10-16

  • 09:04 dcaro: rebooting tools-k8s-worker-45 due to stuck nfs processes

2023-10-13

  • 13:29 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 13:28 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
  • 09:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
  • 08:57 dcaro: rebooting tools-sgeexec-10-8 as the host is stuck/unreachable
  • 07:43 dcaro: rebooting tools-sgeweblight-10-26 as it fails to allocate memory

2023-10-12

  • 15:07 taavi: reboot tools-k8s-worker-70
  • 14:01 taavi: deploy jobs-cli v15 T348250
  • 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-jobs-framework-cli' version '15'
  • 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-jobs-framework-cli' version '15'
  • 12:21 dcaro: rebooting sgeexec-10-17
  • 12:02 taavi: also reboot tools-sgeweblight-10-30
  • 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
  • 11:52 taavi: reboot tools-sgeweblight-10-22, 28

2023-10-11

  • 19:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers
  • 17:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers
  • 14:41 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for weblight nodes (T348634)
  • 14:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.reboot_workers for weblight nodes (T348634)
  • 14:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 14:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 14:19 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 14:19 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 14:16 dcaro: rebooting tools-sgeweblight-10-16 due to stuck NFS (T348634)
  • 12:11 taavi: reboot k8s workers 48, 60, 65, 68, 70, 76 T348634
  • 12:04 taavi: reboot k8s workers 72, 75, 82 T348634
  • 12:01 taavi: reboot tools-sgecron-2 T348634
  • 11:49 taavi: reboot tools-sgeexec-10-19

2023-10-10

  • 08:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api

2023-10-09

  • 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.4.0'
  • 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.4.0'
  • 08:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 08:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 07:14 taavi: deploy jobs-framework-cli v14
  • 07:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-jobs-framework-cli' version '14'
  • 07:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-jobs-framework-cli' version '14'

2023-10-05

  • 09:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-jobs-framework-cli' version '13'
  • 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-jobs-framework-cli' version '13'
  • 07:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
  • 07:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder

2023-10-04

  • 16:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 16:54 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 16:20 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 16:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 13:16 taavi: rollout toolforge-weld 1.3.0
  • 13:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.3.0'
  • 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.3.0'
  • 13:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
  • 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
  • 07:40 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 07:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api

2023-10-03

  • 13:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 13:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
  • 12:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
  • 09:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission
  • 09:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission

2023-10-02

  • 08:38 dcaro: rollout toolforge-cli 0.3.4

2023-10-01

  • 14:43 andrewbogott: rebooting tools-sgegrid-shadow because it's fussing about nfs

2023-09-29

  • 10:48 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers (T347665)
  • 10:20 wm-bot2: taavi@runko END (PASS) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=0) for exec nodes
  • 10:14 wm-bot2: taavi@runko END (PASS) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=0) for weblight nodes
  • 09:59 wm-bot2: taavi@runko START - Cookbook wmcs.toolforge.grid.reboot_workers for exec nodes
  • 09:58 wm-bot2: taavi@runko END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for exec nodes
  • 09:58 wm-bot2: taavi@runko START - Cookbook wmcs.toolforge.grid.reboot_workers for exec nodes
  • 09:57 wm-bot2: taavi@runko END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for exec nodes
  • 09:56 wm-bot2: taavi@runko START - Cookbook wmcs.toolforge.grid.reboot_workers for exec nodes
  • 09:55 wm-bot2: taavi@runko END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for exec nodes
  • 09:52 wm-bot2: taavi@runko END (PASS) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=0) for webgen nodes
  • 09:51 wm-bot2: taavi@runko START - Cookbook wmcs.toolforge.grid.reboot_workers for exec nodes
  • 09:51 wm-bot2: taavi@runko END (FAIL) - Cookbook wmcs.toolforge.grid.reboot_workers (exit_code=99) for exec nodes
  • 09:15 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-76 (T347665)
  • 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-76 (T347665)
  • 09:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-72 (T347665)
  • 09:04 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-72 (T347665)

2023-09-27

  • 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
  • 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config

2023-09-26

  • 00:07 andrewbogott: rebooting tools-puppetdb-1 in case that straightens out the puppet failures

2023-09-25

  • 09:39 dcaro: deploying builds-builder 0.0.71
  • 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
  • 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx

2023-09-22

  • 10:17 taavi: reboot tools-prometheus-6
  • 10:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 09:32 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

2023-09-21

  • 16:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' (T344717)
  • 16:03 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' (T344717)
  • 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0)
  • 16:02 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase
  • 15:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' (T344717)
  • 15:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' (T344717)
  • 15:24 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' (T344717)
  • 15:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' (T344717)

2023-09-20

  • 19:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-webservice' version '0.103'
  • 19:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-webservice' version '0.103'
  • 11:04 taavi: deploying toolforge-webservice 0.102
  • 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-webservice' version '0.102'
  • 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-webservice' version '0.102'
  • 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
  • 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
  • 06:20 taavi: reboot tools-sgebastion-11 due to stuck NFS handles

2023-09-19

  • 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx
  • 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx
  • 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
  • 14:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
  • 09:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)
  • 09:54 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 09:51 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
  • 09:51 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

2023-09-18

  • 10:41 dhinus: restarted stuck pod (webservice stop+start) T346126
  • 07:37 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-64 (T346123)
  • 07:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-64 (T346123)

2023-09-17

  • 18:12 taavi: reboot tools-sgeexec-10-22

2023-09-15

  • 12:32 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
  • 12:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
  • 12:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-65 (T346123)
  • 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-65 (T346123)
  • 11:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-34 (T346123)
  • 11:46 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-34 (T346123)
  • 10:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-52 (T346123)
  • 10:02 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-52 (T346123)
  • 10:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-48 (T346123)
  • 09:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-48 (T346123)
  • 09:52 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-75 (T346123)
  • 09:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-75 (T346123)
  • 09:28 dcaro: rebooting tools-sge-cron-2 (T346123)
  • 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-70 (T346123)
  • 09:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70 (T346123)
  • 09:10 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-69 (T346123)
  • 09:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-69 (T346123)
  • 08:49 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-78 (T346123)
  • 08:48 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-78 (T346123)
  • 08:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-76 (T346126)
  • 08:36 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-76 (T346126)

2023-09-14

  • 16:11 dcaro: increasing secrets quota to 30 (T339916)
  • 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
  • 12:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
  • 12:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer
  • 12:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer
  • 12:01 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission
  • 12:00 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission
  • 10:12 dcaro: deploy bulids-api 0.0.96
  • 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission
  • 09:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission
  • 08:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone (T341084)
  • 08:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone (T341084)

2023-09-13

  • 17:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)
  • 17:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
  • 12:51 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusersNone (T341084)
  • 12:51 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
  • 12:41 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusersNone (T341084)
  • 12:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
  • 12:40 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusersNone (T341084)
  • 12:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
  • 10:41 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone (T341084)
  • 10:41 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
  • 10:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)
  • 10:38 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)
  • 10:35 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone (T341084)
  • 10:34 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone (T341084)

2023-09-12

  • 15:25 andrewbogott: rebooting tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud, oom
  • 09:02 taavi: restart a bunch of sge nodes due to NFS lockups
  • 08:43 taavi: reboot tools-sgebastion-10 due to stuck NFS mounts

2023-09-11

  • 12:34 dcaro: deploy kubernetes-metrics (T341084)

2023-09-05

  • 13:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone (T341462)
  • 13:31 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone (T341462)
  • 11:00 dhinus: restarting mariadb on toolsdb-2 (replica) to test slave_parallel_threads (T345450)

2023-09-01

  • 12:12 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
  • 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 12:12 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
  • 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console
  • 11:54 taavi: reboot unresponsible tools-sgeweblight-10-21
  • 09:03 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99)
  • 09:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console

2023-08-31

  • 13:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0)
  • 13:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors
  • 12:52 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0)
  • 12:51 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.get_cluster_status
  • 09:50 wm-bot2: deployed kubernetes component api-gateway (c0faf0f) (T341462) - cookbook ran by dcaro@urcuchillay
  • 09:49 wm-bot2: deployed kubernetes component jobs-api (9c9bee0) (T341462) - cookbook ran by dcaro@urcuchillay
  • 09:48 wm-bot2: deployed kubernetes component api-gateway (9c9bee0) (T341462) - cookbook ran by dcaro@urcuchillay
  • 09:43 wm-bot2: deployed kubernetes component api-gateway (485046b) (T341462) - cookbook ran by dcaro@urcuchillay
  • 09:41 wm-bot2: deployed kubernetes component api-gateway (c0faf0f) (T341462) - cookbook ran by dcaro@urcuchillay

2023-08-30

  • 10:06 dcaro: upgrade toolforge-weld to 1.2.1 (T344155)
  • 08:59 dcaro: restarting harbor to flush caches (T344435)
  • 08:43 dcaro: cleaning up empty harbor projects (T344435)

2023-08-29

  • 14:17 wm-bot2: deployed kubernetes component jobs-api (485046b) (T341462) - cookbook ran by dcaro@urcuchillay
  • 13:06 wm-bot2: deployed kubernetes component jobs-emailer (6f9c8cf) - cookbook ran by taavi@runko

2023-08-28

  • 14:58 wm-bot2: deployed kubernetes component envvars-api (90055b5) (T344502) - cookbook ran by dcaro@urcuchillay

2023-08-25

  • 02:00 bd808: Reboot of login.toolforge.org hung until a hard reboot was triggered via horizon
  • 01:51 bd808: Scheduled reboot of login.toolforge.org for 2023-08-25 01:56:08 UTC

2023-08-22

2023-08-18

  • 13:46 wm-bot2: deployed kubernetes component envvars-api (06c26be) (T341462) - cookbook ran by dcaro@urcuchillay
  • 12:55 taavi: reboot frozen tools-sgebastion-10
  • 12:48 wm-bot2: deployed kubernetes component builds-api (727e6a7) (T341462) - cookbook ran by dcaro@urcuchillay

2023-08-17

  • 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-25c2b55f

2023-08-15

  • 23:29 bd808: Rebooted tools-db-1.tools.eqiad1.wikimedia.cloud for T344298

2023-07-26

  • 09:30 wm-bot2: deployed kubernetes component image-config (06066ba) - cookbook ran by taavi@runko

2023-07-25

  • 13:03 wm-bot2: deployed kubernetes component image-config (0eb287a) - cookbook ran by taavi@runko
  • 13:03 taavi: add php8.2 image T335352 T335507

2023-07-24

  • 21:45 bd808: Rebuilding container images for refactored config and new PHP 8.2 image (T335352)
  • 17:31 taavi: hard reboot tools-harbor-1, unresponsible

2023-07-23

  • 14:17 taavi: hard reboot tools-sgeexec-10-15

2023-07-20

2023-07-19

  • 16:34 wm-bot2: updating docker-registry.tools.wmflabs.org/toolforge-distroless-base@sha256:77051c1e40d180d0695b5a9ba7a15161ecac7220ea8c1ed6721bd1c8329b1b2f (T321188) - cookbook ran by arturo@nostromo
  • 16:30 wm-bot2: updating docker-registry.tools.wmflabs.org/toolforge-distroless-base@sha256:eebb155bd1116e3b67e2ce43244f9c9958df0cbb75a84c231565fae2ed87c9f4 (T321188) - cookbook ran by arturo@nostromo
  • 16:05 wm-bot2: updating docker-registry.tools.wmflabs.org/toolforge-distroless-base@sha256:c11cf17ee8a54dd3a44908ed3f38ffbfb41f1c8c6a2264de9b3e2f5ef4576006 (T321188) - cookbook ran by arturo@nostromo
  • 15:38 arturo: root@tools-docker-registry-05:~# docker-registry garbage-collect /etc/docker/registry/config.yml (T321188)
  • 15:37 arturo: root@tools-docker-registry-05:~# curl -sS -X DELETE localhost:5000/v2/toolforge-distroless-base/manifests/sha256:2d4d28e45bbe4e38177fd4fdc922dbfaf95e607b06bbc4187a90410d895b4491 (T321188)
  • 15:09 arturo: try to rescue docker-registry.tools.wmflabs.org/toolforge-distroless-base@sha256:eebb155bd1116e3b67e2ce43244f9c9958df0cbb75a84c231565fae2ed87c9f4 back into the registry from a k8s worker local cache (T321188)

2023-07-18

  • 11:02 arturo: redeploy jobs-emailer 0.0.41-20230718103342-3dddcfb8 into k8s (T341084)

2023-07-14

  • 22:45 taavi: reboot tools-sgebastion-11 (dev.toolforge.org) to recover from stuck NFS client causing a high load average
  • 09:48 dcaro: deploy builds-api 0.0.78, ci rebuild

2023-07-13

2023-07-12

  • 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-2ef80a7c (T341084)
  • 10:06 dcaro: deployed api-gateway 0.0.16, no changes, ci rebuild (T341084)

2023-07-11

  • 10:14 dcaro: deploy ingress-admission 0.0.38, ci rebuild (T341084)

2023-07-10

  • 20:39 taavi: freeing up disk space usage on tools docker-registry with `taavi@tools-docker-registry-05:~$ sudo sudo -u docker-registry docker-registry garbage-collect /etc/docker/registry/config.yml --delete-untagged`
  • 13:01 dcaro: deploy envvars-api 0.0.22 (T341462)
  • 09:27 dcaro: deploying calico-0.0.6-20230710081103-dcbbe692, just a rebuild (T341084)

2023-07-09

  • 13:26 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko

2023-07-05

  • 16:32 dcaro: deploy image-config 0.0.14 (no real changes, just ci rebuild)
  • 07:39 taavi: deploying jobs-api 0.0.213-20230705073411-09895639

2023-07-04

  • 17:06 taavi: deploy tools-webservice 0.101 for T341088
  • 16:38 dcaro: deploy volume-admission 0.0.40 (no real changes, just ci rebuild)
  • 11:44 dcaro: deploy jobs-api 0.0.212

2023-07-03

  • 19:09 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git (561b4d9) - cookbook ran by taavi@runko
  • 13:49 dcaro: deploy envvars-api 0.0.21 (no real changes, ci rebuild)
  • 13:29 dcaro: deploy builds-api 0.0.75 (no real changes, just ci rebuild)
  • 13:17 dcaro: deploy envvars-admission 0.0.8
  • 12:17 wm-bot2: Copied Apt package python3-toolforge-weld 1.1.1 to the tools Apt repo on bookworm, bullseye, buster - cookbook ran by taavi@runko
  • 12:16 wm-bot2: Copied Apt package python-toolforge-weld 1.1.1 to the tools Apt repo on - cookbook ran by taavi@runko
  • 12:12 taavi: deploy jobs-api 0.1.5
  • 12:01 dcaro: deploy builds-api 0.0.74
  • 09:24 dcaro: deploy envvars-api 0.0.20

2023-06-30

  • 18:21 taavi: deploy new jobs-api release to fix T340829

2023-06-29

  • 10:19 dcaro: deploy toolforge-cli 0.3.2

2023-06-27

  • 16:48 taavi: building initial set of bookworm based images: node18, ruby31, python311 (T335507)
  • 09:01 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@endurance
  • 08:54 arturo: force-reboot tools-sgeexec-10-15 (unresponsive)

2023-06-23

  • 15:42 dcaro: deploy builds-api 0.3.2 (T337025)

2023-06-22

  • 11:57 taavi: update toolforge-jobs-framework-cli to 12
  • 09:57 dcaro: deploy builds-api 0.3.1
  • 09:32 dcaro: deploy builds-api 0.3.0

2023-06-21

  • 11:57 dcaro: deploy bulids-api 0.2.0 (T337025)

2023-06-20

  • 14:21 taavi: fix gitlab merge settings for tools-webservice to match the agreed values (fast-forward, squash encouraged)
  • 12:11 dcaro: deploy toolforge-envvars-cli (upgrades pthyon3-toolforge-weld) (T337538)
  • 12:04 dcaro: deployed api-gateway with envvars endpoint support (T337538)
  • 11:59 dcaro: deploy buildservice with aptfile support (T336669)

2023-06-16

  • 16:26 andrewbogott: restarting apache2 on toolserver-proxy-01.tools.eqiad1.wikimedia.cloud in hopes of stopping a flapping alert
  • 08:15 dcaro: deployed latest builds-api 0.1.0

2023-06-15

  • 14:05 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by andrew@bullseye

2023-06-13

  • 14:27 dcaro: rebooted tools-harbor-1 as it was not responding

2023-06-12

  • 09:03 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo

2023-06-09

  • 19:57 andrewbogott: rebooting tools-sgeweblight-10-18 to see if it helps with T338644
  • 19:38 andrewbogott: rebooting tools-sgeweblight-10-28 for T337806

2023-06-08

  • 20:21 bd808: Rebuilding container images (T337897)
  • 14:16 dcaro: restart tools-sgeweblight-10-17.tools.eqiad1.wikimedia.cloud due to nfs hiccup
  • 14:07 dcaro: restarting the tools-sgeexec-10-17 node due to nfs hiccup
  • 14:00 dcaro: restarting the tools-sgegrid-master node due to nfs hiccup
  • 12:00 dcaro: powering off tools-k8s-etcd-18 (T334644)
  • 07:18 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (24e7828) - cookbook ran by taavi@runko

2023-06-07

2023-06-05

  • 07:53 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2023-06-01

2023-05-31

  • 02:38 andrewbogott: rebooted tools-sgeweblight-10-16, T337806

2023-05-30

  • 00:22 andrewbogott: rebooted tools-sgeweblight-10-30, oom
  • 00:16 andrewbogott: rebooted tools-sgeweblight-10-24, seems to be oom

2023-05-26

2023-05-24

  • 12:28 dcaro: deploy latest buildservice (T335865)
  • 12:28 dcaro: deploy latest buildservice (T336050)

2023-05-23

2023-05-22

  • 10:06 arturo: hard-reboot tools-sgeexec-10-18 (monitoring reporting it as down)

2023-05-19

  • 13:38 arturo: uncordon tools-k8s-worker-47/48/64/75
  • 08:46 bd808: Building new perl532-sssd/{base,web} images (T323522, T320904)

2023-05-17

  • 16:05 dcaro: release toolforge-cli 0.3.0 (T336225)
  • 12:48 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway (fa8ed2c) (T336225) - cookbook ran by dcaro@vulcanus
  • 12:48 wm-bot2: rebooted k8s node tools-k8s-worker-71 (T316544) - cookbook ran by dcaro@vulcanus
  • 12:45 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway (d1bb238) (T336225) - cookbook ran by dcaro@vulcanus
  • 12:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api (8d21314) - cookbook ran by dcaro@vulcanus
  • 10:54 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-buildpack-admission-controller:7199a9e from https://github.com/toolforge/buildpack-admission-controller (7199a9e) - cookbook ran by fran@wmf3169
  • 08:49 wm-bot2: rebooted k8s node tools-k8s-worker-55 (T316544) - cookbook ran by dcaro@vulcanus
  • 08:33 wm-bot2: rebooted k8s node tools-k8s-worker-64 (T316544) - cookbook ran by dcaro@vulcanus
  • 08:32 wm-bot2: rebooted k8s node tools-k8s-worker-75 (T316544) - cookbook ran by dcaro@vulcanus
  • 08:25 wm-bot2: rebooted k8s node tools-k8s-worker-74 (T316544) - cookbook ran by dcaro@vulcanus
  • 08:17 wm-bot2: rebooted k8s node tools-k8s-worker-61 (T316544) - cookbook ran by dcaro@vulcanus
  • 08:10 wm-bot2: rebooted k8s node tools-k8s-worker-70 (T316544) - cookbook ran by dcaro@vulcanus
  • 08:03 wm-bot2: rebooted k8s node tools-k8s-worker-66 (T316544) - cookbook ran by dcaro@vulcanus
  • 07:54 wm-bot2: rebooted k8s node tools-k8s-worker-72 (T316544) - cookbook ran by dcaro@vulcanus
  • 07:46 wm-bot2: rebooted k8s node tools-k8s-worker-47 (T316544) - cookbook ran by dcaro@vulcanus
  • 07:45 wm-bot2: rebooted k8s node tools-k8s-worker-48 (T316544) - cookbook ran by dcaro@vulcanus
  • 07:42 wm-bot2: rebooted k8s node tools-k8s-worker-69 (T316544) - cookbook ran by dcaro@vulcanus
  • 07:29 wm-bot2: rebooted k8s node tools-k8s-worker-76 (T316544) - cookbook ran by dcaro@vulcanus

2023-05-16

2023-05-15

  • 22:50 bd808: Rebuilding bullseye and buster docker containers to pick up make package addition (T320343)
  • 22:09 wm-bot2: rebooted k8s node tools-k8s-worker-66 (T316544) - cookbook ran by andrew@bullseye
  • 22:07 wm-bot2: rebooted k8s node tools-k8s-worker-65 (T316544) - cookbook ran by andrew@bullseye
  • 22:06 wm-bot2: rebooted k8s node tools-k8s-worker-64 (T316544) - cookbook ran by andrew@bullseye
  • 22:04 wm-bot2: rebooted k8s node tools-k8s-worker-62 (T316544) - cookbook ran by andrew@bullseye
  • 22:02 wm-bot2: rebooted k8s node tools-k8s-worker-61 (T316544) - cookbook ran by andrew@bullseye
  • 21:58 wm-bot2: rebooted k8s node tools-k8s-worker-60 (T316544) - cookbook ran by andrew@bullseye
  • 21:56 wm-bot2: rebooted k8s node tools-k8s-worker-59 (T316544) - cookbook ran by andrew@bullseye
  • 21:54 wm-bot2: rebooted k8s node tools-k8s-worker-58 (T316544) - cookbook ran by andrew@bullseye
  • 21:52 wm-bot2: rebooted k8s node tools-k8s-worker-57 (T316544) - cookbook ran by andrew@bullseye
  • 21:51 wm-bot2: rebooted k8s node tools-k8s-worker-56 (T316544) - cookbook ran by andrew@bullseye
  • 21:50 wm-bot2: rebooted k8s node tools-k8s-worker-55 (T316544) - cookbook ran by andrew@bullseye
  • 21:49 wm-bot2: rebooted k8s node tools-k8s-worker-54 (T316544) - cookbook ran by andrew@bullseye
  • 21:47 wm-bot2: rebooted k8s node tools-k8s-worker-53 (T316544) - cookbook ran by andrew@bullseye
  • 21:44 wm-bot2: rebooted k8s node tools-k8s-worker-52 (T316544) - cookbook ran by andrew@bullseye
  • 21:42 wm-bot2: rebooted k8s node tools-k8s-worker-51 (T316544) - cookbook ran by andrew@bullseye
  • 21:41 wm-bot2: rebooted k8s node tools-k8s-worker-50 (T316544) - cookbook ran by andrew@bullseye
  • 21:40 wm-bot2: rebooted k8s node tools-k8s-worker-49 (T316544) - cookbook ran by andrew@bullseye
  • 21:38 wm-bot2: rebooted k8s node tools-k8s-worker-48 (T316544) - cookbook ran by andrew@bullseye
  • 21:37 wm-bot2: rebooted k8s node tools-k8s-worker-47 (T316544) - cookbook ran by andrew@bullseye
  • 21:33 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by andrew@bullseye
  • 21:16 wm-bot2: rebooted k8s node tools-k8s-worker-45 (T316544) - cookbook ran by dcaro@vulcanus
  • 21:15 wm-bot2: rebooted k8s node tools-k8s-worker-44 (T316544) - cookbook ran by dcaro@vulcanus
  • 21:13 wm-bot2: rebooted k8s node tools-k8s-worker-43 (T316544) - cookbook ran by dcaro@vulcanus
  • 21:12 wm-bot2: rebooted k8s node tools-k8s-worker-42 (T316544) - cookbook ran by dcaro@vulcanus
  • 21:09 wm-bot2: rebooted k8s node tools-k8s-worker-41 (T316544) - cookbook ran by dcaro@vulcanus
  • 21:03 wm-bot2: rebooted k8s node tools-k8s-worker-40 (T316544) - cookbook ran by dcaro@vulcanus
  • 20:58 wm-bot2: rebooted k8s node tools-k8s-worker-39 (T316544) - cookbook ran by dcaro@vulcanus
  • 20:52 wm-bot2: rebooted k8s node tools-k8s-worker-38 (T316544) - cookbook ran by dcaro@vulcanus
  • 20:50 wm-bot2: rebooted k8s node tools-k8s-worker-37 (T316544) - cookbook ran by dcaro@vulcanus
  • 20:49 wm-bot2: rebooted k8s node tools-k8s-worker-36 (T316544) - cookbook ran by dcaro@vulcanus
  • 20:48 wm-bot2: rebooted k8s node tools-k8s-worker-35 (T316544) - cookbook ran by dcaro@vulcanus
  • 20:47 wm-bot2: rebooted k8s node tools-k8s-worker-34 (T316544) - cookbook ran by dcaro@vulcanus
  • 20:42 wm-bot2: rebooted k8s node tools-k8s-worker-33 (T316544) - cookbook ran by dcaro@vulcanus
  • 20:41 andrewbogott: rebooting frozen VMs: tools-k8s-worker-65, tools-sgeweblight-10-27, tools-k8s-worker-45, tools-k8s-worker-36, tools-sgewebgen-10-3 (fallout from earlier nfs outage)
  • 20:36 wm-bot2: rebooted k8s node tools-k8s-worker-32 (T316544) - cookbook ran by dcaro@vulcanus
  • 20:32 wm-bot2: rebooted k8s node tools-k8s-worker-31 (T316544) - cookbook ran by dcaro@vulcanus
  • 20:24 wm-bot2: rebooted k8s node tools-k8s-worker-30 (T316544) - cookbook ran by dcaro@vulcanus
  • 19:04 wm-bot2: rebooted k8s node tools-k8s-worker-67 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:56 wm-bot2: rebooted k8s node tools-k8s-worker-68 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:49 wm-bot2: rebooted k8s node tools-k8s-worker-69 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:46 bd808: Hard reboot tools-static-14 via Horizon per IRC report of unresponsive requests
  • 18:44 wm-bot2: rebooted k8s node tools-k8s-worker-70 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:42 wm-bot2: rebooted k8s node tools-k8s-worker-71 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:39 wm-bot2: rebooted k8s node tools-k8s-worker-72 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:34 wm-bot2: rebooted k8s node tools-k8s-worker-73 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:28 wm-bot2: rebooted k8s node tools-k8s-worker-74 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:22 wm-bot2: rebooted k8s node tools-k8s-worker-75 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:22 taavi: clear mail queue
  • 18:21 wm-bot2: rebooted k8s node tools-k8s-worker-76 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:15 wm-bot2: rebooted k8s node tools-k8s-worker-77 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:08 wm-bot2: rebooted k8s node tools-k8s-worker-80 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:06 wm-bot2: rebooted k8s node tools-k8s-worker-81 (T316544) - cookbook ran by dcaro@vulcanus
  • 18:05 wm-bot2: rebooted k8s node tools-k8s-worker-82 (T316544) - cookbook ran by dcaro@vulcanus
  • 17:57 wm-bot2: rebooted k8s node tools-k8s-worker-83 (T316544) - cookbook ran by dcaro@vulcanus
  • 17:48 wm-bot2: rebooted k8s node tools-k8s-worker-84 (T316544) - cookbook ran by dcaro@vulcanus
  • 17:47 wm-bot2: rebooted k8s node tools-k8s-worker-85 (T316544) - cookbook ran by dcaro@vulcanus
  • 17:38 wm-bot2: rebooted k8s node tools-k8s-worker-86 (T316544) - cookbook ran by dcaro@vulcanus
  • 17:37 wm-bot2: rebooted k8s node tools-k8s-worker-87 (T316544) - cookbook ran by dcaro@vulcanus
  • 17:35 wm-bot2: rebooted k8s node tools-k8s-worker-88 (T316544) - cookbook ran by dcaro@vulcanus
  • 17:34 wm-bot2: rebooting all the workers of tools k8s cluster (64 nodes) (T316544) - cookbook ran by dcaro@vulcanus
  • 17:20 wm-bot2: rebooted k8s node tools-k8s-worker-87 (T316544) - cookbook ran by dcaro@vulcanus
  • 17:19 wm-bot2: rebooted k8s node tools-k8s-worker-88 (T316544) - cookbook ran by dcaro@vulcanus
  • 17:17 bd808: Rebuilding bullseye and buster docker containers to pick up openssh-client package addition (T258841)
  • 17:12 wm-bot2: rebooting the whole tools k8s cluster (64 nodes) (T316544) - cookbook ran by dcaro@vulcanus
  • 17:06 dcaro: rebooting tools-sgegrid-shadow (T316544)
  • 17:00 dcaro: rebooting tools-sgegrid-master (T316544)
  • 16:55 dcaro: rebooting tools-sgeexec-10-20 (T316544)
  • 16:53 dcaro: rebooting tools-sgeweblight-10-18 (T316544)
  • 16:53 dcaro: rebooting tools-sgeweblight-10-25 (T316544)
  • 16:53 dcaro: rebooting tools-sgeweblight-10-20 (T316544)
  • 16:52 dcaro: rebooting tools-sgeweblight-10-21 (T316544)
  • 16:52 dcaro: rebooting tools-sgeexec-10-22 (T316544)
  • 16:51 dcaro: rebooting tools-sgeweblight-10-28 (T316544)
  • 16:50 dcaro: rebooting tools-sgeexec-10-17 (T316544)
  • 16:48 dcaro: rebooting tools-sgeexec-10-21 (T316544)
  • 16:47 dcaro: rebooting tools-sgeexec-10-19 (T316544)
  • 16:45 dcaro: rebooting tools-sgeexec-10-8 (T316544)
  • 16:45 dcaro: rebooting tools-sgeweblight-10-24 (T316544)
  • 16:44 dcaro: rebooting tools-sgewebgen-10-2 (T316544)
  • 16:44 dcaro: rebooting tools-sgeweblight-10-16 (T316544)
  • 16:43 dcaro: rebooting tools-sgeweblight-10-30 (T316544)
  • 16:43 dcaro: rebooting tools-sgeexec-10-18 (T316544)
  • 16:42 dcaro: rebooting tools-sgeexec-10-16 (T316544)
  • 16:42 dcaro: rebooting tools-sgeexec-10-14 (T316544)
  • 16:41 dcaro: rebooting tools-sgeweblight-10-32 (T316544)
  • 16:40 dcaro: rebooting tools-sgeweblight-10-22 (T316544)
  • 16:39 dcaro: rebooting tools-sgeweblight-10-17 (T316544)
  • 16:32 dcaro: rebooting tools-sgeexec-10-13.tools.eqiad1.wikimedia.cloud (T316544)
  • 16:23 dcaro: rebooting tools-sgeweblight-10-26 (T316544)
  • 16:15 bd808: Hard reboot of tools-sgebastion-11 via Horizon (done circa 16:11Z)
  • 16:14 arturo: rebooted a bunch of nodes to cleanup D procs and high load avg because NFS outage (result of T316544)
  • 12:36 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/builds-api:09f3b49-dev from https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api.git (32a8ae9) - cookbook ran by dcaro@vulcanus
  • 09:12 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/volume-admission:c64da5a from https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller (c64da5a) - cookbook ran by dcaro@vulcanus

2023-05-13

  • 09:13 taavi: reboot tools-sgeexec-10-15,17,18,21

2023-05-11

  • 15:48 bd808: Rebooted tools-sgebastion-10 for T336510
  • 15:31 bd808: Sent `wall` for reboot of tools-sgebastion-10 circa 15:40Z

2023-05-09

2023-05-08

  • 09:12 arturo: force-reboot tools-sgeexec-10-13 (reported as down by the monitoring, no SSH)

2023-05-07

  • 16:06 taavi: remove inbound 25/tcp rule from the toolserver legacy server T136225

2023-05-05

2023-05-04

  • 15:15 wm-bot2: removed instance tools-k8s-etcd-15 - cookbook ran by andrew@bullseye
  • 14:13 wm-bot2: removed instance tools-k8s-etcd-14 - cookbook ran by andrew@bullseye

2023-05-03

  • 12:41 wm-bot2: removed instance tools-k8s-etcd-13 - cookbook ran by andrew@bullseye

2023-05-02

2023-05-01

2023-04-28

  • 15:01 arturo: force reboot tools-k8s-worker-79, unresponsive
  • 08:27 dcaro: rebooting tools-sgeweblight-10-28 (T335336)
  • 07:20 dcaro: rebooting tools-sgegrid-shadow due to stale nfs mount
  • 00:09 bd808: `kubectl uncordon tools-k8s-worker-67` (T335543)
  • 00:07 bd808: Hard reboot tools-k8s-worker-67.tools.eqiad1.wikimedia.cloud via horizon (T335543)
  • 00:04 bd808: Rebooting tools-k8s-worker-67.tools.eqiad1.wikimedia.cloud (T335543)

2023-04-27

  • 23:59 bd808: `kubectl drain --ignore-daemonsets --delete-emptydir-data --force tools-k8s-worker-67` (T335543)
  • 20:50 bd808: Started process to rebuild all buster and bullseye based container images again. Prior problem seems to have been stale images in local cache on the build server.
  • 20:42 bd808: Container image rebuild failed with GPG errors in buster-sssd base image. Will investigate and attempt to restart once resolved in a local dev environment.
  • 20:33 bd808: Started process to rebuild all buster and bullseye based container images per https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Building_toolforge_specific_images

2023-04-18

  • 16:46 dcaro: force-rebooting tools-sgeweblight-10-25/26/27 as they got stuck stopping the grid_exec process
  • 16:35 dcaro: rebooting root@tools-sgeweblight-10-27 due to stuck exec daemon not releasing port 6445
  • 16:35 dcaro: rebooting root@tools-sgeweblight-10-25 due to stuck exec daemon not releasing port 6445
  • 16:32 dcaro: rebooting root@tools-sgeweblight-10-26 due to stuck exec daemon not releasing port 6445
  • 16:26 dcaro: rebooting root@tools-sgeexec-10-14 due to stuck exec daemon not releasing port 6445

2023-04-17

  • 13:10 dcaro: rebooting tools-sgegrid-master node (T334847)
  • 02:43 legoktm: manual restart of apache2 on toolserver-proxy-1 to completely pick up renewed TLS cert (alert was flapping)

2023-04-11

2023-04-10

  • 10:46 taavi: patch existing PSP roles to use policy/v1beta1 T331619
  • 09:16 arturo: upgrading k8s cluster to 1.22 (T286856)

2023-04-07

  • 14:34 wm-bot2: drained, depooled and removed k8s control node tools-k8s-control-3 (T333929) - cookbook ran by taavi@runko
  • 14:30 wm-bot2: removed instance tools-k8s-control-2 - cookbook ran by taavi@runko

2023-04-05

  • 15:16 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (5ea5992) - cookbook ran by taavi@runko
  • 15:10 wm-bot2: build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:3569803 from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (3569803) - cookbook ran by taavi@runko
  • 14:56 wm-bot2: Added a new k8s worker tools-k8s-worker-88.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
  • 14:42 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
  • 14:42 wm-bot2: Added a new k8s worker tools-k8s-worker-87.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
  • 14:28 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
  • 14:28 wm-bot2: Added a new k8s worker tools-k8s-worker-86.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
  • 14:15 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
  • 14:15 wm-bot2: Added a new k8s worker tools-k8s-worker-85.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
  • 14:01 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
  • 14:01 wm-bot2: Added a new k8s worker tools-k8s-worker-84.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
  • 13:47 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
  • 13:47 wm-bot2: Added a new k8s worker tools-k8s-worker-83.tools.eqiad1.wikimedia.cloud to the cluster (T333972) - cookbook ran by taavi@runko
  • 13:34 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
  • 13:33 wm-bot2: removed instance tools-k8s-worker-83 - cookbook ran by taavi@runko
  • 13:15 wm-bot2: Adding a new k8s worker node (T333972) - cookbook ran by taavi@runko
  • 13:06 wm-bot2: removing grid node tools-sgeweblight-10-31.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
  • 13:02 wm-bot2: removing grid node tools-sgeweblight-10-29.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
  • 13:00 wm-bot2: removing grid node tools-sgeexec-10-9.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
  • 12:58 wm-bot2: removing grid node tools-sgeweblight-10-15.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
  • 12:54 wm-bot2: removing grid node tools-sgeexec-10-7.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
  • 12:52 wm-bot2: removing grid node tools-sgeweblight-10-13.tools.eqiad1.wikimedia.cloud (T333972) - cookbook ran by taavi@runko
  • 12:34 wm-bot2: drained, depooled and removed k8s control node tools-k8s-control-1 - cookbook ran by taavi@runko
  • 12:07 wm-bot2: Added a new k8s control tools-k8s-control-6.tools.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko
  • 11:53 wm-bot2: Adding a new k8s control node - cookbook ran by taavi@runko
  • 11:51 wm-bot2: removed instance tools-k8s-control-6 - cookbook ran by taavi@runko
  • 11:39 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
  • 11:38 wm-bot2: removed instance tools-k8s-control-6 - cookbook ran by taavi@runko
  • 11:21 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
  • 11:21 wm-bot2: removed instance tools-k8s-control-6 - cookbook ran by taavi@runko
  • 11:09 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
  • 10:53 wm-bot2: removed instance tools-k8s-control-6 - cookbook ran by taavi@runko
  • 10:41 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
  • 10:41 wm-bot2: removed instance tools-k8s-control-6 - cookbook ran by taavi@runko
  • 10:16 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko

2023-04-04

  • 19:00 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
  • 18:59 wm-bot2: removed instance tools-k8s-control-5 - cookbook ran by taavi@runko
  • 18:46 wm-bot2: Adding a new k8s control node (T333929) - cookbook ran by taavi@runko
  • 18:45 wm-bot2: Adding a new k8s CONTROL node (T333929) - cookbook ran by taavi@runko
  • 10:15 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo
  • 09:28 arturo: hard-reboot the 3 k8s control nodes

2023-04-03

  • 17:13 wm-bot2: rebooted k8s node tools-k8s-worker-31 - cookbook ran by taavi@runko
  • 17:11 wm-bot2: rebooted k8s node tools-k8s-worker-32 - cookbook ran by taavi@runko
  • 17:09 wm-bot2: rebooted k8s node tools-k8s-worker-33 - cookbook ran by taavi@runko
  • 17:07 wm-bot2: rebooted k8s node tools-k8s-worker-34 - cookbook ran by taavi@runko
  • 17:05 wm-bot2: rebooted k8s node tools-k8s-worker-35 - cookbook ran by taavi@runko
  • 17:04 wm-bot2: rebooted k8s node tools-k8s-worker-36 - cookbook ran by taavi@runko
  • 17:02 wm-bot2: rebooted k8s node tools-k8s-worker-37 - cookbook ran by taavi@runko
  • 17:00 wm-bot2: rebooted k8s node tools-k8s-worker-38 - cookbook ran by taavi@runko
  • 16:58 wm-bot2: rebooted k8s node tools-k8s-worker-39 - cookbook ran by taavi@runko
  • 16:56 wm-bot2: rebooted k8s node tools-k8s-worker-40 - cookbook ran by taavi@runko
  • 16:55 wm-bot2: rebooted k8s node tools-k8s-worker-41 - cookbook ran by taavi@runko
  • 16:53 wm-bot2: rebooted k8s node tools-k8s-worker-42 - cookbook ran by taavi@runko
  • 16:51 wm-bot2: rebooted k8s node tools-k8s-worker-43 - cookbook ran by taavi@runko
  • 16:49 wm-bot2: rebooted k8s node tools-k8s-worker-44 - cookbook ran by taavi@runko
  • 16:45 wm-bot2: rebooted k8s node tools-k8s-worker-45 - cookbook ran by taavi@runko
  • 16:43 wm-bot2: rebooted k8s node tools-k8s-worker-46 - cookbook ran by taavi@runko
  • 16:41 wm-bot2: rebooted k8s node tools-k8s-worker-47 - cookbook ran by taavi@runko
  • 16:40 wm-bot2: rebooted k8s node tools-k8s-worker-48 - cookbook ran by taavi@runko
  • 16:38 wm-bot2: rebooted k8s node tools-k8s-worker-49 - cookbook ran by taavi@runko
  • 16:36 wm-bot2: rebooted k8s node tools-k8s-worker-50 - cookbook ran by taavi@runko
  • 16:35 wm-bot2: rebooted k8s node tools-k8s-worker-51 - cookbook ran by taavi@runko
  • 16:33 wm-bot2: rebooted k8s node tools-k8s-worker-52 - cookbook ran by taavi@runko
  • 16:31 wm-bot2: rebooted k8s node tools-k8s-worker-53 - cookbook ran by taavi@runko
  • 16:28 wm-bot2: rebooted k8s node tools-k8s-worker-54 - cookbook ran by taavi@runko
  • 16:27 wm-bot2: rebooted k8s node tools-k8s-worker-55 - cookbook ran by taavi@runko
  • 16:25 wm-bot2: rebooted k8s node tools-k8s-worker-56 - cookbook ran by taavi@runko
  • 16:23 wm-bot2: rebooted k8s node tools-k8s-worker-57 - cookbook ran by taavi@runko
  • 16:21 wm-bot2: rebooted k8s node tools-k8s-worker-58 - cookbook ran by taavi@runko
  • 16:20 wm-bot2: rebooted k8s node tools-k8s-worker-59 - cookbook ran by taavi@runko
  • 16:18 wm-bot2: rebooted k8s node tools-k8s-worker-60 - cookbook ran by taavi@runko
  • 16:09 wm-bot2: rebooted k8s node tools-k8s-worker-61 - cookbook ran by taavi@runko
  • 16:07 wm-bot2: rebooted k8s node tools-k8s-worker-62 - cookbook ran by taavi@runko
  • 16:01 wm-bot2: rebooted k8s node tools-k8s-worker-64 - cookbook ran by taavi@runko
  • 16:00 wm-bot2: rebooting the whole tools k8s cluster (58 nodes) - cookbook ran by taavi@runko
  • 15:58 wm-bot2: rebooted k8s node tools-k8s-worker-65 - cookbook ran by taavi@runko
  • 15:56 wm-bot2: rebooted k8s node tools-k8s-worker-66 - cookbook ran by taavi@runko
  • 15:48 wm-bot2: rebooted k8s node tools-k8s-worker-67 - cookbook ran by taavi@runko
  • 15:38 wm-bot2: rebooted k8s node tools-k8s-worker-68 - cookbook ran by taavi@runko
  • 15:36 wm-bot2: rebooted k8s node tools-k8s-worker-69 - cookbook ran by taavi@runko
  • 15:34 wm-bot2: rebooted k8s node tools-k8s-worker-70 - cookbook ran by taavi@runko
  • 15:32 wm-bot2: rebooted k8s node tools-k8s-worker-71 - cookbook ran by taavi@runko
  • 15:30 wm-bot2: rebooted k8s node tools-k8s-worker-72 - cookbook ran by taavi@runko
  • 15:28 wm-bot2: rebooted k8s node tools-k8s-worker-73 - cookbook ran by taavi@runko
  • 15:26 wm-bot2: rebooted k8s node tools-k8s-worker-74 - cookbook ran by taavi@runko
  • 15:24 wm-bot2: rebooted k8s node tools-k8s-worker-75 - cookbook ran by taavi@runko
  • 15:22 wm-bot2: rebooting the whole tools k8s cluster (58 nodes) - cookbook ran by taavi@runko
  • 15:17 wm-bot2: rebooted k8s node tools-k8s-worker-75 - cookbook ran by taavi@runko
  • 15:14 wm-bot2: rebooted k8s node tools-k8s-worker-76 - cookbook ran by taavi@runko
  • 15:12 wm-bot2: rebooted k8s node tools-k8s-worker-77 - cookbook ran by taavi@runko
  • 15:10 wm-bot2: rebooted k8s node tools-k8s-worker-78 - cookbook ran by taavi@runko
  • 15:08 wm-bot2: rebooted k8s node tools-k8s-worker-79 - cookbook ran by taavi@runko
  • 15:06 wm-bot2: rebooted k8s node tools-k8s-worker-80 - cookbook ran by taavi@runko
  • 14:59 wm-bot2: rebooted k8s node tools-k8s-worker-81 - cookbook ran by taavi@runko
  • 14:41 wm-bot2: rebooted k8s node tools-k8s-worker-82 - cookbook ran by taavi@runko
  • 14:38 wm-bot2: rebooting the whole tools k8s cluster (58 nodes) - cookbook ran by taavi@runko
  • 14:13 andrewbogott: test log to see if stashbot is back working
  • 13:19 andrewbogott: forcing puppet run on all toolforge VMs
  • 08:28 taavi: stop exim4.service on tools-sgecron-2 T333477
  • 06:52 taavi: stop jobs-framework-emailer to prevent spam due to NFS being read-only T333477

2023-03-29

2023-03-28

2023-03-27

2023-03-26

  • 20:28 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko

2023-03-24

  • 14:13 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@endurance

2023-03-21

  • 08:11 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko

2023-03-20

  • 13:39 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko
  • 10:57 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@endurance

2023-03-19

  • 09:32 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko

2023-03-17

  • 15:56 andrewbogott: truncating .out, .err, and .log files to 10MB in anticipation of moving the NFS volumes

2023-03-13

2023-03-12

  • 13:40 taavi: restart haproxy on tools-k8s-haproxy-3

2023-03-11

  • 18:38 wm-bot2: removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 18:36 wm-bot2: removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 18:34 wm-bot2: removing grid node tools-sgeexec-10-11.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 18:31 taavi: reboot misbehaving tools-sgeexec-10-11

2023-03-10

2023-03-09

2023-03-08

  • 22:31 bd808: Live hacked user-maintainer clusterrole to work around breakage in T331572

2023-03-07

  • 11:34 wm-bot2: Increased quotas by 2 volumes - cookbook ran by fran@wmf3169
  • 11:09 wm-bot2: Increased quotas by 6 snapshots - cookbook ran by fran@wmf3169
  • 11:07 wm-bot2: Increased quotas by 4000 gigabytes - cookbook ran by fran@wmf3169

2023-03-06

2023-03-05

2023-03-02

2023-03-01

2023-02-28

2023-02-23

2023-02-21

  • 09:37 arturo: hard-reboot tools-sgeexec-10-11 (unresponsive to ssh)

2023-02-20

2023-02-19

  • 09:16 taavi: uncordon tools-k8s-worker-[80-82] after fixing security groups T329378

2023-02-17

2023-02-16

2023-02-15

2023-02-14

  • 15:07 taavi: import cert-manager components to local docker registry T329453
  • 12:12 arturo: the fixed webservicemonitor is starting a bunch of grid webservices (T329611)
  • 12:10 arturo: included tools-manifests 0.25 in tools-buster aptly repo, deploying it now! (T329611, T329467, T244809)

2023-02-13

2023-02-10

  • 15:45 taavi: reboot tools-k8s-worker-82 to troubleshoot network issues
  • 12:44 wm-bot2: Added a new k8s worker tools-k8s-worker-82.tools.eqiad1.wikimedia.cloud to the worker pool (T329357) - cookbook ran by taavi@runko
  • 12:31 wm-bot2: Adding a new k8s worker node (T329357) - cookbook ran by taavi@runko
  • 12:29 wm-bot2: Added a new k8s worker tools-k8s-worker-81.tools.eqiad1.wikimedia.cloud to the worker pool (T329357) - cookbook ran by taavi@runko
  • 12:15 wm-bot2: Adding a new k8s worker node (T329357) - cookbook ran by taavi@runko
  • 11:53 wm-bot2: Adding a new k8s worker node (T329357) - cookbook ran by taavi@runko
  • 11:44 wm-bot2: removing grid node tools-sgeweblight-10-23.tools.eqiad1.wikimedia.cloud (T329357) - cookbook ran by taavi@runko
  • 11:42 wm-bot2: removing grid node tools-sgeexec-10-5.tools.eqiad1.wikimedia.cloud (T329357) - cookbook ran by taavi@runko
  • 11:39 wm-bot2: removing grid node tools-sgeweblight-10-19.tools.eqiad1.wikimedia.cloud (T329357) - cookbook ran by taavi@runko
  • 11:26 wm-bot2: removing grid node tools-sgeweblight-10-12.tools.eqiad1.wikimedia.cloud (T329357) - cookbook ran by taavi@runko
  • 11:24 wm-bot2: removing grid node tools-sgeexec-10-1.tools.eqiad1.wikimedia.cloud (T329357) - cookbook ran by taavi@runko

2023-02-01

2023-01-26

2023-01-24

  • 12:04 taavi: deploying toolforge-jobs-framework-cli v10 T327775
  • 10:07 taavi: publish toolforge-jobs-framework-cli v9

2023-01-23

2023-01-20

  • 23:24 andrewbogott: truncating logfiles with find . -name '*.err' -size +1G -exec truncate --size=100M {} \;
  • 21:24 andrewbogott: truncating logfiles with find . -name '*.out' -size +1G -exec truncate --size=100M {} \;
  • 01:06 andrewbogott: truncating logfiles with find . -name '*.log' -size +1G -exec truncate --size=100M {} \;

2023-01-19

  • 11:46 arturo: `aborrero@tools-k8s-control-1:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff)

2023-01-18

2023-01-17

2023-01-10

2023-01-03

  • 17:17 andrewbogott: find -name '*.log' -size +1G -exec truncate --size=1G {} \;

2022-12-20

  • 09:07 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo

2022-12-12

  • 14:36 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2022-12-09

  • 07:20 taavi: change the canonical tools-mail external hostname to use mail.tools.wmcloud.org and add valid spf to toolforge.org T324809

2022-12-05

  • 11:06 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2022-11-30

2022-11-29

  • 19:52 taavi: clear puppet failure emails from exim queues

2022-11-09

  • 08:58 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by arturo@nostromo

2022-11-05

  • 19:28 andrewbogott: cleaning up nfs share with root@labstore1004:/srv/tools/shared/tools# find -name '*.err' -size +1G -exec truncate --size=1G {} \;
  • 13:26 andrewbogott: cleaning up nfs share with root@labstore1004:/srv/tools/shared/tools# find -name '*.log' -size +1G -exec truncate --size=1G {} \;

2022-11-04

2022-11-01

  • 09:37 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master (T322110) - cookbook ran by dcaro@vulcanus

2022-10-26

  • 08:45 dcaro: depooling and rebooting tools-sgeexec-10-22 to get nfs scratch working again

2022-10-25

  • 16:14 wm-bot2: Increased quotas by 5120 gigabytes - cookbook ran by fran@wmf3169
  • 15:26 dcaro: pushed a newer docker-registry.tools.wmflabs.org/python:3.9-slim-bullseye (from upstream pthyon:3.9-slim-bullseye)

2022-10-20

  • 16:54 andrewbogott: rebooting tools-package-builder-04
  • 16:49 andrewbogott: rebooting redis nodes (one at a time)
  • 10:54 taavi: rebuild mono68-sssd image with the expired DST Root CA X3 removed T311466

2022-10-18

2022-10-17

  • 07:25 taavi: push updated perl532 images T320824

2022-10-14

2022-10-13

  • 15:10 arturo: restart jobs-emailer pod

2022-10-12

  • 23:25 bd808: Rebuilding all Toolforge docker images (T278436, T311466, T293552)
  • 20:43 bd808: Rebuilding all Toolforge docker images to pick up bug and security fix packages. Third try seems to be working. (T316554)
  • 20:31 bd808: Rebuilding all Toolforge docker images to pick up bug and security fix packages after fixing bug in building the bullseye base image. (T316554)
  • 16:26 dcaro: deploy the latest registry admission webhook, now for real (image tag 07bc7db)
  • 12:48 dcaro: deploy the latest registry admission webhook (image tag 07bc7db)
  • 09:26 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus
  • 09:19 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2022-10-11

2022-10-10

2022-10-09

  • 17:29 taavi: kill 10 idle tmux sessions of user 'hoi' on tools-sgebastion-10 T320352

2022-10-07

  • 13:02 taavi: taavi@cloudcontrol1005 ~ $ sudo mark_tool --disable oncall # T320240

2022-10-06

  • 00:39 bd808: Image rebuild failing with debian apt repo signature issue. Will investigate tomorrow. (T316554)
  • 00:36 bd808: Rebuilding all Toolforge docker images to pick up bug and security fix packages. (T316554)
  • 00:04 bd808: Building new php74-sssd-base & web images (T310435)

2022-10-03

2022-09-28

  • 21:23 lucaswerkmeister: on tools-sgebastion-10: run-puppet-agent # T318858
  • 21:22 lucaswerkmeister: on tools-sgebastion-10: apt remove emacs-common emacs-bin-common # fix package conflict, T318858
  • 21:15 lucaswerkmeister: added root SSH key for myself, manually ran puppet on tools-sgebastion-10 to apply it (seemingly successfully)

2022-09-22

  • 12:30 taavi: add TheresNoTime to the 'toollabs-trusted' gerrit group T317438
  • 12:27 taavi: add TheresNoTime as a project admin and to the roots sudo policy T317438

2022-09-10

  • 07:39 wm-bot2: removing instance tools-prometheus-03 - cookbook ran by taavi@runko

2022-09-07

  • 10:22 dcaro: Pushing the new toolforge builder image based on the new 0.8 buildpacks (T316854)

2022-09-06

  • 08:06 dcaro_away: Published new toolforge-bullseye0-run and toolforge-bullseye0-build images for the toolforge buildpack builder (T316854)

2022-08-25

  • 10:40 taavi: tagged new version of the python39-web container with a shell implementation of webservice-runner T293552

2022-08-24

2022-08-20

  • 07:44 dcaro_away: all k8s nodes ready now \o/ (T315718)
  • 07:43 dcaro_away: rebooted tools-k8s-control-2, seemed stuck trying to wait for tools home (nfs?), after reboot came back up (T315718)
  • 07:41 dcaro_away: cloudvirt1023 down took out 3 workers, 1 control, and a grid exec and a weblight, they are taking long to restart, looking (T315718)

2022-08-18

  • 14:45 andrewbogott: adding lucaswerkmeister as projectadmin (T314527)
  • 14:43 andrewbogott: removing some inactive projectadmins: rush, petrb, mdipietro, jeh, krenair

2022-08-17

  • 16:34 taavi: kubectl sudo delete cm -n tool-wdml maintain-kubeusers # T315459
  • 08:30 taavi: failing the grid from the shadow back to the master, some disruption expected

2022-08-16

  • 17:28 taavi: fail over docker-registry, tools-docker-registry-06->docker-registry-05

2022-08-11

  • 16:57 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by taavi@runko
  • 16:55 taavi: restart puppetdb on tools-puppetdb-1, crashed during the ceph issues

2022-08-05

  • 15:08 wm-bot2: removing grid node tools-sgewebgen-10-1.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 15:05 wm-bot2: removing grid node tools-sgeexec-10-12.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 15:00 wm-bot2: created node tools-sgewebgen-10-3.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko

2022-08-03

2022-07-20

  • 19:31 taavi: reboot toolserver-proxy-01 to free up disk space probably held by stale file handles
  • 08:06 wm-bot2: removing grid node tools-sgeexec-10-6.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

2022-07-19

  • 17:53 wm-bot2: created node tools-sgeexec-10-21.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko
  • 17:00 wm-bot2: removing grid node tools-sgeexec-10-3.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 16:58 wm-bot2: removing grid node tools-sgeexec-10-4.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 16:24 wm-bot2: created node tools-sgeexec-10-20.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko
  • 15:59 taavi: tag current maintain-kubernetes :beta image as: :latest

2022-07-17

  • 15:52 wm-bot2: removing grid node tools-sgeexec-10-10.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 15:43 wm-bot2: removing grid node tools-sgeexec-10-2.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 13:26 wm-bot2: created node tools-sgeexec-10-16.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko

2022-07-14

  • 13:48 taavi: rebooting tools-sgeexec-10-2

2022-07-13

  • 12:09 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2022-07-11

  • 16:06 wm-bot2: Increased quotas by {self.increases} (T312692) - cookbook ran by nskaggs@x1carbon

2022-07-07

  • 07:34 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master - cookbook ran by dcaro@vulcanus

2022-06-28

  • 17:34 wm-bot2: cleaned up grid queue errors on tools-sgegrid-master (T311538) - cookbook ran by dcaro@vulcanus
  • 15:51 taavi: add 4096G cinder quota T311509

2022-06-27

  • 18:14 taavi: restart calico, appears to have got stuck after the ca replacement operation
  • 18:02 taavi: switchover active cron server to tools-sgecron-2 T284767
  • 17:54 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0915.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:52 wm-bot2: removing grid node tools-sgewebgrid-generic-0902.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:49 wm-bot2: removing grid node tools-sgeexec-0942.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:15 taavi: T311412 updating ca used by k8s-apiserver->etcd communication, breakage may happen
  • 14:58 taavi: renew puppet ca cert and certificate for tools-puppetmaster-02 T311412
  • 14:50 taavi: backup /var/lib/puppet/server to /root/puppet-ca-backup-2022-06-27.tar.gz on tools-puppetmaster-02 before we do anything else to it T311412

2022-06-23

  • 17:51 wm-bot2: removing grid node tools-sgeexec-0941.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:49 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0916.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:46 wm-bot2: removing grid node tools-sgewebgrid-generic-0901.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:32 wm-bot2: removing grid node tools-sgeexec-0939.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:30 wm-bot2: removing grid node tools-sgeexec-0938.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:27 wm-bot2: removing grid node tools-sgeexec-0937.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:22 wm-bot2: removing grid node tools-sgeexec-0936.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:19 wm-bot2: removing grid node tools-sgeexec-0935.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:17 wm-bot2: removing grid node tools-sgeexec-0934.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:14 wm-bot2: removing grid node tools-sgeexec-0933.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:11 wm-bot2: removing grid node tools-sgeexec-0932.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 17:09 wm-bot2: removing grid node tools-sgeexec-0920.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 15:30 wm-bot2: removing grid node tools-sgeexec-0947.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 13:59 taavi: removing remaining continuous jobs from the stretch grid T277653

2022-06-22

  • 15:54 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0917.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 15:51 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0918.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 15:47 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0919.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 15:45 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0920.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

2022-06-21

  • 15:23 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0914.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 15:20 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0914.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 15:18 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0913.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko
  • 15:07 wm-bot2: removing grid node tools-sgewebgrid-lighttpd-0912.tools.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko

2022-06-03

  • 20:07 wm-bot2: created node tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
  • 19:51 balloons: Scaling webservice nodes to 20, using new 8G swap flavor T309821
  • 19:35 wm-bot2: created node tools-sgeweblight-10-25.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
  • 19:03 wm-bot2: created node tools-sgeweblight-10-20.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
  • 19:01 wm-bot2: created node tools-sgeweblight-10-19.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
  • 19:00 balloons: depooled old nodes, bringing entirely new grid of nodes online T309821
  • 18:22 wm-bot2: created node tools-sgeweblight-10-17.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
  • 17:54 wm-bot2: created node tools-sgeweblight-10-16.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
  • 17:52 wm-bot2: created node tools-sgeweblight-10-15.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
  • 16:59 andrewbogott: building a bunch of new lighttpd nodes (beginning with tools-sgeweblight-10-12) using a flavor with more swap space
  • 16:56 wm-bot2: created node tools-sgeweblight-10-12.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by andrew@buster
  • 15:50 balloons: fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor T309821
  • 15:50 balloons: temp add 1.0G swap to sgeweblight hosts T309821
  • 15:50 balloons: fix fix g3.cores4.ram8.disk20.swap24.ephem20 flavor to include swap. Convert to fix g3.cores4.ram8.disk20.swap8.ephem20 flavor t309821
  • 15:49 balloons: temp add 1.0G swap to sgeweblight hosts t309821
  • 13:25 bd808: Upgrading fleet to tools-webservice 0.86 (T309821)
  • 13:20 bd808: publish tools-webservice 0.86 (T309821)
  • 12:46 taavi: start webservicemonitor on tools-sgecron-01 T309821
  • 10:36 taavi: draining each sgeweblight node one by one, and removing the jobs stuck in 'deleting' too
  • 05:05 taavi: removing duplicate (there should be only one per tool) web service jobs from the grid T309821
  • 04:52 taavi: revert bd808's changes to profile::toolforge::active_proxy_host
  • 03:21 bd808: Cleared queue error states after deploying new toolforge-webservice package (T309821)
  • 03:10 bd808: publish tools-webservice 0.85 with hack for T309821

2022-06-02

  • 22:26 bd808: Rebooting tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud. Node is full of jobs that are not tracked by grid master and failing to spawn new jobs sent by the scheduler
  • 21:56 bd808: Removed legacy "active_proxy_host" hiera setting
  • 21:55 bd808: Updated hiera to use fqdn of 'tools-proxy-06.tools.eqiad1.wikimedia.cloud' for profile::toolforge::active_proxy_host key
  • 21:41 bd808: Updated hiera to use fqdn of 'tools-proxy-06.tools.eqiad1.wikimedia.cloud' for active_redis key
  • 21:23 wm-bot2: created node tools-sgeweblight-10-8.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko
  • 12:42 wm-bot2: rebooting stretch exec grid workers - cookbook ran by taavi@runko
  • 12:13 wm-bot2: created node tools-sgeweblight-10-7.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko
  • 12:03 dcaro: refresh prometheus certs (T308402)
  • 11:47 dcaro: refresh registry-admission-controller certs (T308402)
  • 11:42 dcaro: refresh ingress-admission-controller certs (T308402)
  • 11:36 dcaro: refresh volume-admission-controller certs (T308402)
  • 11:24 wm-bot2: created node tools-sgeweblight-10-6.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko
  • 11:17 taavi: publish jobutils 1.44 that updates the grid default from stretch to buster T277653
  • 10:16 taavi: publish tools-webservice 0.84 that updates the grid default from stretch to buster T277653
  • 09:54 wm-bot2: created node tools-sgeexec-10-14.tools.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by taavi@runko

2022-06-01

  • 11:18 taavi: depool and remove tools-sgeexec-09[07-14]

2022-05-31

  • 16:51 taavi: delete tools-sgeexec-0904 for T309525 experimentation

2022-05-30

  • 08:24 taavi: depool tools-sgeexec-[0901-0909] (7 nodes total) T277653

2022-05-26

2022-05-22

  • 17:04 taavi: failover tools-redis to the updated cluster T278541
  • 16:42 wm-bot2: removing grid node tools-sgeexec-0940.tools.eqiad1.wikimedia.cloud (T308982) - cookbook ran by taavi@runko

2022-05-16

2022-05-14

  • 10:47 taavi: hard reboot unresponsible tools-sgeexec-0940

2022-05-12

2022-05-10

  • 15:18 taavi: depool tools-k8s-worker-42 for experiments
  • 13:54 taavi: enable distro-wikimedia unattended upgrades T290494

2022-05-06

  • 19:46 bd808: Rebuilt toolforge-perl532-sssd-base & toolforge-perl532-sssd-web to add liblocale-codes-perl (T307812)

2022-05-05

  • 17:28 taavi: deploy tools-webservice 0.83 T307693

2022-05-03

  • 08:20 taavi: redis: start replication from the old cluster to the new one (T278541)

2022-05-02

  • 08:54 taavi: restart acme-chief.service T307333

2022-04-25

  • 14:56 bd808: Rebuilding all docker images to pick up toolforge-webservice v0.82 (T214343)
  • 14:46 bd808: Building toolforge-webservice v0.82

2022-04-23

  • 16:51 bd808: Built new perl532-sssd/{base,web} images and pushed to registry (T214343)

2022-04-20

2022-04-16

2022-04-12

  • 21:32 bd808: Added komla to Gerrit group 'toollabs-trusted' (T305986)
  • 21:27 bd808: Added komla to 'roots' sudoers policy (T305986)
  • 21:24 bd808: Add komla as projectadmin (T305986)

2022-04-10

  • 18:43 taavi: deleted `/tmp/dwl02.out-20210915` on tools-sgebastion-07 (not touched since september, taking up 1.3G of disk space)

2022-04-09

  • 15:30 taavi: manually prune user.log on tools-prometheus-03 to free up some space on /

2022-04-08

  • 10:44 arturo: disabled debug mode on the k8s jobs-emailer component

2022-04-05

2022-04-04

2022-03-28

  • 09:32 wm-bot: cleaned up grid queue errors on tools-sgegrid-master.tools.eqiad1.wikimedia.cloud (T304816) - cookbook ran by arturo@nostromo

2022-03-15

2022-03-14

  • 11:44 arturo: deploy jobs-framework-emailer 9470a5f (T286135)
  • 10:48 dcaro: pushed v0.33.2 tekton control and webhook images, and bashA5.1.4 to the local repo (T297090)

2022-03-10

  • 09:42 arturo: cleaned grid queue error state @ tools-sgewebgrid-generic-0902

2022-03-01

  • 13:41 dcaro: rebooting tools-sgeexec-0916 to clear any state (T302702)
  • 12:11 dcaro: Cleared error state queues for sgeexec-0916 (T302702)
  • 10:23 arturo: tools-sgeeex-0913/0916 are depooled, queue errors. Reboot them and clean errors by hand

2022-02-28

  • 08:02 taavi: reboot sgeexec-0916
  • 07:49 taavi: depool tools-sgeexec-0916.tools as it is out of disk space on /

2022-02-17

  • 08:23 taavi: deleted tools-clushmaster-02
  • 08:14 taavi: made tools-puppetmaster-02 its own client to fix `puppet node deactivate` puppetdb access

2022-02-16

  • 00:12 bd808: Image builds completed.

2022-02-15

  • 23:17 bd808: Image builds failed in buster php image with an apt error. The error looks transient, so starting builds over.
  • 23:06 bd808: Started full rebuild of Toolforge containers to pick up webservice 0.81 and other package updates in tmux session on tools-docker-imagebuilder-01
  • 22:58 bd808: `sudo apt-get update && sudo apt-get install toolforge-webservice` on all bastions to pick up 0.81
  • 22:50 bd808: Built new toollabs-webservice 0.81
  • 18:43 bd808: Enabled puppet on tools-proxy-05
  • 18:38 bd808: Disabled puppet on tools-proxy-05 for manual testing of nginx config changes
  • 18:21 taavi: delete tools-package-builder-03
  • 11:49 arturo: invalidate sssd cache in all bastions to debug T301736
  • 11:16 arturo: purge debian package `unscd` on tools-sgebastion-10/11 for T301736
  • 11:15 arturo: reboot tools-sgebastion-10 for T301736

2022-02-10

  • 15:07 taavi: shutdown tools-clushmaster-02 T298191
  • 13:25 wm-bot: trying to join node tools-sgewebgen-10-2 to the grid cluster in tools. - cookbook ran by arturo@nostromo
  • 13:24 wm-bot: trying to join node tools-sgewebgen-10-1 to the grid cluster in tools. - cookbook ran by arturo@nostromo
  • 13:07 wm-bot: trying to join node tools-sgeweblight-10-5 to the grid cluster in tools. - cookbook ran by arturo@nostromo
  • 13:06 wm-bot: trying to join node tools-sgeweblight-10-4 to the grid cluster in tools. - cookbook ran by arturo@nostromo
  • 13:05 wm-bot: trying to join node tools-sgeweblight-10-3 to the grid cluster in tools. - cookbook ran by arturo@nostromo
  • 13:03 wm-bot: trying to join node tools-sgeweblight-10-2 to the grid cluster in tools. - cookbook ran by arturo@nostromo
  • 12:54 wm-bot: trying to join node tools-sgeweblight-10-1.tools.eqiad1.wikimedia.cloud to the grid cluster in tools. - cookbook ran by arturo@nostromo
  • 08:45 taavi: set `profile::base::manage_ssh_keys: true` globally T214427
  • 08:16 taavi: enable puppetdb and re-enable puppet with puppetdb ssh key management disabled (profile::base::manage_ssh_keys: false) - T214427
  • 08:06 taavi: disable puppet globally for enabling puppetdb T214427

2022-02-09

  • 19:29 taavi: installed tools-puppetdb-1, not configured on puppetmaster side yet T214427
  • 18:56 wm-bot: pooled 10 grid nodes tools-sgeweblight-10-[1-5],tools-sgewebgen-10-[1,2],tools-sgeexec-10-[1-10] (T277653) - cookbook ran by arturo@nostromo
  • 18:30 wm-bot: pooled 9 grid nodes tools-sgeexec-10-[2-10],tools-sgewebgen-[3,15] - cookbook ran by arturo@nostromo
  • 18:25 arturo: ignore last message
  • 18:24 wm-bot: pooled 9 grid nodes tools-sgeexec-10-[2-10],tools-sgewebgen-[3,15] - cookbook ran by arturo@nostromo
  • 14:04 taavi: created tools-cumin-1/toolsbeta-cumin-1 T298191

2022-02-07

  • 17:37 taavi: generated authdns_acmechief ssh key and stored password in a text file in local labs/private repository (T288406)
  • 12:52 taavi: updated maintain-kubeusers for T301081

2022-02-04

  • 22:33 taavi: `root@tools-sgebastion-10:/data/project/ru_monuments/.kube# mv config old_config` # experimenting with T301015
  • 21:36 taavi: clear error state from some webgrid nodes

2022-02-03

  • 09:06 taavi: run `sudo apt-get clean` on login-buster/dev-buster to clean up disk space
  • 08:01 taavi: restart acme-chief to force renewal of toolserver.org certificate

2022-01-30

  • 14:41 taavi: created a neutron port with ip 172.16.2.46 for a service ip for toolforge redis automatic failover T278541
  • 14:22 taavi: creating a cluster of 3 bullseye redis hosts for T278541

2022-01-26

  • 18:33 wm-bot: depooled grid node tools-sgeexec-10-10 - cookbook ran by arturo@nostromo
  • 18:33 wm-bot: depooled grid node tools-sgeexec-10-9 - cookbook ran by arturo@nostromo
  • 18:33 wm-bot: depooled grid node tools-sgeexec-10-8 - cookbook ran by arturo@nostromo
  • 18:32 wm-bot: depooled grid node tools-sgeexec-10-7 - cookbook ran by arturo@nostromo
  • 18:32 wm-bot: depooled grid node tools-sgeexec-10-6 - cookbook ran by arturo@nostromo
  • 18:31 wm-bot: depooled grid node tools-sgeexec-10-5 - cookbook ran by arturo@nostromo
  • 18:30 wm-bot: depooled grid node tools-sgeexec-10-4 - cookbook ran by arturo@nostromo
  • 18:28 wm-bot: depooled grid node tools-sgeexec-10-3 - cookbook ran by arturo@nostromo
  • 18:27 wm-bot: depooled grid node tools-sgeexec-10-2 - cookbook ran by arturo@nostromo
  • 18:27 wm-bot: depooled grid node tools-sgeexec-10-1 - cookbook ran by arturo@nostromo
  • 13:55 arturo: scaling up the buster web grid with 5 lighttd and 2 generic nodes (T277653)

2022-01-25

  • 11:50 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo
  • 11:44 arturo: rebooting buster exec nodes
  • 08:34 taavi: sign puppet certificate for tools-sgeexec-10-4

2022-01-24

  • 17:44 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo
  • 15:23 arturo: scaling up the grid with 10 buster exec nodes (T277653)

2022-01-20

  • 17:05 arturo: drop 9 of the 10 buster exec nodes created earlier. They didn't get DNS records
  • 12:56 arturo: scaling up the grid with 10 buster exec nodes (T277653)

2022-01-19

  • 17:34 andrewbogott: rebooting tools-sgeexec-0913.tools.eqiad1.wikimedia.cloud to recover from (presumed) fallout from the scratch/nfs move

2022-01-14

  • 19:09 taavi: set /var/run/lighttpd as world-writable on all lighttpd webgrid nodes, T299243

2022-01-12

  • 11:27 arturo: created puppet prefix `tools-sgeweblight`, drop `tools-sgeweblig`
  • 11:03 arturo: created puppet prefix 'tools-sgeweblig'
  • 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig'

2022-01-04

  • 17:18 bd808: tools-acme-chief-01: sudo service acme-chief restart
  • 08:12 taavi: disable puppet & exim4 on T298501

Archives

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Tools/SAL&oldid=2243644"