18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector
18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector
2024-03-22
11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud
11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud
2024-03-21
14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4
14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4
13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3
13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3
12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0)
11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
2024-03-19
10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
2024-03-18
12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1
12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1
11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud
10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud
2024-03-16
11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8
2024-03-15
10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
2024-03-14
15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01
15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01
13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 (T359638)
13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 (T359638)
13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 (T359638)
13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 (T359638)
13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 (T359638)
12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 (T359638)
12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 (T359638)
12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 (T359638)
12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 (T359638)
12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 (T359638)
12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 (T359638)
12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 (T359638)
12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 (T359638)
12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 (T359638)
12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 (T359638)
12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 (T359638)
12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 (T359638)
12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 (T359638)
12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 (T359638)
12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 (T359638)
12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 (T359638)
12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 (T359638)
11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 (T359638)
11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 (T359638)
10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 (T359638)
10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 (T359638)
10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 (T359638)
10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 (T359638)
10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 (T359638)
10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 (T359638)
10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 (T359638)
10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 (T359638)
10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 (T359638)
10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 (T359638)
09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638)
09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638)
09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638)
09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638)
2024-03-13
16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638)
16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638)
16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638)
16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 (T359638)
15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 (T359638)
15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 (T359638)
2024-03-12
11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 (T359638)
11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 (T359638)
2024-03-11
16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
2024-03-07
14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
2024-03-05
16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
2024-03-04
17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
2024-03-01
21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
2024-02-28
00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
2024-02-26
13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud
13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud
2024-02-22
13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
2024-02-21
17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api
14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api
14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers
13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
2024-02-20
13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6
13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6
13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster
13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster
13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9
13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9
13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9
13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9
13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster
13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster
13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1
11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1
11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1
2024-02-19
18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
2024-02-15
11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5
11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5
11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5
11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8
10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8
10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
2024-02-13
14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4
14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4
14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5
14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5
14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4
14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4
10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster
10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3
10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3
09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster
09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster
09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster
09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8
09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8
09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7
09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7
2024-02-12
10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api
10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api
10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers
10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers
2024-02-09
10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config
10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config
2024-02-08
15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud
15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud
11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster
11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6
11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6
11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6
11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6
11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster
11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster
10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10
10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10
2024-02-06
11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes (T356507)
10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes (T356507)
2024-02-05
09:55 arturo: grant myself member and admin privileges
2024-01-31
13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
2024-01-29
13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud
13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud
2024-01-26
10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster
10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster
10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster
10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster
2024-01-25
12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster
12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster
11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
2024-01-24
11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster
2024-01-23
19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics
19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics
19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics
10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
2024-01-19
15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
2024-01-17
14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
2024-01-12
09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1
2024-01-11
17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
2024-01-09
17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
2024-01-08
10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api
10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api
2024-01-05
14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder
11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder
11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder
2023-12-26
19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable
2023-12-20
18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99)
12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0)
09:36 wm-bot2: deployed kubernetes component api-gateway (c0faf0f) (T341462) - cookbook ran by dcaro@urcuchillay
08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17
08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17
08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17
08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17
08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17
2023-08-30
11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 (T298005) - cookbook ran by arturo@nostromo
11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 (T298005) - cookbook ran by arturo@nostromo
11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 (T298005) - cookbook ran by arturo@nostromo
10:05 dcaro: upgrade toolforge-weld to 1.2.1 (T344155)
08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, T298005T343869
2023-08-29
13:06 wm-bot2: deployed kubernetes component jobs-emailer (6f9c8cf) - cookbook ran by taavi@runko
13:03 wm-bot2: deployed kubernetes component jobs-api (b29193d) (T341462) - cookbook ran by dcaro@urcuchillay
2023-08-28
14:54 wm-bot2: deployed kubernetes component envvars-api (90055b5) (T344502) - cookbook ran by dcaro@urcuchillay
13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo
13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo
13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo
13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo
13:15 arturo: cordoned & drained k8s workers 4 to 7 to force workload to relocate to 8 (T329378)
12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo
12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo
2023-02-10
16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo
16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo
2022-01-25
11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo
2022-01-20
12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster
15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 && git cherry-pick FETCH_HEAD
09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix)
2022-01-17
18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo
18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo
17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo
2022-01-14
11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo
2022-01-12
12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo
11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig`
11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig'
11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo
2022-01-11
11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo
09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo
2021-12-23
13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance
11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance
11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance
11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
2021-12-22
12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance
11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance
11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance
11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance
10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance
2021-12-21
16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance
16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance
12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB (T277653)
2021-12-16
12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance
2021-12-15
14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance
2021-12-08
05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028
2021-11-28
17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance
12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true`
2021-10-25
14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image T292771
2021-10-20
12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories
2021-10-16
07:47 majavah: deployed cert-manager and wave as a test for automating T292238
2021-10-14
15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus
2021-10-13
11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool (T292465) - cookbook ran by dcaro@vulcanus
10:19 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
10:19 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
2021-10-12
16:10 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
14:52 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
14:46 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master
2021-10-11
15:24 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
15:00 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
10:32 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
2021-10-07
14:21 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
14:06 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
13:31 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
12:55 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
12:50 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
12:50 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
08:04 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
07:58 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
2021-10-06
10:36 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
10:13 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
10:08 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
10:07 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
10:05 wm-bot: Adding a new grid webgrid generic node (T292465) - cookbook ran by dcaro@vulcanus
10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-{apiserver,controller-manager}.yaml in all 3 control nodes (T286108)
10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap (T286108)
18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns T277653
2021-04-06
13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number (T267082)
2021-04-01
15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook)
14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs
2021-03-31
18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo T275478
2021-03-24
12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM
11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster (T278303)
11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01
11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` (T278303)
2021-03-23
11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master (T277653)
11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow (T277653)
2021-03-18
18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix
18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project)
18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful
18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM (T277653)
10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) (T271867)
10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) (T271867)
2020-12-22
10:48 arturo: rebase & resolve ugly git merge conflict in labs/private.git
17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) (T267140)
14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage (T267140)
2020-11-06
23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 T263284
21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 T263284
15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild (T267140)
15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup (T267140)
13:18 dcaro: bringin up a new proxy-4 instance as slave (T267140)
13:18 dcaro: bringin up a new proxy-4 instance as slave
2020-11-05
16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 (T267140)
15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera (T267140)
2020-11-04
15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try (T267140)
15:21 dcaro: creating new proxy instance toolsbeta-proxy-03
15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use
15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use
11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix
2020-11-03
16:04 arturo: add dcaro to the toolsbeta.admin LDAP group (T266068)
15:30 dcaro: T267121: Puppetmaster replaced, also removed old puppetdb master from hiera, testing
15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04
10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02
2020-11-02
13:35 arturo: added dcaro as projectadmin & user (T266068)
2020-10-29
22:20 legoktm: switched test tool over to use buildpack image (T265681)
2020-10-28
18:58 andrewbogott: deleting toolsbeta-puppetmaster-03 — seems broken and unused
11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 (T246122)
10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo (T246122)
10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! (T246122)
2020-05-26
16:17 bstorm_: fix incorrect volume name in kubeadm-config T246122
15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons (T246122)
10:04 arturo: add herron as user and projectadmin, we will work on the email setup (T120225)
09:59 arturo: created VM toolsbeta-mail-01 (T120225)
2020-05-04
13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project
2020-04-29
19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta T247455
2020-04-20
14:47 arturo: added joakino to toolsbeta.admin LDAP group
12:06 arturo: installing tools-webservice v0.68 for testing
11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there
10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project
12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required
12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to T234037
2019-10-09
11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster (T235059)
2019-10-08
14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to T234037
12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to T234037
2019-10-07
19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue
2019-09-25
23:31 bd808: Updated user list for "roots" sudoer policy
23:30 bd808: Granted Krenair projectadmin
2019-09-05
{{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}}
2019-08-12
20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for T230147
2019-08-09
10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003
2019-07-24
20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container T228887T215531
19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster
12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets)
12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] T215531
12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs T215531
11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for T215531 (so I have 3 master nodes in this k8s deployment)
11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01
11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01:
11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc
2019-05-07
10:22 arturo: T219362 drop the `toolsbeta-exec` puppet prefix
10:20 arturo: T219362 drop the `toolsbeta-webgrid-generic` puppet prefix
10:19 arturo: T219362 drop the `toolsbeta-webgrid-lighttpd` puppet prefix
2019-04-25
04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver: toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master
2019-04-12
23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1
00:05 andrewbogott: migrating remaining VMs to eqiad1-r
2019-03-25
18:00 bd808: All Trusty instances shutdown and now in process of deleting
17:42 bd808: Preparing to shutdown beta Trusty job grid
2019-03-22
13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing T218126
2019-03-15
10:23 arturo: create VM `arturo-bastion-sssd-test` (T218126)
2019-02-20
14:58 andrewbogott: moving toolsbeta-grid-master and toolsbeta-puppetmaster-02 to labvirt1003
2019-02-14
18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002
13:25 arturo: T210098 install systemd239 from stretch-backports and restart VM
2018-11-08
10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically T207970)
2018-10-22
21:20 bstorm_: launched a stretch/sonofgridengine master server
2018-09-19
20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- T200557
17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for T200557)
17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb
2018-07-22
14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug T156626
2018-07-18
10:46 harej: Deleted toolsbeta-flynn-01
2018-07-12
23:06 bstorm_: Got the grid master running
2018-06-28
16:34 chicocvenancio: adding harej as root for flynn testing
2018-06-27
22:35 chicocvenancio: add harej as project admin to test Flynn stuff
2018-06-22
22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning
09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions
09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating
09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01
08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice
17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 | (while read dir; do id toolsbeta.`basename $dir` 2> /dev/null || sudo rm -rfv $dir; done)`
16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd
2018-05-06
04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd
2018-05-05
19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge T190893
18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml`
2018-05-02
21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera
20:37 bd808: Added Neha16 as a project admin for work on T175768
20:31 zhuyifei1999_: nuke webservice instances and rebuild
20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon T192618
2018-04-20
00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem
2018-04-19
22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :(
21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead
21:43 zhuyifei1999_: Start creating instances for webservice setup T190893
2018-03-30
22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge T190893
2018-03-14
18:07 chicocvenancio: updated paws-beta k8s cluster and nodes to v1.9.4 for T189680
2018-03-05
19:33 chicocvenancio: added Zhuyifei1999 as project admin
2018-02-09
01:11 bd808: Removed Yuvipanda at user request (T186289)
2017-08-07
14:09 andrewbogott: deleted etcd-k8s-CTEST and k8s-master-CTEST
2017-04-26
15:38 madhuvishy: add Madhuvishy as projectadmin
2016-10-07
19:30 valhallasw`cloud: (puppet certs, to be precise)
19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs
2016-10-04
19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for)
2016-09-08
17:11 bd808: Added BryanDavis (self) to project as admin
15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host
2015-10-13
19:21 valhallasw`cloud: started building toolsbeta-bastion.
2015-09-07
18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest...
18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages)
July 4
12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22
July 2
14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting
July 1
19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it?
19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login
June 7
14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo
14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config
May 11
18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages
May 2
11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code
10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera
