Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown
Source code
The source code for the maintain-kubeusers daemon itself is at:
The deployment of the code is done using toolforge-deploy, at:
The prometheus alerts are defined via Toolforge alerts repository:
Logs
This is how you can get the logs for maintain-kubeusers:
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers logs deploy/maintain-kubeusers --timestamps=true
In case you want to check the logs for a previous container restart:
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers logs deploy/maintain-kubeusers --timestamps=true --previous
Get a shell inside the pod
This is how you can get a shell inside the maintain-kubeusers pod:
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers exec -it deploy/maintain-kubeusers exec -- bash
root@maintain-kubeusers-76d6df5465-jgdwt:/app#
[..]
See prometheus metrics inside the container
This is often useful to verify what is being scraped by prometheus:
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers exec -it deploy/maintain-kubeusers exec -- apt-get update
[..]
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers exec -it deploy/maintain-kubeusers exec -- apt-get install curl -y
[..]
user@tools-k8s-control-7:~$ sudo -i kubectl -n maintain-kubeusers exec -it deploy/maintain-kubeusers exec -- curl localhost:9000/metrics
[..]
maintain_kubeusers_run_seconds_count 32.0
maintain_kubeusers_run_seconds_sum 4373.660967367701
maintain_kubeusers_run_finished 1.7175907241729112e+09
Dashboards
See grafana dashboard: