Kubernetes/Kubectl/Cheat Sheet

Introduction

More or less loose collection of kubectl commands that might be helpful (or were helpful at some point)

Commands

The all-namespaces commands need to be run from the admin service which is only accessible to root

kube_env admin codfw

List all images (without registry) running in the cluster

kubectl get pods --all-namespaces --field-selector=status.phase=Running -o jsonpath="{..image}" | tr ' ' '\n' | sed 's/.*\///' | sort -u

List images used per namespace

kubectl get pods --all-namespaces --field-selector=status.phase=Running -o=jsonpath='{range .items[*]}{"\n"}{.metadata.namespace}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'

Check which image version a cronjob is using

kubectl get pod <pod in question> -o jsonpath='{range .spec.containers[*]}{.name}: {.image}{"\n"}{end}'

Run a cronjob manually

Notes: requires the job to have been applied previously. Please keep in mind that the job will probably run as well when its scheduled time comes, so consider changing its schedule to something like @yearly if you don't want it to run while the manual run is in progress.

kube_env admin <cluster>
kubectl create job -n <namespace> --from=cronjob/<cronjob name> <cronjob name>-manual

Peek into a helm deployment config

kubectl get cm staging.v1 -o jsonpath='{.data.release}' | base64 -d | gunzip | vim -

Peek into a k8s secret TLS certificate

kubectl get secret <SECRET> -o jsonpath='{.data.tls\.crt}'  | base64 -d | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -text -noout

List all RBAC "Groups" referenced in the cluster

kubectl get clusterrolebindings.rbac.authorization.k8s.io,rolebindings.rbac.authorization.k8s.io -A -o go-template='{{range $i := .items}}{{range $i.subjects}}{{if eq .kind "Group"}}{{$i.metadata.namespace}}/{{$i.metadata.name}} emerges group: {{.name}}{{"\n"}}{{end}}{{end}}{{end}}'

Get all pods and their priorities

kubectl get po -A -o jsonpath='{range .items[*]}{@.metadata.namespace}/{@.metadata.name} {@.spec.priority}{"\n"}{end}'

Last termination reason and time

kubectl get po -l app=flink-session-cluster-taskmanager -o go-template='{{range .items}}{{$pod := .}}{{range .status.containerStatuses}}{{ if eq .name "flink-session-cluster-main-taskmanager"}}{{$pod.metadata.name}} {{.lastState.terminated.reason}} at {{.lastState.terminated.finishedAt}}{{"\n"}}{{end}}{{end}}{{end}}'

Get elected master component leaders

kubectl -n kube-system get leases.coordination.k8s.io

List all container images available per nodes

Be aware that this lists a maximum of 50 images per node, there might be more!

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .status.images[*]}{"\t"}{.names[0]}{"\n"}{end}{end}'

Force refresh of a cert-manager certificate

kubectl patch certificate/<CERTIFICATE> --type=json --patch='[{"op": "replace", "path": "/spec/renewBefore", "value": "1440h"}]'
# Once it's renewed, delete renewBefore
kubectl patch certificate/<CERTIFICATE> --type=json --patch='[{"op": "remove", "path": "/spec/renewBefore"}]'

Drain a "dead" node

kubectl drain will not work well in case a nodes kubelet is unreachable/unresponsive as kubectl will try to wait for the pods to terminate (which they wont). Disabling the grace-period for pod termination helps instructing the API to mark the Pods as terminated immediately:

kubectl drain --force --delete-emptydir-data --ignore-daemonsets --grace-period=0 kubernetes999.coXfw.wmnet