User:Brouberol
Useful snippets
pcc
This is a bash function I have in my zshrc
# This function allows me to run `pcc` in my puppet repository checkout
# and have it automatically start a PCC run on the gerrit change ID
# associated with the patch change ID, without having me to copy and paste
# anything.
pcc () {
if [[ $PWD != "${WMF_HOME}/puppet" ]]
then
echo "Not in the puppet root dir. Exiting."
else
change_id=$(git show | grep Change-Id | awk '{ print $2 }')
gerrit_change_id=$(curl -s "https://gerrit.wikimedia.org/r/changes/${change_id}" | sed 1d | jq -r "._number")
./utils/pcc ${gerrit_change_id}
fi
}
prepare-commit-msg
I automatically inject this script in my git hooks to avoid having to manually type Bug: TXXXXXX in my commit messages. Note: it relies on the phabricator ticket number to appear in the git branch name.
#!/bin/sh
#
# Inspects branch name and checks if it contains a Phabricator ticket number (i.e. T123456).
# If yes, commit message will be automatically suffixed with "Bug: {num}".
#
# Useful for looking through git history and relating a commit or group of commits
# back to a user story.
#
BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD 2>/dev/null)
# Ensure BRANCH_NAME is not empty and is not in a detached HEAD state (i.e. rebase).
# SKIP_PREPARE_COMMIT_MSG may be used as an escape hatch to disable this hook,
# while still allowing other githooks to run.
if [ ! -z "$BRANCH_NAME" ] && [ "$BRANCH_NAME" != "HEAD" ] && [ "$SKIP_PREPARE_COMMIT_MSG" != 1 ]; then
PREFIX_PATTERN='T[0-9]{6,7}'
[[ $BRANCH_NAME =~ $PREFIX_PATTERN ]]
PREFIX=${BASH_REMATCH[0]}
PREFIX_IN_COMMIT=$(grep -c "Bug: $PREFIX" $1)
# Ensure PREFIX exists in BRANCH_NAME and is not already present in the commit message
if [[ -n "$PREFIX" ]] && ! [[ $PREFIX_IN_COMMIT -ge 1 ]]; then
echo "\nBug: ${PREFIX}" >> $1
fi
fi
How to strace the process running in a container
This allows us to run strace on a running docker container, on a Kubernetes host.
Example: strace-container mpic-staging-5f488948c-2g64g mpic-staging -e 'trace=!epoll_wait'
strace-container() {
local pod_name=$1
local container_name=$2
shift
shift
local container_id=$(sudo docker ps | grep "${container_name}_${pod_name}" | awk '{ print $1 }')
local container_root_pid=$(sudo docker top ${container_id} | grep -v UID | awk '{ print $2 }')
# any additional argument will be passed to `strace`
sudo strace -f -p ${container_root_pid} $@
}
How to run a command into the network namespace of a container
This function allows us to run a command available on a Kubernetes worker host, within a container network namespace. It can be useful to, say, check whether a container process can open a TCP connection to a given host/port.
Example: nsenter-container aqs-http-gateway-main-56945ccf8b-fk8tz aqs-http-gateway-main telnet 10.2.2.38 8082
nsenter-container() {
local pod_name=$1
local container_name=$2
shift
shift
local container_id=$(sudo docker ps | grep "${container_name}_${pod_name}" | awk '{ print $1 }')
local container_root_pid=$(sudo docker top ${container_id} | grep -v UID | awk '{ print $2 }' | head -n 1)
# any additional will be passed to `nsenter`
sudo nsenter --target ${container_root_pid} --net $@
}
How to run tcpdump on a pod IP
tcpdump-pod-ip() {
local pod_ip=$1
shift
# Any additional argument is passed to `tcpdump`
sudo tcpdump -i any host ${pod_ip} $@
}
Link a Calico Network Policy to Pods, IPs and ports
Pods "subscribe" to a Calico Network policy using a selector and a service name, allowing ingress from / egress to that service. However, all of this is hard to inspect, as it requires multiple kubectl describe calls: one to inspect the calico network policy, one to inspect the service and associated endpoints, and one to list the pods matching the selector.
The following script allows to resolve all these selectors to actual pod names and service IP/port data.
brouberol@deploy1002:~$ kube_env superset-deploy dse-k8s-eqiad
brouberol@deploy1002:~$ ./inspect_calico_networkpolicy --help
usage: inspect_calico_networkpolicy [-h] [-n NAMESPACE] [-f {plain,json}]
Inspect the Calico NetworkPolicy objects in a given namespace and link each of
them to the set of attached Pods.
optional arguments:
-h, --help show this help message and exit
-n NAMESPACE, --namespace NAMESPACE
The namespace to inspect (default: the current one)
-f {plain,json}, --format {plain,json}
Output format (default=plain)
brouberol@deploy1002:~$ ./inspect_calico_networkpolicy
[+] NetworkPolicy superset-staging-egress-external-services-cas
Pods:
- superset-staging-8678649994-grb8z
Service cas-idp -> ips=208.80.153.12, 208.80.154.80, 2620:0:860:1:208:80:153:12, 2620:0:861:3:208:80:154:80, port=TCP/443
[+] NetworkPolicy superset-staging-egress-external-services-druid
Pods:
- superset-staging-8678649994-grb8z
Service druid-analytics -> ips=10.64.21.11, 10.64.36.101, 10.64.5.17, 10.64.5.36, 10.64.53.13, 2620:0:861:104:10:64:5:17, 2620:0:861:104:10:64:5:36, 2620:0:861:105:10:64:21:11, 2620:0:861:106:10:64:36:101, 2620:0:861:108:10:64:53:13, port=TCP/8081,TCP/8082,TCP/8083
Service druid-public -> ips=10.64.131.9, 10.64.132.12, 10.64.135.9, 10.64.16.171, 10.64.48.227, 2620:0:861:102:10:64:16:171, 2620:0:861:107:10:64:48:227, 2620:0:861:10a:10:64:131:9, 2620:0:861:10b:10:64:132:12, 2620:0:861:10e:10:64:135:9, port=TCP/8081,TCP/8082,TCP/8083
[+] NetworkPolicy superset-staging-egress-external-services-kerberos
Pods:
- superset-staging-8678649994-grb8z
Service kerberos-kdc -> ips=10.192.48.190, 10.64.0.112, 2620:0:860:104:10:192:48:190, 2620:0:861:101:10:64:0:112, port=UDP/88,TCP/88
[+] NetworkPolicy superset-staging-egress-external-services-presto
Pods:
- superset-staging-8678649994-grb8z
Service presto-analytics -> ips=10.64.138.7, 10.64.142.6, 2620:0:861:100:10:64:138:7, 2620:0:861:114:10:64:142:6, port=TCP/8280,TCP/8281
brouberol@deploy1002:~$ ./inspect_calico_networkpolicy --format json
{
"superset-staging-egress-external-services-cas": {
"pods": [
"superset-staging-8678649994-grb8z"
],
"services": [
{
"name": "cas-idp",
"ips": [
"208.80.153.12",
"208.80.154.80",
"2620:0:860:1:208:80:153:12",
"2620:0:861:3:208:80:154:80"
],
"ports": [
"TCP/443"
]
}
]
},
"superset-staging-egress-external-services-druid": {
"pods": [
"superset-staging-8678649994-grb8z"
],
"services": [
{
"name": "druid-analytics",
"ips": [
"10.64.21.11",
"10.64.36.101",
"10.64.5.17",
"10.64.5.36",
"10.64.53.13",
"2620:0:861:104:10:64:5:17",
"2620:0:861:104:10:64:5:36",
"2620:0:861:105:10:64:21:11",
"2620:0:861:106:10:64:36:101",
"2620:0:861:108:10:64:53:13"
],
"ports": [
"TCP/8081",
"TCP/8082",
"TCP/8083"
]
},
{
"name": "druid-public",
"ips": [
"10.64.131.9",
"10.64.132.12",
"10.64.135.9",
"10.64.16.171",
"10.64.48.227",
"2620:0:861:102:10:64:16:171",
"2620:0:861:107:10:64:48:227",
"2620:0:861:10a:10:64:131:9",
"2620:0:861:10b:10:64:132:12",
"2620:0:861:10e:10:64:135:9"
],
"ports": [
"TCP/8081",
"TCP/8082",
"TCP/8083"
]
}
]
},
"superset-staging-egress-external-services-kerberos": {
"pods": [
"superset-staging-8678649994-grb8z"
],
"services": [
{
"name": "kerberos-kdc",
"ips": [
"10.192.48.190",
"10.64.0.112",
"2620:0:860:104:10:192:48:190",
"2620:0:861:101:10:64:0:112"
],
"ports": [
"UDP/88",
"TCP/88"
]
}
]
},
"superset-staging-egress-external-services-presto": {
"pods": [
"superset-staging-8678649994-grb8z"
],
"services": [
{
"name": "presto-analytics",
"ips": [
"10.64.138.7",
"10.64.142.6",
"2620:0:861:100:10:64:138:7",
"2620:0:861:114:10:64:142:6"
],
"ports": [
"TCP/8280",
"TCP/8281"
]
}
]
}
}
#!/usr/bin/env python3
"""
Inspect the Calico NetworkPolicy objects in a given namespace and link
each of them to the set of attached pod names, service IP and ports.
"""
import argparse
import subprocess
import json
import sys
from collections import defaultdict
from typing import Dict, Optional, List
WARNING = '\033[93m'
ENDC = '\033[0m'
def parse_args():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"-n",
"--namespace",
help="The namespace to inspect (default: the current one)",
type=str,
)
parser.add_argument(
"-f",
"--format",
help="Output format (default=plain)",
choices=("plain", "json"),
default="plain",
)
return parser.parse_args()
def warn(msg: str):
sys.stderr.write(f"{WARNING}{msg}{ENDC}\n")
def kubectl_get(
resource_type: str,
resource_name: Optional[str] = None,
label_selector: Optional[str] = None,
namespace: Optional[str] = None,
) -> List:
cmd = ["kubectl", "get", resource_type]
if label_selector:
cmd.extend(["-l", label_selector])
if namespace:
cmd.extend(["-n", namespace])
cmd.extend(["-o", "json"])
if resource_name:
cmd.append(resource_name)
returncmd = subprocess.run(cmd, capture_output=True)
try:
data = json.loads(returncmd.stdout)
except json.decoder.JSONDecodeError:
return {} if resource_name else []
if resource_name:
return data
return data["items"]
def calico_selector_to_label_selector(calico_selector: str) -> str:
label_selector = []
tokens = calico_selector.split(" && ")
for token in tokens:
selector = token.replace(" == ", "=")
label_selector.append(selector.replace("'", ""))
return ",".join(label_selector)
def main():
output = defaultdict(lambda: defaultdict(dict))
warnings = 0
args = parse_args()
def log(msg):
if args.format == "plain":
print(msg)
network_policies = kubectl_get(
"networkpolicies.crd.projectcalico.org", namespace=args.namespace
)
for network_policy in network_policies:
network_policy_name = network_policy["metadata"]["name"]
namespace = network_policy["metadata"]["annotations"][
"meta.helm.sh/release-namespace"
]
selector = network_policy["spec"]["selector"]
label_selector = calico_selector_to_label_selector(selector)
log(f"[+] NetworkPolicy {network_policy_name}")
pods = kubectl_get(
"pod", label_selector=label_selector, namespace=namespace
)
output[network_policy_name]["pods"] = [pod["metadata"]["name"] for pod in pods]
if not pods:
log(" No pods found")
else:
log(" Pods:")
for pod in pods:
log(f" - {pod['metadata']['name']}")
egress_destinations = network_policy["spec"]["egress"]
output[network_policy_name]['services'] = []
for egress_destination in egress_destinations:
egress_service = egress_destination["destination"]["services"]
egress_service_endpoints = kubectl_get(
"endpoints",
egress_service["name"],
namespace=egress_service["namespace"],
)
if not egress_service_endpoints:
warn(f"The Service {egress_service['name']} does not exist. The network policy won't have any effect.")
warnings += 1
continue
ips = [
addr["ip"]
for addr in egress_service_endpoints["subsets"][0]["addresses"]
]
ports = [
f"{port['protocol']}/{port['port']}"
for port in egress_service_endpoints["subsets"][0]["ports"]
]
log(
f" Service {egress_service['name']} -> ips={', '.join(ips)}, port={','.join(ports)}"
)
service = {
'name': egress_service['name'],
'ips': ips,
'ports': ports
}
output[network_policy_name]['services'].append(service)
if args.format == "json":
print(json.dumps(output, indent=2))
sys.exit(0 if not warnings else 1)
if __name__ == "__main__":
main()