Monitoring/check conntrack

check_conntrack is an Icinga check that ensures that the nf_conntrack table is not full.

Things to do

• Check if the sysctl net.netfilter.nf_conntrack_tcp_timeout_time_wait is correctly set to 65, there's a race connection which sometimes make it apply the 120 seconds default:

https://phabricator.wikimedia.org/T136094. If that's the case, reset it to 65: "sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=65"

• check the relevant "max" value: cat /proc/sys/net/netfilter/nf_conntrack_max
• check the current count: cat /proc/sys/net/netfilter/nf_conntrack_count
• raise the max value: echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max

• permanently change the max value in puppet using sysctl::parameters (example)
• example ticket handling this alert: https://phabricator.wikimedia.org/T131028