Gerrit/Abuse and rate limiting
This page summarizes the available tools against abuse and rate limiting bots/scraping. It is uses GitLab/Abuse and rate limiting as a template.
Access to Gerrit can be throttled or blocked using Requestctl. Start by using the pattern/sites/gerrit and add additional expression. Don't forget to enable and commit the action.
Traffic logs and metrics
Gerrit is behind the load balancing infrastructure, so tools Superset or Turnilo can be used to analyze traffic.
There are some Logstash dashboards which can be used to analyze the traffic as well:
Gerrit has Javamelody which has a page listing ongoing HTTP threads https://gerrit.wikimedia.org/r/monitoring?part=currentRequests
Emergency access
It is possible to tunnel traffic directly to Gerrit using tunnelencabulator. This tool opens a SSH tunnel and modifies the local /etc/hosts file. In case of traffic related incidents this can be a useful tool to access gerrit (probably in combination with blocking certain/all public traffic, if needed). tunnelcabulator is included in Wmf-laptop.
jelto-wmf@x1:~$ curl -v https://gerrit.wikimedia.org
* Trying 2620:0:861:2:208:80:154:151:443...
* Connected to gerrit.wikimedia.org (2620:0:861:2:208:80:154:151) port 443 (#0)
...
HTTP/1.1 302 Found
...
jelto-wmf@x1:~$ tunnelencabulator -s # -s is needed for non-CDN-served services
jelto-wmf@x1:~$ curl -v https://gerrit.wikimedia.org
* Trying 127.149.7.2:443...
* Connected to gerrit.wikimedia.org (127.149.7.2) port 443 (#0)
...
HTTP/1.1 302 Found
...
See also Puppet#Git is down (and requires a puppet change to put it back).
Legacy tooling for Throttling and rate limiting
Some tooling still exists which pre-dates the CDN migration. New throttling should be implemented in Requestctl. For historic reasons documentation for the old tooling can be found below.
Throttling using nftables
In T366882 a new Puppet module profile::firewall::nftables_throttling was implemented which allows basic rate limiting based on a maximum number of parallel TCP connections.
If the threshold is exceeded the offending IP address is blocked for 5 minutes. More information can be found in Firewall#Throttling with nftables.
Metrics for this throttling can be found in Grafana.
Blocked nets
For emergencies it's possible to drop IPs or ranges using nftables directly on the Gerrit host:
sudo nft add rule inet filter throttling ip saddr <IP_range> drop
This adds a drop entry in the throttling chain. Keep in mind to persist this using the profile::firewall::nftables_throttling::abusers hiera key otherwise puppet reverts this after 30 minutes.
Blocking using Apache config
As seen in this patch, it is also possible to block low traffic incidents through Apache configuration. This method is not recommended for high traffic incident. The recommended method in such context would be to use Requestctl.