Phabricator/Slowness

Phabricator is Slow Troubleshooting

Grafana dashboard for Phabricator

Increased network traffic

Grafana link for performance data including network. Typical network traffic is between 3 and 5 Mbit/sec, a significant higher value (>10 Mbit/sec on Dec 15 2020 for example) can be an indication of higher than normal use and it is worth checking the apache access logs on the machine to see if there is a pattern that can be addressed, an IP or a certain distinctive User-Agent to ban.

Database traffic is usually correlated to high Phabricator usage.

Apache Access Logs

The access log is in /var/log/apache2/phabricator_access.log. 400 MB/day is a typical size. Tailing and watching the file might give an indication what requests dominate. To look for repeat accesses: cat /var/log/apache2/phabricator_access.log | awk 'BEGIN {FS="[\t]+"} {a[$19]++}END{for(i in a){print a[i] " " i}}' | sort -n will give a list of IPs and count. Compare to cat /var/log/apache2/phabricator_access.log | wc to judge whether any number is to large. Example: On Dec 15 2020 we had about 560,000 accesses from 2 IPs that represented about 2/3 of all accesses and were responsible for a significant (3x) increase in network traffic.

cat /var/log/apache2/phabricator_access.log | awk 'BEGIN {FS="[\t]+"} {a[$12]++}END{for(i in a){print a[i] " " i}}' | sort -n does the same report for the user-agent. Example: On Dec 15 2020, we had 1,100,00 accesses from user-agent vscode-phabricator.

cat /var/log/apache2/phabricator_access.log | grep vscode-phabricator | awk 'BEGIN {FS="[\t]+"} {a[$1]++}END{for(i in a){print a[i] " " i}}' | sort -n gives an overview of how many requests per second the User-Agent (or IP) in question is generating.

Banning an IP or User-Agent

Best to use edge based banning: [1] IP addresses to ban are specified in puppet/hiera in file common.yaml under phabricator_abusers, similar to varnish blacklists. User-agents are not implemented at the moment

But Phabricator also has a ban list in /etc/apache2/phabbanlist.conf, where IPs and User-Agent can be excluded from access (banned).

phanbanlist.conf is version controlled and managed by puppet. For temporary troubleshooting purposes it is acceptable to edit the local file and restart apache with systemctl reload apache2 but the change needs to be checked in and documented in a phabricator ticket.

Keywords: phab slow dos DDos phabbanlist