Phabricator is Slow Troubleshooting

Grafana dashboard for Phabricator

Increased network traffic

Grafana link for performance data including network. Typical network traffic is between 3 and 5 Mbit/sec, a significant higher value (>10 Mbit/sec on Dec 15 2020 for example) can be an indication of higher than normal use and it is worth checking the apache access logs on the machine to see if there is a pattern that can be addressed, an IP or a certain distinctive User-Agent to ban.

Database traffic is usually correlated to high Phabricator usage.

Apache Access Logs

Logstash link

The access log is in /var/log/apache2/phabricator_access.log. 400 MB/day is a typical size. Tailing and watching the file might give an indication what requests dominate. To look for repeat accesses: cat /var/log/apache2/phabricator_access.log | sed 's/@cee: //' | jq '."client.ip"' | sort -n | uniq -c | sort -n will give a list of IPs and count. Compare to cat /var/log/apache2/phabricator_access.log | wc to judge whether any number is to large. Example: On Dec 15 2020 we had about 560,000 accesses from 2 IPs that represented about 2/3 of all accesses and were responsible for a significant (3x) increase in network traffic.

Banning an IP or User-Agent

Best to use edge based banning: Requestctl IP addresses to ban are specified in requestctl/request-ipblocks/abuse/phabricator_abusers.yaml, User-agents are not implemented at the moment

Keywords: phab slow dos DDos phabbanlist