Phabricator/Slowness
Phabricator is Slow Troubleshooting
Grafana dashboard for Phabricator
Increased network traffic
Grafana link for performance data including network. Typical network traffic is between 3 and 5 Mbit/sec, a significant higher value (>10 Mbit/sec on Dec 15 2020 for example) can be an indication of higher than normal use and it is worth checking the apache access logs on the machine to see if there is a pattern that can be addressed, an IP or a certain distinctive User-Agent to ban.
Database traffic is usually correlated to high Phabricator usage.
Apache Access Logs
The access log is in /var/log/apache2/phabricator_access.log. 400 MB/day is a typical size. Tailing and watching the file might give an indication what requests dominate. To look for repeat accesses: cat /var/log/apache2/phabricator_access.log | sed 's/@cee: //' | jq '."client.ip"' | sort -n | uniq -c | sort -n will give a list of IPs and count. Compare to cat /var/log/apache2/phabricator_access.log | wc to judge whether any number is to large. Example: On Dec 15 2020 we had about 560,000 accesses from 2 IPs that represented about 2/3 of all accesses and were responsible for a significant (3x) increase in network traffic.
cat /var/log/apache2/phabricator_access.log | sed 's/@cee: //' | jq '."user_agent.original"' | sort -n | uniq -c | sort -n does the same report for the user-agent. Example: On Dec 15 2020, we had 1,100,00 accesses from user-agent vscode-phabricator.
Banning an IP or User-Agent
Best to use edge based banning: Requestctl IP addresses to ban are specified in requestctl/request-ipblocks/abuse/phabricator_abusers.yaml, User-agents are not implemented at the moment
Keywords: phab slow dos DDos phabbanlist