User:Giuseppe Lavagetto/How to debug HHVM
We have one HHVM appserver with high load and some thread constantly using up 100% of the CPU, and we want to figure out what's going on.
Here is a recap of my standard procedure, for posterity and for stopping exercising my muscle memory
- Depool the appserver, some of the things we will do are quite disruptive, and also we need to weed out effects of incoming traffic from hanging threads.
- Identify all hanging threads
- Attach to the master process
gdb -p $hhvm_main_pid
- Check out the thread numbers and corresponding PIDs, enter the thread stack, and dump the request headers
(gdb) info threads ... # Find the thread n/system PID correspondence (gdb) thread <N> (gdb) set print elements 0 (gdb) p (*(HPHP::FastCGITransport*)g_context.m_node.m_p.m_transport).m_requestParams # This returns a hash containing all request parameters
- Check what's happening in PHP terms using perf record/perf report on the threadthis could give you an indication of where - if by any chance - in the php code we're spending most time
# record call graph attaching to pid PID perf record -g -p $PID -F 1 sleep 5 chown root /tmp/perf*.map perf report
- If you still don't have enough information, you can gather more from doing a full core dump and attaching gdb to it
hhvm-dump-debug --full # this will create the .bt file, with the full gdb backtrace, and a core file that can be examined later.