User:Giuseppe Lavagetto/How to debug HHVM

From Wikitech

We have one HHVM appserver with high load and some thread constantly using up 100% of the CPU, and we want to figure out what's going on.

Here is a recap of my standard procedure, for posterity and for stopping exercising my muscle memory

  • Depool the appserver, some of the things we will do are quite disruptive, and also we need to weed out effects of incoming traffic from hanging threads.
  • Identify all hanging threads
  • Attach to the master process

gdb -p $hhvm_main_pid

  • Check out the thread numbers and corresponding PIDs, enter the thread stack, and dump the request headers
(gdb) info threads 
... # Find the thread n/system PID correspondence
(gdb) thread <N>
(gdb) set print elements 0
(gdb) p (*(HPHP::FastCGITransport*)g_context.m_node.m_p.m_transport).m_requestParams
# This returns a hash containing all request parameters
  • Check what's happening in PHP terms using perf record/perf report on the thread
    # record call graph attaching to pid PID
perf record -g -p $PID -F 1 sleep 5
chown root /tmp/perf*.map
perf report
    this could give you an indication of where - if by any chance - in the php code we're spending most time
  • If you still don't have enough information, you can gather more from doing a full core dump and attaching gdb to it
    hhvm-dump-debug --full
# this will create the .bt file, with the full gdb backtrace, and a core file that can be examined later.
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=User:Giuseppe_Lavagetto/How_to_debug_HHVM&oldid=1788546"