Debugging in production

From Wikitech
Jump to navigation Jump to search

For general information on HHVM debugging, see https://github.com/facebook/hhvm/blob/master/hphp/doc/debugger.start . In the mwrepl shell, you can use "?" to get help, and "b ?" to get help on breakpoints

Debugging a web request

Externally

Use X-Wikimedia-Debug to make a request bypass Varnish cache and route to a specific debug server.

Locally

When needing to make a request to the local Apache from the same web server, use curl. Below a few examples:

$ curl -v -H 'Host: www.wikimedia.org' 'http://localhost/'
<!DOCTYPE html>
<html lang="mul" dir="ltr">
<head>
<meta charset="utf-8">
<title>Wikimedia</title>
<meta name="description" content="Wikimedia is a global movement whose mission is to bring free educational content to the world.">
[..]
$ curl -v -H 'Host: test.wikipedia.org' 'http://localhost/'
* Connected to localhost (::1) port 80 (#0)
> GET / HTTP/1.1 [..]
> Host: test.wikipedia.org
> 
< HTTP/1.1 301 Moved Permanently [..]
< Server: mwdebug1001.eqiad.wmnet [..]
< Location: http://test.wikipedia.org/wiki/Main_Page

Per T190111, the above does not work currently (due to localhost addresses being reserved for status-check connections). This can be worked around by substituting localhost with $(hostname -i), which makes it connect via the LAN address instead. Be sure to wrap the url string in double quotes for this to work:

$ curl -v -H 'Host: test.wikipedia.org' "http://$(hostname -i)/w/load.php"

Pushing code to a debug server

Developers can put code updates on the hosts serving as the mwdebug1001, mwdebug1002 servers shortly before deploying to the entire production cluster, see Pre-deployment testing in production .

Note that any changes you make this way will be overwritten by cluster-wide deployments. So, long-term changes should go into a block wrapped in an if ( $wgDBname === 'testwiki' ) (to prevent them from accidentally running on all wikis!). Short-term changes (anything not committed to the git repo) should either be committed and rolled out, or reverted as soon as possible.

test.wikipedia.org

Remember that canary servers still communicate with other shared resources such as MySQL (e.g. GlobalUsage or CentralAuth). When staged on a canary server, you should probably still form requests for test.wikipedia.org (instead of production wiki names) to avoid any changes from leaking to shared resources used by other servers and thus affecting production wikis.

Debugging a maintenance script

ssh into mwdebug1001

source /usr/local/lib/mw-deployment-vars.sh
sudo -u "$MEDIAWIKI_WEB_USER" hhvm -m debug "$MEDIAWIKI_DEPLOYMENT_DIR/multiversion/MWScript.php" someScript.php --wiki=testwiki --scriptSpecificParameters "goHere"

Debugging in shell

To open a debugger, run:

mwrepl wikidbname

on mwmaint1001. wikidbname is e.g. eswiki. You can set breakpoints, then call arbitrary MW code.

Debugging action API requests in shell

Sometimes, it is convenient to debug through the action API. Since this is a user-facing entry point, it allows testing user parameters and permission checks.

Start by opening mwrepl as above. Then use:

define( 'MW_API', true ); // Signal this is api.php.

$wgUser = User::newFromName( <Username> );
$wgTitle = Title::makeTitle( NS_SPECIAL, 'Badtitle/dummy title in manual testing' );

$token = $wgUser->getEditToken( '', $wgRequest ); // Although write actions will result in strange logs

$params = [ 'action' => '<action>', 'token' => $token, /* etc */ ];
$request = new DerivativeRequest( $wgRequest, $params, /* $wasPosted = */ true );
$request->setIP( '127.0.0.1' ); // An IP must be set

$context = RequestContext::getMain();
$context->setUser( $wgUser );
$context->setTitle( $wgTitle );
$context->setRequest( $request );

$main = new ApiMain( $context, true );
wfRunHooks( 'ApiBeforeMain', array( &$main ) ); // For CirrusSearch and other extensions
$main->execute();

// Examine $main->getResult()->getResultData() or whatever else you need to do

For code that is not specific to the shell, and more details on internal requests, see mw:API:Calling internally.