Debugging in production

From Wikitech
Jump to: navigation, search

For general information on HHVM debugging, see https://github.com/facebook/hhvm/blob/master/hphp/doc/debugger.start . In the mwrepl shell, you can use "?" to get help, and "b ?" to get help on breakpoints

Request routing

Using X-Wikimedia-Debug a request can bypass Varnish cache and be routed to a specific debug server.

Pushing code to mwtest1001,2

Developers can put code updates on the hosts serving as the mwdebug1001, mwdebug1002 servers shortly before deploying to the entire production cluster, see Pre-deployment testing in production and "Trying tin's code on testwiki" on How to deploy code. Basically: Stage the changes on tin, then run scap pull on mwdebug1001.

Note that any changes you make this way will be overwritten by cluster-wide deployments. So, long-term changes should go into a block wrapped in an if ( $wgDBname === 'testwiki' ) (to prevent them from accidentally running on all wikis!). Short-term changes (anything not committed to the git repo) should either be committed and rolled out, or reverted as soon as possible.

test.wikipedia.org

Remember that canary servers still communicate with other shared resources such as MySQL (e.g. GlobalUsage or CentralAuth). When staged on a canary server, you should probably still form requests for test.wikipedia.org (instead of production wiki names) to avoid any changes from leaking to shared resources used by other servers and thus affecting production wikis.


Debugging a maintenance script

ssh into mwdebug1001

source /usr/local/lib/mw-deployment-vars.sh
sudo -u "$MEDIAWIKI_WEB_USER" hhvm -m debug "$MEDIAWIKI_DEPLOYMENT_DIR/multiversion/MWScript.php" someScript.php --wiki=testwiki --scriptSpecificParameters "goHere"

Debugging in shell

To open a debugger, run:

mwrepl wikidbname

on terbium. wikidbname is e.g. eswiki. You can set breakpoints, then call arbitrary MW code.

Debugging action API requests in shell

Sometimes, it is convenient to debug through the action API. Since this is a user-facing entry point, it allows testing user parameters and permission checks.

Start by opening mwrepl as above. Then use:

define( 'MW_API', true ); // Signal this is api.php.

$wgUser = User::newFromName( <Username> );
$wgTitle = Title::makeTitle( NS_SPECIAL, 'Badtitle/dummy title in manual testing' );

$token = $wgUser->getEditToken( '', $wgRequest ); // Although write actions will result in strange logs

$params = [ 'action' => '<action>', 'token' => $token, /* etc */ ];
$request = new DerivativeRequest( $wgRequest, $params, /* $wasPosted = */ true );
$request->setIP( '127.0.0.1' ); // An IP must be set

$context = RequestContext::getMain();
$context->setUser( $wgUser );
$context->setTitle( $wgTitle );
$context->setRequest( $request );

$main = new ApiMain( $context, true );
wfRunHooks( 'ApiBeforeMain', array( &$main ) ); // For CirrusSearch and other extensions
$main->execute();

// Examine $main->getResult()->getResultData() or whatever else you need to do

For code that is not specific to the shell, and more details on internal requests, see mw:API:Calling internally.