Debugging in production

From Wikitech
(Redirected from Debugging)
Jump to navigation Jump to search

For general information on HHVM debugging, see . In the mwrepl shell, you can use "?" to get help, and "b ?" to get help on breakpoints

Debugging a web request


Use X-Wikimedia-Debug to make a request bypass Varnish cache and route to a specific debug server.


You can make a request to the local Apache from any web server by using cURL, for example:

mwdebug1001$ curl -H 'Host:' 'http://mwdebug1001/'
<!DOCTYPE html>
<html lang="mul" dir="ltr">
<meta charset="utf-8">
<meta name="description" content="Wikimedia is a global movement whose mission is to bring free educational content to the world.">

.. and with verbose logging:

mwdebug1001$ curl -v -H 'Host:' 'http://mwdebug1001/'
* Connected to ( port 80 (#0)
> GET / HTTP/1.1 [..]
> Host:
< HTTP/1.1 301 Moved Permanently [..]
< Server: mwdebug1001.eqiad.wmnet [..]
< Location:

Note: You do need to use the server's hostname or IP address in the URL. Prior to 2015, it was also possible to simply use http://localhost but per T190111 that now uses the VirtuallHost for status-check connections, which doesn't expose MediaWiki. If you want a universal snippet that works for any production server, use $HOSTNAME or hostname -i instead:

$ curl -v -H 'Host:' "http://$HOSTNAME/w/load.php"
$ curl -v -H 'Host:' "http://$(hostname -i)/w/load.php"

Pushing code to a debug server

Developers can put code updates on one of the mwdebug hosts, before deploying to the entire production cluster, see Pre-deployment testing in production .

Conditional code

Note that any changes you make this way will be overwritten by cluster-wide deployments. So, long-term changes should go into a block wrapped in an if ( $wgDBname === 'testwiki' ) (to prevent them from accidentally running on all wikis!). Short-term changes (anything not committed to the git repo) should either be committed and rolled out, or reverted as soon as possible.

PHP7 Opcache

When editing files on a debug server directly, remember to clear the PHP7 opcache afterwards. Without this, changes to files on disk might not take affect.

mwdebug1001$ php7adm /opcache-free

When using Scap to pull down a change from the deployment host, this happens automatically.

Testing it

Use X-Wikimedia-Debug in a browser to route one of your regular web requests to the debug server you have staged code on.

Remember that canary servers still communicate with other shared resources such as MySQL (e.g. GlobalUsage or CentralAuth). When staged on a canary server, you should probably still form requests for (instead of production wiki names) to avoid any changes from leaking to shared resources used by other servers and thus affecting production wikis.

Debugging a maintenance script

ssh to a mwdebug host, then:

source /usr/local/lib/
sudo -u "$MEDIAWIKI_WEB_USER" hhvm -m debug "$MEDIAWIKI_DEPLOYMENT_DIR/multiversion/MWScript.php" someScript.php --wiki=testwiki --scriptSpecificParameters "goHere"

Debugging in shell

To open a debugger, run:

mwrepl wikidbname

on mwmaint1002. wikidbname is e.g. eswiki. You can set breakpoints, then call arbitrary MW code.

Debugging action API requests in shell

Sometimes, it is convenient to debug through the action API. Since this is a user-facing entry point, it allows testing user parameters and permission checks.

Start by opening mwrepl as above. Then use:

define( 'MW_API', true ); // Signal this is api.php.

$wgUser = User::newFromName( <Username> );
$wgTitle = Title::makeTitle( NS_SPECIAL, 'Badtitle/dummy title in manual testing' );

$token = $wgUser->getEditToken( '', $wgRequest ); // Although write actions will result in strange logs

$params = [ 'action' => '<action>', 'token' => $token, /* etc */ ];
$request = new DerivativeRequest( $wgRequest, $params, /* $wasPosted = */ true );
$request->setIP( '' ); // An IP must be set

$context = RequestContext::getMain();
$context->setUser( $wgUser );
$context->setTitle( $wgTitle );
$context->setRequest( $request );

$main = new ApiMain( $context, true );
wfRunHooks( 'ApiBeforeMain', array( &$main ) ); // For CirrusSearch and other extensions

// Examine $main->getResult()->getResultData() or whatever else you need to do

For code that is not specific to the shell, and more details on internal requests, see mw:API:Calling internally.