Debugging in production

From Wikitech
(Redirected from Debugging)
Jump to navigation Jump to search

Debugging a web request

Externally

Use X-Wikimedia-Debug to make a request bypass Varnish cache and route to a specific debug server.

Locally

You can make a local self request from any web server by using curl, like so, for a regular MediaWiki request over HTTPS:

mwdebug1002$ curl -i --connect-to "::$HOSTNAME" 'https://test.wikipedia.org/w/load.php'
HTTP/1.1 200 OK
Server: mwdebug1002.eqiad.wmnet
…
/* This file is theWeb entry point for MediaWiki's ResourceLoader: … */

Or over HTTP:

mwdebug1002$ curl -i --connect-to "::$HOSTNAME" 'http://test.wikipedia.org/wiki/Main_Page'
HTTP/1.1 302 Found
Server: mwdebug1002.eqiad.wmnet
Location: https://test.wikipedia.org/wiki/Main_Page
mwdebug1001$ curl -i --connect-to "::$HOSTNAME" 'http://www.wikimedia.org/'
HTTP/1.1 200 OK
Server: mwdebug1001.eqiad.wmnet
…
<!DOCTYPE html>
<html lang="mul" dir="ltr">
<head>
<meta charset="utf-8">
<title>Wikimedia</title>
<meta name="description" content="Wikimedia is a global movement whose mission is to bring free educational content to the world.">
…

And over HTTP as if from an external HTTPS request (This is currently the only way to debug in Beta Cluster, since internal HTTPS is not available there):

deployment-mediawiki11$ curl -i --connect-to "::$HOSTNAME" -H 'X-Forwarded-Proto: https' 'http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page'
HTTP/1.1 200 OK
Server: deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud
…
<!DOCTYPE html>
…


Note about Host header: Prior to 2015, the more traditional approach of using curl 'http://localhost/wiki/Main_Page' -H 'Host: test.wikipedia.org' was supported, but per T190111 this is no longer possible because connections via "localhost" are handled by a higher priority VirtualHost in Apache that serves responses for the health status checks (not related to MediaWiki).

Note about FQDN address: Prior to 2019, it was common to workaround the above "localhost" issue by using a FQDN address instead, with $HOSTNAME or $(hostname -f), e.g. like curl -i -H 'Host: test.wikipedia.org' "http://$HOSTNAME/w/load.php". While this still works today for HTTP requests, it does not work reliably for HTTPS requests since the web server in question has no certificate for the internal hostname, though this could be bypassed with curl -k.

Note about --resolve option: Prior to 2020, other documentation pages recommended --resolve as the main strategy, e.g. curl -i --resolve "test.wikipedia.org:443:$(hostname -i)" 'https://test.wikipedia.org/w/load.php'. This still works perfectly today and is functionally equivalent to the current recommendation with --connect-to. The --resolve option is no longer recommended because it is too easy to misuse and not realize that it was silently ignored. For example, if it specifies a different hostname than the real URL, then curl will connect to the main production edge instead which is easy to miss if you don't enable verbose -v mode and check what server it connected to. This can be mitigated by using a wildcard hostname like --resolve "*:443:$(hostname -i)" but that still requires getting the port right, which means over HTTP, it would get silenly ignored again, plus it requires the IP address and thus the extra hostname command. The --connect-to option has the benefit of allowing both host and port to be unspecified, and supports a hostname as destination (instead of IP address), thus allowing the simpler and more memorable "::$HOSTNAME" form.

Pushing code to a debug server

Developers can put code updates on one of the mwdebug hosts, before deploying to the entire production cluster, see Pre-deployment testing in production .

Conditional code

Note that any changes you make this way will be overwritten by cluster-wide deployments. So, long-term changes should go into a block wrapped in an if ( $wgDBname === 'testwiki' ) (to prevent them from accidentally running on all wikis!). Short-term changes (anything not committed to the git repo) should either be committed and rolled out, or reverted as soon as possible.

PHP7 Opcache

When editing files on a debug server directly, remember to clear the PHP7 opcache afterwards. Without this, changes to files on disk might not take affect.

mwdebug1001$ php7adm /opcache-free

When using Scap to pull down a change from the deployment host, this happens automatically.

Testing it

Use X-Wikimedia-Debug in a browser to route one of your regular web requests to the debug server you have staged code on.

Debugging a maintenance script

ssh to a mwdebug host, then:

source /usr/local/lib/mw-deployment-vars.sh
sudo -u "$MEDIAWIKI_WEB_USER" php -m debug "$MEDIAWIKI_DEPLOYMENT_DIR/multiversion/MWScript.php" someScript.php --wiki=testwiki --scriptSpecificParameters "goHere"

Debugging logs

To locally debug messages sent to Logstash from MediaWiki or php-fpm, read Application_servers/Runbook#Logging.

Debugging in shell

To open a debugger, run:

mwrepl wikidbname

on mwmaint1002. wikidbname is e.g. eswiki. You can set breakpoints, then call arbitrary MW code.

Debugging action API requests in shell

Sometimes, it is convenient to debug through the action API. Since this is a user-facing entry point, it allows testing user parameters and permission checks.

Start by opening mwrepl as above. Then use:

define( 'MW_API', true ); // Signal this is api.php.

$wgUser = User::newFromName( <Username> );
$wgTitle = Title::makeTitle( NS_SPECIAL, 'Badtitle/dummy title in manual testing' );

$token = $wgUser->getEditToken( '', $wgRequest ); // Although write actions will result in strange logs

$params = [ 'action' => '<action>', 'token' => $token, /* etc */ ];
$request = new DerivativeRequest( $wgRequest, $params, /* $wasPosted = */ true );
$request->setIP( '127.0.0.1' ); // An IP must be set

$context = RequestContext::getMain();
$context->setUser( $wgUser );
$context->setTitle( $wgTitle );
$context->setRequest( $request );

$main = new ApiMain( $context, true );
wfRunHooks( 'ApiBeforeMain', array( &$main ) ); // For CirrusSearch and other extensions
$main->execute();

// Examine $main->getResult()->getResultData() or whatever else you need to do

For code that is not specific to the shell, and more details on internal requests, see mw:API:Calling internally.