WikimediaDebug is a set of tools for debugging and profiling MediaWiki web requests in a production environment.
You can use WikimediaDebug through the accompanying browser extension, or from the command-line using the
The following options exist to configure the backend request:
- Backend selection: force Varnish to pass your request to a specific backend application server.
- Caching behavior: requests that bear an X-Wikimedia-Debug header are always handled as a Varnish cache miss.
- Request profiling: record a trace of request-processing code for performance analysis.
- Verbose logs: enable MediaWiki's verbose debug log group.
- Read-only mode: request a page with MediaWiki set to read-only mode.
Install the extension on supported browsers: Firefox or Chromium/Chrome. It should add a button with a grayscale version of the Wikimedia icon, usually near the upper right corner of the browser window.
To make a web request to one of the debug servers, click the icon, select the right server (if you’re not deploying a change yourself, the deployer should have told you which server to select), then toggle the on/off switch. When the extension is enabled, the button should show an “ON” badge. (See also the screenshot above.)
Force Varnish to skip the cache and pass request to mwdebug1001.eqiad.wmnet:
$ curl -H 'X-Wikimedia-Debug: backend=mwdebug1001.eqiad.wmnet' https://meta.wikimedia.org/wiki/Main_Page
Same as above, but profile request using Tideways XHProf and log profile to XHGui:
$ curl -H 'X-Wikimedia-Debug: backend=mwdebug1001.eqiad.wmnet; profile' https://meta.wikimedia.org/wiki/Main_Page
Request a page with MediaWiki configured for read-only mode:
$ curl -H 'X-Wikimedia-Debug: backend=mwdebug1001.eqiad.wmnet; readonly' https://meta.wikimedia.org/wiki/Main_Page
This section will walk you through a common X-Wikimedia-Debug debugging workflow.
It worked on your machine. It worked on the Beta Cluster. And the unit tests pass. But will it work in production? You can reduce the risk of unexpected breakage by deploying your change to a debugging server in production and using the X-Wikimedia-Debug header to verify its behavior.
Follow the instructions on How to deploy code, but stop after you have completed step 2 ("Get the code on the deployment host"). Now sync your change to one of the debug backends in a new tap by SSHing into it and running
you@laptop:~$ ssh mwdebug1001.eqiad.wmnet you@mwdebug1001:~$ scap pull
Now use the X-Wikimedia-Debug extension to enable header injection, selecting the same backend you just synced your code to. Your requests will be routed to the backend on which you staged your changes, allowing you to verify that is working correctly.
profile attribute in the X-Wikimedia-Debug header will use XHProf to capture a profile and send it to XHGui. If you are using the browser extension, there will be a link in the page footer with the label "Find in XHGui". Clicking or tapping on it will take you directly to a list of profiles matching your Request ID. Then click on the method (e.g. "GET") or timestamp link to show the recorded profile.
The Firefox add-on adds a menu item named
X-Wikimedia-Debug profile to the context menu, shown when you right-click on the page body.
- Overview dashboard (e.g. listing functions that used the most time and/or most memory).
- Complete call tree with timing, counts, sub/parent for each function.
- Callgraph visualisation.
- and more...
To get an idea of what XHGui provides, view this example profile.
Plaintext request profile
In addition to obtaining a profiler via XHGui (see above), it is also possible to get a plaintext-format profile directly from the web server.
Enable X-Wikimedia-Debug with the
forceprofile attribute ("Inline profile" option). On any MediaWiki web request (including index.php, api.php, and load.php), this will result in a plain text profile being appended to the web response.
With an X-Wikimedia-Debug header present, view this example request. Toward the end of the response is a call-graph summary, that should look roughly like this:
/* 100.00% 1437.125 1 - main() 87.21% 1253.268 1 - ResourceLoader::respond 79.31% 1139.756 1509 - ResourceLoaderModule::getVersionHash 77.88% 1119.292 3 - ResourceLoader::getCombinedVersion ..
Example using cURL:
# Production $ curl -H 'X-Wikimedia-Debug: 1; forceprofile' 'https://test.wikipedia.org/w/load.php?modules=startup&only=scripts&raw=1' # Beta Cluster $ curl -H 'X-Wikimedia-Debug: 1; forceprofile' 'https://en.wikipedia.beta.wmflabs.org/w/load.php?modules=startup&only=scripts&raw=1'
For maintenance scripts, this can be triggered using the
--profiler=text option. For both web requests profiling and CLI profiling, the debug tools are only installed on mwdebug servers, so be sure to run mwscript from an mwdebug server (in Beta Cluster, any MW server will do).
mwdebug:~$ mwscript showJobs.php --wiki testwiki --profiler text 0 <!-- 100.00% 282.967 1 - main() 89.56% 253.419 1 - section.Setup.php 57.12% 161.627 139 - AutoLoader::autoload 49.59% 140.324 1 - ExtensionRegistry::loadFromQueue ..
This feature requires a WMF or Volunteer NDA Wikimedia developer account.
log attribute in the X-Wikimedia-Debug header (“Verbose log” checkbox in the extension) will cause MediaWiki to be maximally verbose, recording all log messages on all channels (regardless of whether or not they are otherwise enabled in wmf-config).
These messages will end up in Logstash at https://logstash.wikimedia.org/app/dashboards#/view/mwdebug and in
/srv/mw-log/XWikimediaDebug.log on mwlog1002 (mwlog2002 during data center switch). See Logs#mw-log for more information.
To view the logs of a specific web request only, the browser extension adds a "Find in Logstash" link in the footer on any wiki as well, which opens goes to Logstash with a filter for the request ID of the current page view. You can also construct this URL manually, by navigating to https://logstash.wikimedia.org/app/dashboards#/view/x-debug and entering a search query like
The following application servers are reserved for X-Wikimedia-Debug use:
- mwdebug1001.eqiad.wmnet (reimaged to buster on 2020-02-16)
- mwdebug1002.eqiad.wmnet (reimaged to buster on 2020-02-16)
- mwdebug2001.codfw.wmnet (reimaged to buster on 2020-02-12)
- mwdebug2002.codfw.wmnet (reimaged to buster on 2020-02-12)
- k8s-experimental (mediawiki on kubernetes project mw-on-k8s
It's now also possible to reach an experimental MediaWiki installation running on kubernetes using the `k8s-experimental` server name from the list. Beware that most of the features described in this wiki page is not available at the moment on that installation, and it's also still not necessarily working correctly at the moment. Please use another backend unless you really need to test things on kubernetes.
X-Wikimedia-Debug does not currently work on the Wikitech wiki, which is served by servers separated from the rest of the Wikimedia Foundation production cluster. Try it on Wikipedia instead!
If it appears that your request are not producing any results, check if XHProf/tideways is properly installed and called:
- verify that
php-tideways-xhprofis installed on the target host
- ensure that the extension is loaded, i.e. look for
- check if
auto_prepend_fileis defined in
- Maintained by Performance Team
- Live chat (IRC): #wikimedia-perf connect
- Issue tracker: Phabricator (Report an issue)
Documented in README.md.