X-Wikimedia-Debug

From Wikitech
Jump to: navigation, search

X-Wikimedia-Debug is an HTTP request header you can use to aid problem determination in production. You can use the header to control several aspects of backend request processing:

  • Backend selection: force Varnish to pass your request to a specific backend application server.
  • Caching behavior: requests that bear an X-Wikimedia-Debug header are always handled as a Varnish cache miss.
  • Request profiling: record a trace of request-processing code for performance analysis.
  • Debug logs: enable MediaWiki's verbose debug log group.
  • Read-only mode: request a page with MediaWiki set to read-only mode.

There is a X-Wikimedia-Debug browser extension for any WebExtension-supported browser that make working with the header easier: Chromium and Firefox (source code).

Command-line usage

Force Varnish to skip the cache and pass request to mwdebug1002.eqiad.wmnet:

$ curl -H 'X-Wikimedia-Debug: backend=mwdebug1002.eqiad.wmnet' https://meta.wikimedia.org/wiki/Main_Page

Same as above, but profile request using XHProf and log profile to XHGui:

$ curl -H 'X-Wikimedia-Debug: backend=mwdebug1002.eqiad.wmnet; profile' https://meta.wikimedia.org/wiki/Main_Page

Request a page with MediaWiki configured for read-only mode:

$ curl -H 'X-Wikimedia-Debug: backend=mwdebug1002.eqiad.wmnet; readonly' https://meta.wikimedia.org/wiki/Main_Page

Staging changes

This section will walk you through a common X-Wikimedia-Debug debugging workflow.

It worked on your machine. It worked on the Beta Cluster. And the unit tests pass. But will it work in production? You can reduce the risk of unexpected breakage by deploying your change to a debugging server in production and using the X-Wikimedia-Debug header to verify its behavior.

Follow the instructions on How to deploy code, but stop after you have completed step 2 ("Get the code on the deployment host"). Now sync your change to one of the debug backends in a new tap by SSHing into it and running scap pull:

you@laptop:~$ ssh mwdebug1002.eqiad.wmnet
you@mwdebug1002:~$ scap pull

Now use the X-Wikimedia-Debug extension to enable header injection, selecting the same backend you just synced your code to:

X-Wikimedia-Debug demo.png

Your requests will be routed to the backend on which you staged your changes, allowing you to verify that is working correctly.

Request profiling

The "Profiling Data" link is in the footer.

Setting the profile attribute in the X-Wikimedia-Debug header will use XHProf to capture a profile and send it to XHGui. If you are using the browser extension, there will be a link in the page footer with the label Profiling Data. Clicking or tapping on it will take you directly to a list of profiles matching your Request ID. Then click on the method (e.g. "GET") or timestamp link to show the recorded profile.

The Firefox add-on adds a menu item named X-Wikimedia-Debug profile to the context menu, shown when you right-click on the page body.

XHGui feature:

  • Overview dashboard (e.g. listing functions that used the most time and/or most memory).
  • Complete call tree with timing, counts, sub/parent for each function.
  • Flamegraph visualisation. (example)
  • Callgraph visualisation.
  • and more...

To get an idea of what XHGui provides, view this example request.

Debug logging

Setting the log attribute in the X-Wikimedia-Debug header will cause MediaWiki to be maximally verbose, forwarding all log messages on all channels (regardless of whether or not they are otherwise enabled) to Logstash and /a/mw-log/XWikimediaDebug.log on mwlog1001. See Logs#mw-log for more information.

If you are using the browser extension, there will be a Debug Logs link in the footer that navigates to the Logstash view of log messages generated in the course of preparing the page.

You can also construct the URL for the Logstash view manually, by typing the following code into your browser's JavaScript console:

'https://logstash.wikimedia.org/#/dashboard/elasticsearch/request-id?id=' + mw.config.get('wgRequestId');

Available backends

The following application servers are reserved for X-Wikimedia-Debug use:

  • mwdebug1001.eqiad.wmnet
  • mwdebug1002.eqiad.wmnet
  • mw2017.codfw.wmnet
  • mw2099.codfw.wmnet

Browser extensions

Screenshot of WikimediaDebug extension
Main interface of the WikimediaDebug extension

Source code

See also