Measure Performance

From Wikitech
Jump to: navigation, search

We continuously measure the performance from our users using https://github.com/wikimedia/mediawiki-extensions-NavigationTiming and our WebPageTest instance but you can also test it yourself by either use the performance APIs our browsers support or use tools to collect metrics.

Getting metrics from your browser

Modern browsers has built in support for performance measurements. We use some of these metrics to collect data from real users to know the performance of Wikipedia. You can get those metrics yourself by running JavaScript in your browser console.

One important thing: Most of these metrics are browser focused instead of user focused.

Navigation Timing API

The Navigation Timing API is supported in all major browsers. From the API you get information how the browser is processing the page and all the assets. See https://www.w3.org/TR/navigation-timing/#process for the full picture.

In version 1 all time metrics is measured in milliseconds since midnight of January 1, 1970 (UTC). With version 2 all metrics are relative to navigation start.

If your browser supports version 1 you can get the information by:

var t = window.performance.timing;
console.log({
    navigationStart: 0,
    unloadEventStart: t.unloadEventStart > 0 ? t.unloadEventStart - t.navigationStart : undefined,
    unloadEventEnd: t.unloadEventEnd > 0 ? t.unloadEventEnd - t.navigationStart : undefined,
    redirectStart: t.redirectStart > 0 ? t.redirectStart - t.navigationStart : undefined,
    redirectEnd: t.redirectEnd > 0 ? t.redirectEnd - t.navigationStart : undefined,
    fetchStart: t.fetchStart - t.navigationStart,
    domainLookupStart: t.domainLookupStart - t.navigationStart,
    domainLookupEnd: t.domainLookupEnd - t.navigationStart,
    connectStart: t.connectStart - t.navigationStart,
    connectEnd: t.connectEnd - t.navigationStart,
    secureConnectionStart: t.secureConnectionStart ? t.secureConnectionStart - t.navigationStart : undefined,
    requestStart: t.requestStart - t.navigationStart,
    responseStart: t.responseStart - t.navigationStart,
    responseEnd: t.responseEnd - t.navigationStart,
    domLoading: t.domLoading - t.navigationStart,
    domInteractive: t.domInteractive - t.navigationStart,
    domContentLoadedEventStart: t.domContentLoadedEventStart - t.navigationStart,
    domContentLoadedEventEnd: t.domContentLoadedEventEnd - t.navigationStart,
    domComplete: t.domComplete - t.navigationStart,
    loadEventStart: t.loadEventStart - t.navigationStart,
    loadEventEnd: t.loadEventEnd - t.navigationStart
});

If your browser supports version 2 you can run it like this to get all the entries:

window.performance.getEntriesByType('navigation').forEach(entry => console.log(entry));

User Timing API

The User Timing APIis also supported by all major browsers. The API let us developers define measurements on the page. We use it today to measure the JavaScript startup time for MediaWiki and in the future this API will be more important to us if we do a single page application.

We currently only use marks and not measurements, so to get the marks we create:

console.log(window.performance.getEntriesByType('mark'));

Resource Timing API

The Resource Timing API is about getting information about when all resources are downloaded for a page. In version 2 the size of the resource is also included. To get information from resources on different domains you need to add the response header Timing-Allow-Origin (we do that on Wikipedia domains).

console.log(window.performance.getEntriesByType('resource'));

First paint(ish)

First paint means when the browser starts to paint something on the screen. In the past Chrome and IE has been the only one supporting this non standard but Firefox is coming along. What's interesting about first paint is that it is related to user experience.

One important thing about first paint is that this is when the browser thinks pixels are painted on the screen, it's not 100% that it matches when it really happens.

Chrome
console.log(window.performance.getEntriesByType('paint'));
IE
console.log(window.performance.timing.msFirstPaint - window.performance.timing.navigationStart);
Firefox

In Firefox firstPaint is called timeToNonBlankPaint and are at the moment behind a preference (you need to turn it on in your browser). You need to set dom.performance.time_to_non_blank_paint.enabled to true for it to work.

console.log(window.performance.timing.timeToNonBlankPaint - window.performance.timing.navigationStart);

Testing performance on your local machine

There's two ways of testing performance on your local machine:

  • Use developer tools to get in deep information about JavaScript and layout. This is good to use to look into bottlenecks or concrete problems.
  • Collect First Visual Change and Speed Index to make sure that a change you do doesn't impact those values (by testing each change x amount of times).

Using developer tools to investigate performance

Using developer tools is perfect for finding in deep information about JavaScript or CSS performance.

Chrome

Chrome has a long history of strong developer tools. You should checkout the performance tab where you can investigate JavaScript and CSS performance. The best way to get it going is looking at perf audits made by Google Engineers. Checkout Paul Irish investigating CNET, Time and Wikipedia or look at Sam Saccone Profiling Paint Perf or Identifying the JavaScript slowdown.

Firefox

Firefox is in the making of releasing Firefox 57 (Quantum) in November this year. When that's happened I'm gonna look for good examples on how to use Firefox devtools, but you can use https://developer.mozilla.org/en-US/docs/Tools as a start for now.

Edge

TODO (do we have a Edge user who can help us here?).

Collecting Visual Metrics like SpeedIndex and First Visual Change

One problem with the metrics that you collect from the browser are that they are browser centric instead of user centric.

To get user centric metrics, we need to record a video of the screen and analyze the result. To do that we use FFMPEG, ImageMagick and couple of Python libraries. The easiest way to get that all to work is to use a ready made Docker container containing all the software you need.

Setup with Docker

The Docker container comes ready made with Firefox and Chrome.

  1. Install Docker
  2. Download the container to your local machine.
    docker pull sitespeedio/browsertime
    
  3. Run the container against your localhost (on Mac OS X the 192.168.65.1 is the way the container access localhost on your machine).
    docker run sitespeedio/browsertime http://192.168.65.1:8080
    
Setting connectivity

Running tests on you localhost will be super fast and will not be the same experience as the user. To simulate that you need to slow down your connection. It's hard to do that inside of Docker, since it's depending on the host you run on, there's simpler to change the connectivity outside of Docker.

On Mac OS X you can do that with pfctl and on Linux you can use tc. If you want help to simulate slow networks you can use throttle.

Install

To get Throttle up and running you need latest LTS release of NodeJS and then install it:

npm install throttle -g
Linux (tc based)

If you test on your localhost on Linux, you need to specify that to Throttle. You can add delay/latency with the rtt-switch. To add 100 ms latency on all traffic on localhost run:

throttle --localhost -rtt 100

To remove the latency you stop Throttle by:

throttle --stop
Mac OS X

On Mac you can specify RTT and up/download speed on all network interfaces (and not only on localhost):

throttle --up 330 --down 780 --rtt 200

And to stop it you:

throttle --stop

The HAR file

When you run your test, you will a HAR file. To view it you can use http://www.softwareishard.com/har/viewer/ or https://micmro.github.io/PerfCascade/

Comparing HAR files

If you want to compare two difference HAR files you can use https://compare.sitespeed.io