Performance/Essay/Synthetic Measurement Experiment (2019)

Background

The current testing we do is mainly focused on testing Wikipedia with a cold browser cache: for each URL we test, we start the browser with a new session. We also have a couple of tests where we log in the user but we have never focused on what's the difference in performance if a user is logged in vs logged out vs a user that has some of the assets for the pages in the cache/local storage.

Goal

The main goal for this experiment it to share some light on the difference user experience if you are logged in vs logged out vs assets in cache. It is good to have one reference point so that we can compare and do the same in the future.

Test setup

We run the test on AWS. We use c5.xlarge (that the same size we use in the other synthetic testing). One server is located in India (Mumbai) and one in Sweden. We run the test in Chrome (v72) and for the mobile URLs we run Chrome in emulated mobil mode for a iPhone 6. The test was done in https://phabricator.wikimedia.org/T215489

Test scenarios

We run the same test scenarios from both places but with different URLs:

We access five URLs after each other, where we close the browser between each URL and start a new browser session with a cleared cached between each URL.
We access the same five URLs after each other, waiting 20 s between we hit each page with the browser. We don't close the browser between each URL so the browser cache can be filled. This way we want to simulate a user reading a couple of pages at Wikipedia.
We login the user and then access all five URLs after each other (keeping the browser open). We wait 20 s between each URL.

Tested URLs

We test English URLs from India since we get a lot of traffic to the English version. From Sweden we test the Swedish Wikipedia. We used a couple of pages: some are more light weight, some are more heavy weight.

India

Desktop

https://en.wikipedia.org/wiki/Main_Page
https://en.wikipedia.org/wiki/List_of_Bollywood_films_of_2018
https://en.wikipedia.org/wiki/Goods_and_Services_Tax_(India)
https://en.wikipedia.org/wiki/List_of_Chief_Ministers_of_Uttar_Pradesh
https://en.wikipedia.org/wiki/India

Mobile

https://en.m.wikipedia.org/wiki/Main_Page
https://en.m.wikipedia.org/wiki/List_of_Bollywood_films_of_2018
https://en.m.wikipedia.org/wiki/Goods_and_Services_Tax_(India)
https://en.m.wikipedia.org/wiki/List_of_Chief_Ministers_of_Uttar_Pradesh
https://en.m.wikipedia.org/wiki/India

Sweden

Desktop

https://sv.wikipedia.org/wiki/Portal:Huvudsida
https://sv.wikipedia.org/wiki/Astrid_Lindgren
https://sv.wikipedia.org/wiki/Sm%C3%A5land
https://sv.wikipedia.org/wiki/Stockholm
https://sv.wikipedia.org/wiki/Huvudstad

Mobile

https://sv.m.wikipedia.org/wiki/Portal:Huvudsida
https://sv.m.wikipedia.org/wiki/Astrid_Lindgren
https://sv.m.wikipedia.org/wiki/Sm%C3%A5land
https://sv.m.wikipedia.org/wiki/Stockholm
https://sv.m.wikipedia.org/wiki/Huvudstad

Metrics

We collect a lot of different metrics:

Visual metrics - metrics that are collected when analyzing a video of the screen. Here we can see when the first/last visual change happens within the viewport, when the largest image is painted.
CPU time spent metrics - we collect where Chrome spend time: parsing Javascript, rendering and painting the screen
Transfer size - how much data each page send of the internet wire.
Amount of requests - how many requests that is used for each different test

Result

Let us go through the results and see if we can find any patterns. Remember that we only tested at two different connectivity types and a limited set of pages.

Slow connection (3g) mobile in India

We test a couple of different URLs, some URLs need to do more requests and some fewer. We start by taking a look at https://en.m.wikipedia.org/wiki/Goods_and_Services_Tax_(India) that got most request cached when the user has accessed another Wikipedia page before. The first visual change looks like this:

Here we can see that users that are not logged in but have visited pages before are the fastest. We can also see that logged in users have a faster first visual change than a user that with a cold cache.

How much differs the data that is downloaded? The page is pretty lightweight and difference looks like this.

And the difference in actual JavaScript and CSS:

Another way of looking at the metrics is visual progress of the viewport. When 100% is painted the user see everything in the viewport. This is what the progress looks like if we compare a total cold cache (green) with some items cached (red).

And then comparing logged users (green) with a total cold cache (red).

And then finally logged users (green) and users with a warm cache (items in the cache).

If we look at another page https://en.m.wikipedia.org/wiki/List_of_Bollywood_films_of_2018 that makes more requests the Visual progress looks like this (mobile is on the right):

For mobile it is the same pattern.

And the progress (green is cold cache and red logged in):

Cable connection desktop in India

The desktop site has an other patterns for first visual change: Warm cache is always fastest and then sometimes logged in is faster and sometimes no cache.

And the visual progress looks like this:

Slow connection (3g) mobile in Sweden

On mobile from Sweden we see the exact same behavior on all pages on mobile as from India. Warm cache is always fastest, then a logged in user (with a warm cache) and then cold cache.

Cable connection desktop in Sweden

On desktop we also see the same thing as with the tests from India. Warm cache always fastest. Logged in/cold cache depends on the page.

Summary

Here are the patterns we can see with the testing we done so far:

A user that is not logged in and have visited other Wikipedia pages before, always have the fastest experience. This is true on both mobile and desktop.
On mobile a logged in user has a better experience than a user that has a cold cache.
On desktop a logged in user and a user with cold cache seems to have roughly the same experience.

Remember that we only tested at two different connectivity types and a limited set of pages. We should try to see what kind of patterns we can find with the users we collect from real users. Also note that we emulated the mobile by using a desktop browser in emulated mode, that is not the same as running the same tests on a real mobile phone.

Raw results

Sweden First Visual Change

Page weight and requests desktop Sweden

Page weight and requests mobile Sweden

Measuring the login step Sweden

India First Visual Change

Page weight and requests desktop India

Page weight and requests mobile India

Measuring the login step India

Scripts

Hitting URLs with a cache:

module.exports = async function (context, commands) {
    await commands.measure.start('https://en.wikipedia.org/wiki/Main_Page');
    await commands.wait.byTime(20000);
    await commands.js.run('document.body.innerHTML = ""; document.body.style.backgroundColor = "white";');
    await commands.measure.start('https://en.wikipedia.org/wiki/List_of_Bollywood_films_of_2018 ');
    await commands.wait.byTime(20000);
    await commands.js.run('document.body.innerHTML = "";document.body.style.backgroundColor = "white";');
    await commands.measure.start('https://en.wikipedia.org/wiki/Goods_and_Services_Tax_(India)');
    await commands.wait.byTime(20000);
    await commands.js.run('document.body.innerHTML = ""; document.body.style.backgroundColor = "white";');
    await commands.measure.start('https://en.wikipedia.org/wiki/List_of_Chief_Ministers_of_Uttar_Pradesh ');
    await commands.wait.byTime(20000);
    await commands.js.run('document.body.innerHTML = ""; document.body.style.backgroundColor = "white";');
    await commands.measure.start('https://en.wikipedia.org/wiki/India');
};

Hitting pages as a logged in user:

module.exports = async function (context, commands) {
    await commands.navigate(
        'https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page '
    );

    try {
        // Add text into an input field, finding the field by id
        await commands.addText.byId('USERNAME', 'wpName1');
        await commands.addText.byId('PASSWORD', 'wpPassword1');

        // Start the measurement and give it the alias login
        // The alias will be used when the metrics is sent to
        // Graphite/InfluxDB
        await commands.js.run('document.body.style.display = "none";');
        await commands.wait.byTime(1000);
        await commands.measure.start('login');

        // Find the submit button and click it and wait for the
        // page complete check to finish on the next loaded URL
        await commands.click.byIdAndWait('wpLoginAttempt');
        // Stop and collect the metrics
        await commands.measure.stop();
        await commands.wait.byTime(20000);
        await commands.js.run('document.body.style.display = "none";');
        await commands.measure.start('https://en.wikipedia.org/wiki/List_of_Bollywood_films_of_2018');
        await commands.wait.byTime(20000);
        await commands.js.run('document.body.style.display = "none";');
        await commands.measure.start('https://en.wikipedia.org/wiki/Goods_and_Services_Tax_(India)');
        await commands.wait.byTime(20000);
        await commands.js.run('document.body.style.display = "none";');
        await commands.measure.start('https://en.wikipedia.org/wiki/List_of_Chief_Ministers_of_Uttar_Pradesh');
        await commands.wait.byTime(20000);
        await commands.js.run('document.body.style.display = "none";');
        return commands.measure.start('https://en.wikipedia.org/wiki/India');
    } catch (e) {
    }
};