From Wikitech
Jump to: navigation, search

WebPageTest is a web performance tool that uses real browsers to access web pages and collect timing metrics. The killer feature of WebPageTest is the metric called SpeedIndex – a measure of how fast the above-the-fold content is displayed. WMF's Web Performance Team runs a private instance of WebPageTest at http://wpt.wmftest.org on AWS, and you can view the metrics we collect at https://grafana.wikimedia.org/dashboard/db/webpagetest.


There are two standard ways of collecting timing metrics for web pages today: RUM (Real User Measurement) and synthetic testing. Collecting data using RUM means adding some JavaScript to pages. The script runs in users' browsers, collects some metrics (eg. Navigation Timing and User Timing), and beacons back the data to our servers. The upside of RUM metrics is that they are from real users, the downside is that it varies quite much and it's hard to isolate changes and correlate things. Furthermore, the metrics depend on latency and connection type. We are also missing good ways of measuring things that really matters like first paint (only in Chrome & IE11) and when content is finished loading within the current view port.

Synthetic testing on the other hand tries to minimize the different factors that can impact the metrics to let us pinpoint the correlation between code changes and metrics impact. Synthetic testing tries to run from the same location, same latency, same browser and measuring the same way. Using WebPageTest as our synthetic tool also has another advantage: SpeedIndex - the best way today to measure when the above the fold content is ready for the user. Synthetic testing's downside is that the metrics aren't from real users.

We use the NavigationTiming extension to add our own script to collect RUM metrics and we run a private instance of WebPageTest to collect synthetic testing.


The current setup looks like this:

WebPageTest setup


You can see the agents we use here: http://wpt.wmftest.org/getLocations.php

Install the CLI

We have use a small wrapper script that talks to the WebPageTest API using the https://github.com/marcelduran/webpagetest-api. It collects the metrics and reports the metrics as csv/json or to graphite/statsv.

You can clone the project https://github.com/wikimedia/wpt-reporter or install the wrapper with

npm install wpt-reporter -g

Running a test using the CLI

If you have installed the project, you can simply run it like this:

wpt-reporter --webPageTestKey WPT_API_KEY --reporter json --webPageTestHost wpt.wmftest.org --location us-west-1:Chrome https://en.wikipedia.org/wiki/Facebook

It will send a request to our WebPageTest instance and start a test. Make sure to change the WPT_API_KEY value to your secret key.


If you want to see what you can configure and the default values, run:

wpt-reporter --help

All parameters that you send to the tool are passed to the WPT API, there are a lot of things you can configure and here is the full list.

Send to statsv

Sending statistics to statsv is disabled by default. Turn it on like this (by setting the reporter to statsv and setting a valid endpoint):

wpt-reporter --webPageTestKey WPT_API_KEY --endpoint YOUR_STATSV_ENDPOINT --reporter statsv --webPageTestHost wpt.wmftest.org --location us-west-1:Chrome https://en.wikipedia.org/wiki/Facebook
Store as CSV

If you want to verify that your changes are faster than your current version, you can use WebPageTest and test the changes and store the result as a CSV file. You can choose where CSV data will be stored with the file option. If the file doesn't exist, it will add one line with all the column names of the metrics. If the file exists, it will just append the new metrics on a new line.

wpt-reporter --webPageTestKey WPT_API_KEY --webPageTestHost wpt.wmftest.org --reporter csv --file myresult.csv --location us-west-1:Chrome https://en.wikipedia.org/wiki/Facebook
Mobile phone

If you want to test mobile pages, you can fake the user agent and set viewport and size using our own WebPageTest instance. If you want to test with a real mobile phone, you can use the Motorola G phones that are available on WebPageTest.org.

Set view port and screen size

You can fake a mobile user agent (only when using) Chrome. By using emulateMobile you will use a Chrome mobile user agent, 640x960 screen, 2x scaling and fixed viewport.

wpt-reporter --webPageTestKey WPT_API_KEY --webPageTestHost wpt.wmftest.org --emulateMobile true --reporter json --location us-west-1:Chrome https://en.m.wikipedia.org/wiki/Denali%E2%80%93Mount_McKinley_naming_dispute
Set your User Agent string

If you want to set your own user agent, use --userAgent (it will only work if you use Chrome).

wpt-reporter --webPageTestKey WPT_API_KEY  --webPageTestHost wpt.wmftest.org --location us-west-1:Chrome --userAgent "Mozilla/5.0(iPhone;U;CPUiPhoneOS4_0likeMacOSX;en-us)AppleWebKit/532.9(KHTML,likeGecko)Version/4.0.5Mobile/8A293Safari/6531.22.7" --reporter json https://en.m.wikipedia.org/wiki/Denali%E2%80%93Mount_McKinley_naming_dispute
Use a real mobile phone

If you want to use a real mobile phone, you can do that by using the public instance of WebPageTest.org and use the Motorola 5. Make sure that you use the WebPageTest key for the public instance (and not the key for our private instance).

wpt-reporter  --webPageTestKey WPT_ORG_API_KEY --webPageTestHost http://www.webpagetest.org --location "Dulles_MotoG:Motorola G - Chrome" --reporter json https://en.m.wikipedia.org/wiki/Facebook
Test with 2g connectivity

The current version of the WebPageTest API doesn't have a short handle for 2g but you can set the connectivity yourself and simulate 2g. Just make sure to also increase the timeout time, because running multiple tests on 2g takes time.

wpt-reporter --webPageTestKey WPT_API_KEY --webPageTestHost wpt.wmftest.org --bandwidthDown 35000 --bandwidthUp 32000 --latency 1300 --timeout 2400 --connectivity custom --reporter json --emulateMobile true --location us-west-1:Chrome https://en.m.wikipedia.org/wiki/Facebook

If you want to try out your own settings, you can checkout WebPageTest ini file for inspiration.

Test multiple URLs

You can test multiple URLs by choosing the --batch option. Running a batch will fetch all the URLs and the configuration from a file. You supply the path and name of the file and the file needs to have one URL/run on each line. It looks like this:

## Test the Facebook page 15 times
--webPageTestKey <%WMF_WPT_KEY> --webPageTestHost wpt.wmftest.org --runs 15 --median SpeedIndex --reporter json --location us-west-1:Chrome  https://en.wikipedia.org/wiki/Facebook

## And then test Barack 31 and use SpeedIndex as median pick
--webPageTestKey <%WMF_WPT_KEY> --webPageTestHost wpt.wmftest.org --runs 31 --median SpeedIndex --reporter json --location us-west-1:Chrome https://en.wikipedia.org/wiki/Barack_Obama

Look closely at the webPageTestKey value. The value <%WMF_WPT_KEY> will be replaced by the environment variable named WMF_WPT_KEY. Running in node, the value will be replaced with process.env.WMF_WPT_KEY. If the variable isn't found, there will be an error logged. You can create your own variables in the script files and feed the values with environment variables following the same pattern.

Changing and testing a batch file

When we run the tests in Jenkins we run batch scripts located here: https://github.com/wikimedia/performance-WebPageTest/tree/master/scripts/batch

When you want to add a URL to test or change anything, you need to test it locally. You can easily do that by following the example in each batch, however you need to make sure that you run it on another WebPageTest location than us-east-1 because that location is busy running the Jenkins test. You can see all available locations: http://wpt.wmftest.org/getLocations.php. You can safely use us-west-1 when you test.

Here's an example of running a batch locally.

WMF_WPT_KEY=OUR_SECRET_KEY STATSV_ENDPOINT= WPT_RUNS=1 WMF_WPT_LOCATION=us-west-1 bin/index.js --batch scripts/batch/desktop.txt

When you add a new test, make sure to check out the screenshots on WebPageTest to see that it worked as expected (the user is logged in etc). There's also a bash script that will test all batch files, so if you make changes to the current batch scripts, please run it like this before you commit your changes:


When your changes has been approved and Jenkins pull them, make sure that the jobs in Jenkins work fine:



Run on Jenkins

We use Jenkins to continuously run the tests (https://integration.wikimedia.org/ci/job/performance-webpagetest-wmf/ and https://integration.wikimedia.org/ci/job/performance-webpagetest-wpt-org/). Jenkins runs the batch files and to be able to do that, you need to do three things:

  • Git clone the project: https://gerrit.wikimedia.org/r/p/performance/WebPageTest.git and use the master branch refs/heads/master
  • Add the binding for the environment variables using Bindings and share them as Secret Text. You need to setup WMF_WPT_KEY, WPT_ORG_WPT_KEY, WPT_USER & WPT_USER_PASSWORD
  • Run the tests in an execute shell build step:
# These tests runs on our own WebPageTest instance
# Securely set beforehand: WMF_WPT_KEY, WPT_USER, WPT_USER_PASSWORD
export STATSV_ENDPOINT="https://www.wikimedia.org/beacon/statsv"
export WPT_RUNS="5"
export WPT_MOBILE_RUNS="5"
export WMF_WPT_LOCATION="us-east-1"
npm install --production
./bin/index.js --batch ./scripts/batch/mobile.txt
./bin/index.js --batch ./scripts/batch/desktop.txt
./bin/index.js --batch ./scripts/batch/login-mobile.txt
./bin/index.js --batch ./scripts/batch/login-desktop.txt
./bin/index.js --batch ./scripts/batch/second-view-mobile.txt
./bin/index.js --batch ./scripts/batch/second-view-desktop.txt
exit $RESULT
Find results

We automatically run wpt a couple of times a day to collect metrics and send them to statsv. If you want to look at a specific test, go to http://wpt.wmftest.org/testlog/30/ and choose Show tests from all users. You will then look at all the test runs for the last 30 days. You can change the time span by changing the View and choose Update list.

Test as a authenticated user (use scripting)

In the world of WebPageTest you can either test a specific URL or write a script that perform a couple of interactions; like access a page, log in the user and access another page. And you can choose when to start collecting metrics. You can read more about the setup: https://sites.google.com/a/webpagetest.org/docs/using-webpagetest/scripting

Using scripts, it follows the same pattern as running batch scripts, meaning you can have variables in your script that will be replaced with the value of environment variables at run time. Name your variable  <%YOUR_NAME_HERE> and it will be replaced with process.env.YOUR_NAME_HERE.

Using custom metrics

You can collect custom metrics using Javascript that will run in the browser when the page has been loaded. That's nothing we use today (meaning we don't support sending these metrics to statsv) but it's a pretty neat feature and we will have the data/metrics on the WebPageTest result pages.


How to test new features/changes

One idea with using WebPageTest is that we also can test new features, changes and have a good way of measuring metrics before and after a change. If you want to collect metrics for a change, it is important that you run the test many times so that we have a median value that can reflect the change. Many times means at least 31 times :) Always choose an odd number, that way a real run is picked as the median.

Choose to report the result as CSV, then you will have one file with all the URLs you test in one condition, switch the condition and make another CSV and import the two CSV files into a program that handles CSV files. The CSV file will on each row contain the tested URL and the different metrics. This is a good approach:

  1. Create a batch file with the URLs you want to test. Make sure to parameterize things like connectivity, number of runs and so on. One line in your batch file can look like this:
    --webPageTestKey <%WMF_WPT_KEY> --webPageTestHost wpt.wmftest.org --median SpeedIndex --location <%WMF_WPT_LOCATION>:Chrome --label chrome --runs <%WPT_RUNS> --connectivity <%WPT_CONNECTIVITY> --reporter csv --file <%CSV_FILE_NAME%> --timeout <%WPT_TIMEOUT> https://en.wikipedia.org/wiki/Main_Page
  2. Add the rest of the URLs that you want to test in the file by copy/pasting the row and changing the URL at the end.
  3. Setup your environment so you export the environment variables. First time your run it: make sure to only use one run and good connectivity like cable to run through the test fast and see that all URLs and configuration are ok. The timeout time is high by default but if you test with 2g for example, the test will take a long time to finish, so you need to increase the timeout value (default is 20 minutes and you define it in seconds, so default timeout is 1200).
  4. Verify the CSV file. Does it look the way you need it? Good, keep going.
  5. Change the number of runs to the number you want (and the connectivity) and run the script again. This can take time. Keep your terminal open, each and every test will be submitted when the last one is finished.
  6. Ok we have some numbers, now configure the change so that when you access the test URL we have the new feature/change. Make sure to change the name of the CSV file, so we keep two different ones.
  7. Run the test again.
  8. Now you have the numbers! Lets compare them. If you need help to understand them, talk to the performance team and we will help you analyze them.

Caution: Choose what to see on the result page

By default WebPageTest will pick the median run of pageLoadTime. That's not optimal because we want to focus on SpeedIndex or start render time. By adding parameters to the start result page, you can choose what run and metric that will picked up as the median run. Choose which metric to use and if you want the median or the fastest run:





WebPageTest and AWS

WebPageTest consists of two separate entities: a server (that runs on Linux or Windows) and agent(s) (Windows only). On AWS there are ready made AMI:s (prepared images) for the two, so it is an easy to click and deploy. Setting up the server, will automatically spin up agents when requested.

WebPageTest can run headless or not. Headless in this context meaning no GUI available to start a test, you then need to use the API to submit tests.

Setup the server

It can be hard finding the right AMI, for a server in us-west we use AMI id ami-d7bde6e7

  1. Find the right AMI (ami-d7bde6e7) under Images/AMI and pick it (make sure to choose Public images).
  2. Choose Launch and use type t2.micro. Make sure to choose Next: Configure Instance Details
  3. Go the the Advanced Details section and add the configuration for the server. Make sure you change all the secret placeholders to the real values and choose Next: Choose storage
  4. Nothing you need to do here, choose Next: Tag Instance
  5. Add the tag Name with the value: WebPageTest Server and then Next: Configure Security Group
  6. Change the SSH access to only be our IP range.
  7. Add access for HTTP by choosing Add Rule - use the dropdown and choose HTTP and keep the rest of the values default.
  8. Choose Review and launch and then Launch. You will be asked to choose an existing key pair or use an existing. Create a new pair (name it webpagetest) and download it. You will need the keys to be able to SSH to the server so make sure to download them.
  9. Attach the new server the Elastic Path IP: NETWORK & SECURITY/Elastic IPs and choose the IP and Associate Address (we use ip
  10. Use the tag WebPageTest Server in the instance field and choose Associate.
  11. You should now be able to access http://wpt.wmftest.org and see the "headless" start page.
Login to the server

ssh -i webpagetest.pem ubuntu@wpt.wmftest.org


These are the configuration details that you use in the Advanced Details section.


; the key used when starting tests

; no GUI for submitting tests, but we can check the results

; Define maximums runs per URL

; Quality of images, lets define this to something good

; save png full-resolution screen shots

; automatically update the agent when a new version is available

; needed for autoscaling

; keep an instance up and running

; how long to keep tests locally before sending them to S3

; archiving to s3 (using the s3 protocol, not necessarily just s3)
Setup S3

WebPageTest can automatically store the test results on S3 and that is perfect for us so we can drop the server instance whenever we want.

To setup S3 (these are the instructions to do it the first time):

  1. Log into the AWS console and choose S3
  2. Choose Create a bucket
  3. Add a Bucket name and name it wpt-wikimedia (the bucket name needs to correspond to the property archive_s3_bucket when you configure the server).
  4. Add the Region. We use Oregon and that matches the configuration property archive_s3_server key.
  5. Choose Create and we have created the bucket.
  6. Next step is to setup the properties on the bucket, meaning giving access for HTTP traffic and the server to upload the test results.
  7. Choose your bucket (webpagetest) and choose Properties/Permissions.
  8. Choose Add more permissions and add Authenticated Users as Grantee and give it Upload/Delete permissions.
  9. Then we need to configure how long time we want to store the data.
  10. Choose Lifecycle and Add rule.
  11. Apply the rule for the whole bucket
  12. Choose Permanently Delete and 370 days.
  13. Choose Review and add a Rule Name: Permanently remove tests after 370 days
  14. Choose Create and Activate Rule

We are now finished setting up S3.

Setup EBS

WebPageTest is stateless and stores everything on file. To be able to find old tests, WebPageTest uses a log file. The log file is not backed up to S3, so to be able to find old tests if the server is dropped, we need to store the logs on a separate disk.

  1. Choose Elastic Block Store / Volumes
  2. Choose Create Volume
  3. Choose a Size in GiB (the lowest 30GB will do fine)
  4. Choose Availability Zone. Use the same as the server
  5. Leave everything else as the default and choose Create.
  6. Choose the radio button for the newly created volume and the Tag label.
  7. Add a new Tag with the key Name and the value WebPageTest logs and choose Save.
  8. Make sure the volume is selected with the radio button and choose Action/Attach Volume.
  9. Choose WebPageTest server as the instance and use the default Device and choose Attach.
  10. The volume is now attached to our server, the next step is to login to the server and make sure that the logs are stored on the device.
  11. Use the pem-file for the server and login: ssh -i NAME.pem ubuntu@SERVER_IP (change name of the pem file to your pem file and the SERVER_IP to the real IP and follow these instructions: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
  12. Follow the instructions and mount the device to /data
  13. Now your volume is mounted, the next step is to change WebPageTest log dir to a symbolic link to a directory that exists on the volume. If you haven't done any tests, the directory should be empty except for a .htaccess file.
  14. Make your new directory on the mounted device: sudo mkdir /data/logs
  15. Move the access file: sudo mv /var/www/webpagetest/www/logs/.htaccess /data/logs
  16. Remove the old one: sudo rm -fR /var/www/webpagetest/www/logs/
  17. Make the symbolic: sudo ln -s  /data/logs /var/www/webpagetest/www/logs
  18. Make sure we have the right owner for the directory: sudo chown -h www-data:www-data /data/logs
Setup relay server

We use WebPageTest.org to run some test on real devices (iPhone 6, iPadMini2, Moto G) and setup that server as a relayServer. In the WebPageTest world, that means our own instances sends jobs to WebPageTest.org and when it finished, the result is collected and visible on our server. We can then also do searches on our instance and show the result from WebPageTest.org in our search result.

Start with open /var/www/webpagetest/www/settings/ec2_locations.ini and add the following:

// In [locations] add our new instances. If the current [locations] ends on 28, start on 29 etc.

// Then map the instance against the relay configuration


label="Dulles Iphone6"

label="Dulles Ipad Mini"

// Setup the relay server(s)
// Change the secret key to our key at WebPageTest.org
label="WebPageTest.org Dulles"

browser="Motorola G - Chrome, Motorola G - UC Mini, Motorola G - Opera Mini"
label="WebPageTest.org Dulles Moto G"

label="WebPageTest.org Dulles Iphone 6"

label="WebPageTest.org Dulles Ipad Mini2"

Access http://wpt.wmftest.org/getLocations.php to see that the new locations work (they will be at the end of the page).

Note: We don't use the Dulles location because of this error: https://github.com/WPO-Foundation/webpagetest/issues/694

Connectivity profiles

Depending on the AMI image, it could be that we are missing connectivity profiles: 3GFast, 3GSlow and 2G. If they are missing, you should add them in /var/www/webpagetest/www/settings/connectivity.ini

label="Mobile 3G - Fast (1.6 Mbps/768 Kbps 150ms RTT)"

label="Mobile 3G - Slow (780 Kbps/330 Kbps 200ms RTT)"

label="Mobile 2G (280 Kbps/256 Kbps 800ms RTT)"

The username and password for the master AWS account is recorded in iron:/srv/passwords/aws-webpagetest. Please avoid using this account directly. Ask an existing maintainer to create an IAM user for you instead. The IAM users sign-in link for this account is https://wikimedia.signin.aws.amazon.com/console .


If something isn't working for you on the WebPageTest server instance you can find the logs here (yep they are on different locations)

Restart the server

If for some reason you want to restart the server (normally if you manually changed settings in /var/www/webpagetest/www/settings/settings.ini) restart nginx:

sudo service nginx restart
Archive old tests

Old test data will automatically be sent to S3. But we also need to remove old tests, easiest way to do it is to run the archive page. We do it in the crontab. Edit the crontab (as the ubuntu user): crontab -e

0 * * * * curl -sS http://wpt.wmftest.org/cli/archive.php >> /tmp/cron.txt

We then run archiving every our.

Cloud watch

We use Amazon cloud watch to keep track of the disk space of the WebPageTest server. You need to install a couple if libraries to get it up and running, follow the instructions. Then add one line to your crontab to start sending the metrics to Amazon.

*/5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --disk-space-used --disk-space-avail --disk-path=/ --from-cron

Then setup an alarm for the disk space. The current alarm warns (sends an email to the web perf list) when we only have 2 gb disk free.

Install the wrapper on the server

You only need to do this if you don't want to run the wrapper on Jenkins. First we install node & git to be able to get and run our wrapper:

sudo apt-get update && sudo apt-get install -y git nodejs npm && sudo ln -s /usr/bin/nodejs /usr/local/bin/node

Install the latest version of the wrapper

npm install wpt-reporter -g
Add jobs to run automatically

To schedule jobs we use Jenkins.

Setup the agents

You don't need to, they are automatically created by the server.

Connect to an agent

Normally you should never need to connect to an agent, in rare cases it could be good just to debug errors. The agent run on Windows, so log into AWS, download the Remote Desktop file. Go to the server in AWS and change the security group (you should have a group that opens TCP and UDP on port 3389 and open only for your IP). Then connect through your Remote Desktop client and log in as Administrator with the password that you can find https://github.com/WPO-Foundation/webpagetest/blob/master/docs/EC2/Server%20AMI.md.

Change agent Amazon instance type

You do that on the server, by configuring the location and size in /var/www/webpagetest/www/settings/ec2_locations.ini

You find the location/browser in the configuration file and add the line size=AMAZON_IMAGE_SIZE, so for us it will look like this:

label="US East (N. Virginia)"

We run our automatic test on us-east-1. If you change the size, you need to manually kill the instance in the AWS console, and WebPageTest will spin up a new instance with the new size.

Timeout and agents not responding

We have seen that a couple of times one of the agents just stop working. You can see that by that all tests timeout and if you go to http://wpt.wmftest.org/ and check the latest finished results, the report will say that the agent couldn't be contacted by the server. To fix that, you need to login to the AWS console and go to EC2 management and make sure you are on "US East" region, choose the agent (it is named WebPagetest Agent), and choose Instance state -> Terminate. The main WPT server will then automatically spin up a new instance when it's time to run the next test.

Bulk test

If you want to test changes before and after it's super important to test it many times to get correct values, use WPTBulkTest for that.

Using http://wpt.wmftest.org/

At the moment our test instance is busy running our continuously performance tests that we graph on https://grafana.wikimedia.org/dashboard/db/webpagetest. We run one test agent to minimize the costs. If you want to use wpt.wmftest.org to run your own one shot tests, I (phedenskog@wikimedia.org) can help you with that. You will need the key for the instance and choose which location you wanna use and then I can help you verify that the location is setup with the correct instance type.

Setting up a Linux agent

We are on our way to remove the Windows agent and run our tests on Linux (that will make it cheaper and we will hopefully get rid of a lot of problems we've had).

We have an agent up and running on us-east-1 on EC2 and it is called us-east. That's the one that we will use for now and if you need to setup your own instance, you need to do like this:

  1. In the AWS console, choose to Launch instance and choose Community AMIs. There you can find the prepared AMI. For us-east-1 I used ami-1502116f. You can find all prepared AMIs here (you need a specific per location).
  2. Choose the instance size c4.large.
  3. Then you need to make sure that your instance talk to the WebPageTest server, you do that by adding the configuration in the Advanced Details text field. The SECRET key is available on your WebPageTest server. wpt_server=wpt.wmftest.org wpt_loc=us-east wpt_url=http://wpt.wmftest.org/work/ wpt_key=SECRET
    WebPageTest Agent AMI setup..png

Then you can start your agent.

The next step is to configure the WebPageTest server so that it knows about the agent. You do that in /var/www/webpagetest/www/settings/locations.ini. That file is parsed with the ec2_locations.ini file and the result is the configured agents.


label="Linux US east 1"

browser=Chrome,Chrome Beta,Chrome Canary,Firefox,Firefox Nightly,Opera,Opera Beta,Opera Developer
label="Linux US east"

You can read more about how to configure the locations.ini file at https://github.com/WPO-Foundation/webpagetest/blob/master/www/settings/locations.ini.sample

When you started your agent, and changed the locations.ini file, restart nginx on the WebPageTest server:

sudo service nginx restart

And then verify that you can see your instance at http://wpt.wmftest.org/getLocations.php?f=html