WebPageTest is a web performance tool that uses real browsers to access web pages and collect timing metrics. The killer feature of WebPageTest is the metric called SpeedIndex – a measure of how fast the above-the-fold content is displayed. The Wikimedia Performance Team runs a private instance of WebPageTest at http://wpt.wmftest.org on AWS, and you can view the metrics we collect at https://grafana.wikimedia.org/dashboard/db/webpagetest.
Synthetic testing on the other hand tries to minimize the different factors that can impact the metrics to let us pinpoint the correlation between code changes and metrics impact. Synthetic testing tries to run from the same location, same latency, same browser and measuring the same way. Using WebPageTest as our synthetic tool also has another advantage: SpeedIndex - the best way today to measure when the above the fold content is ready for the user. Synthetic testing's downside is that the metrics aren't from real users.
We use the NavigationTiming extension to add our own script to collect RUM metrics, we run a private instance of WebPageTest to collect synthetic testing and we run Browsertime/WebPageReplay to collect synthetic testing under isolated premisses.
The current setup looks like this:
Login to the servers
Login to the WebPageTest server:
ssh -i webpagetest.pem firstname.lastname@example.org
Login to the WebPageTest job runner:
Login to the WebPageTest agent:
ssh -i WebPageTestAgent.pem email@example.com
We use our WebPageTest instance to continuously monitoring a couple of URLs. You can find the URLs we test here. If you want to add more URLs to monitor, you can do that to our Gerrit repo. If you want to spontaneously test a URL you can do that on the public WebPageTest instance.
Find results using Grafana
You can find each run by looking in Grafana.
First make sure that you have turned on the annotations for "Show each test".
When that is turned on, you will see vertical green lines that represents each time WebPageTest collected metrics. Use your mouse to hover of one of the annotations and you will see links and screenshots from that run. Click on the result link to get to the result. Click on WebPageTest to get to the WebPageTest instance with the result.
Find results using search
We automatically run WebPageTest continuously to collect metrics and send them to Graphite. If you want to look at a specific test, go to http://wpt.wmftest.org/testlog/30/ and choose Show tests from all users. You will then look at all the test runs for the last 30 days. You can change the time span by changing the View and choose Update list.
Caution: Choose what to see on the result page
By default WebPageTest will pick the median run of pageLoadTime. That's not optimal because we want to focus on SpeedIndex or start render time. By adding parameters to the start result page, you can choose what run and metric that will picked up as the median run. Choose which metric to use and if you want the median or the fastest run:
WebPageTest and AWS
WebPageTest consists of two separate entities: a server and agent(s). On AWS there are ready made AMI:s (prepared images) for the two, so it is an easy to click and deploy.
WebPageTest can run headless or not. Headless in this context meaning no GUI available to start a test, you then need to use the API to submit tests.
Setup the server
It can be hard finding the right AMI, for a server in us-west we use AMI id ami-d7bde6e7
- Find the right AMI (ami-d7bde6e7) under Images/AMI and pick it (make sure to choose Public images).
- Choose Launch and use type t2.micro. Make sure to choose Next: Configure Instance Details
- Go the the Advanced Details section and add the configuration for the server. Make sure you change all the secret placeholders to the real values and choose Next: Choose storage
- Nothing you need to do here, choose Next: Tag Instance
- Add the tag Name with the value: WebPageTest Server and then Next: Configure Security Group
- Change the SSH access to only be our IP range.
- Add access for HTTP by choosing Add Rule - use the dropdown and choose HTTP and keep the rest of the values default.
- Choose Review and launch and then Launch. You will be asked to choose an existing key pair or use an existing. Create a new pair (name it webpagetest) and download it. You will need the keys to be able to SSH to the server so make sure to download them.
- Attach the new server the Elastic Path IP: NETWORK & SECURITY/Elastic IPs and choose the IP and Associate Address (we use ip 18.104.22.168).
- Use the tag WebPageTest Server in the instance field and choose Associate.
- You should now be able to access http://wpt.wmftest.org and see the "headless" start page.
Login to the server
ssh -i webpagetest.pem firstname.lastname@example.org
These are the configuration details that you use in the Advanced Details section.
email@example.com ec2_key=SECRET ec2_secret=SECRET ; the key used when starting tests api_key=SECRET ; no GUI for submitting tests, but we can check the results headless=1 ; Define maximums runs per URL maxruns=51 ; Quality of images, lets define this to something good iq=80 ; save png full-resolution screen shots pngss=1 ; automatically update the agent when a new version is available agentUpdate=http://cdn.webpagetest.org/ ; needed for autoscaling EC2.default=us-east-1 ; keep an instance up and running EC2.us-east-1.min=1 EC2.us-east-1.max=2 ; how long to keep tests locally before sending them to S3 archive_days=0 ; archiving to s3 (using the s3 protocol, not necessarily just s3) archive_s3_server=s3.amazonaws.com archive_s3_key=SECRET archive_s3_secret=SECRET archive_s3_bucket=wpt-wikimedia
WebPageTest can automatically store the test results on S3 and that is perfect for us so we can drop the server instance whenever we want.
To setup S3 (these are the instructions to do it the first time):
- Log into the AWS console and choose S3
- Choose Create a bucket
- Add a Bucket name and name it wpt-wikimedia (the bucket name needs to correspond to the property archive_s3_bucket when you configure the server).
- Add the Region. We use Oregon and that matches the configuration property archive_s3_server key.
- Choose Create and we have created the bucket.
- Next step is to setup the properties on the bucket, meaning giving access for HTTP traffic and the server to upload the test results.
- Choose your bucket (webpagetest) and choose Properties/Permissions.
- Choose Add more permissions and add Authenticated Users as Grantee and give it Upload/Delete permissions.
- Then we need to configure how long time we want to store the data.
- Choose Lifecycle and Add rule.
- Apply the rule for the whole bucket
- Choose Permanently Delete and 370 days.
- Choose Review and add a Rule Name: Permanently remove tests after 370 days
- Choose Create and Activate Rule
We are now finished setting up S3.
WebPageTest is stateless and stores everything on file. To be able to find old tests, WebPageTest uses a log file. The log file is not backed up to S3, so to be able to find old tests if the server is dropped, we need to store the logs on a separate disk.
- Choose Elastic Block Store / Volumes
- Choose Create Volume
- Choose a Size in GiB (the lowest 30GB will do fine)
- Choose Availability Zone. Use the same as the server
- Leave everything else as the default and choose Create.
- Choose the radio button for the newly created volume and the Tag label.
- Add a new Tag with the key Name and the value WebPageTest logs and choose Save.
- Make sure the volume is selected with the radio button and choose Action/Attach Volume.
- Choose WebPageTest server as the instance and use the default Device and choose Attach.
- The volume is now attached to our server, the next step is to login to the server and make sure that the logs are stored on the device.
- Use the pem-file for the server and login: ssh -i NAME.pem ubuntu@SERVER_IP (change name of the pem file to your pem file and the SERVER_IP to the real IP and follow these instructions: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
- Follow the instructions and mount the device to /data
- Now your volume is mounted, the next step is to change WebPageTest log dir to a symbolic link to a directory that exists on the volume. If you haven't done any tests, the directory should be empty except for a .htaccess file.
- Make your new directory on the mounted device: sudo mkdir /data/logs
- Move the access file: sudo mv /var/www/webpagetest/www/logs/.htaccess /data/logs
- Remove the old one: sudo rm -fR /var/www/webpagetest/www/logs/
- Make the symbolic: sudo ln -s /data/logs /var/www/webpagetest/www/logs
- Make sure we have the right owner for the directory: sudo chown -h www-data:www-data /data/logs
Depending on the AMI image, it could be that we are missing connectivity profiles: 3GFast, 3GSlow and 2G. If they are missing, you should add them in /var/www/webpagetest/www/settings/connectivity.ini
[3GFast] label="Mobile 3G - Fast (1.6 Mbps/768 Kbps 150ms RTT)" bwIn=1600000 bwOut=768000 latency=150 plr=0 timeout=120 [3GSlow] label="Mobile 3G - Slow (780 Kbps/330 Kbps 200ms RTT)" bwIn=780000 bwOut=330000 latency=200 plr=0 timeout=240 [2G] label="Mobile 2G (280 Kbps/256 Kbps 800ms RTT)" bwIn=280000 bwOut=256000 latency=800 plr=0 timeout=300
The username and password for the master AWS account is recorded in
iron:/srv/passwords/aws-webpagetest. Please avoid using this account directly. Ask an existing maintainer to create an IAM user for you instead. The IAM users sign-in link for this account is https://wikimedia.signin.aws.amazon.com/console .
If something isn't working for you on the WebPageTest server instance you can find the logs here (yep they are on different locations)
/var/www/webpagetest/www/cli/archive.log /var/www/webpagetest/www/logs/ /var/www/webpagetest/www/ec2/log /var/www/webpagetest/www/log /var/log/nginx/error.log
Restart the server
If for some reason you want to restart the server (normally if you manually changed settings in /var/www/webpagetest/www/settings/settings.ini) restart nginx:
sudo service nginx restart
Archive old tests
Old test data will automatically be sent to S3. But we also need to remove old tests, easiest way to do it is to run the archive page. We do it in the crontab. Edit the crontab (as the ubuntu user): crontab -e
SHELL=/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin 0 * * * * curl -sS http://wpt.wmftest.org/cli/archive.php >> /tmp/cron.txt
We then run archiving every our.
We use Amazon cloud watch to keep track of the disk space of the WebPageTest server. You need to install a couple of libraries to get it up and running, follow the instructions. Then add one line to your crontab to start sending the metrics to Amazon.
AWS_CREDENTIAL_FILE=~/.aws/awscreds.txt */5 * * * * ~/aws-scripts-mon/mon-put-instance-data.pl --disk-space-used --disk-space-avail --disk-path=/ --from-cron
Then setup an alarm for the disk space. The current alarm warns (sends an email to the web perf list) when we only have 2 gb disk free.
Install the wrapper on the server
You only need to do this if you don't want to run the wrapper on Jenkins. First we install node & git to be able to get and run our wrapper:
sudo apt-get update && sudo apt-get install -y git nodejs npm && sudo ln -s /usr/bin/nodejs /usr/local/bin/node
Install the latest version of the wrapper
npm install wpt-reporter -g
Add jobs to run automatically
To schedule jobs we use Jenkins.
Setup the agents
We have an agent up and running on us-east-1 on EC2 and it is called us-east. That's the one that we will use for now and if you need to setup your own instance, you need to do like this:
- In the AWS console, choose to Launch instance and choose Community AMIs. There you can find the prepared AMI. You can find all prepared AMIs here (you need a specific per location). Make sure to choose the Linux AMIm
- Choose the instance size c5.xlarge.
- Then you need to make sure that your instance talk to the WebPageTest server, you do that by adding the configuration in the Advanced Details text field. The SECRET key is available on your WebPageTest server. wpt_server=wpt.wmftest.org wpt_loc=us-east wpt_url=http://wpt.wmftest.org/work/ wpt_key=SECRET
Then you can start your agent.
The next step is to configure the WebPageTest server so that it knows about the agent. You do that in /var/www/webpagetest/www/settings/locations.ini. That file is parsed with the ec2_locations.ini file and the result is the configured agents.
[locations] 1=Hosted_Linux default=Hosted_Linux [Hosted_Linux] 1=us-east label="Linux US east 1" default=us-east group=Desktop [us-east] browser=Chrome,Chrome Beta,Chrome Canary,Firefox,Firefox Nightly,Opera,Opera Beta,Opera Developer label="Linux US east"
You can read more about how to configure the locations.ini file at https://github.com/WPO-Foundation/webpagetest/blob/master/www/settings/locations.ini.sample
When you started your agent, and changed the locations.ini file, restart nginx on the WebPageTest server:
sudo service nginx restart
And then verify that you can see your instance at http://wpt.wmftest.org/getLocations.php?f=html
Connect to an agent
You can ssh to the agent with the WebPageTestAgent.pem file. You can find the IP of the agent on AWS.
Timeout and agents not responding
We have seen that a couple of times one of the agents just stop working. You can see that by that all tests timeout and if you go to http://wpt.wmftest.org/ and check the latest finished results, the report will say that the agent couldn't be contacted by the server. To fix that, you need to login to the AWS console and go to EC2 management and make sure you are on "US East" region, choose the agent (it is named WebPagetest Agent), and choose Instance state -> Restart.
Add a new URL to test
All configuration files exists in our synthetic monitoring tests repo. Clone the repo and go into the tests folder:
git clone ssh://USERNAME@gerrit.wikimedia.org:29418/performance/synthetic-monitoring-tests.git cd synthetic-monitoring-tests/tests
All test files lives in that directory. WebPageTest tests exists in two directories:
- desktop - the tests that test desktop URLs. All these tests runs on one machine.
- emulatedMobile - the test URLs that gets tested for emulated mobile. All these tests runs on one machine.
The directory structure looks like this. Each file contains the URLs that is tested for that wiki (emulated mobile only tests on enwiki):
. └── webpagetest └── webpagetest ├── desktop │ └── urls │ ├── webpagetest.beta.txt │ ├── webpagetest.enwiki.txt │ ├── webpagetest.ruwiki.txt │ └── webpagetest.wikidata.txt └── emulatedMobile └── urls └── webpagetestEmulatedMobile.txt
Let have a look at webpagetest.enwiki.txt, it contains of three URLs:
https://en.wikipedia.org/wiki/Facebook https://en.wikipedia.org/wiki/Barack_Obama https://en.wikipedia.org/wiki/Sweden https://en.wikipedia.org/wiki/Metalloid https://www.wikipedia.org/ https://en.wikipedia.org/speed-tests/Banksy.enwiki.872156204/ Banksy
If you want to add a new URL to be tested on the English Wikipedia, you open webpagetest.enwiki.txt, add the URL on a new line and commit the result. When the commit passed through the Gerrit review, that URL will be picked up on the next iteration automatically by the test agent.
If you want to test changes before and after it's super important to test it many times to get correct values, use WPTBulkTest for that. Make sure to setup a new agent for your bulk test!
At the moment our test instance is busy running our continuously performance tests that we graph on https://grafana.wikimedia.org/dashboard/db/webpagetest. We run one test agent to minimize the costs. If you want to use wpt.wmftest.org to run your own one shot tests, I (firstname.lastname@example.org) can help you with that. You will need the key for the instance and choose which location you wanna use and then I can help you verify that the location is setup with the correct instance type.
We run automatic tests every hour (you can find the tests here. We test mainly test the English Wikipedia: 3 desktop URLs using Chrome, the same URLs using Firefox and the mobile version on emulated mobile. You can see how we graph the metrics (and alert on regression): https://grafana.wikimedia.org/dashboard/db/webpagetest-alerts
Our mainly focus is testing on empty browser cache but we also run test with multiple page views (first hit one and then another) and as authenticated users. The metrics are too unstable at the moment to add alerts but we hope we can do that in the future.