WebPageTest/WPTBulkTester
Replaced by WebPageReplay.
WPT Bulk Tester is a Open Source script to bulk test many URLs using WebPageTest and get the result in a Google spreadsheet. It's created by Andy Davies.
The bulk tester is perfect when we want to test changes in a systematic way and use our own instance to test each URL many many times (WebPageTest.org has a limit of 9 runs per URL).
Start
You can see Andy's script and read his blog post. Do that before you start. If you follow the instructions there, it works. We also have a template that is pre-filled with API keys and server names and example how to test Wikipedia, contact the Performance Team to get help.
Gotchas
There are some things I learned the hard way so read it before you start.
Connectivity
If you want to set the connectivity on your run (and you sure want to do that) the recommended way according to the docs is adding it to the location. Say for example that we run our tests from eu-west-1:Chrome and want to run it using 2G then we change the location to eu-west-1:Chrome.2G and the connectivity will be 2G. Well, in the GUI it's 2G but checking the actual metrics like start rendering tells another story. Running on WebPageTest.org our page on 2G start renders after 12 s, setting it to 2G using the bulk tester we got rendering in 4 seconds so there's a mismatch in the latency and down/up-loading speed.
But you can fix that. Checkout the source code configuration in WebPageTest and take the bwIn, bwOut and latency from there and add it to your run. Then it will work.
Verify
Before you start testing your changes with 30+ runs make sure that it works as expected first. Always test with one run, go to the WebPageTest result page, verify that the URL is the right one, that the screenshots are ok, the video looks good and the metrics seems to be ok compared with how you configured the runs.
Then when you made sure one run created the metrics you wanted, change the number of runs to a large number.
Scripted tests
You can script your tests, that works perfectly fine. The URL field will not have meaning so just skip that. A scripted test can look like this (set a specific header, navigate to that page).
addHeader X-Wikimedia-Debug: 1
navigate https://en.wikipedia.org/speed-tests/pages.html