Performance/Mobile Device Lab

The performance device lab is on hold as of July 2023 until we find a new provider. The old documentation below describes our previous provider, kept to have a jump start for when we push to find a new provider.

Wikimedia infrastructure

Data centers

Networking

Global traffic routing

MediaWiki SRE

MediaWiki Engineering

Multimedia

Data Engineering

Search

SRE Data Persistence

SRE Infra Foundations

SRE Observability

Wikimedia Performance

• Performance dictionary
• Web Perf Hero award
• Performance review
• Guides:
  • Frontend best practices
  • Backend best practices
  • more…
• Tools:
  • Arc Lamp
  • AS Report
  • Excimer UI
  • Fresnel
  • performance.wikimedia.org
• Metrics:
  • Metric definitions
  • Real user monitoring
  • User Journey
• Synthetic testing
  • Run a test
  • Dedicated server setup
  • WebPageReplay
  • On demand testing
  • Graphite (synthetic instance)
• Runbooks
  • WebPageReplay
  • On demand testing
  • Graphite
  • Result storage

Event Platform

Release Engineering

[]

The Mobile Device Lab is provided by the Performance Team for executing tests on real mobile devices, in a variety of network environments. We use this to measure how changes impact mobile users.

Summary

In the path to get more realistic performance metric run tests on real mobile devices. That makes it easier to find performance regressions. We use BitBar as the provider of our devices. All tasks are reported under the Performance Device Lab tag in Phabricator.

Performance testing on mobile phones

Running tests on mobile phones we want a stable environment that do not change, so we can see performance regressions. To make that happen we use:

• A stable network: We throttle the connection to look like a 3g or 4g connection.By limiting the upload/download speed and adding delay, we make the phone get requests in a more realistic scenario. By making sure the network is the same all the time, the network will not affect our metrics.
• Low phone temperature: We measure the battery temperature as a proxy for CPU temperature. Android phones change behavior when the CPU gets warm and we want to avoid that. Some of our phones are rooted to try to make sure the phone has the same performance characteristics. We use the same settings as the Mozilla performance team to setup the rooted phones. We measure the temperature before and after we start a test.

Setup

We use five mobile phones, a server and two wifi:s setup with throttled connection to simulate 4g traffic. The wifi connections is provided by two Raspberry Pis 4 running humble.

Setup showing the mobile performance device lab.

The workflow: The jobs are started on the server that runs sitespeed.io that drives the phones using WebDriver. The configuration and URLs to tests exists in a public Git repo. The tests runs on the phones, access Wikipedia and we record a video of the screen and analyze the result to get visual metrics. The metrics is sent to our Graphite instance and the test results (screenshots, HTML result pages) are stored on S3. We also run one tests using WebPageReplay where we record and replay Wikipedia locally on the server to try to get as stable metrics as possible between runs.

Performance device lab at BitBar.

Setup the phones

BitBar is handling the setup of phones. If we need to change anything we need to contact them.

We have five phones running at the Performance Device Lab with the following setup.

Id	Type	Internet connection	Extras	OS version	Usage
ZY322DJBW9	Motorola Moto G5 #1	Simulated 4g (wifi wikimedia93)	Root	8.1.0	Used for testing
ZY322GXQN8	Motorola Moto G5 #2	Simulated 4g (wifi wikimedia94)	Root	8.1.0	Direct Wikipedia
ZY322H9XKL	Motorola Moto G5 #3	Simulated 4g (wifi wikimedia93)	Root	8.1.0	Direct Wikipedia
R58NC31FK3Y	Samsung A51	WebPageReplay	Root	11	Using WebPageReplay
	Samsung A51	WebPageReplay	Root	11	Using WebPageReplay

Using rooted phones makes it possible to stabilise CPU and GPU performance by configuring governors.

The the Moto G5 #2 and #3 are setup as one phone as one group. We then add tests to that group and the first available phone start the tests. Both Samsung phones are also in the same group, meaning they act as one and as soon as one phone is available, it will take on jobs.

BitBar setup

At BitBar our test use a generic setup with a start bash file (run-tests.sh), a secrets.json file and a Slack bash file (slack.sh) that are uploaded in the BitBar GUI in a zip file. The bash file is called when a test is started and looks like this:

# We unpack tests.zip that contains the secrets.json configuration file
unzip tests.zip

# Clone the git repo where we have all the code
git clone https://github.com/wikimedia/performance-mobile-synthetic-monitoring-tests.git
cd performance-mobile-synthetic-monitoring-tests

# There's a hack on BitBar where you can pass on parameters
touch sitespeed.log
../slack.sh sitespeed.log "$1" "$2" "$3" &
./start.sh "$1" "$2" "$3" 2>&1 |  tee sitespeed.log

And the secrets.json file contains configuration to sitespeed.io to be able to send metrics to our Graphite instance and send data to S3. All tests then extends that configuration file and we can have those configurations file open to the public in our Git repo. We also have a bash script that just reads the log from the test and report all errors to a Slack channel:

#!/bin/bash
tail -n0 -F "$1" | while read LINE; do
  (echo "$LINE" | grep -A 3 -e "ERROR:") && curl -X POST --silent --data-urlencode \
  "payload={\"text\": \"[bitbar $2] $(echo $LINE | sed "s/\"/’/g")\"}" https://hooks.slack.com/services/OUR_SERVICE;
done

We then have cronjob calling the API using CURL with settings to use our zip file . At the moment the job is fired from gpsi.webperf.eqiad1.wikimedia.cloud using the bitbar user. The job is kicked off by the bitbarcaller.sh script that lives in the repo.

We can pass on max five parameters from the API to our scripts, today we use three of them. They are named: PARAM1, PARAM2, PARAM3, PARAM4 nd PARAM5.

Performance tests

All configuration and setups for our tests lives in Gerrit in performance/mobile-synthetic-monitoring-tests. To add or change tests you clone the repo and send in your change for review.

Add a test

All configuration files exists in our synthetic monitoring tests repo. Clone the repo and go into the tests folder:

git clone ssh://USERNAME@gerrit.wikimedia.org:29418/performance/mobile-synthetic-monitoring-tests.git

cd mobile-synthetic-monitoring-tests/tests

Change configuration

If you need to change the secrets.json file, the easiest way to do that it to do like this:

1. Upload a new file to BitBar and check the id for that file.
2. Update the bitbarcaller.sh script to use the new file (use the id).
3. Done.

Alert and error reporting

We have a two alerts setup to verify that the network and tests work as they should. The alerts verify that we get metrics into Graphite for the phones that uses the 4g and WebPageReplay tests. The alerts exist in: https://grafana.wikimedia.org/d/frWAt6PMz/synthetic-tool-alerts

We also report error log message to the #synthetic-tests-error-reporting Slack channel (you need to be invited by the performance team). Here we can see smaller kind of failures like one of the runs didn't work, instability in Firefox/Chrome. We use those errors to know if there's something that needs to be tuned or reported upstream.

Troubleshooting

All phones trouble shooting is handled by BitBar. If one phone is "offline", "online dirty" or do not have an internet connection we contact BitBar using the BitBar/Wikimedia Slack channel.

Dashboards

The data is reported under the android key in Graphite. Make sure type is android in the dashboard.

Outstanding issues

There's a problem at BitBar if one phone loses the internet connection. With their current setup that phone still take on jobs. Tracked in T334710.

One thing to keep track of is that maxAutoRetriesCount needs to be set to 0 (not null as it is default) so that BitBar can debug if something fails, else the failing test is just deleted.