Performance/Synthetic testing/OnDemandTesting

On demand performance testing

We have setup for running on demand performance testing The big win with this setup is that we can easily re-use the hardware we use for performance monitoring for on demand tests. The performance monitoring tests use the hardware and continuously runs the test. With the on demand GUI/API you can add one off tests that will be the next job to run in the hardware.

At the moment we are testing the setup running desktop/emulates mobile tests. The plan is to have the setup also run Android tests.

The setup looks like this.

The on demand synthetic test setup.

Servers

We have three servers (we can add more test runner servers):

• The queue queue.webperformancetest.eqiad1.wikimedia.cloud that runs KeyDB and handles the job queue.
• The database db.webperformancetest.eqiad1.wikimedia.cloud that runs PostgreSQL and a webserver front end for adding tests through the GUI.

One machine at Hetzner (138.201.135.103) that runs the actual tests. At the moment this server kicks of the jobs known as the direct tests (testing direct against Wikipedia servers) against the on demand testing framework.

The queue

The queue run KeyDB (an Open Source version of Redis). It stores jobs in memory.

When you login to the server, you will get information on how restart and how to change configuration.

The configuration for KeyDB exists in keydb.conf. The configuration file holds the password, set a max memory limit (to make sure the server do not run out of memory) and configuration to store the tests on disk using an append only file (aof) to make sure we can restart KeyDB without loosing any tests in the queue.

The KeyDB is setup as a service. You can restart the service using: sudo systemctl restart keydb-queue

The Database

The database store meta data about all the tests. The database uses PostgreSQL.

There's a user: database that have access to everything. You can restart the database with:

sudo systemctl restart database.service

The GUI

The GUI run on the database server with the database user.

You can restart the GUI using sudo systemctl restart sitespeedserver.service

The test runner

At the moment there's one test runner/test server that runs the tests. The same server runs the tests for the direct tests for monitoring. If you pass on a test through the GUI, that test will be the first in line in the queue.