Performance/Synthetic testing/Bare metal

Synthetic tests on a physical server

We run our tests synthetic tests on a physical server ("bare metal") to get as stable metrics as possible. You can read about how we evaluated running on a physical server in T203060.

The physical server gives us the following advantage:

1. We don't get the noisy neighbour effect, meaning we get stable metric over time
2. We can adjust the CPU frequency on the machine to match the speed of our users.

At the moment we run our desktop tests with the CPU frequency of our 90 percentile of users in India. For emulated mobile we match the 90/95 percentile. By matching those users, we know that we can pickup regressions that will be visible for them.

Synthetic tests

We run tests using WebPageReplay on our physical server. That is replay proxy that tries to remove the noice of internet, We have one server running those tests.

Setup a bare metal server

This document explains how the bare metal server at Hetzner is setup. Through the Hetzner setup, you can choose base OS. We use Ubuntu 22.04 to be able to run Chrome tests directly without using Docker in the future.

Start by updating the machine:

sudo apt-get update 
sudo apt-get upgrade

Install Docker

Install Docker (we use Docker for the WebPageReplay tests):

sudo apt install apt-transport-https curl gnupg-agent ca-certificates software-properties-common -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
sudo apt install docker-ce docker-ce-cli containerd.io -y

Setup the user

Create a new user that will run the tests and make sure that user can use sudo (needed for using Linux traffic control (tc)

adduser sitespeedio 
usermod -aG sudo sitespeedio
su - sitespeedio
echo "sitespeedio ALL=(ALL:ALL) NOPASSWD:ALL" | sudo tee "/etc/sudoers.d/sitespeedio"

Make sure the new user can run Docker without using sudo:

sudo usermod -aG docker ${USER} 
su - ${USER}

Install dependencies for sitespeed.io

Install dependencies to be able to run sitespeed.io (use NodeJS LTS version):

curl -sL https://deb.nodesource.com/setup_18.x -o nodesource_setup.sh
sudo bash nodesource_setup.sh
sudo apt install -y nodejs

To be able to record a video we need FFPMEG, a couple of Python libs and xvfb. Net-tools is needed to use Linux traffic control

sudo apt-get update -y && sudo apt-get install -y ffmpeg
python -m pip install pyssim OpenCV-Python Numpy
sudo apt-get install -y xvfb
sudo apt-get install -y net-tools

Install sitespeed.io

Make sure you can install using npm without using sudo. Checkout Sindre Sorhus guide and then install sitespeed.io

npm install sitespeed.io --location=global

Install browsers

If you want to run without Docker you need to install the browsers manually. Install Chrome:

wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
sudo apt update
sudo apt install -y google-chrome-stable

To install Firefox without using Snap on Ubuntu 22.04 follow this guide.

Setup unattended-upgrades

At the moment we run the unattended upgrades make sure we keep the machine up to date.

sudo apt-get install unattended-upgrades -y
sudo dpkg-reconfigure -plow unattended-upgrades

Pin the CPU frequency and use performance governor

We want the machine to use the same CPU frequency all the time since that will make our metrics more stable. Make sure you are root and install cpufrequtils

sudo apt-get install cpufrequtils

Then you can check the current setup with cpufreq-info command. Then you which governor that is used for each CPU and the min and max limit. The box we use at the moment has 8 CPUs and minimum 800 MHZ and max 4 GHZ. We set it up to use the performance governor and run at 1 GHz. You need to configure that for each and every CPU.

cpufreq-set -d 1.00Ghz -u 1.00Ghz -g performance -c 0
cpufreq-set -d 1.00Ghz -u 1.00Ghz -g performance -c 1
cpufreq-set -d 1.00Ghz -u 1.00Ghz -g performance -c 2
cpufreq-set -d 1.00Ghz -u 1.00Ghz -g performance -c 3
cpufreq-set -d 1.00Ghz -u 1.00Ghz -g performance -c 4
cpufreq-set -d 1.00Ghz -u 1.00Ghz -g performance -c 5
cpufreq-set -d 1.00Ghz -u 1.00Ghz -g performance -c 6
cpufreq-set -d 1.00Ghz -u 1.00Ghz -g performance -c 7

You can verify that it worked by running cpufreq-infoagain.

The CPU benchmark is the at 200 ms. That matches almost our 90 percentile of desktop users in India. For the emulated mobile tests we slow down the CPU some more using the Chrome built in CPU throttler so the the CPU benchmark is at 420 ms that matches somewhere like 90/95 percentile for mobile users in India.

Set DNS

The machine comes setup with using a local DNS server. You can change that following these instructions and use 1.1.1.1 instead (that has given us more stable metrics).

Open Graphite firewall for the machine

You need to open the firewall on AWS for the new machine. You do that using the AWS GUI but first you need to know your public ip:

curl http://ipinfo.io/ip

Enable the firewall

Enable the firewall by accepting only incoming SSH traffic.

sudo ufw default allow outgoing
sudo ufw default deny incoming
sudo ufw allow ssh
sudo ufw enable