Performance/WebPageReplay

From Wikitech
Jump to: navigation, search

Background

In the path to have more stable metrics in our synthetic testing we have been trying out mahimahi and WebPageReplay to record and replay Wikipedia. For mahimahi we have used patched version fixed by Gilles over Benedikt Wolters HTTP2 version of https://github.com/worenga/mahimahi-h2o. With WebPageReplay we use the default version. The work has been done in T176361.

We have put mahimahi on ice because it is too much of hack to get HTTP/2 to work at the moment and WebPageReplay works out of the box with HTTP/2.

Replaying vs non replying

Let us compare what the metrics looks like comparing WebPageTest vs WebPageReplay (Chrome).

Compare emulated mobile First Visual Change on Obama
Compare emulated mobile Speed Index on Obama
First Visual Change on Desktop using WPT vs WebPageReplay
Compare Speed Index on Desktop using WPT vs WebPageReplay

WebPageReplay setup

The current version run that collect the data for https://grafana.wikimedia.org/dashboard/db/webpagereplay is a Docker container with this setup:

https://github.com/soulgalore/browsertime-replays/tree/master/webpagereplay and the setup looks like this:

WebPageReplay setup

Running on AWS (instance type c4.large) we get stable metrics. On desktop we can use 30 frames per second for the video and then get a change of 33 ms for first visual change. Speed Index got more variance but still ok (less than 50 points but it depends on the content). For emulated mobile we can run 60 frames per second and get the same first visual change and Speed Index somewhere of lower 50 points. We run the both desktop and mobile on 100 ms latency.

Server setup

Here are the details of our current setup. We run it today on a C4.large on AWS using Ubuntu 16.

First time install

To make it work, we need to install three things:

  1. Install Docker
  2. Install NodeJS/npm (latest LTS)
  3. Install bttostatsv: npm install bttostatsv -g
Access

Access the server (TODO: set up a server name):

ssh -i "webpagereplay.pem" ubuntu@50.19.169.203

Job setup

We run this job as an infinite loop and when we wanna update the script, we remove the control file.

#!/bin/bash
exec > /tmp/browsertime.log 2>&1

CONTROL_FILE=/home/ubuntu/browsertime.run

if [ -f "$CONTROL_FILE" ]
then
  echo "$CONTROL_FILE exist, do you have running tests?"
  exit 1;
else
  touch $CONTROL_FILE
fi

LATENCY=100
CHROME_RUNS=11
FIREFOX_RUNS=11
MOBILE_RUNS=7
FF_RUNS=11
CHROME_FRAMERATE=30
MOBILE_FRAMERATE=60
FIREFOX_FRAMERATE=60

CONTAINER=sitespeedio/browsertime:2.1.1-wpr-alpha

function cleanup() {
  docker system prune --all --volumes -f
  docker pull $CONTAINER
}

function control() {
  if [ -f "$CONTROL_FILE" ]
  then
    echo "$CONTROL_FILE found. Make another run ..."
  else
    echo "$CONTROL_FILE not found - stopping after cleaning up ..."
    cleanup
    echo "Exit"
    exit 0;
  fi
}

while true
do
  declare -a DESKTOP_URLS=(https://en.wikipedia.org/wiki/Barack_Obama https://en.wikipedia.org/wiki/Facebook https://en.wikipedia.org/wiki/Sweden)

  declare -a MOBILE_URLS=(https://en.m.wikipedia.org/wiki/Barack_Obama https://en.m.wikipedia.org/wiki/Facebook https://en.m.wikipedia.org/wiki/Sweden)

  echo "Run Chrome tests"
  FRAMERATE=$CHROME_FRAMERATE
  RUNS=$CHROME_RUNS
  GRAPHITE_PREFIX=browsertime.enwiki.desktop.chrome.anonymous.replay.$LATENCY
  for URL in "${DESKTOP_URLS[@]}"
  do
    docker run --cap-add=NET_ADMIN --shm-size=1g --rm -v "$(pwd)":/browsertime -e REPLAY=true -e LATENCY=$LATENCY $CONTAINER -b chrome -n $RUNS --resultDir result --cacheClearRaw --videoParams.framerate $FRAMERATE $URL
    GRAPHITE_KEY=$(basename $URL)
    bttostatsv result/browsertime.json $GRAPHITE_PREFIX.$GRAPHITE_KEY https://www.wikimedia.org/beacon/statsv
    sleep 3
    sudo rm -fR result
    control
  done

  echo "Run Firefox tests"
  FRAMERATE=$FIREFOX_FRAMERATE
  RUNS=$FIREFOX_RUNS
  GRAPHITE_PREFIX=browsertime.enwiki.desktop.firefox.anonymous.replay.$LATENCY
  for URL in "${DESKTOP_URLS[@]}"
  do
    docker run --cap-add=NET_ADMIN --shm-size=1g --rm -v "$(pwd)":/browsertime -e REPLAY=true -e LATENCY=$LATENCY $CONTAINER --resultDir result -n $RUNS -b firefox --skipHar --cacheClearRaw --videoParams.framerate $FRAMERATE $URL
    GRAPHITE_KEY=$(basename $URL)
    bttostatsv result/browsertime.json $GRAPHITE_PREFIX.$GRAPHITE_KEY https://www.wikimedia.org/beacon/statsv
    sleep 3
    sudo rm -fR result
    control
  done

  echo "Run emulates mobile tests"
  FRAMERATE=$MOBILE_FRAMERATE
  RUNS=$MOBILE_RUNS
  GRAPHITE_PREFIX=browsertime.enwiki.mobile.chrome.anonymous.replay.$LATENCY
  for URL in "${MOBILE_URLS[@]}"
  do
    docker run --cap-add=NET_ADMIN --shm-size=1g --rm -v "$(pwd)":/browsertime -e REPLAY=true -e LATENCY=$LATENCY $CONTAINER --resultDir result -b chrome -n $RUNS --cacheClearRaw --videoParams.framerate $FRAMERATE --viewPort 376x668 $URL
    GRAPHITE_KEY=$(basename $URL)
    bttostatsv result/browsertime.json $GRAPHITE_PREFIX.$GRAPHITE_KEY https://www.wikimedia.org/beacon/statsv
    sleep 3
    sudo rm -fR result
    control
  done

  cleanup
done