WebPageReplay

Wikimedia infrastructure

Data centers

Networking

Global traffic routing

MediaWiki SRE

MediaWiki Engineering

Multimedia

Data Engineering

Search

SRE Data Persistence

SRE Infra Foundations

SRE Observability

Wikimedia Performance

• Performance dictionary
• Web Perf Hero award
• Performance review
• Guides:
  • Frontend best practices
  • Backend best practices
  • more…
• Tools:
  • Arc Lamp
  • AS Report
  • Excimer UI
  • Fresnel
  • performance.wikimedia.org
• Metrics:
  • Metric definitions
  • Real user monitoring
  • User Journey
  • Synthetic testing
    • Run a test
    • Dedicated server setup
    • WebPageReplay
    • Graphite (synthetic instance)

Event Platform

Release Engineering

[]

Background

In the path to have more stable metrics in our synthetic testing we have been trying out mahimahi, mitmproxy and WebPageReplay to record and replay Wikipedia. For mahimahi we have used patched version fixed by Gilles over Benedikt Wolters HTTP2 version of https://github.com/worenga/mahimahi-h2o. With mitmproxy and WebPageReplay we use the default version. The work has been done in T176361.

We have put mahimahi on ice because it is too much of hack to get HTTP/2 to work at the moment and WebPageReplay works out of the box with HTTP/2. mitmproxy worked fine but offered no clear benefit over WebPageReplay.

Replaying vs non replaying

Let us compare what the metrics looks like comparing WebPageTest vs WebPageReplay (Chrome).

Compare emulated mobile First Visual Change on Obama

Compare emulated mobile Speed Index on Obama

First Visual Change on Desktop using WPT vs WebPageReplay

Compare Speed Index on Desktop using WPT vs WebPageReplay

WebPageReplay setup

The current version run that collects the data for https://grafana.wikimedia.org/d/IvAfnmLMk/page-drilldow is a Docker container with this setup:

WebPageReplay setup

We run all tests on a couple of bare metal servers. We have tried running the same code on AWS, WMCS and Google Cloud and in all those cases the metrics stability over time was at least 2 to 4 times worse than AWS and over time the metrics are most stable on bare metal.

Servers

We run tests from four server at the moment:

• hetzner1 (195.201.12.112) - Run Chrome tests for desktop against enwiki.
• hetzner2 (138.201.207.48) - Run Chrome tests for desktop against group 0 and group 1.
• hetzner3 (136.243.175.100) - Run Firefox tests for desktop against group 0, group 1 and enwiki.
• hetzner4 (88.99.26.157) - Run emulated mobile tests for group 0, group 1 and enwiki.

Upgrade to a new version

Checkout WebPageReplay/Runbook#Update to new version.

Alerts

We also run alerts on the metrics we collect from WebPageReplay. Checkout Performance/Guides/WebPageReplay alert.

Maintenance

Debug missing metrics

Checkout WebPageReplay/Runbook#Debug missing metrics

See also

• WebPageReplay/Runbook
• Performance/Guides/WebPageReplay alert