Performance/WebPageReplay/Alerts

From Wikitech
Jump to navigation Jump to search

We collect metrics using Browsertime and WebPageReplay and use Grafana alerts and Icinga to get automated alerts when we have a front end performance regression.

Mobile

We test the mobile site using Chrome running in emulated mode.

Setup

At the moment we test three pages for each wiki we test. We have alerts for the following:

enwiki
beta
group 0
group 1

We do 5 runs against WebPageReplay per URL, record a video in 60 fps and take the median metrics. Since we run against a local proxy (WebPageReplay) we set the latency to 100ms to slowdown the rendering.

We collect Visual Metrics (SpeedIndex/First Visual Change) and CPU metrics (time spent in Layout/Scripting/Loading/Painting) that we get from parsing the Chrome trace log.

Alerts

Alerts for First Visual Change and Speed Index looks for a 20 ms or 20 points regression on all three pages and then alert.

Alerts for Layout/Scripting/Loading/Painting looks for a regression in 20 ms on all three pages for one of the metrics.

Alert dashboard

https://grafana.wikimedia.org/dashboard/db/webpagereplay-mobile-alerts

Desktop

We use Chrome and Firefox to collect metrics for desktop. At the moment the Firefox metrics is too unstable to use for alerting, so we focus on Chrome.

Setup

At the moment we test three pages for each wiki we test. We have alerts for the following:

enwiki
beta
group 0
group 1

We do 9 (or 7) runs against WebPageReplay per URL, record a video in 60 fps for Firefox and 30 fps with Chrome and take the median metrics. Since we run against a local proxy (WebPageReplay) we test with 100ms latency to slowdown the rendering.

We collect Visual Metrics (SpeedIndex/First Visual Change) and CPU metrics (time spent in Layout/Scripting/Loading/Painting Chrome only) that we get from parsing the Chrome trace log.

Alerts

Alerts for First Visual Change and Speed Index looks for a 40 ms or 40 points regression on all three pages and then alert.

Alerts for Layout/Scripting/Loading/Painting looks for a regression in 20 ms on all three pages for one of the metrics.

Firefox

The current version of Firefox seems really sensitive to content changes so we have a little higher limit there for regressions. Alerts for First Visual Change and Speed Index looks for a 5% regression on all three pages and then alert. At the moment Firefox don't support getting CPU metrics (tracked in 1250290).

Alert dashboard

https://grafana.wikimedia.org/dashboard/db/webpagereplay-desktop-alerts