The AS Report (Autonomous Systems performance report) is a monthly-generated dataset anonymised from sampled visits to Wikipedia and other Wikimedia Foundation sites. The ranking is generated based on normalized RUM navtiming metrics measured in the web browser from sampled page views.
Performance metrics are collected using the Navigation Timing API.
Our CPU microbenchmark (source code) is run for a short time in a separate Worker thread, to avoid disrupting their experience. The benchmark assesses the overall performance of the device at the time of the measurement. This is important, as background memory usage and battery levels can greatly influence the performance of mobile devicess at any given time. In the wild, we have observed wide ranges of CPU scores for the exact same device model.
Using this information, we only keep the client metrics for a range of CPU scores. We then compile medians for that device range for each autonomous system, based on the visitors' IP addresses.
Mobile vs desktop
We separate mobile and desktop experiences, as they have significantly different median page weights. As such, scores aren't comparable between mobile and desktop for a given country. Just as they are not comparable between countries either, due to different CPU benchmark score slices being selected for each. Reports are per country, as Internet services tend to be sold to consumers on a national basis, and networks in the same country face the same infrastructural challenges.
Mobile networks are those with at least one sampled visit having a "cellular" connection type, as reported by the Network Information API. When calculating the scores for mobile, we only consider data from sampled pageviews to the mobile site. Desktop networks are those with at least one sampled visit having a "wifi" connection type. When calculating scores for desktop, we only consider data from desktop pageviews.
We don't want an AS corresponding to an ISP selling femtocell devices widely to be unfairly advantaged or disadvantaged due to differences between the mobile and desktop experience, resulting in lighter or heavier pages on average.
Only AS organizations for which we found sufficient data are retained. Finally, only countries with sufficient data from sampled page views are reported.
We report the medians for 2 core RUM metrics measures by the visitors' browsers: Time to first byte and Page Load Time, collected using the Navigation Timing API.
Time to first byte (TTFB) is how long it takes between the client requesting the page and it receiving the first byte of data from us. Page load time (PLT) is how long it takes to load the whole page, including all images and critical styles/scripts.
TTFB is the metric closest to latency, which is something ASOs might improve by peering with us or tweaking their routes to us. PLT is the metric that correlates the most to the visitors' perception of performance, as shown by research we've conducted. It's what captures the ASOs' quality of service to their customers the best.
We also report the median transferSize as reported by the Navigation Timing API as a sanity check, to ensure that the RUM metrics comparison is fair between ASOs, and that the differences aren't caused by visitors using a particular ASO accessing much smaller or much bigger pages on average.
In order to respect the privacy of our visitors, we only report ASOs for which we aggregate more than 500 unique pageviews to generate scores.
- Puppet class: statistics::performance.
- Current host: stat1007 (controlled by
How it works
asoranking.py script run on the first of every month from an Analytics Team stat machine.
Puppet provisions and schedules this the Python script via Scap from a deployment server. It runs as a monthly cron job under the analytics-privatedata user (a requirement to have access to the Kerberos keytab).
The script reads queries Hive to read the Navigation Timing data that we originally collected from web browsers via EventLogging.
The reports are expected to be generated once a month. Our monitoring alert checks that the latest published dataset should be less than 32 days old. The alert also checks that the report is at least 1KB in size.
When used as follows:
python asoranking.py --publish
This will output the generated ranking for the previous calendar month in the form of a TSV file, published to
As per Analytics/Web publication, these are publicly web-accessible at https://analytics.wikimedia.org/published/datasets/performance/autonomoussystems/.
- Analytics/Web publication
- Performance/Runbook/Webperf-processor services#site
- Introducing: Autonomous Systems report (2019), by Gilles Dubuc, techblog.wikimedia.org.