Analytics/Data Lake/Traffic/referrer daily
The table referrer_daily
(available in the wmf
database on Hive) contains pre-aggregated counts of how many Wikipedia pageviews were referred from common search engines on a given day. They split the data by country, language edition, browser family, and OS family. Given that this table contains sensitive geographic content, a privacy threshold of 500 is enforced such that any set of facets (search engine, country, language, OS family, browser family) that did not refer at least 500 pageviews is represented by other
. This retains accurate complete counts of search engine referrals while reducing privacy risks.
Schema
hive (default)> DESCRIBE wmf.referrer_daily; # col_name data_type comment country string Reader country per IP geolocation lang string Wikipedia language -- e.g., en for English browser_family string Browser family from user-agent os_family string OS family from user-agent search_engine string One of ~20 standard search engines (e.g., Google) num_referrals int Number of pageviews from the referral source year int Unpadded year of request month int Unpadded month of request day int Unpadded day of request # Partition Information # col_name data_type comment year int Unpadded year of request month int Unpadded month of request day int Unpadded day of request
Search Engines
As of September 2022, the following search engines are tracked by this dataset: Google, Yahoo, Bing, Yandex, Baidu, DuckDuckGo, Ecosia, Startpage, Naver, Docomo, Qwant, Daum, MyWay, Seznam, AU, Ask, Lilo, Coc Coc, AOL, Rakuten, Brave, Petal, and VK. You can see the regexes that are used for each search engine here. Periodically externally-referred traffic is checked to identify any new search engines that should be captured (example but the above search engines appear to capture the vast vast majority of search-engine-based traffic. Changes to the list can be found in the general changes / issues table for the webrequest table from which this data is derived. Note that Wikimedia does not have any data on search queries that come via voice assistants such Amazon Alexa or Apple Siri and thus they are not part of this dataset.
Availability
Beyond Hive, the data is available in various other places.
Stat machines
Any stat machine with access to Hadoop can access daily TSV dumps of the data at /mnt/hdfs/wmf/data/archive/referrer/daily
.
Dashboard
Given the many, orthogonal facets to this data -- e.g., one person may want to aggregate by country while another might want to aggregate by language -- this data is also made available via a prototype public Turnilo instance. See Dashboard main page for more information.
Privacy
For more details and discussion around the privacy risks of this dataset, see task T270140.
See also
- The code that generates it:
- https://github.com/wikimedia/analytics-refinery/tree/master/oozie/referrer
- https://github.com/wikimedia/analytics-refinery/blob/master/oozie/referrer/daily/referrer.hql
- https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/SearchEngine.java
- Related datasets and dashboards:
- Aggregate search engine traffic: https://discovery.wmflabs.org/external/#traffic_by_engine
- Browser and OS pageview breakdowns: https://analytics.wikimedia.org/dashboards/browsers/
- Clickstream dataset: https://dumps.wikimedia.org/other/clickstream/readme.html