Analytics/Data Lake/Edits/Mediawiki page history
This page describes the data set that stores the page history of WMF's wikis. It lives in Analytic's Hadoop cluster and is accessible via the Hive/Beeline external table
wmf.mediawiki_page_history. For more detail of the purpose of this data set, please read Analytics/Data Lake/Page and user history reconstruction. Also visit Analytics/Data access if you don't know how to access this data set.
|enwiki, dewiki, eswiktionary, etc.
|Id of the page, as in the page table.
|Generated Id for deleted pages without real Id.
|Creation timestamp of the page.
|Timestamp of the page\'s first revision (can be before page_creation in restore/merge cases).
|Historical page title, with spaces replaced by underscores.
|Page title as of today, with spaces replaced by underscores.
|Whether the historical namespace is categorized as content
|Namespace as of today.
|Whether the current namespace is categorized as content
|In revision/page events: whether the page is currently a redirect
|Whether the page is rebuilt from a delete event
|Timestamp from where this state applies (inclusive).
|Timestamp to where this state applies (exclusive).
|Event that caused this state (create, move, delete or restore).
|ID from the user that caused this state.
|Name of the user that caused this state
|Whether the user that caused this state was anonymous
|If non-NULL, some fields have been inferred from an inconsistency in the source data.
|ID of the logging table row that caused this state
|Comment of the logging table row that caused this state
|Parameters of the logging table row that caused this state, parsed as a map
|Versioning information to keep multiple datasets (YYYY-MM for regular labs imports)
snapshot field: It is a Hive partitions. It explicitly maps to snapshot folders in HDFS. Since the full data is present in every snapshot up to the snapshot date, you should always specify a snapshot partition predicate in the
where clause of your queries.
Changes and known problems
|Schema changes: Addition of
|Schema changes (no breaking change, only new fields): Addition of
Change in how delete/restore are handled: restore was supposed to always create a new page_id, when it actually doesn't - It either restores a page that was deleted if no page is present with the given title, or do nothing if a page already exist with the given title (restore-into --> merge revisions from a previously deleted page with the given title into an existing page).
|For pairs of fields that give current and historical versions of a value, rename the fields so that
_historical is appended to the historical field rather than
_latest to the current one.
|The dataset contains data for simplewiki and enwiki until september 2016. Still we need to productionize the automatic updates to that table and import all the wikis.
snapshot partition, allowing to keep multiple versions of the page history. Data starts to flow regularly (every month) from labs.