Data Platform/Data Lake/Events

Data Platform

Discover data

• Explore datasets in DataHub
• Data Lake
  • Traffic data
  • Edits data
  • Content data
  • Events data
• Analytics Query Service (AQS)

Access, query, and analyze data

• Get access to internal data
• Analytics tools
  • Jupyter notebooks
  • Superset
  • Spark
  • Presto
• Quickstart notebook
• Internal API requests

Transform and publish data

• Get help or file a request
• Plan data lifecyle
• Build tables and datasets
  • Dataset creation process
  • Data modeling guidelines
  • Airflow developer guide
  • Hive
  • Iceberg
  • Druid
• Share data and dashboards
  • Data publication guidelines
  • Turnilo
  • Superset
  • analytics.wikimedia.org
  • Web publication guide
  • Dashiki
• Manage published data
  • Data Incident management
  • Data Issue reporting
  • Data Retention Guidelines
  • Event data retention
  • Event Sanitization
  • Dataset archiving and deletion

Collect data

• Metrics platform
• Instrumentation tutorial
• Event Platform

---

Data Platform infrastructure and operations

• Systems overview
• Data pipelines
• Search
  • Using search for new features
  • Search Platform
  • Wikidata Query Service (WDQS)
• Operations and team processes
  • Ops week
  • Team pages on Wikitech
  • Team and project pages on MediaWiki.org

[]

Accessing Event Data

As of September 2020, you have a choice of three engines that can run SQL queries against the Data Lake: Presto, Hive, and Spark. If you're not sure which to choose, Hive is good to start with. All three engines can be used from the Analytics clients.

• WMF maintains 2 separate git schema repositories:
  • schemas/event/primary
  • schemas/event/secondary

• Schemas and schema status for the old EventLogging system are documented at Research:Schemas

• Event vs event_sanitizied*
  • The eventdatabase stores original (unsanitized) events within a 90 day retention period, while event_sanitized database is an archive of sanitized events, beyond the 90 day retention period.
  • Sanitized event data is processed per WMF’s Privacy Policy and Data Retention Guidelines.