Data Platform/Data Lake/Events

Data Platform

Discover data

• Explore datasets in DataHub
• Data Lake
  • Traffic data
  • Edits data
  • Content data
  • Events data
• Analytics Query Service (AQS)

Access, query, and analyze data

• Get access to internal data
• Analytics tools
  • Jupyter notebooks
  • Superset
  • Spark
  • Presto
• Quickstart notebook
• Internal API requests

Transform and publish data

• Get help or file a request
• Plan data lifecyle
• Build tables and datasets
  • Dataset creation process
  • Data modeling guidelines
  • Airflow developer guide
  • Hive
  • Iceberg
  • Druid
• Share data and dashboards
  • Data publication guidelines
  • Turnilo
  • Superset
  • Web publication guide
  • Dashiki
• Manage published data
  • Data incident management
  • Data issue reporting
  • Data retention guidelines
  • Event data retention
  • Event sanitization
  • Dataset archiving and deletion

Collect data

• Test Kitchen
• Instrumentation tutorial
• Event Platform

---

Data Platform infrastructure and operations

• Systems overview
• Data pipelines
• Search
  • Using search for new features
  • Search Platform
  • Wikidata Query Service (WDQS)
• Operations and team processes
  • Ops week
  • Team pages on Wikitech
  • Team and project pages on MediaWiki.org

[]

Data from Event Platform data stream is available in the Data Lake in the event and event_sanitized databases.

The event database stores the past 90 days of raw (unsanitized) events for all streams. The event_sanitized database stores events sanitized to protect privacy only for streams where sanitization has been manually configured in the sanitization allowlist.

Schemas for these data streams are stored in one of the two schema repositories:

• schemas/event/primary
• schemas/event/secondary

The the old EventLogging system used schemas stored on Meta-Wiki. Any schemas still being used were migrated to the secondary schema repository, but the pages on Meta-Wiki may be useful for historical information.