Data Engineering/Systems/DataHub

Overview

We run an instance of DataHub which acts as a centralized data catalog, intended to facilitate the following:

Discovery by potential users of the various data stores operated by WMF.
Documentation of the data structures, formats, access rights, and other associated details.
Governance of these data stores, including details of retention, sanitization, recording changes over time.

Currently our DataHub should be considered at an MVP stage, so functionality and content may change rapidly as the service develops.

Accessing DataHub

The URL for the web interface for DataHub is: https://datahub.wikimedia.org

This requires a Wikimedia developer account and access is currently limited to members of the wmf or nda LDAP groups.

When logging in, enter your shell login name and your wikitech password in order to gain access.

Please note that the username should be entered in lower case. Please see T309382 for more information on case-sensitivity in usernames and rights assignment..

If you are in any doubt as to your shell login name you can go to Special:Preferences as shown in the following screenshot.

n.b. We intend to use your Developer Account username in due course, but currently there is a bug preventing this from working.

The DataHub Metadata Service is: https://datahub-gms.discovery.wmnet:30443

This service is not public facing and it is only available from our private networks. Currently we have not enabled authentication on this interface, although it is planned.

Related Pages

Upgrading

Service Overview

DataHub Components

The DataHub instance is composed of several components, each built from the same codebase

a metadata server (or GMS)
a frontend web application
an mce consumer (metadata change event)
an mae consumer (metadata audit event)

All of these components are stateless and currently run on the Wikikube Kubernetes clusters.

Their containers are built using the Deployment pipeline and the configuration for this is in the wmf branch of our fork of the datahub repository:

Backend Data Tiers

The stateful components of the system are:

a MariaDB database on the analytics-meta database
an Opensearch cluster running on three VMs named datahubsearch100[1-3].eqiad.wmnet
an instance of Karapace, which acts as a schema registry, running on karapace1001.eqiad.wmnet
serveral Kafka topics running on the kafka-jumbo cluster

Our Opensearch cluster fulfils two roles of:

a search index
a graph database

The design document for the DataHub service is here (restricted to WMF staff)

We had previously carried out a Data Catalog Application Evaluation and subsequently the decision was taken to use DataHub and to implement an MVP deployment.

Metadata Sources

We have several key sources of metadata.

Ingestion

Currently ingestion can be performed by any machine on our private networks, including the stats servers.

Automated Ingestion

We are moving to automated and regularly scheduled metadata ingestion using Airflow. Please check back soon for updated documentation on this topic.

Manual Ingestion Example

The following procedure should help to get started with manual ingestion.

Select a stats server for your use.
Activate a stacked anaconda environment.
Configure the HTTP proxy servers in your shell
Install the necessary python modules

# it's very important to install the same version CLI as server that's running,
# otherwise ingestion will not work
pip install 'acryl-datahub[datahub]'~=0.8.32.0
datahub version
datahub init
server: https://datahub-gms.discovery.wmnet:30443

Until this bug is fixed, we will also need to use:

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

Then create a recipe file, install more plugins if required. datahub ingest -c recipe.yaml

Some examples of recipes, including Hive, Kafka, and Druid are avaiable on this ticket