We run an instance of DataHub which acts as a centralized data catalog, intended to facilitate the following:
- Discovery by potential users of the various data stores operated by WMF.
- Documentation of the data structures, formats, access rights, and other associated details.
- Governance of these data stores, including details of retention, sanitization, recording changes over time.
Currently our DataHub should be considered at an MVP stage, so functionality and content may change rapidly as the service develops.
The URL for the web interface for DataHub is: https://datahub.wikimedia.org
This requires a Wikimedia developer account and access is currently limited to members of the
nda LDAP groups.
When logging in, enter your shell login name and your wikitech password in order to gain access.
Please note that the username should be entered in lower case. Please see T309382 for more information on case-sensitivity in usernames and rights assignment..
If you are in any doubt as to your shell login name you can go to Special:Preferences as shown in the following screenshot.
n.b. We intend to use your Developer Account username in due course, but currently there is a bug preventing this from working.
The DataHub Metadata Service is: https://datahub-gms.discovery.wmnet:30443
This service is not public facing and it is only available from our private networks. Currently we have not enabled authentication on this interface, although it is planned.
The DataHub instance is composed of several components, each built from the same codebase
- a metadata server (or GMS)
- a frontend web application
- an mce consumer (metadata change event)
- an mae consumer (metadata audit event)
All of these components are stateless and currently run on the Wikikube Kubernetes clusters.
Their containers are built using the Deployment pipeline and the configuration for this is in the wmf branch of our fork of the datahub repository:
Backend Data Tiers
The stateful components of the system are:
- a MariaDB database on the analytics-meta database
- an Opensearch cluster running on three VMs named
- an instance of Karapace, which acts as a schema registry, running on
- serveral Kafka topics running on the kafka-jumbo cluster
Our Opensearch cluster fulfils two roles of:
- a search index
- a graph database
The design document for the DataHub service is here (restricted to WMF staff)
We had previously carried out a Data Catalog Application Evaluation and subsequently the decision was taken to use DataHub and to implement an MVP deployment.
We have several key sources of metadata.
Currently ingestion can be performed by any machine on our private networks, including the stats servers.
We are moving to automated and regularly scheduled metadata ingestion using Airflow. Please check back soon for updated documentation on this topic.
Manual Ingestion Example
The following procedure should help to get started with manual ingestion.
- Select a stats server for your use.
- Activate a stacked anaconda environment.
- Configure the HTTP proxy servers in your shell
- Install the necessary python modules
# it's very important to install the same version CLI as server that's running, # otherwise ingestion will not work pip install 'acryl-datahub[datahub]'~=0.8.32.0 datahub version datahub init server: https://datahub-gms.discovery.wmnet:30443
Until this bug is fixed, we will also need to use:
Then create a recipe file, install more plugins if required.
datahub ingest -c recipe.yaml
Some examples of recipes, including Hive, Kafka, and Druid are avaiable on this ticket