Event Platform/Producer Requirements
In order to ensure that Event Platform based event streams are automatically integratable with all consumers and downstream data systems, producers must ensure that the events satisify specific requirements before producing them. If you are using a supported WMF Event Platform producer service or library, these requirements should be satisified already. However, if you are working in a language or area that does not have a Event Platform client library, the you will be producing events directly to Kafka yourself.
This page describes the requirements that any Event Platform producer should satisfy.
- Events MUST have an event schema in a WMF Schema Repository.
- Events MUST have required event fields set.
- Events MUST have
meta.streamset to the name of the stream they are destined to.
- Event streams MUST be declared in WMF Event Stream Config, with
schema_titleset to the title that matches the event schema's
titlethey will conform to.
A producer will need to interact with and lookup Event Stream Configuration and WMF Schema Repositories. The URI locations from which to look up event stream configuration and event schemas should be configurable. E.g. A user should be able to provide your producer library a local schema repository base path for development.
All generic producer clients and libraries:
- MUST look up the event's schema via its $schema URI field in the configured base schema repository URIs.
- MUST look for any event stream configuration defined for the stream the event will be produced to. The stream the event will be produced to should be in the event's meta.stream field.
- MUST ensure the event is allowed in its destination stream, as specified in the meta.stream field. This can be determined by checking that the stream's configured schema_title matches the event's schema's title.
- MUST ensure that the event has a dt field set specifying its ISO-8601 event time.
- SHOULD set the meta.dt field to indicate the event's ISO-8601 'system ingestion time' by the library.
- MUST ensure the event is valid according to the schema it declares in its $schema field.
- MAY automatically add datacenter prefixes to the stream name to make the destination Kafka topic name.
- MAY produce to the datacenter specific Kafka topic.
Supported Event Platform producer clients
- EventGate - An http producer proxy. At WMF, this takes events to produce over HTTP and produces them to Kafka.
- wikimedia-event-utilities Java library - A java library for working with Event Platform events and streams. Has APIs for preparing events for production to Kafka.
- eventutilities-python - A python library mostly (as of 2023-05) for building PyFlink + Event Platform applications
As new clients and libraries are implemented, please link to them here.