Event Platform/Instrumentation How To

From Wikitech
Jump to navigation Jump to search

Wikimedia's Event Platform supports both production 'tier 1' events, as well as analytics 'tier 2' events. To support both, the backend systems that handle these events are more reliable and scalable, but also more complex than legacy EventLogging. This page will provide an instrumentation event stream lifecycle example for developing, deploying, evolving, and decommissioning MediaWiki client side JS instrumentation event streams from the mw:Extension:WikimediaEvents via mw:Extension:EventLogging.

Differences from EventLogging the legacy backend

The EventLogging extension was originally built as an all in one system to capture MediaWiki analytics events. It managed schemas, client side event submission, server side event validation and server side event ingestion (into e.g. MySQL). The Event Platform program was conceived to unify event collection for production and analytics events. EventLogging's tier 2 and analytics focus and breadth was not suitable to support this unification. Many of the features of WMF's Event Platform are the same as the legacy EventLogging system, but are more modular and scalable. From an instrumentation only perspective, it may not be clear why things have to be different, but there are good engineering reasons for all of these changes.

The EventLogging extension has been repurposed as an MediaWiki instrumentation event producer library only. On wiki schemas and backend validation are no longer supported by EventLogging.


EventLogging legacy Event Platform
Schema repositories EventLogging schemas were stored as centralized wiki pages on metawiki, and all environments (development, beta, production, etc.) had to use this same schema repository. Event Platform schema are in decentralized git repositories. (Analytics instrumentation schemas are in the schemas/event/secondary repository. Schema repositories are also readable at https://schema.wikimedia.org/#!/ )
Streams, not schemas EventLogging schemas were single use. Each schema corresponded to only one instrumentation, and eventually only one downstream SQL table. Event Platform schemas are like data types for a dataset. A realtime event data set is called an 'event stream' (or just 'stream' for shorthand). Each stream must specify its schema, and a schema may be used by multiple streams.
Schema versions EventLogging schema versions were wiki page revisions. Each event specified its schema name and revision. Event Platform schemas are semantically versioned, and each event declares its schema and version in a $schema URI.
Schema compatibility Each EventLogging schema revision could change the schema in any way, which lead to backwards incompatible changes. Event Platform schemas versions must be backwards compatible; i.e. only adding new optional fields is allowed.
Stream config None. Changes to the way events were emitted (like sampling rate) required a code deployment. Streams are configured in mediawiki-config and can be modified via a Backport window deployment.


Event Streams and Schemas

Both event schemas and streams must be declared before they can be used by EventLogging. Doing so will require three changes:

  • To create or modify a schema, you will create and edit a current.yaml JSONSchema file in the schemas/event/secondary repository. You can read more in depth about how Event Platform schemas work at Event_Platform/Schemas. Please also read Event_Platform/Schemas/Guidelines before creating or modifying scheams.
  • To declare an event stream, you will edit the mediawiki-config repository and add a stream config entry to $wgEventStreams (in wmf-config/InitialiseSettings.php and/or wmf-config/InitialiseSettings-labs.php).
  • To tell EventLogging that is should look up stream config for a stream (and that is is allowed to produce that stream), you will add an entry to $wgEventLoggingStreamNames

Instrumentation Event Stream Lifecycle Example

Event Platform's instrumention development lifecycle is still a work in progress, so please be patient as we work to improve this. Feedback and ideas are very welcome!

Assuming you'll be using the WikimediaEvents extension to produce your events, you'll need to make changes to 3 git repositories: schemas/events/secondary, WikimediaEvents, and finally for deployment (in beta and production) mediawiki-config.

This lifecycle example will demonstrate creating a new stream to log whenever a user hovers over an interwiki link. We'll create a new event stream called 'analytics.interwiki_link_hover' that conforms to a new 'analytics/link_hover' schema.

Development

Setup

In MediaWiki Vagrant

Enabling the wikimediaevents role will also include the eventlogging role for you, and set up other Event Platform backend components on MediaWiki Vagrant including EventGate and Kafka.

$ vagrant roles enable wikimediaevents --provision
$ vagrant git-update

This will clone WikimediaEvents into mediawiki/extensions/WikimediaEvents and the schemas/event/secondary repository at srv/schemas/event/secondary (and also install its npm dependencies for schema materialization and tests)

Mediawiki Vagrant's EventLogging and EventGate setup will allow events of any schema into any stream. For now, you will not have to think about stream config in your development environment.

In your local dev environment with eventgate-devserver

If you aren't using Mediawiki-Vagrant, or you'd rather have more manual control over your development environment, EventLogging comes with an 'eventgate-devserver' that will accept events and write them to a local file. Clone mediawiki/extensions/EventLogging and run

$ npm install --skip-optional
$ npm run eventgate-devserver

This should download EventGate and other dependencies and run the eventgate-devserver accepting events at http://localhost:8192 and writing them to ./events.json. See the devserver/README.md for more info.

Creating a new schema

EventLogging instrumentation schemas will be added to the schemas/event/secondary repository in the jsonschema/analytics namespace. We want to create a new schema that represents a link hover event. Because this type of event can be modeled pretty generically, we are going to create a generic link hover event schema, not one that is specific to interwiki links. This will allow the schema to possibly be reused by other types of link hover events.

Create a new file in at in schemas/event/secondary at jsonschema/analytics/link_hover/current.yaml:

$ cd srv/schemas/event/secondary
$ mkdir -p jsonschema/analytics/link_hover

Create jsonschema/analytics/link_hover/current.yaml with this content:

title: analytics/link_hover
description: Represents an html link mouseover hover event
$id: /analytics/link_hover/1.0.0
$schema: https://json-schema.org/draft-07/schema#
type: object
allOf:
  - $ref: /fragment/analytics/common/1.0.0#
  # event data fields.
  - properties:
      link_href:
        type: string
        description: href attribute of http anchor link
      link_title:
        type: string
        description: title attribute of http anchor link

examples:
  - {$schema":{$ref: "#/$id"},"meta":{"dt":"2020-04-02T19:11:20.942Z" ,"stream":"analytics.interwiki_link_hover"},"event_dt": "2020-04-02T19:11:20.942Z", "link_href": "mw:Extension:EventLogging", "link_title": "mw:Extension:EventLogging"}

NOTE: Event_Platform/Schemas/Guidelines has rules and conventions for schemas.

Now git add and git commit this new file. jsonschema-tools will automatically dereference and materialize this file for you.

$ git add jsonschema/analytics/link_hover/current.yaml

$ git commit -m 'Add new analytics/link_hover schema'

[2020-05-20 18:47:33.865 +0000]: Looking for modified current.yaml schema files in ./jsonschema/
[2020-05-20 18:47:33.946 +0000]: Materializing ./jsonschema/analytics/link_hover/current.yaml...
[2020-05-20 18:47:33.957 +0000]: Dereferencing schema with $id /analytics/link_hover/1.0.0 using schema base URIs ./jsonschema/,https://schema.wikimedia.org/repositories/primary/jsonschema/
[2020-05-20 18:47:34.005 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.0.0.yaml.
[2020-05-20 18:47:34.006 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.0.0.json.
[2020-05-20 18:47:34.007 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.yaml -> 1.0.0.yaml.
[2020-05-20 18:47:34.008 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.json -> 1.0.0.json.
[2020-05-20 18:47:34.008 +0000]: Created extensionless symlink ./jsonschema/analytics/link_hover/1.0.0 -> 1.0.0.yaml.
[2020-05-20 18:47:34.009 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest -> 1.0.0.yaml.
[2020-05-20 18:47:34.011 +0000]: New schema files have been materialized. Adding them to git: ./jsonschema/analytics/link_hover/1.0.0.yaml,./jsonschema/analytics/link_hover/latest.yaml,./jsonschema/analytics/link_hover/1.0.0,./jsonschema/analytics/link_hover/latest,./jsonschema/analytics/link_hover/1.0.0.json,./jsonschema/analytics/link_hover/latest.json
[analytics_link_hover bd4496b] Add new analytics/link_hover schema
 7 files changed, 248 insertions(+)
 create mode 120000 jsonschema/analytics/link_hover/1.0.0
 create mode 100644 jsonschema/analytics/link_hover/1.0.0.json
 create mode 100644 jsonschema/analytics/link_hover/1.0.0.yaml
 create mode 100644 jsonschema/analytics/link_hover/current.yaml
 create mode 120000 jsonschema/analytics/link_hover/latest
 create mode 120000 jsonschema/analytics/link_hover/latest.json
 create mode 120000 jsonschema/analytics/link_hover/latest.yaml

As you can see, you now have many more files committed to git than just current.yaml. Event Platform will use the statically versioned schema files to validate your events.

If you want to materialize a current.yaml schema file without committing it to git, you run the command to materialize it manually:

$ ./node_modules/.bin/jsonschema-tools materialize jsonschema/analytics/link_hover/current.yaml

You can read more about how and why we materialize schema versions over at Event_Platform/Schemas.

Writing instrumentation code

Assuming you already have a ResourceLoader package module in WikimediaEvents, you need to add code that calls the mw.eventLog.submit function. We want to fire events whenever an interwiki link is hovered over.

// 'a.extiw' will match anchors that have a extiw class.  extiw is used for interwiki links.
$( '#content' ).on( 'mouseover', 'a.extiw', function ( jqEvent ) {
	var link = jqEvent.target;
	var linkHoverEvent = {
		// $schema is required and must be set to the exact value of $id that you set in your schema.
		$schema: '/analytics/link_hover/1.0.0',
		link_href: link.href,
		link_title: link.title,
	};
	var streamName = 'analytics.interwiki_link_hover';
	mw.eventLog.submit( streamName, linkHoverEvent );
} );

Now when you over over a link, mw.eventLog.submit will be called and an event will be sent to the 'analytics.interwiki_link_hover' stream.

In summary:

  • Make sure your event data includes $schema which should match $id and is set to the path (starting with /) and (extensionless) version. This tells EventGate which schema and specifically which version of that schema the instrumentation conforms to.
  • mw.eventLog.submit needs: (1) the stream name (this must match what will be configured in production in wgEventStreams stream config, and (2) the event data must include $schema.

Deployment

Once your schema and instrumentation code have been reviewed and merged, you are ready for deployment. You will make some changes to the mediawiki-config repository to configure your new stream, as well as register it for use by the EventLogging extension.

Clone mediawiki-config and edit wmf-config/InitialiseSettings.php. NOTE: You can configure these same settings for beta only by editing wmf-config/InitialiseSettings-labs.php instead. InitialiseSettings.php will be used in both beta and production if no corresponding values are found in InitialiseSettings-labs.php.

Stream Configuration

First declare your stream in the wgEventStreams config variable.

'wgEventStreams' => [
	'default' => [
		// ...
		[
			'stream' => 'analytics.interwiki_link_hover',
			'schema_title' => 'analytics/link_hover',
			'canary_events_enabled' => true,
			'destination_event_service' => 'eventgate-analytics-external',
		],
	],
],

This is the minimal stream config required. This config is used to ensure that only events that have a schema with a title that matches the schema_title value here are allowed in the stream. (Note that this is NOT the same as the schema's $id field. The $id field is a versioned URI. Each event data's $schema URI field will be used to lookup the schema at that schema URI.)

Future work will add support for other stream configs, like adjusting sampling rate without having to deploy code.

Register your stream for use by EventLogging

Now list your stream in wgEventLoggingStreamNames so that EventLogging will get the config for your stream and be able to produce these events.

'wgEventLoggingStreamNames' => [
	'default' => [
		// ...
		'analytics.interwiki_link_hover',
	],
],

If you've made these changes in InitialiseSettings-labs.php, you can find a reviewer to just merge your change and the config will be automatically synced to the beta cluster. If your schema change and instrumentation code change are also merged, you'll be able to send these events in beta.

If you've made these changes in InitialiseSettings.php, you'll need to schedule a Backport window deployment to sync out your config change to the production cluster. See Deployments and Backport windows for instructions.

Per-wiki configurations

The examples above (and many streams currently in production) are deployed to 'default' (all wikis), but what's important is that they're always at least deployed to metawiki – because EventGate uses the stream configuration from meta.wikimedia.org for checking which streams are active and which schema titles they're associated with.

Suppose you have deployed an instrument to production and that instrument produces events to 'analytics.interwiki_link_hover', which is not yet configured. You want to "activate" this instrumentation on a smaller wiki (e.g. cawiki) first to gauge volume of events.

You'll need to:

  1. declare the stream in wgEventStreams for all wikis ('default')
  2. declare the stream with overrides for specific wikis, e.g. 'cawiki'
  3. Only register the stream with EventLogging for wikis that should actually produce the stream
'wgEventStreams' => [
	'default' => [
		// ...
		[
			'stream' => 'analytics.interwiki_link_hover',
			'schema_title' => 'analytics/link_hover',
			'destination_event_service' => 'eventgate-analytics-external',
		],
	],
	'cawiki' => [
		// ...
		[
			'stream' => 'analytics.interwiki_link_hover',
			'schema_title' => 'analytics/link_hover',
			'destination_event_service' => 'eventgate-analytics-external',
			'sampling' => [
				'rate' => 0.5,
			],
		],
	],
],

and

'wgEventLoggingStreamNames' => [
	'cawiki' => [
		'analytics.interwiki_link_hover'
	],
],

The stream is registered on all wikis (importantly on metawiki), but overrides are applied for the sampling rate on cawiki. EventLogging is configured to only register the stream for use on cawiki.

(Note: if you don't need any stream config overrides, you can omit the 'cawiki' entry in wgEventStreams.)

Viewing and querying events

In beta

TODO: set up a GUI consuming from EventStreams in beta that helps here.

Until that happens (hopefully Q2 2020!) you can view events in kafka.

sudo apt-get install kafkacat # if needed
kafkacat -C -b deployment-kafka-jumbo-2.deployment-prep.eqiad1.wikimedia.cloud -t eqiad.<your_stream_name>

In production

Events in production eventually make their way into the Analytics/Data_Lake where they are ingested into a Hive table in the event Hive database. From there they are queryable using Hive, Spark, or Presto, and also available for dashboarding in Superset via Presto.

The Hive table name will be a normalized version of the stream name. Our example stream's Hive table will be event.analytics_interwiki_link_hover.

Events are also available realtime by consuming them directly from Kafka. The stream name is prefixed with the source datacenter for the Kafka topic. WMF has two main datacenters: 'eqiad' and 'codfw'. To consume all events from Kafka for your stream, you should consume from both of these topics. Our example's Kafka topics will be 'eqiad.analytics.interwiki_link_hover and 'codfw.analytics.interwiki_link_hover'.

Viewing schema validation errors

Events that do not validate with their schema will not be produced. For events that are produced via EventLogging, validation error events will be produced into the stream eventgate-analytics-external.error.validation and will be ingested into Hive in the event.eventgate_analytics_external_error_validation table. These validation error events are also ingested into logstash and can be viewed in Kibana using this saved search.

Evolving

Let's say our 'analytics.interwiki_link_hover' stream is operating fine in production, and now we want to also log the link's text value. We'll need to add a new link_text field to the 'analytics/link_hover' schema.

Edit jsonschema/analytics/link_hover/current.yaml again and add the field and bump the version number in the $id. The new content of current.yaml should look like:

title: analytics/link_hover
description: Represents an html link mouseover hover event
$id: /analytics/link_hover/1.1.0
$schema: https://json-schema.org/draft-07/schema#
type: object
allOf:
  - $ref: /fragment/analytics/common/1.0.0#
  # event data fields.
  - properties:
      link_href:
        type: string
        description: href attribute of http anchor link
      link_title:
        type: string
        description: title attribute of http anchor link
      link_text:
        type: string
        description: text value of http anchor link
examples:
  - {$schema":{$ref: "#/$id"},"meta":{"dt":"2020-04-02T19:11:20.942Z" ,"stream":"analytics.interwiki_link_hover"},"event_dt": "2020-04-02T19:11:20.942Z", "link_href": "mw:Extension:EventLogging", "link_title": "mw:Extension:EventLogging", "link_text": "EventLogging extension"}

Now, add and commit current.yaml again and jsonschema-tools will materialize new 1.1.0 version files.

$ git add jsonschema/analytics/link_hover/current.yaml
$ git commit -m 'analytics/link_hover - add link_text field and bump to version 1.1.0'
[2020-05-20 19:52:39.162 +0000]: Looking for modified current.yaml schema files in ./jsonschema/
[2020-05-20 19:52:39.229 +0000]: Materializing ./jsonschema/analytics/link_hover/current.yaml...
[2020-05-20 19:52:39.235 +0000]: Dereferencing schema with $id /analytics/link_hover/1.1.0 using schema base URIs ./jsonschema/,https://schema.wikimedia.org/repositories/primary/jsonschema/
[2020-05-20 19:52:39.264 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.1.0.yaml.
[2020-05-20 19:52:39.271 +0000]: Materialized schema at ./jsonschema/analytics/link_hover/1.1.0.json.
[2020-05-20 19:52:39.275 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.yaml -> 1.1.0.yaml.
[2020-05-20 19:52:39.275 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest.json -> 1.1.0.json.
[2020-05-20 19:52:39.276 +0000]: Created extensionless symlink ./jsonschema/analytics/link_hover/1.1.0 -> 1.1.0.yaml.
[2020-05-20 19:52:39.278 +0000]: Created latest symlink ./jsonschema/analytics/link_hover/latest -> 1.1.0.yaml.
[2020-05-20 19:52:39.279 +0000]: New schema files have been materialized. Adding them to git: ./jsonschema/analytics/link_hover/1.1.0.yaml,./jsonschema/analytics/link_hover/latest.yaml,./jsonschema/analytics/link_hover/1.1.0,./jsonschema/analytics/link_hover/latest,./jsonschema/analytics/link_hover/1.1.0.json,./jsonschema/analytics/link_hover/latest.json
[analytics_link_hover 3c4f8c7] analytics/link_hover - add link_text field and bump to version 1.1.0
 7 files changed, 227 insertions(+), 4 deletions(-)
 create mode 120000 jsonschema/analytics/link_hover/1.1.0
 create mode 100644 jsonschema/analytics/link_hover/1.1.0.json
 create mode 100644 jsonschema/analytics/link_hover/1.1.0.yaml

New analytics/link_hover/1.1.0 files have been created, and the jsonschema/analytics/link_hover/latest symlinks have been updated to point to the 1.1.0 versions.

Now edit your instrumentation code to produce the event data with the link_text field and the updated versioned $schema URI. Your instrumentation code should now look like this:

$( '#content' ).on( 'mouseover', 'a.extiw', function ( jqEvent ) {
	var link = jqEvent.target;
	var linkHoverEvent = {
		// $schema is required and must be set to the exact value of $id that you set in your schema.
		// We've added link_text, which was added in schema verison 1.1.0, so we need to specify that this
		// event should be validated using schema verison 1.1.0.
		$schema: '/analytics/link_hover/1.1.0',
		link_href: link.href,
		link_title: link.title,
		link_text: link.text,
	}
	var streamName = 'analytics.interwiki_link_hover'
	mw.eventLog.submit( streamName, linkHoverEvent );
} );

Note that only backwards compatible changes are allowed. This means that the only type of change you can do to a schema is add new optional fields. jsonschema-tools will ensure that all schema versions are backwards compatible (as well as ensuring that the schema repository is in good shape). Jenkins will run CI tests when you push your schema change to gerrit, but you can also run the tests manually:

$ npm test
...
  Schema Compatibility in Repository ./jsonschema/
    analytics/link_hover
      Major Version 1
        ✓ 1.1.0 must be compatible with 1.0.0
...

Decommissioning

When deploying a new instrumentation event stream, you should also plan to decommission it one day. A schema should never be deleted, but all of the stream related code and configuration can be removed at anytime to stop producing an event stream. In the rare cases where the presence of the schema itself is problematic, we can delete it but it requires a coordinated effort to blacklist it and prevent alarms along the data pipeline.