Event Platform/Instrumentation How To

From Wikitech

Wikimedia's Event Platform supports both production "Tier 1" events, as well as analytics "Tier 2" events. To support both, the backend systems that handle these events are more reliable and scalable, but also more complex than legacy EventLogging. This page will provide an instrumentation event stream lifecycle example for developing, deploying, evolving, and decommissioning MediaWiki client side JS instrumentation event streams from the mw:Extension:WikimediaEvents via mw:Extension:EventLogging.

You can learn more about the differences between how Event Platform works and how the legacy EventLogging backend systems work on the EventLogging legacy page.

This lifecycle example will demonstrate creating a new stream to log whenever a user hovers over an interwiki link. We'll create a new event stream called mediawiki.interwiki_link_hover that conforms to a new analytics/link_hover schema.

Creating a new schema

To create or modify a schema, you will create and edit a current.yaml JSONSchema file in the schemas/event/secondary repository. You can read more in depth about how Event Platform schemas work at Event Platform/Schemas. Please also read Event Platform/Schemas/Guidelines before creating or modifying schemas.

EventLogging instrumentation schemas will be added to the schemas/event/secondary repository in the jsonschema/analytics namespace. We want to create a new schema that represents a link hover event. Because this type of event can be modeled pretty generically, we are going to create a generic link hover event schema, not one that is specific to interwiki links. This will allow the schema to possibly be reused by other types of link hover events.

To do this, we create jsonschema/analytics/link_hover/current.yaml with this content: 

title: analytics/link_hover
description: Represents an html link mouseover hover event
$id: /analytics/link_hover/1.0.0
$schema: https://json-schema.org/draft-07/schema#
type: object
allOf:
  - $ref: /fragment/analytics/common/2.0.0#
  # event data fields.
  - properties:
      link_href:
        type: string
        description: href attribute of http anchor link
      link_title:
        type: string
        description: title attribute of http anchor link

examples:
  - {$schema":{$ref: "#/$id"},"meta":{"dt":"2020-04-02T19:11:20.942Z" ,"stream":"mediawiki.interwiki_link_hover"},"dt": "2020-04-02T19:11:20.942Z", "link_href": "mw:Extension:EventLogging", "link_title": "mw:Extension:EventLogging"}

NOTE: Event Platform/Schemas/Guidelines has rules and conventions for schemas.

Run npm run build-new and then git commit this new file. 

$ npm run build-new jsonschema/analytics/link_hover/current.yaml
# ...

$ git add jsonschema/analytics/link_hover/*

$ git commit -m 'Add new analytics/link_hover schema'

As you can see, you now have many more files committed to Git than just current.yaml. Event Platform will use the statically-versioned schema files to validate your events.

You can read more about how and why we materialize schema versions over at Event Platform/Schemas.

Setting up for local testing

In MediaWiki Docker

See MediaWiki Docker EventLogging Configuration recipe.

In MediaWiki Vagrant

Enabling the wikimediaevents role will also include the eventlogging role for you, and set up other Event Platform backend components on MediaWiki Vagrant including EventGate. 

$ vagrant roles enable wikimediaevents --provision
$ vagrant git-update

This will clone WikimediaEvents into mediawiki/extensions/WikimediaEvents and the schemas/event/secondary repository at srv/schemas/event/secondary (and also install its npm dependencies for schema materialization and tests)

MediaWiki Vagrant's EventLogging and EventGate setup will allow events of any schema into any stream. For now, you will not have to think about stream config in your development environment.

Events will be written to /vagrant/logs/eventgate-events.json. eventgate logs, including validation errors, are in /vagrant/logs/eventgate-wikimedia.log.

To verify that eventgate is working properly, you can force a test event to be produced by curl-ing http://localhost:8192/v1/_test/events. You should see a test event logged into eventgate-events.json.

In your local dev environment with eventgate-devserver

If you aren't using Mediawiki-Vagrant, or you'd rather have more manual control over your development environment, EventLogging comes with an 'eventgate-devserver' that will accept events and write them to a local file. Clone mediawiki/extensions/EventLogging and run 

$ cd extensions/EventLogging/devserver
$ npm install --no-optional
$ npm run eventgate-devserver

This should download EventGate and other dependencies and run the eventgate-devserver accepting events at http://localhost:8192 and writing them to ./events.json. See the devserver/README.md for more info.

Writing MediaWiki instrumentation code

Per the instrumentation DACI (and soon to succeed it data collection guidelines [internal draft]) before starting to instrument the data collection, it is important that your plans to collect data are reviewed and approved by Legal, Security, and Safety. Wikimedia Foundation staff can use the L3SC to submit a request to have the plans reviewed and approved. For parties (such as WMDE) working on features and instruments that are deployed on Foundation infrastructure, a process is being developed to allow obtaining the necessary approval. It is recommended to perform this step before starting instrumentation because during the review process you may learn that you cannot collect certain data you were planning to collect, so you will save yourself time by not writing code that you will have to remove. For more information about what your request should contain, refer to this draft guide on measurement plans and instrumentation specifications.

JavaScript

Assuming you already have a ResourceLoader package module in WikimediaEvents, you need to add code that calls the mw.eventLog.submit function. We want to fire events whenever an interwiki link is hovered over. 

// 'a.extiw' will match anchors that have a extiw class.  extiw is used for interwiki links.
$( '#content' ).on( 'mouseover', 'a.extiw', function ( jqEvent ) {
	var link = jqEvent.target;
	var linkHoverEvent = {
		// $schema is required and must be set to the exact value of $id that you set in your schema.
		$schema: '/analytics/link_hover/1.0.0',
		link_href: link.href,
		link_title: link.title,
	};
	var streamName = 'mediawiki.interwiki_link_hover';
	mw.eventLog.submit( streamName, linkHoverEvent );
} );

Now when you hover over a link, mw.eventLog.submit will be called and an event will be sent to the 'mediawiki.interwiki_link_hover' stream.

In summary:

  • Make sure your event data includes $schema which should match $id and is set to the path (starting with /) and (extensionless) version. This tells EventGate which schema and specifically which version of that schema the instrumentation conforms to.
  • mw.eventLog.submit needs: (1) the stream name (this must match what will be configured in production in wgEventStreams stream config, and (2) the event data must include $schema.

PHP

The EventLogging PHP interface is the same as the JavaScript one. If you were building a server side event that will be send with the EventLogging extension, you'd use the EventLogging::submit function. 

$exampleEvent = [
	'$schema' => '/analytics/example_schema/1.0.0'
	'field_a' => 'value_a'
	// ... Other event data fields from /analytilcs/example_schema/1.0.0
];

$streamName = 'mediawiki.example.stream';
EventLogging::submit( $streamName, $exampleEvent );

Deployment

Once your planned data collection activity has been approved by Legal, Security, and Safety teams and your schema and instrumentation code have been reviewed and merged, you are ready for deployment. You will make some changes to the mediawiki-config repository to configure your new stream, as well as register it for use by the EventLogging extension.

Clone mediawiki-config and edit wmf-config/ext-EventLogging.php. NOTE: You can configure these same settings for beta only by editing wmf-config/InitialiseSettings-labs.php instead. ext-EventLogging.php will be used in both beta and production if no corresponding values are found in InitialiseSettings-labs.php.

Configuring the stream

First declare your stream in the wgEventStreams config variable. 

'wgEventStreams' => [
	'default' => [
		// ...
		'mediawiki.interwiki_link_hover' => [
			'schema_title' => 'analytics/link_hover',
			'destination_event_service' => 'eventgate-analytics-external',
		],
	],
],

This is the minimal stream config required. This config is used to ensure that only events that have a schema with a title that matches the schema_title value here are allowed in the stream. (Note that this is NOT the same as the schema's $id field. The $id field is a versioned URI. Each event data's $schema URI field will be used to lookup the schema at that schema URI.)

Future work will add support for other stream configs, like adjusting sampling rate without having to deploy code.

See Event_Platform/Stream_Configuration#Common_Settings_Documentation for documentation on common stream config settings.

Registering the stream with EventLogging

If producing events using the EventLogging extension (most likely) You need to list your stream in wgEventLoggingStreamNames so that EventLogging will get the config for your stream and be able to produce these events. 

'wgEventLoggingStreamNames' => [
	'default' => [
		// ...
		'mediawiki.interwiki_link_hover',
	],
],

If you've made these changes in InitialiseSettings-labs.php, you can find a reviewer to just merge your change and the config will be automatically synced to the beta cluster. If your schema change and instrumentation code change are also merged, you'll be able to send these events in beta.

If you've made these changes in ext-EventLogging.php, you'll need to schedule a Backport window deployment to sync out your config change to the production cluster. See Deployments and Backport windows for instructions.

Configuring data retention

By default, stream data is retained in the Data Lake only for 90 days. To change this, you must edit the sanitization allowlist.

Viewing and querying events

Beta

Event Platform components are set up in the Cloud VPS deployment-prep project (AKA 'beta') similar to how they run in production. The EventLogging extension there is configured to send events to https://intake-analytics.wikimedia.beta.wmflabs.org/v1/events?hasty=true. If you are not using EventLogging (e.g. from a mobile app), you will have to configure your 'beta' installation of the app to also POST events to this URL.

An EventStreams instance in the Cloud VPS deployment-prep project (AKA 'beta') exists that allows you to consume any stream declared in stream config. As this instance is in beta, it only has events that are generated in beta. You can consume this stream via an EventSource/SSE client (see https://stream-beta.wmflabs.org/?doc) or in your browser using the EventStreams GUI at https://stream-beta.wmflabs.org/v2/ui.

Alternatively, you can log into a host in the deployment-prep Cloud VPS project and consume events directly from Kafka using a Kafka client like kafkacat: 

sudo apt-get install kafkacat # if needed
kafkacat -C -b deployment-kafka-jumbo-9.deployment-prep.eqiad1.wikimedia.cloud -t eqiad.<your_stream_name>

Production

Data Lake

Events in production eventually make their way into the Analytics/Data_Lake where they are ingested into a Hive table in the event Hive database. From there they are queryable using Hive, Spark, or Presto, and also available for dashboarding in Superset via Presto.

The Hive table name will be a normalized version of the stream name. Our example stream's Hive table will be event.mediawiki_interwiki_link_hover.

Kafka

Events are also available in real time by consuming them directly from Kafka. The stream name is prefixed with the source datacenter for the Kafka topic. WMF has two main datacenters: 'eqiad' and 'codfw'. To consume all events from Kafka for your stream, you should consume from both of these topics. Our example's Kafka topics will be eqiad.mediawiki.interwiki_link_hover and codfw.mediawiki.interwiki_link_hover.

Legacy EventLogging streams are named in a different way and not split by data center: for example eventlogging_InukaPageView. Note that Kafka topic names are case sensitive, so make sure you use the correct capitalization, as listed in the wgEventStreams configuration variable.

Here's a useful command which fetches and pretty-prints the last 5 messages from a topic:

kafkacat -C -b kafka-jumbo1009.eqiad.wmnet:9092 -t {{topic name}} -o -5 -e -q | jq .

If you want to listen for new events (perhaps when deploying new instrumentation), you can use this instead:

kafkacat -C -b kafka-jumbo1009.eqiad.wmnet:9092 -t {{topic name}} -o end | jq .

EventStreams

Production has an internal EventStreams instance that allows you to consume any stream declared in our stream config. eventstreams-internal.discovery.wmnet is not publicly accessible, so you have to access it through an SSH tunnel: 

# Tunnel local traffic to port 4992 to eventstreams-internal.discovery.wmnet:4992 via bast1003.wikimedia.org
# Replace bast1003.wikimedia.org with your preferred bastion or other accessible production host.
ssh -N -L4992:eventstreams-internal.discovery.wmnet:4992 bast1003.wikimedia.org

Then, in your browser, you can navigate to https://localhost:4992/v2/ui/#/ to use the GUI. Note that you will probably have to add a security exception to do so, since the site uses an "unofficial" HTTPS certificate. You can also consume with your own EventSource/SSE client (see https://localhost:4992/?doc).

Viewing schema validation errors

Events that do not validate with their schema will not be produced. For events that are produced via EventLogging, validation error events will be produced into the stream eventgate-analytics-external.error.validation and will be ingested into Hive in the event.eventgate_analytics_external_error_validation table.

These validation error events are also ingested into logstash and can be viewed in Kibana using this dashboard.

The rate of validation errors per stream can be seen in the EventGate Grafana dashboard.

The eventgate-analytics-external.error.validation stream is a stream like any other, so you can also view the stream using an EventStreams GUI in beta or production (see above).

Evolving your schema

Let's say our mediawiki.interwiki_link_hover stream is operating fine in production, and now we want to also log the link's text value. We'll need to add a new link_text field to the analytics/link_hover schema.

Edit jsonschema/analytics/link_hover/current.yaml again and add the field and bump the version number in the $id field. The new content of current.yaml should be 

title: analytics/link_hover
description: Represents an html link mouseover hover event
$id: /analytics/link_hover/1.1.0
$schema: https://json-schema.org/draft-07/schema#
type: object
allOf:
  - $ref: /fragment/analytics/common/1.0.0#
  # event data fields.
  - properties:
      link_href:
        type: string
        description: href attribute of http anchor link
      link_title:
        type: string
        description: title attribute of http anchor link
      link_text:
        type: string
        description: text value of http anchor link
examples:
  - {$schema":{$ref: "#/$id"},"meta":{"dt":"2020-04-02T19:11:20.942Z" ,"stream":"mediawiki.interwiki_link_hover"},"dt": "2020-04-02T19:11:20.942Z", "link_href": "mw:Extension:EventLogging", "link_title": "mw:Extension:EventLogging", "link_text": "EventLogging extension"}

Now, run npm run build-modified and commit the new 1.1.0 version files. 

$ npm run build-modified
# ...
$ git add jsonschema/analytics/link_hover/*
$ git commit -m 'analytics/link_hover - add link_text field and bump to version 1.1.0'

New analytics/link_hover/1.1.0 files have been created, and the jsonschema/analytics/link_hover/latest symlinks have been updated to point to the 1.1.0 versions.

Now edit your instrumentation code to produce the event data with the link_text field and the updated versioned $schema URI. Your instrumentation code should now look like this: 

$( '#content' ).on( 'mouseover', 'a.extiw', function ( jqEvent ) {
	var link = jqEvent.target;
	var linkHoverEvent = {
		// $schema is required and must be set to the exact value of $id that you set in your schema.
		// We've added link_text, which was added in schema verison 1.1.0, so we need to specify that this
		// event should be validated using schema verison 1.1.0.
		$schema: '/analytics/link_hover/1.1.0',
		link_href: link.href,
		link_title: link.title,
		link_text: link.text,
	}
	var streamName = 'mediawiki.interwiki_link_hover'
	mw.eventLog.submit( streamName, linkHoverEvent );
} );

Note that only backwards compatible changes are allowed. This means that the only type of change you can do to a schema is add new optional fields. jsonschema-tools will ensure that all schema versions are backwards compatible (as well as ensuring that the schema repository is in good shape). Jenkins will run CI tests when you push your schema change to gerrit, but you can also run the tests manually: 

$ npm test
...
  Schema Compatibility in Repository ./jsonschema/
    analytics/link_hover
      Major Version 1
        ✓ 1.1.0 must be compatible with 1.0.0
...

Backwards incompatible schema changes

In general, backwards incompatible schema changes are not allowed. This is because they are not possible to do without manual intervention. Your code may be the only producer of this data, but there may be many consumers, and making a backwards incompatible change requires coordination with all of them. For instrumentation, likely the only uses of your schema will be by EventGate for validation, and Hive for querying your data. If you absolutely need to make a backwards incompatible change to an instrumenation schema, the procedure is:

  • Make a Phabricator ticket and tag Data-Engineering that describes your request.
  • Manual Steps Data-Engineering:
    • Merge the schema change, wait at least 30 minute (or run puppet on schema* nodes to apply merge)
    • Drop the corresponding Hive table(s).

As long as the schema change is a new version (not an editing an existent version), eventgate does not need a restart. If for some forsaken reason you need to edit a change an existent schema version, you'll need to rolling restart the eventgate-analytics-external service.

Overriding event stream config settings

In production for specific wikis / groups

Because EventStreamConfig uses mediawiki-config, we can override settings for beta, wiki groups, or even specific wikis. Unfortunetly, this ONLY works for clients that request their config via the specific wiki you are overriding.

EventGate is a service that is not wiki specific, so it requests all stream configs from meta.wikimedia.org. This means that you MUST add entries in wgEventStreams for at least metawiki (usually done so in the 'default' configs).

EventLogging is a MediaWiki extension that produces events, and is usually executed within the context of a specific wiki. It will get configuration for the specific wiki it is being run in. That means you can override configs that EventLogging uses for specific wikis.

Example: EventLogging respects a sample setting only produce a sample of events. We only want to send a 1/10 events on enwiki. We've already got the mediawiki.interwiki_link_hover stream declared in wgEventStreams default. Let's add an override for the sample setting on enwiki. 

'wgEventStreams' => [
	'default' => [
		// ...
		'mediawiki.interwiki_link_hover' => [
			'schema_title' => 'analytics/link_hover',
			'destination_event_service' => 'eventgate-analytics-external',
		],
	],

    // Add the overrides for enwiki.
    // The '+' prefix indicates that these settings should be recursively merged with 'default' settings.
    // See: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/src/StaticSiteConfiguration.php#64
    '+enwiki' => [
        'mediawiki.interwiki_link_hover' => [
            'sample' => [
                'rate' => 0.1,
            ],
        ],
    ]
],

When enwiki MediaWiki JavaScript requests event stream config for the mediawiki.interwiki_link_hover stream, it will get something like 

{
  "streams": {
    "mediawiki.interwiki_link_hover": {
      "schema_title": "analytics/link_hover",
      "destination_event_service": "eventgate-analytics-external",
      "sample": {
        "rate": 0.1,
      },
    }
  }
}

with the default config merged together.

In beta

Beta AKA deployment-prep configs are in InitialiseSettings-labs.php. For the most part, overrides here work exactly as described above. However, the 'default' configs will not be merged if you set them. That means that you cannot use the 'default' config section in wgEventStreams in InitialiseSettings-labs.php. You can however merge overrides in from InitialiseSettings.php 'default' section if you use the per wiki or group merge override as described above. E.g. adding the following in InitialiseSettings-labs.php will override the sample rate for en.wikipedia.beta.wmflabs.org: 

'wgEventStreams' => [
     
    // Add the overrides for enwiki in beta
    // The '+' prefix indicates that these settings should be recursively merged with 'default' settings from InitialiseSettings.php.
    '+enwiki' => [
        'mediawiki.interwiki_link_hover' => [
            'sample' => [
                'rate' => 1.0,
            ],
        ],
    ]
],

Note that backend services (like EventGate) use the metawiki API (in beta or in prod) as the canonical source of all stream configuration. Your stream MUST be declared for metawiki, even if you are overriding wiki specific settings. If you are developing a new stream in beta, and have not yet declared your stream in the production default $wgEventStreams, then you will need to add a +metawiki override that declares your stream with minimal stream configs:

// In InitialiseSettings-labs.php

'wgEventStreams' => [
    '+metawiki' => [
        'my_new_stream' => [
            'schema_title' => 'my/new/schema',
            'destination_event_service' => 'eventgate-analytics-external'
        ],
    ],
    // ...
    '+enwiki' => [
        'my_new_stream' => [
            // override settings for enwiki here
        ],
    ],

]

Decommissioning

When deploying a new instrumentation event stream, you should also plan to decommission it one day. A schema should never be deleted, but all of the stream related code and configuration can be removed at anytime to stop producing an event stream. In the rare cases where the presence of the schema itself is problematic, we can delete it but it requires a coordinated effort to exclude it and prevent alarms along the data pipeline.

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Event_Platform/Instrumentation_How_To&oldid=2170643"