Test Kitchen/Maintainer introduction

This page provides an introduction to working with Test Kitchen for maintainers.

Component Catalog

As explained in Test Kitchen#About, Test Kitchen provides an standardized toolkit to implement experiments or instruments on Wikimedia-hosted wikis. This toolkit is composed of various resources, libraries or extensions:

• MediaWiki extensions: Hosts the PHP and JS SDKs and integrate them with MediaWiki, and enrolls logged-in users into experiments
• UI/GrowthBook: The UI you use to configure and analyze experiments and instruments
• Schemas: Common data contracts used to validate events that your experiments and instruments send
• Streams: Shared streams that your experiments and instruments send events to
• Dashboards: Dashboards that allow us to monitor the health of the system
• Bots: Bots that alert us to changes in the system

You will likely work on all of these components while maintaining Test Kitchen. This section will give you an overview of all of the components and jumping off points to help you start working on them.

MediaWiki Extensions

TestKitchen

The TestKitchen extension hosts and integrates the Test Kitchen SDKs with MediaWiki, and enrolls logged-in users into experiments.

• Homepage: https://www.mediawiki.org/wiki/Extension:TestKitchen
• Repository: https://gerrit.wikimedia.org/g/mediawiki/extensions/TestKitchen

EventBus

The Test Kitchen PHP SDK uses the EventBus extension to send events.

• Homepage: https://www.mediawiki.org/wiki/Extension:EventBus
• Repository: https://gerrit.wikimedia.org/r/admin/repos/mediawiki/extensions/EventBus

EventStreamConfig

The EventBus extension requires the EventStreamConfig extension.

The EventStreamConfig extension is a utility extension that provides library functions for retrieving event stream configuration and an API for exporting it.

• Homepage: https://www.mediawiki.org/wiki/Extension:EventStreamConfig

• Repository: https://gerrit.wikimedia.org/r/admin/repos/mediawiki/extensions/EventStreamConfig

WikimediaEvents (Optional)

WikimediaEvents is the extension where almost all experiments and instruments reside.

• Homepage: https://www.mediawiki.org/wiki/Extension:WikimediaEvents
• Repository: https://gerrit.wikimedia.org/g/mediawiki/extensions/WikimediaEvents

EventLogging (Optional)

Test Kitchen doesn't depend on the EventLogging extension (EventLogging). However, when the Experiment Platform team was the Metrics Platform team, they introduced a lot of functionality to EventLogging that would later become the Test Kitchen SDKs. The Experiment Platform team are planning on tidying up a lot of that functionality.

• Homepage: https://www.mediawiki.org/wiki/Extension:EventLogging
• Repository: https://gerrit.wikimedia.org/g/mediawiki/extensions/EventLogging

Test Kitchen UI

Test Kitchen UI allows you to configure and activate experiments and instruments. GrowthBook allows you to do the same and to also analyze experiment results.

• Repository: https://gitlab.wikimedia.org/repos/data-engineering/test-kitchen

Schemas

Schemas are part of the Event Platform, which Test Kitchen depends on.

Schemas are the data contracts used to validate events that your experiments and instruments send. https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-secondary contains all of the schemas that have been used by EventLogging, the Event Platform (FKA Modern Event Platform), Metrics Platform, and now Test Kitchen. The Test Kitchen schemas and schema fragments are available.

Streams

Streams are part of the Event Platform, which Test Kitchen depends on.

Test Kitchen uses two streams by default, product_metrics.web_base and product_metrics.app_base. If a custom schema is required, then a new stream must be created and registered with Test Kitchen UI and GrowthBook.

Test Kitchen doesn't store experiment or instrument configuration in stream configuration. However, when the Experiment Platform team was the Metrics Platform team, they introduced a lot of functionality that was configured using stream configuration. The Experiment Platform team are planning on tidying up a lot of that functionality.

• Documentation: Event Platform/Stream Configuration
• Reposity: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/refs/heads/master/wmf-config/ext-EventStreamConfig.php

Dashboards

• The Test Kitchen MediaWiki Extension dashboard gives an overview of the health of the TestKitchen MediaWiki extension and the Test Kitchen UI as observed by the extension
• The Experiment Status dashboard gives an overview of the status of an individual experiment
• The ATS Backends <-> Origin Servers Overview (analytics-external) dashboard gives an overview of the health of the Event Platform Event Intake Service, EventGate, from the perspective of the Apache Traffic Server backends