Test Kitchen/Glossary

This page documents terms used in the Test Kitchen documentation.

Terms

Analytics sampling

Analytics sampling is the logic that determines whether an event is included in data collection for an event stream, including the sampling unit and rate. See Analytics sampling.

SDK

The methods provided by the Test Kitchen client libraries to submit events. See Test Kitchen/SDK.

Base schema

A reusable default schema provided by Test Kitchen to use with web or mobile app instruments

Baseline

See mw:Product Analytics/Glossary#Baseline.

Bucketing

See #Experiment enrolment sampling.

Contextual attributes

Fields in an event data object that provide information about the performer who triggered the event and the wiki where the event occurred. For example, the page title and page language. The values of contextual attributes that are included in the stream configuration are populated automatically by Metrics Platform when the event is generated. See Metrics Platform/Contextual attributes.

Data contract

Refers to the Test Kitchen base schemas and the conventions for creating custom schemas. See Test Kitchen/Schemas.

Experiment

See mw:Product Analytics/Glossary#Experiment.

Experiment enrolment sampling

Experiment enrolment sampling is the act of enrolling users into experiments (also called an A/B tests) and consistently assigning an enrolled user a variant of the feature that is being experimented on. Experiment enrolment sampling is also known as bucketing. See Experiment enrolment sampling.

Experiment orchestration

See mw:Product Analytics/Glossary#Experiment orchestration.

Test Kitchen UI

Test Kitchen tool that allows you to create and manage experiments and baseline instruments, available at mpic.wikimedia.org. Formerly known as MPIC or xLab.

Event

A data object that represents a user interaction happening at a definite time. For example, a user clicking a button. Instruments work by sending event data objects to an event stream in response to user actions or app conditions.

Event stream

A continuous series of structured event data^[1]. An instrument publishes events to an event stream. Contextual attributes and analytics sampling are configured per event stream. To configure an event stream manually using MediaWiki's configuration files, see Test Kitchen/Stream configuration.

Fragment

A partial schema designed to be combined with other fragments to build custom schemas. See Test Kitchen/Custom schemas.

Guardrail metric

See mw:Product Analytics/Glossary#Guardrail metric.

Hive

Datasource where event stream data is available for analysis. See Data Platform/Systems/Hive.

Interaction data

Properties supported by the Test Kitchen schemas that can be customized to include data specific to an instrument. For example, action_context and element_id. See Test Kitchen/Schemas.

L3SC review

A process in which new data collection activities are reviewed by Privacy Engineering and Privacy Legal depending on risk tier. See foundation:Legal:Data Collection Guidelines.

Leading indicator

See mw:Product Analytics/Glossary#Leading indicator.

Monotable

Instruments using the same base schema can use the same event stream. The resulting data is available in a shared Hive table. This shared table and this architectural approach is sometimes referred to as the monotable. See Metrics Platform/Decision Records/Single Table Per Base Schema.

MPIC

Former name for #Test Kitchen UI.

xLab

Former name for #Test Kitchen UI.

Primary metric

See mw:Product Analytics/Glossary#Primary metric.

Product health metric

See mw:Product Analytics/Glossary#Product health metric.

Rolling metric

See mw:Product Analytics/Glossary#Rolling metric.

Sampling

Can refer to analytics sampling or experiment enrolment sampling. See Test Kitchen/Sampling.

Secondary metric

See mw:Product Analytics/Glossary#Secondary metric.

Schema

A data structure used by Test Kitchen to validate event data. Test Kitchen schemas as structured as JSON Schemas and written as YAML files. See Test Kitchen/Schemas.

Stream

See #Event stream.

Trailing (lagging) indicator

See mw:Product Analytics/Glossary#Trailing (lagging) indicator.

References

↑ Event Platform/EventStreams HTTP Service

[1] Event Platform/EventStreams HTTP Service

[1]