Test Kitchen/Conduct an experiment
An experiment is a test of a hypothesis designed to provide trustworthy and generalizable data. It imposes an intervention on subjects with the intention of observing what outcome that intervention leads to.[1] This page describes how to conduct an experiment using Test Kitchen, our experimentation platform.
This guide describes the process for implementing a web experiment using Test Kitchen SDKs (for JavaScript and PHP), configuring the experiment in the GrowthBook UI, analyzing it, and the next steps after obtaining results.
The steps you should expect to follow are to:
- Plan: define your questions and metrics
- Code: write code using the SDKs to show your variants and collect data
- Launch: configure which wikis and how many users will be in your experiment
- Evaluate: review your results and decide!
For the process of creating an instrument for product analytics instead of an experiment, see Create an instrument. If you are unsure which process to use, reach out in Slack to #talk-to-experiment-platform
Plan
In the planning stage, you'll define your hypothesis, what the intervention you want to test is, and what metrics you'll use to measure the results. The definition of your experiment hypothesis and variations should be described in a measurement plan.
You can define your metrics and the events needed to produce them in an instrumentation specification. This will be used in the Code stage where you add code to generate the events.
Measurement plan
Experiments should have a measurement plan to describe the impact an intervention is intended to have, and how we will evaluate its impact. Templates for experimentation plans are available on the main Test Kitchen documentation page.
The sections below have information to help with the decisions you will need to make while designing your measurement plan. If you have questions, reach out in Slack to #talk-to-experiment-platform.
Experiment design: identifier type
When configuring the experiment you will decide identifier type that you wish to use, and this is a major component of your experiment design. Currently Test Kitchen supports experiments on two sets of users: all user traffic (where enrollment is based on Edge Unique cookie) and logged-in users only (where enrollment is based on CentralAuth global ID, including temporary accounts).
If you choose all user traffic, you will also need to decide whether your experiment will split the cache or not. Splitting the cache means that the cache needs to have multiple versions of a single page to run your experiment. This puts strain on the cache. Due to this, experiments which split the cache have lower maximum enrollments than those which do not. See the user traffic per wiki section for more information.
The options are:
mw-user- The experiment enrollment authority is MediaWiki and it enrolls users using their CentralAuth global ID.
- When using
mw-user, the user will be consistently enrolled and assigned across all wikis and across all their sessions. The user may log out and log back in, and their enrollment and assignment will not change. - Best for: when it's important to get accurate data about logged-in traffic, or server-side instrumentation is needed.
- NOTICE: This also includes Temporary Accounts.
edge-unique- The experiment enrollment authority is our caching system, Varnish, and it enrolls clients using anonymous cookies (wmf-uniq cookie).
- When using
edge-unique, the user will be consistently enrolled and assigned within top-level domain (e.g. wikipedia.org) for the lifetime of their wmf-uniq cookie. If the cookie is cleared manually or automatically – by, for example, a privacy enhanced browser – the client is issued a new wmf-uniq cookie, and they may or may not end up enrolled or even assigned to the same group. - Best for: experiments on logged-out users which do not meet the requirements for
edge-unique (no cache split) - NOTICE: Experiments using this identifier type cannot use server-side instrumentation. Due to the privacy design of Edge Uniques, MediaWiki does not have access to that cookie. These experiments can only collect analytics data using client-side instrumentation.
edge-unique (no cache split)
- As with
edge-unique, the experiment enrollment authority is our caching system, Varnish, and it enrolls clients using anonymous cookies (wmf-uniq cookie). Varnish will consistently enroll and assign users for experiments usingedge-unique (no cache split). However, Varnish will not split the cache based on those enrollments and assignments. - Best for: experiments on logged-out users where the feature being experimented on is implemented entirely in JavaScript and CSS, and no variation in the server-rendered HTML is required.
- NOTICE: Experiments using this identifier type cannot use server-side instrumentation or feature toggling. These can only be done using the JS SDK.
When to use one or the other: When using edge-unique, a small proportion of subjects in your experiment will be logged-in users. Depending on when and how frequently those users clear their cookies (things to consider: users who only log in inside an incognito window), a volunteer may come in and out of the experiment and even switch groups – because their enrollment is based on the wmf-uniq cookie. For that reason we split analysis of experiments by user authentication status: All, Logged-in only, and Logged-out only. If you need to have trustworthy insights about the effect of your change on logged-in user behavior, we recommend conducting a separate experiment (either in parallel or as a follow-up) specifically on logged-in users, using mw-user as the identifier type.
Experiment design: user traffic per wiki
Max traffic allocation rules exist when using edge-unique identifier type:
| User identifier type: | mw-user-id |
edge-unique (no cache split) |
edge-unique (cache split) |
| English wikipedia: | no max | Reader experiment: 1% max traffic Editing experiment: 25% max traffic |
1% max traffic |
| Any other wiki: | no max | Reader experiment: 10% max traffic Editing experiment: 25% max traffic |
10% max traffic |
By default, experiments are assumed to be reader experiments, and the reader experiment traffic limits apply. If your experiment will target editing or account creation, you can request a higher max traffic allocation for the experiment by creating a Phabricator ticket in the Test Kitchen project with the name of the experiment, the limit requested (editing or account creation), and a reasoning for why the experiment qualifies for the higher limit.
The maximum traffic allocation on English Wikipedia for cache splitting experiments is 1% – which, given the volume of reader traffic, would still be approximately 333K unique devices/day.
The math: English Wikipedia gets about 1B unique devices per month.[2] 1B/30 days is 33.33M/day, and 1% of that is 333K/day. For comparison, 0.5% of traffic is ~167K/day; 0.01% of traffic is 3.3K/day).
Domains are not a reliable differentiator of mobile vs desktop (e.g. user can use Minerva Neue (mobile) skin while on the desktop domain), so mobile and desktop variants of domains are bundled. Experiments target wikis, not domains. If you need to target mobile or desktop experiences specifically, you will need to do this with your feature using checks such as MobileFrontend's shouldDisplayMobileView() in PHP. Refer to MobileFrontend's FAQ for more information about detecting mobile view in frontend code.
Experiment design: experiment duration
If you want to know the exact sample size to target with your experiment and ensure a certain trustworthiness of results, we recommend working with a specialist from the Product Analytics team to conduct a power analysis. To do this the analyst will need:
- A baseline measurement of the primary metric that will be used for evaluating the experiment, including that metric's variance
- A minimally detectable effect (MDE), which is usually 5% or 10%. Smaller MDE will require a larger experiment. If you powered your experiment enough to detect a 5% lift – because practically speaking that's the smallest lift you care about – but the true lift is 3% or 4%, the experiment would be underpowered for that. It does not mean that you will not be able to detect it at all, but the probability of detecting it is lower.
- A desired statistical power, which is usually 80%. It is the probability of correctly rejecting null hypothesis (that there is no effect/impact/lift) when there actually is an effect/impact/lift and the null hypothesis should be rejected. More power will require a larger experiment.
- A significance level, which is usually 5%. It is the probability of incorrectly rejecting null hypothesis (that there is no effect/impact/lift) when there actually is no effect/impact/lift and the null hypothesis should not be rejected. It is the false positive rate, and a smaller significance level will require a larger experiment.
Scenario: Imagine you run 100 experiments of the same feature, all powered to 80% to detect an MDE of 7% with significance level 0.05 (common threshold for p-values) and there actually is a 7% lift from exposure to the change. We can expect about 80% of those experiments to produce results that suggest you ship the change, and about 20% of those experiments to produce results that suggest you do not ship the change. When you conduct an experiment, you do not know if your experiment-powered-to-80% will be one of the 80% or one of the 20%.
Our recommendation is to run the experiment for 1 week, or 2 weeks if you need more data. A week-long experiment will account for weekday and weekend seasonalities. You may have very different profiles of users who use your feature on weekdays than you do on weekends, so if you conduct an experiment on a Tuesday, you may get very different results than if you were to conduct it on a Friday or a Saturday. By conducting the experiment for a full week, it allows you to include both profiles of user for more trustworthy results.
Note for all user traffic experiments (using edge uniques for enrollment): If your experiment's primary metric relies on the edge unique-derived subject identifier, that metric (e.g. reader retention rate) can only be measured within an experiment. This means that one of the key ingredients for power analysis – a baseline measurement of the primary metric – is not readily available. To address this challenge we recommend conducting a pilot study: a small scale A/A test (an experiment where the treatment group is a second control group and no change is tested) on the same set of wikis you are planning to target with your experiment. This will yield a baseline (with its variance) and a rate of user traffic, which you can then use to determine the duration your experiment should be to obtain a desired sample size.
Instrumentation specification
Once you have a measurement plan, the next step is to create an instrumentation specification ("spec") (template). The instrumentation spec defines all the data you'll collect with your instrument. The spec is also a useful tool for engineers to ensure that all events are being produced and received correctly. For a template and examples of instrumentation specs, see the folder on Google Drive.
Key pieces of information:
- machine-readable experiment name
- Instrumentation will use this to determine if user is a subject enrolled in the experiment
- This will be recorded under
experiment.enrolledin event data - This is needed when registering experiments for automated analysis
- machine-readable treatment group name
- Instrumentation will use this to determine if subject is assigned to the group
- This will be recorded under
experiment.assignedin event data
Data collection risk level assessment
Please refer to Test Kitchen/Regulation section.
Code
You can write your experiment code in the WikimediaEvents extension or in your product codebase.
Your code will need to:
- Check for group assignment in your experiment
- Choose which variation to display based on group membership
- Send exposure events for both control and treatment users
- Send events for metrics to be measured in your experiment
The Test Kitchen SDK API Docs explain how how to use the SDKs to help with these tasks. There are examples below of how to handle exposure logging and instrumentation.
Scenario for code examples below:
- The experiment is named "Larger default font size", with machine-readable name "larger-default-font-size"
- The treatment group is named "X Large", with machine-readable name "x-large"
CSS classes
To make it easy to test appearance changes we automatically add the following classes to the <body> element, no matter which skin is in use:
test-kitchen-experiment-{experiment_name}test-kitchen-experiment-{experiment_name}-{group_name}
Where {experiment_name} and {group_name} are machine-readable names of the experiment (that the user is enrolled in) and group (that the user is assigned to), respectively.
Based on the scenario, you can then have CSS that uses Codex's font size design tokens and looks something like:
.test-kitchen-experiment-larger-default-font-size-x-large {
font-size: var( --font-size-x-large );
}
Users assigned to the treatment group would automatically have that style applied to <body> on every page they view.
We reserve an explicit control group for all experiments. Control group should have the same experience as users who are not enrolled in the experiment have. So in this scenario, users in the control group would have a <body> with class test-kitchen-experiment-larger-default-font-size-control.
Exposure logging
Exposure logging is the mechanism Test Kitchen has to express that a user has been exposed to a variation (either control or treatment) in the experiment. The most important thing to do is try to only log exposure when exposure actually happens (a true positive). Logging exposure when there has not been an exposure (a false positive) or not logging an exposure when there actually was one (a false negative) can negatively affect analysis and yield untrustworthy experiment results. For best practices and recommended patterns – including how to reliably log exposure for all user traffic experiments – please see our guide to logging experiment exposure.
Both PHP and JS SDKs support logging exposure via Experiment#sendExposure() method which produces an experiment_exposure event that is decorated with a curated set of contextual attributes, which then allow us to offer dimensions – such as user authentication status (logged-out, temporary, or registered) – for digging into experiment results.
const experiment = await mw.testKitchen.getExperiment( EXPERIMENT_NAME );
// The user has been exposed to a variation we are testing
experiment.sendExposure();
. . .
$experimentManager = MediaWikiServices::getInstance()->getService( 'TestKitchen.ExperimentManager' );
. . .
$experiment = $experimentManager->getExperiment( EXPERIMENT_NAME );
. . .
// The user has been exposed to a variation we are testing
$experiment->sendExposure();
. . .
sendExposure() calls in your instrumentation, as we have a mechanism which minimizes the volume of exposure events that experiments produce.
Benefits of logging exposure
Experiment exposure events enable GrowthBook to know:
- Which variation subjects were assigned to
- What attributes those subjects had throughout the experiment – this is used to enable dimensions
- What is the earliest exposure each subject had to their variation – this is used to exclude any data collected about subjects before their first exposure, which improves the strength and accuracy of the results
- Whether any subject was exposed to more than one variation – this is used to assess experiment health
This is why exposure logging are a requirement for experiments to be analyzable in GrowthBook.
Server-side instrumentation
To retrieve enrollment and assignment information:
const EXPERIMENT_NAME = 'larger-default-font-size';
const TREATMENT_GROUP_NAME = 'x-large';
$experimentManager = MediaWikiServices::getInstance()->getService( 'TestKitchen.ExperimentManager' );
$experiment = $experimentManager->getExperiment( EXPERIMENT_NAME );
Server-side feature toggling
edge-unique identifier type for enrollment, data collection is limited to client-side but feature toggling works server-side. For everyone experiments using edge-unique (no cache split), data collection and feature toggling are limited to client-side.
When you implement the change to your feature server-side, you condition it on the user being enrolled in the experiment and assigned to the treatment group which will receive the change:
if ( $experiment->isAssignedGroup( TREATMENT_GROUP_NAME ) ) {
// Code that only runs for subjects in the treatment group
} else {
// Code that runs for subjects in the control group and
// users who are not enrolled in experiment
}
This works for all experiments where the cache is split – those using edge-unique and mw-user identifiers for enrollment.
Server-side analytics
edge-unique or edge-unique (no cache split) identifier types for enrollment due to the privacy design of the edge unique cookie (it is not accessible by MediaWiki). For experiments conducted on all user traffic you can only collect data using client-side instrumentation.
To collect data for the experiment:
// Example of logging a page_visit action
$experiment->send( 'page_visit' );
If the user is enrolled in the experiment, this will produce an event to the product_metrics.web_base stream using the latest version of the web base schema (2.0.0 at the time of writing). This event will have all the experiment membership information, such as the name of the experiment and the group they are assigned to. If the user is not enrolled in the experiment, this will do nothing.
Please refer to this guide on interaction data for best practices and data modeling recommendations as you design your analytics instrumentation.
Client-side instrumentation
To retrieve enrollment and assignment information:
const EXPERIMENT_NAME = 'larger-default-font-size';
const TREATMENT_GROUP_NAME = 'x-large'; // only needed for feature toggling, not analytics
const experiment = await mw.testKitchen.getExperiment( EXPERIMENT_NAME );
Client-side feature toggling
When you implement the change to your feature client-side, you condition it on the user being enrolled in the experiment and assigned to the treatment group which will receive the change:
if ( experiment.isAssignedGroup( TREATMENT_GROUP_NAME ) ) {
// Code that only runs for subjects in the treatment group
} else {
// Code that runs for subjects in the control group and
// users who are not enrolled in experiment
}
Client-side analytics
edge-unique identifier type for enrollment) due to the privacy design of the cookie (it is not accessible within MediaWiki).
To collect data for the experiment:
// Example of logging a page_visit action
experiment.send( 'page_visit' );
If the user is enrolled in the experiment, this will produce an event to the product_metrics.web_base stream using the latest version of the web base schema (2.0.0 at the time of writing). This event will have all the experiment membership information, such as the name of the experiment and the group they are assigned to. If the user is not enrolled in the experiment, this will do nothing.
Please refer to this guide on interaction data for best practices and data modeling recommendations as you design your analytics instrumentation.
Using custom schema and/or stream
By default, events produced by experiments instrumented with TK SDKs use the analytics/product_metrics/web/base schema and flow into the product_metrics.web_base stream.
If your experiment requires collecting data that is not supported by the base schema, you will need to create a custom schema and a custom stream; refer to this guide on custom schemas for instructions.
Considerations
- Preferably, the custom stream is opted out of User-Agent string collection (using this mechanism).
- Preferably, the custom stream does not include
performer_namecontextual attribute, because we try not to associate usernames with subject IDs. - The custom stream may need to be configured for edge uniques (if your experiment will use that for enrollment).
- The resulting table in Hive would not be allowlisted for event sanitization and would thus be subject to to 90 day data retention.
Stream configuration
use_edge_uniques. Otherwise experiment.subject_id will be set to "awaiting" for all events flowing into the stream.
Backport a MediaWiki configuration change as follows.
- Configure your stream in wmf-config/ext-EventStreamConfig.php:
'wgEventStreams' => [
'default' => [
// ...
// Stream name.
'mediawiki_product_metrics_translation_mint_for_readers_experiments' => [
// Schema name.
'schema_title' => 'analytics/product_metrics/web/translation',
'destination_event_service' => 'eventgate-analytics-external',
'producers' => [
'eventgate' => [
'enrich_fields_from_http_headers' => [
// Collect the user agent.
// Disabled by default.
'http.request_headers.user-agent' => 'user-agent',
],
// Target logged-out users.
'use_edge_uniques' => true,
],
'metrics_platform_client' => [
// Contextual attributes.
// See https://wikitech.wikimedia.org/wiki/Experimentation_Lab/Contextual_attributes
// NOTE `agent_client_platform` and `agent_client_platform_family`
// are automatically added.
'provide_values' => [
// We recommend collecting `performer_is_logged_in` and `performer_is_temp` at the same time.
// See https://phabricator.wikimedia.org/T374812#10953216
'performer_is_logged_in',
'performer_is_temp',
'performer_pageview_id',
'mediawiki_database',
],
],
],
],
],
],
Example scenario
We want to run experiments with the MinT for Wiki Readers feature/initiative. Since we need to collect special data such as source and target languages, we will use the analytics/product_metrics/web/translation schema. Now, there is already a mediawiki.product_metrics.translation_mint_for_readers stream configured, so why should we not use it? Because:
- It has not yet been opted out of User-Agent string collection
- That stream is configured to collect
performer_namewhich we would rather avoid for experiments using edge uniques for enrollment - Stream has not been enabled for edge uniques
- The corresponding table
mediawiki_product_metrics_translation_mint_for_readershas been allowlisted for event sanitization, so the data is retained in perpetuity, albeit in a sanitized way (with some columns cleared and some identifiers hashed).
So in this case the best thing to do would be to create a separate stream just for experiments, reassess which contextual attributes we need for experiments, and include additional configuration to make it suitable for data collection from all user traffic experiments.
Instrumentation
Next we need to override the default schema ID when we initialize experiment (the stream name can be set when registering the experiment in Test Kitchen UI):
const experiment = await mw.testKitchen.getExperiment( EXPERIMENT_NAME );
experiment.setSchema( '/analytics/product_metrics/web/translation/1.4.2' );
When experiment.send() is called, the events will declare our custom schema and be produced to our custom stream.
Test
This section covers testing of experiments. Refer to Event Platform/Instrumentation How To § Setting up for local testing for general instructions for testing analytics instrumentation locally.
A complete testing workflow should target 3 environments:
- local development via #HTTP header
- PatchDemo and Beta cluster via #Enrollment override
- production wikis via #Enrollment override
HTTP header
The X-Experiment-Enrollments: {experiment_machine_readable_name}={assigned_group}; HTTP request header mocks enrollment in the given experiment. Example value: larger-default-font-size=x-large.
Instructions:
- on Google Chrome, install Inssman
- add a rule that sets the HTTP header
- hit Create Rule, then Modify Header
- If Request - Contain -
http://localhost - Operator - Set - Request -
X-Experiment-Enrollments-{experiment_machine_readable_name}={assigned_group} - hit Create
- go to your feature URL
You can change {assigned_group} to switch between treatment and control groups.
The experiment.sampling_unit event field will be set to edge-unique and experiment.subject_id will be set to awaiting, but that's not important for testing variations of your feature. We use this mechanism for testing locally even if the plan/intention is to conduct the experiment on logged-in users only. Rest assured that the experiment properties will be set correctly by our system when the experiment is configured and deployed to production.
When using this method you can avoid using local instances of Test Kitchen UI/API and GrowthBook by filling $wgTestKitchenExperiments in your LocalSettings.php file with the minimal experiment config you need. For example:
$wgTestKitchenExperiments = [
[
'name' => 'my-experiment',
'user_identifier_type' => 'edge-unique',
'phase_index' => 0,
'stream_name' => 'product_metrics.web_base',
'contextual_attributes' => []
]
];
For a logged-in experiment the configuration is a little different:
$wgTestKitchenExperiments = [
[
'name' => 'my-experiment',
'user_identifier_type' => 'mw-user',
'groups' => [ 'treatment' ],
'sample' => [
'rate' => 1.0,
],
// 'start' => '2026-10-01T00:00:00Z', // Optional, for use with T435990
'phase_index' => 0,
'stream_name' => 'product_metrics.web_base',
'contextual_attributes' => []
]
];
Enrollment override
action, interactionData and per-event contextualAttributes as per Experiment#send's parameters in the browser's JavaScript console.
You have 2 options:
- call
mw.testKitchen.overrideExperimentGroup()in the #Browser's console - set the
?mpo={experiment_machine_readable_name}:{assigned_group}#URL query parameter
Browser's console
mw.testKitchen.overrideExperimentGroup() overrides enrollment when you are logged in. The override will persist between page views within your session. Sessions on the Web are… complicated. Usually a session ends when the browser process is terminated. See Data Platform/Sessions#Web for more details.
Invoke the method in your browser's JavaScript console. For example:
mw.testKitchen.overrideExperimentGroup( 'larger-default-font-size', 'x-large' );
Manually clear the override through mw.testKitchen.clearExperimentOverride(). For example:
mw.testKitchen.clearExperimentOverride( 'larger-default-font-size' );
Or clear all overrides through mw.testKitchen.clearExperimentOverrides():
mw.testKitchen.clearExperimentOverrides();
URL query parameter
The ?mpo={experiment_machine_readable_name}:{assigned_group} URL query parameter overrides enrollment when you are logged in or out.
The override will not persist between page views: if you navigate to another page, you'll have to set the parameter again.
Append the query parameter to your feature URL. For example:
?mpo=larger-default-font-size:x-large
The ?mpo= parameter also works on PatchDemo wikis where Test Kitchen is enabled by default. This lets you demo an upcoming experiment on a PatchDemo instance - the override applies the treatment (if overriding with the treatment group) and adds the corresponding CSS classes to the <body> element similar to production.
WikimediaDebug
WikimediaDebug can force your requests into specific Test Kitchen experiment enrollment groups. Because the extension forces a Varnish cache miss on every request, these overrides reach the app server reliably including for "everyone" cache-splitting experiments and multi-page workflows such as login and signup.
Useful for:
- Previewing an experiment before it starts
- Testing a specific group without waiting to be bucketed
- Exercising cache-splitting "everyone" experiments
- Testing experiments spanning multiple pageviews
How it works:
On popup open, WikimediaDebug fetches current experiments from https://test-kitchen.wikimedia.org/api/v1/experiments (cached 10 min) for autocomplete, and encodes your pairs in the header as comma-separated key-value pairs:
X-Wikimedia-Debug: backend=k8s-mwdebug; experiments=experiment-name-1:treatment,experiment-name-2:treatment2
TestKitchen's RequestEnrollmentsProcessor reads it and forces each assignment, recorded as overridden (coordinator: forced) and excluded from analytics.
Steps:
- Install/update latest version of the WikimediaDebug extension
- Open a production/Beta Wikimedia page
- In the Test Kitchen experiment overrides popup, type in an experiment (autocomplete or custom) and a group
- Click
+Add experiment /xfor more rows - one experiment each (note that duplicates are flagged/excluded) - Pick a backend - k8s-mwdebug
- Toggle ON and reload
Save/share:
- Export JSON copies the pairs to the clipboard
- Import JSON pastes a JSON object and replaces the rows (invalid entries skipped).
Verify:
- Open DevTools, go to the Network tab, look at the document request
- View Request Headers
- Confirm seeing the
X-Wikimedia-Debug: … experiments=…header - Run
mw.tk.getAssignments();in the console - Confirm the experiment and group assignments you selected in the WikimediaDebug popup
Launch
Once your experiment code has been deployed to production, create a configuration for your experiment in GrowthBook to describe when it should start, which wikis to run on, how much traffic to allocation to each variation, and which metrics to calculate. The GrowthBook guide describes how to create an experiment.
Experiments are active if they are turned on and if the current time is between start date's noon (UTC) and end date's noon (UTC).
For example: if an experiment is scheduled for 2025-06-02 – 2025-06-09 and has been turned on, it will only be active between 2025-06-02T12:00:00Z and 2025-06-09T12:00:00Z
Monitor event production
Once an instrument or experiment is live on production, you can take a look at the following Grafana dashboards to see if it's sending already events:
Also, you can use kcat to review events flowing in:
kcat -C -b kafka-jumbo1010.eqiad.wmnet:9092 \
-t eqiad.product_metrics.web_base -o end | \
jq 'select(.experiment.enrolled == "YOUR_EXPERIMENT_NAME")'
(Assuming the experiment produces to the default product_metrics.web_base stream.) Refer to Event Platform/Producing events § Directly from Kafka for more details on viewing produced events.
If your experiment won't start: validation errors
When you start an experiment, GrowthBook runs inline validation. If launch is blocked, the message explains what to fix.
Common cases are leaving variation ids as default integers, leaving Risk Level as "Risk assessment pending", or targeting an invalid/empty set of wikis. This inline check is the only validation, so every error must be cleared before the experiment can be run. See Configuring experiments: Resolve validation errors for the full list and step-by-step fixes.
Emergency shutdown
If something is wrong, you can turn the experiment off in Test Kitchen UI using the same menu you used to turn the experiment on.
For everyone (all user traffic based on edge unique cookie) experiments, the change will take about 3 minutes to fully propagate through our caching centers and for no new clients to be enrolled.
For logged-in users experiments, the change will be almost immediate (<1min) in MediaWiki.
Evaluate
Refer to the GrowthBook user guide, including the guide to defining metrics.
Aftermath
Make decision
If the experiment informed a decision, please document it at metawiki:List of experiments in Product and Technology for transparency, visibility, and because we use that log to track our success of enabling data-informed decision making through shared experimentation tools and infrastructure.
Share results
The experiment health pane on the dashboard will let you know whether the results of the experiment are suitable for publication on wiki or other public location, depending on number of subjects in the experiment. Please refer to data publication guidelines for more information about publishing data.
Clean up
Remember to decommission the instrument/experiment (using this guide) when you are done.
If you decided to not ship the change based on the results of the experiment, remember to remove the change from the codebase.
Convert experiment to baseline instrument
If you decide to ship the change and collect data on an ongoing basis to measure the metric(s) as a product health metric(s), you can convert the experiment to a baseline instrument:
- const EXPERIMENT_NAME = 'larger-default-font-size';
- const experiment = mw.testKitchen.getExperiment( EXPERIMENT_NAME );
- experiment.send( 'page_visit' );
+ const INSTRUMENT_NAME = 'page-visits'
+ const instrument = mw.testKitchen.getInstrument( INSTRUMENT_NAME );
+ instrument.send ( 'page_visit' );
And then configure the instrument "Page Visits" (machine-readable name: page-visits) in Test Kitchen UI. Please refer to this guide for more information on measuring product health.
Case study: account creation form improvements V1 A/B test
This section collects links and information related to the Contributor Growth team's experiment to test form changes on Special:CreateAccount on English Wikipedia.
| Phabricator | Documentation | Implementation |
|---|---|---|
| Epic, V1 | Artifacts related to documentation:
|
Artifacts related to implementation:
|
Notably, because of the way authentication (account creation and login) system works on Wikimedia sites and how edge uniques work, this experiment did not target English Wikipedia. Edge uniques do not cross 2LD boundaries,[3] so the same visitor on *.wikipedia.org sites has one edge unique cookie and another edge unique cookie on auth.wikimedia.org (and another on meta.wikimedia.org, because wikimedia.org subdomains are special cased for privacy reasons). Instead, this experiment targeted auth.wikimedia.org and performed a check for wgDBname (which would be "enwiki" for visitors landing on the auth domain from English Wikipedia).
While this experiment's setup is somewhat unconventional (due requirements & constraints on Test Kitchen and MediaWiki side), this experiment's instrumentation follows all of the best practices as of the time of writing and makes for a good example:
- Exposure logging follows our recommendations:
- It logs exposure with
sendExposure()after checking for targeting/eligibility conditions (mobile users coming toauth.wikimedia.orgfrom English Wikipedia only), aligned with the server-side implementation. - Server-side implementation serves mobile enwiki users enrolled in the experiment either a control version of the form or the treatment version of the form, so exposure happens when the page is served the user
- Instrumentation only runs when
mw.config.get( 'wgCanonicalSpecialPageName' ) === 'CreateAccount' - After the other targeting conditions are met, instrumentation calls
sendExposure()and trusts it to send a minimal number of exposure events (see T414738 for more details).
- Instrumentation only runs when
- It logs exposure with
- Event names use present tense and snake_case (e.g.
page_visit, notpage-visited) per our guidelines and recommendations. - It is configured entirely in Test Kitchen UI (TK UI) and uses the default
product_metrics.web_basestream, with the experiment's contextual attributes selected via TK UI rather than stream configuration.- However, since this experiment was analyzed in GrowthBook it did not need any contextual attributes besides the ones automatically collected with exposure events (to enable segmentation of experiment results on dimensions).
Case study: synthetic A/A test
This section collects links and information related to the Experiment Platform team's first end-to-end test of the Test Kitchen – and edge unique-based enrollment of subjects – for reference. If you conducting an experiment for the first time, it may be helpful to review all the artifacts. Note that since this was an A/A test, not an A/B test, we did not vary the user experience and did not need to implement any feature toggling (making execution of some code conditional on user being enrolled in experiment and assigned to treatment group).
| Phabricator | Documentation | Instrumentation | Experiment | |
|---|---|---|---|---|
| Epic | Measurement Plan (WMF only) | Instrumentation Spec (WMF only) | Task, Code | Configuration, Results (TK AA) |
Analytics instrument was added to WikimediaEvents in this patch. Note that in addition to the instrument code itself, it also had to be registered with ResourceLoader – see ResourceLoader/Developing with ResourceLoader § JavaScript for more details.
The "Page visits" metric was defined in this patch, where the experiment was also registered for automated analysis. It is okay to register an experiment for automated analysis in advance because the analysis will not be attempted until the experiment has actually started.
The experiment was configured in Test Kitchen UI. The machine-readable name of the experiment (specified in the instrumentation specification) is the most important identifier and had to match the experiment's entry in the registry when it was registered for automated analysis.