Test Kitchen/GrowthBook user guide/Configuring experiments

Managing experiments and other assets requires the user to have CustomElevatedAccess role or greater. See Test Kitchen/GrowthBook user guide/Access for more information.

This part of the GrowthBook user guide focuses on creating, starting, and analyzing experiments.

How do I…?

Import an experiment managed externally for analysis

Soon GrowthBook will become the control plane for experiments conducted with Test Kitchen. TK UI will remain available for the few experiments that target constructive edit rate, as analysis of that metric is only available in TK AA. The Experiment Platform team is planning to make constructive edit rate available in GrowthBook in FY26/27 Q1. Also until the team implements a traffic limit exception mechanism (T429648).

GrowthBook allows you to import experiments that are managed outside of GrowthBook but collecting data using Test Kitchen experimentation SDKs or adhering to the system's expectations:

• a well-populated experiment property in the event data, and
• experiment_exposure events produced to product_metrics.web_base stream.

Once the experiment has started collecting data and that data has been processed (approx. 3 hours), you will be able to import it into GrowthBook for analysis:

1. On the Experiments page, click Add > Import Existing Experiment.
2. Click Get New Data if you don't see your experiment in the list. Make sure you're importing from the Presto Analytics Iceberg data source.
3. Select your experiment.
4. Populate the fields (experiment name, description, hypothesis, Phabricator link).
5. Add external imported tag. This will let Test Kitchen's GrowthBook integration layer know to bypass validation of the experiment's configuration that the system performs on all GrowthBook-managed experiments. If you don't add this tag, your imported experiment's status will always be changed to Draft and you will not be able to analyze it.
6. On the Targeting screen, update the variation names (pre-populated from observed experiment.assigned values).

The experiment will only be analyzed if you add metrics. If the metrics you need to add are not defined yet, you can finish importing the experiment and add them later. Follow our guide to metrics in GrowthBook to define your metrics and then add them to the experiment at any time within 90 days of the experiment's start date.

Create/schedule an experiment

STATUS: We recommend configuring your experiment in GrowthBook, unless the experiment requires higher traffic limit or can only be analyzed in TK AA. Currently, experiments configured in TK UI will continue to run, but new experiment creation in TK UI will be disabled in the future.

In Test Kitchen's integration of GrowthBook, experiments cannot be started right away – they can only be scheduled for the a valid deployment window. The Test Kitchen deployment windows are at 14:30 UTC Monday through Thursday, chosen because that is the time most engineers (product and SRE) are likely to be available in case something goes wrong and the experiment is found to be the root cause. This also means that an experiment cannot be scheduled to start on a Friday. If you start or schedule an experiment outside a window, it will be activated during the next window.

To create a new experiment:

Click Create > Experiment on the GrowthBook homepage or from Experimentation > Experiments click Add > Create New Experiment.

Step 1: Specify experiment details

The Overview screen is where you fill out details of your experiment and make it easier for yourself and others to find:

• Project should be Wikimedia.
• Please use a descriptive experiment name. Some guidelines:
  • Tracking Key (or "Experiment Key") identifies your experiment and what you use with getExperiment() in TK SDKs.
  • The key should start with a letter and can only use lowercase letters, numbers, or dashes.
• Please provide a description and, ideally, also your hypothesis for the experiment.
• Make sure to have these tags:
  • A tag for your team, which will help you find all experiments conducted by your team
  • One of these tags: OKR Work or Essential Work – these will help us understand how many experiments are essential work vs in support of objectives and key results
  • One of these tags: aa test or ab test
• If this is an OKR-related experiment, please specify the Related OKRs with format FY##/## OKR #.# (e.g. "FY25/26 SDS 2.3", "FY26/27 DE 1.1").
• Selecting which contextual attributes to collect at experiment level:
  • Reasons for collecting contextual attributes at experiment level:
    • You plan to analyze the experiment manually and need an easy way of segmenting the data.
    • Most metrics don't require contextual attributes but if you plan to analyze a metric which does require a specific attribute for some events but your instrumentation does not record it at event level. For example: Permanent account creation rate requires account_created events to have performer.is_temp but the instrumentation does not record it at event level.
  • Experiment#sendExposure() collects a predefined set of contextual attributes with just experiment exposure events which enable our set of Dimensions. See EXPOSURE_CONTEXTUAL_ATTRIBUTES in Experiments.js for latest list.
• Leave Test Kitchen UI Link field blank since you are creating this experiment in GrowthBook.

Step 2: Specify how to enroll and how many

The Traffic screen is where you choose how to enroll users into the experiment and how many to enroll:

• Assign Variation by Attribute is where you select the identifier type:
  • MW user ID (mw-user in TK UI)
  • Edge Unique (Cache splitting) (edge-unique in TK UI)
  • Edge Unique (Cache Non-splitting) (edge-unique (no cache split) in TK UI)
• For Traffic Percent & Variations:
  • Traffic will be relative to the maximum allowed traffic based on selected identifier type. For example:
    • For MW user ID-based experiments:
      • 100% in GB will mean 100% of logged-in users (temporary & permanent)
      • 50% in GB. will mean 50% of logged-in users (temporary & permanent)
    • For Edge Unique (Cache splitting) and Edge Unique (Cache Non-splitting)-based experiments:
      • 100% in GB will mean 1% on English Wikipedia (if targeted) and 10% on all other wikis.
      • 50% in GB will mean 0.5% on English Wikipedia (if targeted) and 5% on all other wikis.
  • Use descriptive human-friendly variation (group) names, not machine-friendly variation names – you will set those later. For example:
    • Control vs Treatment
    • Dark blue vs Light blue
    • No <feature> vs Has <feature>

Step 3: Specify where to enroll

The targeting screen is where you select which wikis the experiment will be live on. For now you can only select wikis (by specifying MW database codes) but eventually we hope to offer other attributes such as country, platform, or user authentication status. To select which wikis to target:

1. Under Target by Attributes click + Add attribute targeting.
2. Select the attribute MediaWiki Database Code (MW user ID is selected by default)
3. If your experiment will be on:
   • Only one wiki: you can keep is equal to selected
   • Multiple wikis: select is any of instead
4. List the database codes of the target wikis (e.g. "enwiki" for English Wikipedia, "commonswiki" for "Wikimedia Commons")
   • You must select at least one wiki.
   • The maximum number of targetable wikis is 100.

The text input supports pasting an inline comma-separated list (useful if copy-pasting from a Phabricator task) and pasting rows from a spreadsheet (useful if you're experiment planning in Google Docs/Sheets).

Target by Saved Groups is currently not supported in Test Kitchen. We plan to support it in the near future and offer saved groups such as "Top 50 Wikipedias by pages".

Step 4: Specify how to analyze the data

The metrics screen is optional for configuring an experiment. The experiment will only be analyzed if you add metrics. If the metrics you need to add are not defined yet, you can finish creating the experiment and add them later. Follow our guide to metrics in GrowthBook to define your metrics and then add them to the experiment at any time within 90 days of the experiment's start date.

Make sure your experiment has these settings:

• Data Source should be Presto Analytics Iceberg (default)
• Experiment Assignment Table will be Experiment subjects (Identifier type: subject_id)
  • The subject identifier will be derived from the identifier based on the type you selected (e.g. a MW user's global user ID) and the experiment (tracking) key.

Please consult the section below to help you decide which metrics should be goal, secondary, or guardrail.

We recommend not changing any of the Advanced Settings at this time.

Step 5: Complete the Pre-Launch Checklist

The team is currently working on a feature which will automatically check off various tasks on the checklist as part of validating the experiment's configuration (T429420). Until we deploy that feature you will need to manually check off the tasks on the checklist.

Our Pre-Launch Checklist includes:

• The Experiment Key is unique among all existing Test Kitchen UI experiments
• The Experiment Key meets the required pattern: /^[A-Za-z0-9][-_.A-Za-z0-9]{7,62}$/, which just means:
  • Key must start with a letter or a number
  • Key must be at least 7 characters
  • Key cannot be longer than 62 characters
  • Key can only use letters, numbers, and special characters -_.
• A Description has been defined
• The Owner Team has been defined as a tag
• Must edit Variation IDs (defaults to integers) - Must correspond to feature code
• Traffic per wiki has been defined (at least one wiki has been added)
• A specific Risk Level has been defined according to the specific combination of contextual attributes
• A link to Security and Legal Review has been provided (it's required for Medium and High Risk Level)

Step 6: Edit Variation IDs

By default GrowthBook will use integer-based variation IDs, which you will need to update to align with your experiment's instrumentation specification:

1. In the Implementation section, Edit the Variations.
2. Click Switch to advanced mode.
3. Now you can replace the integer-based default values (e.g. 0 for Control) with correct values that correspond to code (e.g. control for Control) – what you use with Experiment#isAssignedGroup().
4. We recommend also adding a Description for each variation.

While you are here in this section, we recommend adding images (e.g. screenshots) of the variations – this will greatly improve your experiment's documentation in GrowthBook and make it easier for others to know what change you tested.

Step 7: Start the experiment

Our integration layer constantly validates all Running experiments' configurations. If at any point your experiment's configuration fails to validate, the experiment's status will change to Draft. We are working on a Phabricator bot which will post on the experiment's task to notify of any validation errors.

Once you have completed all the tasks on the Pre-Launch Checklist, you can Start Experiment.

Please ignore the first item "Add at least one Linked Feature…" as we currently cannot hide/remove it and are waiting for a fix upstream. Until that is fixed, when you click Start Experiment, GrowthBook will warn you that you have some tasks left to complete. Assuming you have done all of the other tasks, you can safely click Start Anyway.

The experiment will launch at the next deployment window (14:30 UTC, chosen to maximize availability of staff in case there are problems). If you start your experiment after 14:30 UTC, the experiment will start the following day (at 14:30 UTC).

Decide which metrics should be goal, secondary, or guardrail

Metric type	Product Analytics glossary	GrowthBook guidance
Goal (Primary)	The main outcome you are trying to impact through your intervention. This is what you are primarily using for evaluating your hypothesis and for deciding whether to deploy an intervention more widely or iterate and test again or abandon the idea entirely.	The primary metrics you are trying to improve with this experiment.
Secondary	Used to learn about additional impact of the intervention in an experiment, but are not primary targets of the intervention. They reveal side effects (both positive and negative) of trying to improve the Primary Metric with the intervention.	Additional metrics to learn about experiment impacts, but not primary objectives.
Guardrail	Metric that we monitor in addition to the Goal Metric(s) and any Secondary Metric(s) to make sure that the intervention we are testing does not have a negative impact in ways we care about – that is, to make sure we are not going off the rails. These are assurances that the intervention does not have unintended negative side effects on the feature or more generally.	Metrics you want to monitor, but are NOT specifically trying to improve.

Best practices: a single goal metric of interest. Everything else should be a secondary metric or a guardrail. The choice of metric should (1) be supported by a theory of change for the product and user behavior, and (2) be sensitive and timely to the planned intervention.

If you have multiple goal metrics (no more than 5), you should consider the potential tradeoffs and try to make them explicit. What if the experiment has a meaningfully positive impact on one primary metric but a meaningfully negative impact on another, rather than positively impacting both?

The more goal metrics you have, the more difficult it becomes to know when to stop your experiment.

Furthermore, our GrowthBook installation uses experiment decision framework which uses goal metrics and guardrail metrics to produce recommendations of decisions. For example, if any of the guardrail metrics are significantly bad (<0.01% Chance To Win) then the system will recommend you to rollback because the treatment is having a statistically significant negative effect on a metric you do not want to negatively affect.