Test Kitchen/GrowthBook user guide/Configuring experiments

Managing experiments and other assets requires the user to have CustomElevatedAccess role or greater. See Test Kitchen/GrowthBook user guide/Access for more information on roles and how to request access.

This part of the GrowthBook user guide focuses on creating, starting, and analyzing experiments.

How do I…?

Decide which metrics should be goal, secondary, or guardrail

Metric type	Product Analytics glossary	GrowthBook guidance
Goal (Primary)	The main outcome you are trying to impact through your intervention. This is what you are primarily using for evaluating your hypothesis and for deciding whether to deploy an intervention more widely or iterate and test again or abandon the idea entirely.	The primary metrics you are trying to improve with this experiment.
Secondary	Used to learn about additional impact of the intervention in an experiment, but are not primary targets of the intervention. They reveal side effects (both positive and negative) of trying to improve the Primary Metric with the intervention.	Additional metrics to learn about experiment impacts, but not primary objectives.
Guardrail	Metric that we monitor in addition to the Goal Metric(s) and any Secondary Metric(s) to make sure that the intervention we are testing does not have a negative impact in ways we care about – that is, to make sure we are not going off the rails. These are assurances that the intervention does not have unintended negative side effects on the feature or more generally.	Metrics you want to monitor, but are NOT specifically trying to improve.

Best practices: a single goal metric of interest. Everything else should be a secondary metric or a guardrail. The choice of metric should (1) be supported by a theory of change for the product and user behavior, and (2) be sensitive and timely to the planned intervention.

If you have multiple goal metrics (no more than 5), you should consider the potential tradeoffs and try to make them explicit. What if the experiment has a meaningfully positive impact on one primary metric but a meaningfully negative impact on another, rather than positively impacting both?

The more goal metrics you have, the more difficult it becomes to know when to stop your experiment.

Furthermore, our GrowthBook installation uses experiment decision framework which uses goal metrics and guardrail metrics to produce recommendations of decisions. For example, if any of the guardrail metrics are significantly bad (<0.01% Chance To Win) then the system will recommend you to rollback because the treatment is having a statistically significant negative effect on a metric you do not want to negatively affect.

Create an experiment

This section has been updated for the new experiment creation flow in GrowthBook 5.0. If you prefer the pre-5.0 flow, click Switch to old experience (not recommended) and refer to an older revision of this section.

To create a new experiment, click Create > Experiment on the GrowthBook homepage or from Experimentation > Experiments click Add > Create New Experiment.

If you used our experimentation templates, you should already have a Phabricator Task or a Google Doc with all of the necessary details so most of this is just copying those into GrowthBook.

Step 1: Specify experiment details

Please provide a description and, ideally, also your hypothesis for the experiment.

1. Please use a descriptive experiment name that makes it clear what change is being tested.
2. Project should be Wikimedia.
3. The hypothesis should be specific to the change you are testing with the experiment.
   • This is not necessarily the same as the hypothesis in the P&T OKR framework.
   • If the experiment is an A/A test, there isn't a hypothesis being tested so leave it blank. You will have a chance to add a Description on the Overview page.
4. For Assignment Attribute:
   • MW user ID is for conducting the experiment on all logged-in users, which includes both Permanent users and Temporary users.
   • Edge Unique (Cache splitting) is for conducting the experiment on all traffic, which includes logged-in and logged-out users. Since splitting the cache creates stress for our caching infrastructure, these experiments have low traffic limits.
   • Edge Unique (Cache Non-splitting) is same as the other option, but does not split the cache. Page contents cannot be varied with PHP SDK, only JS SDK. Since this option does not create stress for our caching infrastructure, these experiments have higher traffic limits.
5. If this is an OKR-related experiment, please specify the Related OKRs with format FY##/## OKR #.# (e.g. "FY25/26 SDS 2.3", "FY26/27 DE 1.1").
6. Selecting which contextual attributes to collect at experiment level:
   • Reasons for collecting contextual attributes at experiment level:
     • You plan to analyze the experiment manually and need an easy way of segmenting the data.
     • Most metrics don't require contextual attributes but if you plan to analyze a metric which does require a specific attribute for some events but your instrumentation does not record it at event level. For example: Permanent account creation rate requires account_created events to have performer.is_temp but the instrumentation does not record it at event level.
   • Experiment#sendExposure() collects a predefined set of contextual attributes with just experiment exposure events which enable our set of Dimensions. See EXPOSURE_CONTEXTUAL_ATTRIBUTES in Experiments.js for latest list.
7. For Risk Level and Security and Legal Review, please refer to Test Kitchen/Regulation section.
8. Leave Test Kitchen UI Link field blank since you are creating this experiment in GrowthBook.

As of GrowthBook 5.0, the rest of the steps have been moved to the Overview page.

Step 2: Update Experiment Key

GrowthBook 5.0+ automatically generates an Experiment Key based on the experiment name. Please review it (at the top) and change if needed, making sure to match the experiment key in your experiment plan and/or instrumentation specification.

Experiment Key identifies your experiment and what you use with getExperiment() in TK SDKs.

Some guidelines:

• The key should start with a letter and can only use lowercase letters, numbers, or dashes.
• The key must match the experiment name in code.

Step 3: Update Variations

In the Implementation section, click Pencil button to open the Edit Traffic & Variations modal. Now you can edit variation names and add descriptions.

In general we recommend using descriptive names like "Redesigned feed" and "Shown banner", but if the experiment name is clear and descriptive enough about the change being tested, generic names like "Control" and "Treatment" also work.

In Test Kitchen we use variation IDs like control and treatment for readability, but GrowthBook defaults to integer-based IDs (e.g. 0, 1 which must be manually changed.

To update the Variation IDs:

1. Click Switch to advanced mode (in the Edit Traffic & Variations modal).
2. Now you can replace the integer-based default values (e.g. 0 for Control) with correct values that correspond to code (e.g. control for Control) – what you use in code with Experiment#isAssignedGroup().
3. We recommend also adding a Description for each variation while you're here.

We recommend adding images (e.g. screenshots) of the variations – this will greatly improve your experiment's documentation in GrowthBook and make it easier for others to know what change you tested.

Step 4: Update Traffic Allocation

Test Kitchen currently only supports targeting wikis, but the Experiment Platform team plans to offer more targeting attributes in the future.

To select which wikis your experiment will be active on:

1. In the Implementation > Targeting section, click the Pencil button to open the Edit Targeting modal.
2. In Target by Attributes, click Add attribute targeting.
3. The INCLUDE IF rule should be:
   • MediaWiki database code
   • is any of
   • Your target wikis (e.g. "enwiki" for English Wikipedia, "commonswiki" for Wikimedia Commons)

You must select at least one wiki, and the maximum number of targetable wikis is 100.

The text input supports pasting an inline comma-separated list (useful if copy-pasting from a Phabricator task) and pasting rows from a spreadsheet (useful if you're experiment planning in Google Docs/Sheets).

By default, experiments use 100% of traffic, which is always relative to maximum allowed traffic the selected Assignment Attribute.

For example:

• For MW user ID-based experiments:
  • 100% in GB will mean 100% of logged-in users (temporary & permanent)
  • 50% in GB. will mean 50% of logged-in users (temporary & permanent)
• For Edge Unique (Cache splitting) and Edge Unique (Cache Non-splitting)-based experiments:
  • 100% in GB will mean 1% on English Wikipedia (if targeted) and 10% on all other wikis.
  • 50% in GB will mean 0.5% on English Wikipedia (if targeted) and 5% on all other wikis.

Step 5: Add Tags

We use Tags to organize experiments and make them easy to filter and find, so please add:

• A tag for your team, which will help you find all experiments conducted by your team
• One of these tags: OKR Work or Essential Work – these will help us understand how many experiments are essential work vs in support of objectives and key results
• One of these tags: aa test or ab test

Step 6: Schedule and start the experiment

In Test Kitchen's integration of GrowthBook, experiments cannot be started right away – they can only be scheduled for the a valid deployment window. The Test Kitchen deployment windows are at 14:30 UTC Monday through Thursday, chosen because that is the time most engineers (product and SRE) are likely to be available in case something goes wrong and the experiment is found to be the root cause. This also means that an experiment cannot be scheduled to start on a Friday. If you start or schedule an experiment outside a window, it will be activated during the next window.

Click + Add Schedule and pick a start date and time. Once you have scheduled your experiment, click Approve for Scheduled Start at the top.

Please ignore the first item "Add at least one Linked Feature…" as we currently cannot hide/remove it and are waiting for a fix upstream. Until that is fixed, when you click Start Experiment, GrowthBook will warn you that you have some tasks left to complete. Assuming you have done all of the other tasks, you can safely click Start Anyway.

Avoid starting the experiment on Thursday if the code for it is riding the MediaWiki deployment train that week, since Group 2 wikis (e.g. English Wikipedia) get deployed to sometime on Thursday – usually after the 14:30 UTC Test Kitchen deployment/activation window.

Step 7: Update Analysis Settings

We do not recommend modifying Statistics Engine, CUPED, Post-Stratification settings or any of the Advanced Settings at this time.

The experiment will only be analyzed if you add metrics. If the metrics you need to add are not defined yet, you can finish creating the experiment and add them later. Follow our guide to metrics in GrowthBook to define your metrics and then add them to the experiment at any time within 90 days of the experiment's start date.

Make sure your experiment has these settings:

• Data Source should be Presto Analytics Iceberg (default)
• Experiment Assignment Table will be Web experiments (Identifier type: subject_id)
• Activation Metric should be None. (Because experiments should only be logging exposure when user is actually exposed to a variation.)

When you select a Data Source in the Analysis Settings modal and see the field for Tracking Key, that is an opportunity to verify that your experiment in GrowthBook is using the same experiment key as what you have in your Phabricator task or Google Doc (created with our experimentation templates). See also Step 2.

Please consult the section at the top to help you decide which metrics should be goal, secondary, or guardrail.

Check allotted traffic

Once your experiment has started, you can check that the allotted traffic is accurate by checking the API responses associated with your experiment.

If it's an edge-unique experiment, look for your experiment name as a key in https://test-kitchen.wikimedia.org/api/v1/experiments?format=config&authority=varnish

For logged-in experiments, check https://test-kitchen.wikimedia.org/api/v1/experiments

Import an externally managed experiment for analysis

Web experiments should be managed entirely in GrowthBook (see section below on creating experiments), so this section is now exclusively about importing app experiments for analysis.

We have a video demonstration/walkthrough of the import process for reference, for Wikimedia Foundation staff only.

GrowthBook allows you to import experiments that are managed outside of GrowthBook but collecting data adhering to the system's expectations.

Once the experiment has started collecting data and that data has been processed (approx. 3 hours), you will be able to import it into GrowthBook for analysis:

1. On the Experiments page, click Add > Import Existing Experiment.
2. Click Get New Data if you don't see your experiment in the list. Make sure you're importing from the Presto Analytics Iceberg data source.
3. Select your experiment.
4. Populate the fields (experiment name, description, hypothesis, Phabricator link).
5. Add external imported tag. This will let Test Kitchen's GrowthBook integration layer know to bypass validation of the experiment's configuration that the system performs on all GrowthBook-managed experiments. If you don't add this tag, your imported experiment's status will always be changed to Draft and you will not be able to analyze it.
6. On the Targeting screen, update the variation names (pre-populated from observed experiment.assigned values from experiment_exposure events).

The experiment will only be analyzed if you add metrics. If the metrics you need to add are not defined yet, you can finish importing the experiment and add them later. Follow our guide to metrics in GrowthBook to define your metrics and then add them to the experiment at any time within 90 days of the experiment's start date. Please refer to the section below to help you choose which metrics should be goal, secondary, or guardrail.

Resolve validation errors

When you save or start an experiment, inline validation checks are run in GrowthBook. If a check fails, GrowthBook shows a message and either blocks the action or flags it:

• Errors (blocking): the experiment cannot be saved or started until you fix it.
• Warnings (non-blocking): the experiment can still start but the flagged issue should be addressed.

These checks are the only validation of your experiment. They run in GrowthBook as you save and start your experiment. Every error needs to be resolved before starting an experiment.

Some checks run while the experiment is in draft mode. Other checks run only when the experiment is started. An experiment that saves cleanly as a draft can still be blocked at launch.

Test Kitchen reads experiments from GrowthBook, applies deployment-window date normalization, and serves them.

Validation errors: quick reference

#	Message (verbatim)	Type	Fires on	Section
1	The Experiment Key must start with a letter or number; contain only letters, numbers, hyphens, underscores, or periods; and must be between 8 and 63 characters long	Error	Draft + Start	Tracking key
2	variation ids must be changed from the default integers	Error	Start	Variations
3	variation ids must start with a letter or number; contain only letters, numbers, hyphens, underscores, or periods; and must be between 1 and 63 characters long	Error	Draft + Start	Variations
4	Missing Description and Hypothesis	Warning	Start	Details
5	must be tagged OKR Work or Essential Work	Warning	Start	Tags
6	must have a team tag	Warning	Start	Tags
7	Risk Level is Risk assessment pending: experiments cannot run until resolved	Error	Start	Regulation
8	A Security Review URL is required when Risk Level is <level>	Error	Draft + Start	Regulation
9	risk_level <declared> is lower than <required> required by the selected contextual attributes	Error	Draft + Start	Regulation
10	wiki count must be 1–100, got <n>	Error	Draft + Start	Targeting (wikis)
11	invalid wiki IDs: <ids>	Error	Draft + Start	Targeting (wikis)
12	Invalid operators: When adding wikis the only valid operators are "is equal to" and "is any of"	Error	Draft + Start	Targeting (wikis)
13	You must select a datasource and Experiment Assignment Table for this experiment	Error	Start	Analysis settings
14	You have not added yet a goal metric	Warning	Start	Metrics

Fix each validation failure

Tracking key

1. "The Experiment Key must start with a letter or number; contain only letters, numbers, hyphens, underscores, or periods; and must be between 8 and 63 characters long"

• Why: The experiment's tracking key (slug) doesn't match the required pattern. It must be 8–63 characters, start with a letter or number, and use only letters, numbers, hyphens (-), underscores (_), or periods (.) — no spaces or other symbols.
• Fix:
  1. Open the experiment in GrowthBook.
  2. Edit the key field — labeled Tracking Key in the new-experiment form, and Experiment Key in the "Edit info" modal of an existing experiment.
  3. Enter a value like mobile-font-size-v2 (≥ 8 chars, valid characters only). Save.
• Good: desktop-search-v1, android.toolbar_test, readmore_2026
• Bad: test (too short), my experiment (space), 2 (too short / integer-like).

Variations

2. "variation ids must be changed from the default integers"

• Why: GrowthBook seeds every new variation with an integer key (0, 1, 2, …). A bare integer means the variation hasn't been mapped to its feature code yet. This check fires when you start the experiment and blocks launch if any variation key is still a default integer.
• Fix:
  1. Open the experiment and go to the Variations section; click Edit Variations.
  2. Each variation row has a Key column — this is the variation id the hook checks.
  3. If the Key fields aren't editable, click the Switch to advanced mode link (it reads Advanced mode once enabled) to turn on id editing.
  4. Change each variation's Key from the default integer to the identifier your client/feature code checks for — e.g. control, treatment, variant-a. (The adjacent Variation Name is just a human label and does not satisfy this check — you must change the Key.)
  5. Save, then start the experiment again.
• Note: The hook can only confirm the keys are not bare integers; it cannot verify they match your code. Make sure the keys you set are exactly what your instrument/feature reads.

3. "variation ids must start with a letter or number; contain only letters, numbers, hyphens, underscores, or periods; and must be between 1 and 63 characters long"

• Why: A variation key is malformed — it contains a space or illegal character, is empty, or is over 63 characters. (This is the format rule; it runs in both draft and at start. Error #2 is the separate "still an integer" rule.) The pattern is enforced because the edge/Varnish layer expects well-formed keys.
• Fix:
  1. Variations → Edit Variations (use Switch to advanced mode if the Key fields aren't editable, as in #2).
  2. Rename each Key to 1–63 characters, starting with a letter or number, using only letters, numbers, -, _, ..
  3. Save.
• Good: control, treatment, variant-a, test_1, v2.0
• Bad: variant a (space), empty, *treatment (leading symbol).

Details & tags

(These are warnings — they don't block launch, but resolve them for governance/quality.)

4. "Missing Description and Hypothesis" (warning)

• Why: At start time, both the Description and the Hypothesis are empty. You need at least one of them.
• Fix: Open the experiment and fill in the Description and/or Hypothesis field. Having either one clears the warning.

5. "must be tagged OKR Work or Essential Work" (warning)

• Why: Every running experiment must declare how it's funded with a type-of-work tag: exactly one of OKR Work (maps to a specific objective) or Essential Work (baseline upkeep/maintenance).
• Fix:
  1. Open the experiment's Tags field.
  2. Add the tag OKR Work or Essential Work (exact text, case-sensitive).
  3. If OKR Work, record the specific objective(s) in the related_okrs custom field (WMF custom-field display name — confirm against the live instance).

6. "must have a team tag" (warning)

• Why: Beyond the type-of-work tag (#5), a running experiment also needs a team tag identifying the owning team. Practically this means you need at least two tags: one type-of-work tag + one team tag. (The hook detects this as "only the OKR/Essential tag present, or no tags at all.")
• Fix: In the Tags field, add your team's tag (e.g. Web, Android, Growth) in addition to the OKR Work/Essential Work tag.

Regulation (risk & privacy)

7. "Risk Level is Risk assessment pending: experiments cannot run until resolved"

• Why: The Risk Level custom field is still at its GrowthBook default, Risk assessment pending. An unresolved risk assessment blocks launch.
• Fix:
  1. Open the experiment's Risk Level custom field.
  2. Change it from Risk assessment pending to one of:
     • Tier 1: High risk
     • Tier 2: Medium risk
     • Tier 3: Low risk
  3. Pick the tier that matches the attributes you collect (see #9). Save and start again.

8. "A Security Review URL is required when Risk Level is <level>"

• Why: You set Risk Level to Tier 1 (High) or Tier 2 (Medium) but left the Security Review URL empty. Medium/high-risk experiments require a link to the security & legal review. (Low risk does not.)
• Fix:
  1. Open the Security Review URL custom field.
  2. Paste the URL of the completed security/legal review (ticket or doc). Save.
  3. If a review hasn't happened yet, start one before launching. Or if the experiment is genuinely low-risk and collects no sensitive attributes, correct the Risk Level to Tier 3: Low risk.

9. "risk_level <declared> is lower than <required> required by the selected contextual attributes"

• Why: The contextual attributes you're collecting demand a higher minimum risk level than you declared. Risk must be proportional to privacy sensitivity. The minimum is computed from which categories of attribute you selected:

Attribute categories selected	Minimum risk level
Only one category	Tier 3: Low risk
Personal Info + Unique ID	Tier 2: Medium risk
Long-term Viewing History + Unique ID	Tier 2: Medium risk
Personal Info + Unique ID + Long-term Viewing History	Tier 1: High risk

Categories (current attribute → category mapping):

• Personal Information: agent_ua_string
• Unique ID: performer_id, performer_name, agent_app_install_id
• Long-term Viewing History: page_id, page_title
• App schemas automatically include agent_app_install_id (a Unique ID attribute), which can raise the required tier even if you didn't select it explicitly.

• Fix (choose one):
  1. Raise the risk level to meet or exceed the required tier from the table (then satisfy #8 if that pushes you to Tier 1/2), or
  2. Remove the sensitive contextual attributes you don't actually need, so the required minimum drops.
• Reference: Contextual attributes § Privacy considerations

Targeting (wikis)

10. "wiki count must be 1–100, got <n>"

• Why: Your targeting condition resolves to 0 wikis or more than 100. An experiment must target 1–100 wikis (duplicates are de-duplicated before counting).
• Fix: Edit the phase Targeting condition on the MediaWiki database code attribute so it lists between 1 and 100 wikis. If you need a broader rollout, split it or talk to the Test Kitchen team.

11. "invalid wiki IDs: <ids>"

• Why: The targeting condition contains wiki codes that aren't valid MediaWiki database codes. The message lists the offending codes.
• Fix:
  1. Edit the Targeting condition.
  2. Replace each listed code with a valid database code (e.g. enwiki, frwiki, eswiki, jawiki, commonswiki). Check Special:SiteMatrix / the database-code list if unsure.
  3. Save. (The valid-wiki list is maintained in the hook from wikis.tsv; if a legitimately new wiki is rejected, the list may need npm run load:wikis — flag the Test Kitchen team.)

12. "Invalid operators: When adding wikis the only valid operators are "is equal to" and "is any of""

• Why: Wiki targeting only supports two operators. You picked a different one from the condition dropdown (e.g. is not equal to, is none of, is greater than, matches regex, etc.).
• Fix: In the Targeting builder for the MediaWiki database code attribute, set the operator dropdown to only one of:
  • is equal to — a single wiki, e.g. eswiki
  • is any of — multiple wikis, e.g. enwiki, frwiki
  • (These are the exact operator labels in the GrowthBook condition dropdown; they map to $eq and $in.)
• JSON equivalents: {"MediaWiki database code": "eswiki"} or {"MediaWiki database code": {"$in": ["enwiki", "frwiki"]}}

Analysis settings

13. "You must select a datasource and Experiment Assignment Table for this experiment"

• Why: When you start the experiment, no Data Source is selected. GrowthBook needs a Data Source (and its Experiment Assignment Table) to query exposure data and analyze results.
• Fix: In the experiment's Analysis Settings, set the Data Source (e.g. Presto Analytics Iceberg) and the Experiment Assignment Table (e.g. Web experiments). Save and start again (GrowthBook won't let you pick a Data Source without also picking an Assignment Table, so setting the Data Source resolves both).

Metrics

14. "You have not yet added a goal metric" (warning)

• Why: The experiment has no goal metric. This is a warning, not a blocker. You can start without one and add it later. A goal metric is required before results can be analyzed.
• Fix: Add at least one metric under Goal Metrics in the experiment's metrics section. It can be added after launch, but add it before you expect to read results.