Jump to content

Test Kitchen/GrowthBook user guide

From Wikitech
For users with CustomElevatedAccess role: When viewing an experiment's results, please DO NOT click the Update button if the start date of the experiment was more than 90 days ago as we do not retain raw experiment data longer than 90 days. Triggering an update when some or all of the underlying data has been deleted will overwrite the saved results with incomplete results (if some data still exists) or erase the saved results entirely.

This page provides guidance and resources for learning how to use Test Kitchen's GrowthBook installation (growthbook.wikimedia.org) for exploring and managing experiments.

This is a user-focused guide; GrowthBook system administrators should refer to Data Platform/Systems/GrowthBook.

What is GrowthBook?

GrowthBook is an open source feature flagging and experimentation software integrated into Test Kitchen (but we are only using it for experimentation). Currently our installation only allows analysis of imported experiments that were configured in Test Kitchen UI, but the Experiment Platform team is working to make it the way to configure experiments.

Our installation of GrowthBook is entirely on-premises – GrowthBook queries our Data Lake using our Presto cluster and then analyzes the data in a statistics engine running on our servers.

How do I…?

Access GrowthBook

Refer to Test Kitchen/GrowthBook user guide/Access for a comprehensive guide on this topic.

View and understand experiment results

We have a 4 minute introductory video available for WMF and WMDE staff (with requests).

Refer to docs.growthbook.io/using/experimenting#understanding-results for a comprehensive guide on this topic.

Our GrowthBook installation calculates and reports Bayesian results by default and we do not recommend switching to frequentist. You can learn more about our rationale at Test Kitchen/Decision Records/Keep Bayesian engine as default in GrowthBook.

Share results

If you use the built-in Share feature, know that only authorized users will be able to view the results even if you select "Anyone with the link" under "View access" because only authorized users can access growthbook.wikimedia.org.

The Experiment Platform team is considering solutions for easily sharing/publishing experiment results. In the meantime, you may use the Export CSV feature to download the experiment results. Please follow the data publication guidelines if you wish to share the results publicly.

Exclude bots from results

For logged-in subjects (permanent users and temporary users) in web experiments we already exclude known bots (as determined by membership in bot user group at the time of an exposure).[1]

For logged-out subjects we do not currently perform any form of bot / automated agent detection, but we know that web experiment owners would like to have that as a dimension to see results differ between "likely human" vs "likely automated" vs "hard to tell" (for example) and what the proportion of each is in their experiment.

Dig deeper with dimensions

GrowthBook lets you break down results by a dimension, which you can select under Unit Dimension dropdown.

Unit Dimensions available for app experiments
Dimension Values (user segments)
app_platform Android, iOS
app_theme Light, Sepia, Dark, Black
app_primary_language_code en, zh-cn, es, etc.
user_auth_status_first_exposure Logged-out user, Temporary user, Permanent user

Status observed when the app install was first exposed to a variation in the experiment.

user_auth_status_highest_observed_90d Logged-out user, Temporary user, Permanent user

Highest status observed in the last 90 days, where: Permanent user > Temporary user > Logged-out user

country_name United States, Germany, etc.
country_code US, DE, etc.
Unit Dimensions available for web experiments
Dimension Values (user segments)
project_family Wikipedia, etc.
language English, French, etc.
wiki_id enwiki, frwiki, etc.
wiki_name English Wikipedia, French Wikipedia, etc.
user_platform Desktop, Mobile
user_auth_status_first_exposure Logged-out user, Temporary user, Permanent user

Status observed when subject was first exposed to a variation in the experiment.

user_auth_status_highest_observed Logged-out user, Temporary user, Permanent user

Highest status observed during the experiment, where: Permanent user > Temporary user > Logged-out user

user_100_edit_threshold_first_exposure 0 edits, 1-100 edits, >100 edits
user_platform_auth_status_100_edit_threshold_first_exposure Yields about 16 possible user segments such as
  • Desktop, Permanent user, >100 edits
  • Mobile, Temporary user, 1-100 edits
  • Mobile, Logged-out user, 0 edits

Note: Implemented in service of FY26/27 DE 1.1 KR which targets registered (permanent) users with ≤100 cumulative edits on mobile web.

To request more dimensions contact the Experiment Platform team's Product Manager.

Compare treatments

For experiments with multiple treatments (e.g. A/B/C tests) results are initially shown relative to control (no treatment). To compare one treatment against another when viewing results of a multi-treatment experiment:

  • When you view the results of an A/B/C test, the Control group is the default baseline.
  • Use the dropdown where it says Control to select a different baseline.
    Use the dropdown where it says Control to select a different baseline.
  • Use the other dropdown to deselect Control as one of the variations to show.
    Use the other dropdown to deselect Control as one of the variations to show.
  • Congratulations! You are now comparing the second treatment against the first.
    Congratulations! You are now comparing the second treatment against the first.

This is especially useful if the tested treatments have treatment-only metrics that are not applicable to the control group, such as when a new feature (rather than a variation on an existing feature) is being introduced.

Interpret paradoxical results

It is possible to encounter results which may seem paradoxical. As a concrete example, in June 2026 the Reader Experience ran an A/B/C test to test two variations of badges for donors – a simple version and a fancy version – on English and German Wikipedias. Midway through the experiment we temporarily saw counter-intuitive results for the goal metric (21-day reader retention rate): Treatment B (simple version) had slightly higher variation value than Treatment C (fancy version), but also a lower estimated lift and lower chance to win.

This was due to the Post-Stratification feature of GrowthBook, which is a variance reduction technique that's meant to increase sensitivity (make it easier to detect small effects). With post-stratification (enabled by default in our installation), the weighted average gives greater weight to observations from the strata that are underrepresented in the sample. If a sample is imbalanced for some covariate such as wiki, the weighted average estimate would correct for it.[2]

When we dug deeper with dimensions and examined the results broken down by language, we saw that C had a greater lift than B for German Wikipedia readers & donors and a smaller lift than C for English Wikipedia readers & donors. However, despite only enrolling 1% of English Wikipedia visitors and 10% of German Wikipedia visitors, the English stratum was overrepresented in the sample. Post-Stratification gave German stratum's lift greater weight in the estimation of overall lift, thus making the estimated lift higher for C than B, even though B's variation value for the metric was slightly higher than C.

Create/import and analyze an experiment

Refer to Test Kitchen/GrowthBook user guide/Configuring experiments for a comprehensive guide on this topic.

Assess experiment's health

A report on the experiment's health will only be available once the experiment has been analyzed using at least one metric. GrowthBook automatically performs a number of data quality checks, which you can learn more about at docs.growthbook.io/using/experimenting#health-page.

Experiments with very large sample sizes (e.g. 10M users) are likely to have Sample Ratio Mismatches (SRM) detected even when the split appears to be nearly perfect. The very large sample size shrinks the p-value of the SRM check so much that it produces a false positive.

Stop an experiment

Most experiments should not be longer than 30 days and Test Kitchen automatically stops an experiment after 90 days. (Data is automatically deleted after 90 days due to the data retention guidelines.)

When you are ready to stop your experiment, there are two ways:

  • Stop Experiment button is our recommended way because you have to pick a Conclusion (Did Not Finish, Inconclusive, Lost, Won) and write a short blurb. Remember: there are no losers in A/B testing, only learnings.
  • ⋮ → Edit status link lets you change the status from Running to Stopped. The experiment will have status "Stopped: Awaiting decision" and you will be able to pick a conclusion and write a short blurb at a later time by visiting the experiment page and clicking the pencil icon in the "Experiment stopped" bar.

Know when to stop the experiment

Refer to docs.growthbook.io/using/experimenting#deciding-ab-test-results.

Our installation uses the Experiment Decision Framework feature to generate recommendations of what decisions to make when the experiment has collected enough data for the results to be reliable:

  • ship the treatment when it is clear that the treatment won
  • roll back when it is clear that the treatment lost
  • review the results

If the experiment has not reached target power, the system estimates how many more days are needed until the results are reliable. The estimation is based on target MDEs of the goal metrics, how many goal metrics there are (the fewer the better), and how many new subjects the experiment acquires daily.

Define a new metric

Refer to Test Kitchen/GrowthBook user guide/Metrics for a comprehensive guide on this topic.

Organize information and assets

GrowthBook has two systems for organizing information: Projects and Tags.

Projects

The Experiment Platform team reserves the use of Projects, and there is just one: Wikimedia. All assets (fact tables, metrics, and experiments) must be under the Wikimedia project.

In the future we may use other projects to isolate data sources, assets, and access – such as a separate project for Fundraising.

Tags

Test Kitchen uses tags to filter experiments for validation of configuration. For experiments managed outside of GrowthBook (e.g. via Test Kitchen UI) and imported for analysis:

  • The special external imported tag allows the experiment to bypass validation.
  • If an imported experiment is not tagged as external imported and does not have a valid configuration in GrowthBook, the system will change its status from Running to Draft.

We also use some optional tags to help us organize experiments and make them easier to filter and find:

  • Like in Phabricator, the tags Essential Work and OKR Work make it easier to find which experiments supported objectives and key results or were essential work.
    • For OKR Work-tagged experiments, please provide Related OKRs in the experiment overview using the format FY##/## OKR #.# (e.g. FY25/26 SDS 2.3)
  • The tags ab test and aa test make it easier to filter out various A/A tests present in the system (logged-out web reader retention rate measurements, synthetic tests) to list just controlled experiments.

See also

References

  1. ↑ Data_Platform/Systems/GrowthBook#Experiment_assignment_query
  2. ↑ Huizhi Xie and Juliette Aurisset. 2016. Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '16). Association for Computing Machinery, New York, NY, USA, 645–654. https://doi.org/10.1145/2939672.2939733
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Test_Kitchen/GrowthBook_user_guide&oldid=2452568"