Test Kitchen/Decision Records/Keep Bayesian engine as default in GrowthBook

Status: Accepted

Author: Mikhail Popov

Deciders: Mikhail Popov, Julie van der Hoop

Date authored: 2026-05-22

Context: By the end of June 2026, experiments conducted with Test Kitchen are expected to be analyzed in GrowthBook (replacing Test Kitchen Automated Analytics MVP). The GrowthBook software allows for experiments to be analyzed using two different statistical inference methodologies: Bayesian and frequentist. The software ships with Bayesian as the default statistical engine. Administrators may change the organizational default to be the Frequentist engine. Individual experiment owners may also change their experiment's engine to be different than the organizational default.

Problem Statement: Which statistical engine should Test Kitchen's GrowthBook instance use as the organizational default?

Decision Outcome: We are keeping Bayesian engine as the organizational default, and while experiment owners may switch engines for their specific experiment's analysis, we recommend they stick with the default.

Rationale: GrowthBook documentation includes the following rationale for setting Bayesian engine as the default, and we agree with it:

Bayesian methods provides a few key advantages over frequentist approaches.

First, they provide more intuitive results. Instead of p-values and confidence intervals, you get probabilities and distributions of likely outcomes. These values allow you to make statements like "there’s a 95% chance this new button is better and a 5% chance it’s worse" while there is no direct analog in a frequentist framework.

Second, Bayesian methods allow you to write down your prior knowledge about experiment effects to ensure that you do not over-interpret small sample sizes and to benefit from old knowledge to reduce uncertainty in new experiments.

Third, Bayesian results are still valid even if you stop an experiment early. While they can suffer from the same "peeking" problems as frequentist statistics, at least the main probabilities and statistical results that you see are not invalidated by stopping early. However, this is something of a difference without a distinction, as the decision to stop an experiment early can still result in inflated false positive rates.

Furthermore, p-values in frequentist statistics are so often misused or misunderstood that there is a Wikipedia article on the phenomenon, and in our experience there is a lot of critical nuance that people who engage with statistical analysis results are unaware of.

Bayesian methods do not have the multiple comparisons problem. When an experiment with multiple variations (2+ treatments) is analyzed using frequentist methodology and the variations are compared not just with the baseline (control) but with each other, there is a greater probability of a false discovery. False discoveries – statistically significant effects when the true effect is null – pose a risk in two ways:

• There may be costs associated with switching (shipping/deploying the winning treatment to all users) and we rely on the discovery that the benefit – expected impact – outweighs the cost of switching.
• As Berman et al. (2022) write "deploying false discoveries may harm efforts to instill a test-and-learn culture and the analytics team may worry about their credibility within the organization."

Finally, the Bayesian analysis approach enables us to have informative priors to provide additional information to the analysis and what kinds of results are likely based on what we have observed or what we know or believe. As GrowthBook's documentation correctly states:

Priors can be very helpful in reducing uncertainty in small sample sizes and in ensuring that you do not over-interpret results that are unlikely to be reliable.

GrowthBook ships with improper, uninformative priors as the default for the Bayesian engine, so enabling and specifying proper priors is outside the scope of this decision and will be decided on separately (T426619).