Test Kitchen/Decision Records/Logged in user opt out
Status: Decided, Nov 6, 2025
Decide: Marshall Miller
Agree: Chris Albon, Rita Ho, Kate Zimmerman
Recommend/Drive: Julie van der Hoop
Input: Mikhail Popov (Data Scientist), Experiment Platform Team, Brandon Black, Valentin Gutierrez, Kelsi Stine-Rowe, Eliza Blackorby, Olga Vasileva, Sucheta Salgaonkar, Peter Pelberg, Sonja Perry, Sarai Sanchez, Chris Danis, Stef Dunlap, Pablo Aragon, Legal Team, PTAC members
Perform: Experiment Platform Team
Phabricator: https://phabricator.wikimedia.org/T420655
What we're planning
We're introducing an opt-out preference for A/B testing for logged-in users. This will allow editors to choose whether to participate in product experiments conducted through Test Kitchen, our experimentation platform. This responds to community requests, and aligns with WMF's commitment to transparency and user agency, while maintaining our ability to make evidence-based product improvements.
What this looks like
For logged-in users:
- New preference in user settings: "A/B Testing Participation"
- Clear explanation of how experiments work to improve the platform
- Single, global opt-out affecting all experiments run through Test Kitchen
- Links to public documentation of active experiments
What's included:
- Experiments conducted through Test Kitchen (our primary platform)
- Applies to logged-in users only
What's not included:
- Infrastructure and security-related tests (these cannot be opted out)
- Experiments run on legacy systems (these will migrate to Test Kitchen over time)
- Logged-out users (technically complex; may be considered in future)
Our approach: Default participation with opt-out
We evaluated five options (from opt-in required to disclosure-only). We decided on default participation with opt-out for logged-in users as this balances community trust and data quality best:
- Maintains sufficient sample sizes for valid testing (historical data shows only 0.8% opt-out rates), resulting in lower selection bias than opt-in models
- Most users want experiments to improve the product. Users who care strongly about test enrolment can easily opt out
- Focus on logged-in users leverages existing Preferences infrastructure and serves our most engaged community members who've expressed strongest interest
Communication
We'll announce this through Village Pump, Tech News, and Diff posts. We’ll document on Meta alongside our experiment registry. We are not introducing banners or pop-ups; communication will happen through established community channels where we can provide proper context.
Signals
- How many users navigate to and change these settings
- Opt-out rates and their impact on experiment validity
- Community feedback and concerns
FAQ
Why not include logged-out users? Technical complexity (requires cookie-based tracking) and feedback from PTAC members suggests it's sufficient to focus on editors initially. Logged-out users can create accounts to set Preferences.
What about users already enrolled? The opt-out will take immediate effect, removing users from any active tests.
How is this different from Beta features? Beta features are opt-in early access to specific features you can try. A/B testing is how we decide whether features achieve their intended goals and should be released at all. Opting out of experiments doesn't mean you won't get new features—it means you won't be part of the testing process.