Experimentation Lab/Decision Records/Experiment end behaviour

Context: When an experiment reaches its end date, the treatment ends, and all users will go back to the original experience. Will that change in experience for the treatment group be jarring? For small enough changes (e.g. button colour) users in the treatment group may never notice. But consider larger initiatives, e.g. reading lists: users in the treatment group will have created reading lists, added content, and then the feature disappears, only to return weeks later if/when we officially launch reading lists.

Key Question: Should experiments be allowed to continue running after their scheduled end date when the treatment significantly impacts user experience, in order to maintain continuity?

Main Considerations:

Community Engagement: The period between experiment completion and final implementation decisions is key for community engagement. There needs to be time built in for community feedback, statistical reporting, and collaborative discussion of results between experiment end and making shipping decision.

Technical Feasibility: The platform can support end-date extension, but the team currently has to modify end dates (it is blocked in the UI). A task has been created to allow experiment owners to change the end date on their own.

Risks: Introducing a mechanism to extend experiment end dates could unintentionally delay decision-making, slow shipping timelines, and potentially lead to different conclusions as new data come in. To mitigate these risks, we'll want to be clear on decision-points and next steps needed to release.

Automated Analysis: The automated analytics system would continue to analyze the extended experiment, overwriting the results that were used to make the decision. We do not expect the results (and conclusions drawn from them) to change so drastically as to invalidate the decision, but the statistical results (e.g. estimated lift, Chance To Win, p-value, credible/confidence interval) at the time of the decision should be included in the decision record and that snapshot should be treated as the source of truth for the experiment results.

Decision Outcome: The Experiment Platform will allow end-users to extend experiment duration at the decision point, to maintain treatment consistency and avoid disrupting user experience while community consultation occurs. Extensions should be considered when treatments are substantial enough to significantly alter user experience, with the scope and circumstances of each experiment factored into the community consultation approach.

Deciders: Julie van der Hoop, Marshall Miller, Mikhail Popov

Consulted: Clare Ming, Santiago Faci, Sam Smith, Dan Andreescu, Steph Toyofuku, Katy Graessle, Adam Baso, Chris Albon, Rita Ho, Nat Baca

Date Decided: 2025-09-08

Date authored: 2025-09-15

Phabricator: T402308