Jump to content

Test Kitchen/Experimenting with Special:CreateAccount

From Wikitech
Although this page was written about Special:CreateAccount, the patterns apply to Special:Login as well since both are under the auth domain.

This page captures patterns and tricks for conducting experiments on Special:CreateAccount, which falls in the intersection of two highly complex and constrained systems: Single Unified Login (SUL) and Edge Uniques.

Why it is challenging

When a logged out user clicks Create account (on desktop website) or ≡ (hamburger menu) → Log in → Join Wikipedia (on mobile website), they are redirected to auth.wikimedia.org/{dbcode}/wiki/Special:CreateAccount. See Auth.wikimedia.org for more information.

Edge Uniques does not across 2LD boundary, so a user who came from a Wikipedia and landed on that account creation page would have two wmf-uniq cookies:

  • One scoped to *.wikipedia.org
  • One scoped to auth.wikimedia.org because each wikimedia.org subdomain is handled separately (e.g. meta.wikimedia.org is a separate cookie from commons.wikimedia.org)

Therefore, it is impossible to target both a language of Wikipedia AND the auth domain with the same experiment and have consistent edge unique-based enrollment across domains.

In addition to enrollment challenges, to analyze the experiment and the target metric we need:

  • An experiment_exposure event (usually produced by Experiment#sendExposure()) when the user is exposed to a variation of the Special:CreateAccount page, which only happens when the user lands on the page.
  • An account_created event, which can only be:
    • Produced when the user is redirected the auth domain to the origin domain/wiki.
    • Produced client-side via Test Kitchen JS SDK because Test Kitchen PHP SDK cannot produce (server-side) events due to lack of access to the wmf-uniq cookie from MediaWiki side for security and privacy reasons.

Ideally, we would be able to log account creation server-side (the most reliable) but we need the edge unique-derived subject ID to be in the event data for the experiment to be analyzable. We also cannot assume that a click on the link is equivalent to actually arriving at the linked destination.

How FY25/26 WE 1.8.3 overcame the challenges

In FY25/26 Q4, the Contributor Growth team ran an experiment as part of WE 1.8 Account creation experiments KR in which they tested a redesign of the account creation form to see if it would improve account creation rate (definition in GrowthBook): the proportion of subjects who produced an account_created event out of the total subjects exposed to a particular variation.

Configuration

The experiment was configured in Test Kitchen UI as a non-cache splitting (NCS) experiment: https://test-kitchen.wikimedia.org/experiment/we-1-8-account-creation-form-v2, targeting 50% of English Wikipedia visitors on the web.

Some considerations:

Instrumentation

Entry point

Unlike cache splitting experiments where enrollment information is available at request time – available to both the JS and PHP Test Kitchen SDKs – enrollment information for a NCS experiment becomes available asynchronously after the page load. The instrument experimentDecorateCreateAccountLinks.js used the async JS SDK to decorate the login/create account links' href with URL query parameter that encoded experiment name and assigned group. This allowed the instrumentation running on en.wikipedia.org to pass enrollment information to instrumentation running on the auth domain by utilizing onAuthPreserveQueryParams (see ConfirmEmailHooks.php).

Experiment exposure

To assist with instrumenting this experiment, the team developed a UrlEnrolledExperiment class which replicated some of Test Kitchen's Experiment class functionality. This was used by experimentInstrumentation.js which then produced exposure events from instrumentation running on auth.wikimedia.org to https://en.wikipedia.org/evt-103e/v2/events?hasty=true so they include the wikipedia.org edge unique cookie. Notably:

  • Sending cross-domain events is done using navigator.sendBeacon() and manually constructed events, making sure to adhere to /analytics/product_metrics/web/base/2.0.0 schema.
  • The content type of the payload is set to text/plain so that the beacon request does not trigger CORS restrictions.
  • As with TK JS SDK, experiment.subject_id (in the event data) is set to "awaiting" and would be filled out at event ingestion time by EventGate after receiving enrollment information from Varnish.

The presence of the cookie is very important because when the beacon request is processed, Varnish uses the attached cookie to construct a X-Experiment-Enrollments header that includes the edge unique-derived subject ID, which is then used by EventGate to extract the subject ID to populate experiment.subject_id (replacing "awaiting").

Account creation

The instrumentation in server-side AccountCreationHandler.php relies on onCentralAuthPostLoginRedirect to set an accountJustCreated URL query parameter which then tells the client-side useAccountCreationInstrument.js to produce an account_created event.

Since the user is back on the origin domain where they are enrolled, the instrumentation can use Test Kitchen's Experiment#send() as experiments normally would.

Analysis

The experiment's results are available in GrowthBook: https://growthbook.wikimedia.org/experiment/exp_10ipnmqb88wy4#results

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Test_Kitchen/Experimenting_with_Special:CreateAccount&oldid=2430102"