Data Platform/Systems/GrowthBook

GrowthBook is one component of Test Kitchen, Wikimedia's experimentation platform. Test Kitchen also includes the Test Kitchen SDKs and other components; our instance of GrowthBook now handles experiment configuration (replacing Test Kitchen User Interface, TK UI) and analysis (replacing Test Kitchen Automated Analytics, TK AA). Both are owned by the Experiment Platform team.

For information on how to use our instance of GrowthBook, consult the GrowthBook user guide.

The instance is self-hosted and uses Postgres and FerretDB (open source MongoDB alternative) for data storage.

Instances

Staging

growthbook-next
Attribute	Value
Owner	DPE SRE
Kubernetes Cluster	`dse-k8s-eqiad`
Kubernetes Namespace	`growthbook-next`
Chart	https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/charts/growthbook
Helmfiles	https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/helmfile.d/dse-k8s-services/growthbook-next/helmfile.yaml
Docker image	https://gitlab.wikimedia.org/repos/data-engineering/growthbook/
Internal service DNS	`growthbook-next.discovery.wmnet`
Public service URL	https://growthbook-next.wikimedia.org
Logs	{{{logsUrl}}}
Metrics	{{{metricsDashboardUrl}}}
Monitors	{{{monitorsUrl}}}
Application documentation	https://docs.growthbook.io/
Paging	false
Deployment Phabricator ticket	https://phabricator.wikimedia.org/T405749

Production

growthbook
Attribute	Value
Owner	DPE SRE
Kubernetes Cluster	`dse-k8s-eqiad`
Kubernetes Namespace	`growthbook`
Chart	https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/charts/growthbook
Helmfiles	https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/helmfile.d/dse-k8s-services/growthbook/helmfile.yaml
Docker image	https://gitlab.wikimedia.org/repos/data-engineering/growthbook/
Internal service DNS	`growthbook.discovery.wmnet`
Public service URL	https://growthbook.wikimedia.org
Logs	{{{logsUrl}}}
Metrics	{{{metricsDashboardUrl}}}
Monitors	{{{monitorsUrl}}}
Application documentation	https://docs.growthbook.io/
Paging	false
Deployment Phabricator ticket	https://phabricator.wikimedia.org/T405749

Upgrading

Readiness checklist

Paste the following into a Phabricator task for updating/upgrading the deployed version of GrowthBook:

Staging:

[ ] Loads, can log in
[ ] Connections still work (Data Source > Presto Analytics Test Hive|Iceberg > View SQL Explorer)
  [ ] Can query `SELECT 1` in Presto Analytics Test Hive
  [ ] Can query `SELECT 1` in Presto Analytics Test Iceberg
[ ] No requests to third parties introduced (use Network tab in your browser's Developer Tools to verify)

Production:

[ ] Any currently running experiment's results are successfully refreshed (use Update button)

Mainly this helps us verify that the new build still works with our setup which uses the Presto connector with Kerberos authentication.

When to deploy

Deployments to staging environment (https://growthbook-next.wikimedia.org) can be done at will to test new features:

A single task with the checklist above will be filed for the deployment to both staging and production environments

Deployments to production environment (https://growthbook.wikimedia.org) will be done in a controlled way:

There is a Test Kitchen UI deployment window, set every Tuesday at 14:00h UTC (7:00 PDT)

How to deploy

Before running the deployment itself, the specific docker image has to be built from source, based on either an official release or a specific commit. There is a specific gitlab pipeline for that purpose. Take a look at https://gitlab.wikimedia.org/repos/data-engineering/growthbook to know how to build and publish a new docker image for GrowthBook.

Note that:

growthbook-next is the staging environment
growthbook is the production environment

# Do the following once you have prepared the change and the change has been reviewed and merged
ssh deployment.eqiad.wmnet
cd /srv/deployment-charts
git log -n 1
# Check that the change has been pulled
cd helmfile.d/dse-k8s-services/<environment>
# Check changes and deploy
helmfile -e dse-k8s-eqiad diff
helmfile -e dse-k8s-eqiad -i apply

Access management

If you are interested in how to get access to GrowthBook, see the GrowthBook user guide. The page you are on right now is for administering and maintaining the software.

Our GrowthBook installations work with 3 distinct levels of access, mapped to 3 distinct LDAP groups:

LDAP group	GrowthBook Role	Description
growthbook-readonly	Read Only	Can only observe data and experiments in a read-only fashion. This would be the bulk of user seats, and is the default role.
growthbook-customelevatedaccess	CustomElevatedAccess	Meant for people who need to drive experiment and data analysis configuration. It's expected this will usually be A/B test fluent people with software/data/research engineer/scientist/analyst titles
growthbook-admin	Admin	Full R/W access to all growthbook features and settings

Users request access via https://idm.wikimedia.org/permissions. Requests are approved by the current managers of the LDAP groups as defined in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/idm/templates/idm-django-settings.erb (look for growthbook-readonly through growthbook-admin configuration stanzas).

See T419622 Verify GrowthBook access approach for more background on the design of access control

Approver flow

Target response turnaround: 2 full business days.

Julie or Katherine will review the request generated by the system (double checking the URL before visiting it), verifying:
1. That the WMF or WMDE staff member request is for a user with an email domain of wikimedia.org or wikimedia.de (this can be verified at https://ldap.toolforge.org/user/<username>)
2. The presence of the WMF or WMDE staff member or contractor's membership in the wmf or wmde LDAP group (this can be verified via https://ldap.toolforge.org/user/<username> ).
3. That the WMF or WMDE staff member or contractor's ID in data.yaml is active (i.e., has ensure: present).
4. The presence of the WMF or WMDE staff member or contractor's membership in the analytics_privatedata_users group in data.yaml. This is required for all GrowthBook users. Membership in this group is sufficient for growthbook-readonly access.
5. If growthbook-admin is requested, additionally the user must be a member of Experiment Platform Team or Data Platform Engineering where this access level is useful and necessary and comes with heightened expectations. This is for people who are fluent in the system , and who otherwise could be tasked with configuration of data sources, dimensions, SDK configuration (e.g., Attributes), and system Settings. Confirmation of this level of access need should be coordinated on an org-facing DPE IM channel.
Assuming everything checks out, the approver will grant the access in Wikimedia IDM ("Bitu"). Julie and Katherine can approve both Read Only and CustomElevatedAccess requests. Katherine can approve Admin requests. See the idm-django-settings.erb file linked above for alternate approvers in case Julie or Katherine or both are unavailable.
The approver should follow up with the requester on email (Doc: Email for new IDM GrowthBook Approvals).

Synchronization of Growthbook user role with LDAP groups

See T420691 for the whole context.

We regularly reconcile our POSIX user list with several GrowthBook-related LDAP groups (growthbook-admin, growthbook-customeleveatedaccess and growthbook-readonly) with the GrowthBook users themselves.

Any users belonging to growthbook-admin are granted the global Admin role on the whole instance, when members of the two other groups are respectively granted the CustomElevated and Readonly role on the Wikimedia project (and No Access globally). This synchronization is managed through the following DAGs: https://airflow-test-k8s.wikimedia.org/home?tags=growthbook

Troubleshooting access issues

If someone has an access issue:

First check is if they've waited long enough (they might not have seen or read the notice)
Second check is the Airflow logs for the LDAP sync job
- Look for whether the job ran successfully or if it failed
- The logs include reasons why a user's permissions have not been updated – for example: missing_privatedata_gate, meaning they are not in analytics-privatedata-users POSIX group.

For clarity, the user flow is:

User is added to one of the LDAP groups
User goes to growthbook.wikimedia.org
If they're in one of the LDAP groups they land and GB creates an application user account for them
At this point the user can't see anything because they don't have any permissions
The LDAP sync job runs at :00 and at :30
1. It checks if user is in analytics-privatedata-users as a prerequisite for all access
2. It updates the user's permissions based on their LDAP group
Next time user goes to growthbook.wikimedia.org their GB user account has the permissions and they can see everything in Wikimedia project

Settings

Experiment assignment query

Web experiments

Assignment queries are one half of the queries that generate experiment results.^[1] The query that our installation uses is:

SELECT
  subject_id,
  timestamp,
  experiment_id,
  variation_id,
  -- Dimensions:
  user_auth_status AS user_auth_status_first_exposure,
  user_platform,
  wiki_name,
  wiki_id,
  language,
  project_family,
  -- Edit count-based dimensions (T433381):
  CASE
    WHEN user_edit_count IS NULL OR user_edit_count = 0 THEN '0 edits'
    WHEN user_edit_count <= 100 THEN '1-100 edits'
    ELSE '>100 edits'
  END AS user_100_edit_threshold_first_exposure,
  CONCAT(
    user_platform, ', ',
    user_auth_status, ', ',
    -- same as user_100_edit_threshold_first_exposure above:
    CASE
      WHEN user_edit_count IS NULL OR user_edit_count = 0 THEN '0 edits'
      WHEN user_edit_count <= 100 THEN '1-100 edits'
      ELSE '>100 edits'
    END
  ) AS user_platform_auth_status_100_edit_threshold_first_exposure
FROM
  wmf_experiments.experiment_assignment_v1
WHERE
  user_auth_status != 'Bot'
  AND timestamp BETWEEN from_iso8601_timestamp('{{ startDateISO }}') AND from_iso8601_timestamp('{{ endDateISO }}')
  AND experiment_id LIKE '{{ experimentId }}'

The experiment_assignment_v1 table uses experiment_exposure events from experiment_event_v1 table and joins with information in canonical_data.wikis to generate dimensions like wiki_name – enabling experiment owners to compare results between "English Wikipedia" and "German Wikipedia" rather than "enwiki" vs "dewiki".^[2]

App experiments

App experiments have 2 experiment assignment queries/tables: one for development (dev) release data (including Beta version of the app) and one for production (prod) release data:

SELECT
  app_install_id,
  timestamp,
  experiment_id as experiment_id,
  variation_id as variation_id,
  -- Dimensions:
  user_auth_status AS user_auth_status_first_exposure,
  app_platform,
  app_theme,
  app_primary_language_code,
  country_name,
  country_code
FROM wmf_apps.app_experiment_assignment_v1
WHERE timestamp BETWEEN from_iso8601_timestamp('{{ startDateISO }}') AND from_iso8601_timestamp('{{ endDateISO }}')
  AND experiment_id LIKE '{{ experimentId }}'
  AND app_release_status = 'prod'

The two assignment queries use different app_release_status. This allows the experiment owner to import their experiment into GB using data collected during development and Beta testing, verify that everything looks good, and then switch the experiment assignment table (in analysis settings) to use production data.

The app_experiment_assignment_v1 table uses experiment_exposure events from wmf_apps.app_instrument_event_v1 table.

Unit Dimensions

user_auth_status_highest_observed

The query below uses zero-width spaces (not visible in rendered code block) to force an ordering, ensuring that Permanent user > Temporary user > Logged-out user.

SELECT DISTINCT
  subject_id,
  CASE user_auth_status
    WHEN 'Temporary user' THEN 'Temporary user'
    WHEN 'Permanent user' THEN 'Permanent user'
    WHEN 'Logged-out user' THEN 'Logged-out user'
  END AS value
FROM wmf_experiments.experiment_assignment_v1
WHERE NOT user_auth_status IN('Bot', 'Unknown')
  AND experiment_id LIKE '{{ experimentId }}'

References

[1] ttps://docs.growthbook.io/app/datasources#experiment-assignment-queries

[2] ttps://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/blob/main/growthbook/experiment_assignment_query_iceberg.hql?ref_type=heads

[1]

[2]