Jump to content

Data Platform/Systems/GrowthBook

From Wikitech

We run an instance of GrowthBook intended to facilitate the following aspects of Wikimedia's experimentation platform Test Kitchen:

  • Configuration of experiments, replacing Test Kitchen User Interface (TK UI).
  • Analysis of experiments, replacing Test Kitchen Automated Analytics (TK AA).

The instance is self-hosted and uses Postgres and FerretDB (open source MongoDB alternative) for data storage.

Instances

Staging

growthbook-next
Attribute Value
Owner DPE SRE
Kubernetes Cluster dse-k8s-eqiad
Kubernetes Namespace growthbook-next
Chart https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/charts/growthbook
Helmfiles https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/helmfile.d/dse-k8s-services/growthbook-next/helmfile.yaml
Docker image https://gitlab.wikimedia.org/repos/data-engineering/growthbook/
Internal service DNS growthbook-next.discovery.wmnet
Public service URL https://growthbook-next.wikimedia.org
Logs {{{logsUrl}}}
Metrics {{{metricsDashboardUrl}}}
Monitors {{{monitorsUrl}}}
Application documentation https://docs.growthbook.io/
Paging false
Deployment Phabricator ticket https://phabricator.wikimedia.org/T405749


Production

growthbook
Attribute Value
Owner DPE SRE
Kubernetes Cluster dse-k8s-eqiad
Kubernetes Namespace growthbook
Chart https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/charts/growthbook
Helmfiles https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/master/helmfile.d/dse-k8s-services/growthbook/helmfile.yaml
Docker image https://gitlab.wikimedia.org/repos/data-engineering/growthbook/
Internal service DNS growthbook.discovery.wmnet
Public service URL https://growthbook.wikimedia.org
Logs {{{logsUrl}}}
Metrics {{{metricsDashboardUrl}}}
Monitors {{{monitorsUrl}}}
Application documentation https://docs.growthbook.io/
Paging false
Deployment Phabricator ticket https://phabricator.wikimedia.org/T405749


Upgrading

Readiness checklist

Paste the following into a Phabricator task for updating/upgrading the deployed version of GrowthBook: 

Staging:

[ ] Loads, can log in
[ ] Connections still work (Data Source > Presto Analytics Test Hive|Iceberg > View SQL Explorer)
  [ ] Can query `SELECT 1` in Presto Analytics Test Hive
  [ ] Can query `SELECT 1` in Presto Analytics Test Iceberg
[ ] No requests to third parties introduced (use Network tab in your browser's Developer Tools to verify)

Production:

[ ] Any currently running experiment's results are successfully refreshed (use Update button)

Mainly this helps us verify that the new build still works with our setup which uses the Presto connector with Kerberos authentication.

Access management

If you are interested in how to get access to GrowthBook, see the GrowthBook user guide. The page you are on right now is for administering and maintaining the software.

Our GrowthBook installations work with 3 distinct levels of access, mapped to 3 distinct LDAP groups:

LDAP group GrowthBook Role Description
growthbook-readonly Read Only Can only observe data and experiments in a read-only fashion. This would be the bulk of user seats, and is the default role.
growhtbook-customelevatedaccess CustomElevatedAccess Meant for people who need to drive experiment and data analysis configuration. It's expected this will usually be A/B test fluent people with software/data/research engineer/scientist/analyst titles
growthbook-admin Admin Full R/W access to all growthbook features and settings

Users request access via https://idm.wikimedia.org/permissions. Requests are approved by the current managers of the LDAP groups as defined in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/idm/templates/idm-django-settings.erb (look for growthbook-readonly through growthbook-admin configuration stanzas).

See T419622 Verify GrowthBook access approach for more background on the design of access control

Approver flow

Target response turnaround: 2 full business days.

  1. Julie or Katherine will review the request generated by the system (double checking the URL before visiting it), verifying:
    1. That the WMF or WMDE staff member request is for a user with an email domain of wikimedia.org or wikimedia.de (this can be verified at https://ldap.toolforge.org/user/<username>)
    2. The presence of the WMF or WMDE staff member or contractor's membership in the wmf or wmde LDAP group (this can be verified via https://ldap.toolforge.org/user/<username> ).
    3. That the WMF or WMDE staff member or contractor's ID in data.yaml is active (i.e., has ensure: present).
    4. The presence of the WMF or WMDE staff member or contractor's membership in the analytics_privatedata_users group in data.yaml. This is required for all GrowthBook users. Membership in this group is sufficient for growthbook-readonly access.
    5. If growthbook-admin is requested, additionally the user must be a member of Experiment Platform Team or Data Platform Engineering where this access level is useful and necessary and comes with heightened expectations. This is for people who are fluent in the system , and who otherwise could be tasked with configuration of data sources, dimensions, SDK configuration (e.g., Attributes), and system Settings. Confirmation of this level of access need should be coordinated on an org-facing DPE IM channel.
  2. Assuming everything checks out, the approver will grant the access in Wikimedia IDM ("Bitu"). Julie and Katherine can approve both Read Only and CustomElevatedAccess requests. Katherine can approve Admin requests. See the idm-django-settings.erb file linked above for alternate approvers in case Julie or Katherine or both are unavailable.
  3. The approver should follow up with the requester on email (Doc: Email for new IDM GrowthBook Approvals).

Synchronization of Growthbook user role with LDAP groups

See T420691 for the whole context.

We regularly reconcile our POSIX user list with several GrowthBook-related LDAP groups (growthbook-admin, growthbook-customeleveatedaccess and growthbook-readonly) with the GrowthBook users themselves.

Any users belonging to growthbook-admin are granted the global Admin role on the whole instance, when members of the two other groups are respectively granted the CustomElevated and Readonly role on the Wikimedia project (and No Access globally). This synchronization is managed through the following DAGs: https://airflow-test-k8s.wikimedia.org/home?tags=growthbook

Troubleshooting access issues

If someone has an access issue:

  • First check is if they've waited long enough (they might not have seen or read the notice)
  • Second check is the Airflow logs for the LDAP sync job
    • Look for whether the job ran successfully or if it failed
    • The logs include reasons why a user's permissions have not been updated – for example: missing_privatedata_gate, meaning they are not in analytics-privatedata-users POSIX group.

For clarity, the user flow is:

  1. User is added to one of the LDAP groups
  2. User goes to growthbook.wikimedia.org
  3. If they're in one of the LDAP groups they land and GB creates an application user account for them
  4. At this point the user can't see anything because they don't have any permissions
  5. The LDAP sync job runs at :00 and at :30
    1. It checks if user is in analytics-privatedata-users as a prerequisite for all access
    2. It updates the user's permissions based on their LDAP group
  6. Next time user goes to growthbook.wikimedia.org their GB user account has the permissions and they can see everything in Wikimedia project

Settings

Experiment assignment query

Assignment queries are one half of the queries that generate experiment results.[1] The query that our installation uses is:

SELECT
  subject_id,
  timestamp,
  experiment_id,
  variation_id,
  -- Dimensions:
  user_auth_status AS user_auth_status_first_exposure,
  user_platform,
  wiki_name,
  wiki_id,
  language,
  project_family,
  mediawiki_skin
FROM
  wmf_experiments.experiment_assignment_v1
WHERE
  user_auth_status != 'Bot'
  AND timestamp BETWEEN from_iso8601_timestamp('{{ startDateISO }}') AND from_iso8601_timestamp('{{ endDateISO }}')
  AND experiment_id LIKE '{{ experimentId }}'

The experiment_assignment_v1 table uses experiment_exposure events from experiment_event_v1 table and joins with information in canonical_data.wikis to generate dimensions like wiki_name – enabling experiment owners to compare results between "English Wikipedia" and "German Wikipedia" rather than "enwiki" vs "dewiki".[2]

Unit Dimensions

user_auth_status_highest_observed

The query below uses zero-width spaces (not visible in rendered code block) to force an ordering, ensuring that Permanent user > Temporary user > Logged-out user.

SELECT DISTINCT
  subject_id,
  CASE user_auth_status
    WHEN 'Temporary user' THEN '​​Temporary user'
    WHEN 'Permanent user' THEN '​​​Permanent user'
    WHEN 'Logged-out user' THEN '​Logged-out user'
  END AS value
FROM wmf_experiments.experiment_assignment_v1
WHERE NOT user_auth_status IN('Bot', 'Unknown')
  AND experiment_id LIKE '{{ experimentId }}'

References

  1. ↑ https://docs.growthbook.io/app/datasources#experiment-assignment-queries
  2. ↑ https://gitlab.wikimedia.org/repos/product-analytics/data-pipelines/-/blob/main/growthbook/experiment_assignment_query_iceberg.hql?ref_type=heads
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Data_Platform/Systems/GrowthBook&oldid=2448291"