Test Kitchen/GrowthBook user guide/Metrics

Managing fact tables and metrics requires the user to have CustomElevatedAccess role or greater. See Test Kitchen/GrowthBook user guide/Access for more information.

This part of the GrowthBook user guide focuses on metrics:

How to define new metrics
Using and creating fact tables
Guidelines and best practices

Overview

The flow of experiment data in Test Kitchen is:

Experiment sends events to product_metrics.web_base stream.
Events are refined and land in event.product_metrics_web_base Hive table, along with events from other instrumentation.
Just the experiment events are extracted into wmf_experiments.experiment_event_v1 Iceberg table (via Airflow DAG).
- Note: experiment_exposure events are used to populate wmf_experiments.experiment_assignment_v1 Iceberg table and enable Dimensions and Health analysis in GrowthBook.
When an experiment's results are updated (the experiment is analyzed), experiment-specific data from wmf_experiments.experiment_event_v1 is transformed into fact tables for metric calculation.
Metrics are computed from the fact table(s) and are reported on in the experiment's results.

Defining a new metric

All fact tables and metrics should use:

Projects: Wikimedia
Data source: Presto Analytics Iceberg

Always use Presto Analytics Iceberg data source (default). Do not use Presto Analytics Hive data source.

A metric requires these components to be analyzable in GrowthBook:

An instrumentation specification, a document which prescribes what events must be produced by an experiment's instrumentation.
An implementation of that specification (the experiment's instrumentation), the software which actually produces those events and collects data for the experiment.
A fact table (in GrowthBook), a query which pulls those events, transforming the collected data into a structure that makes it possible to define metrics.
A definition (in GrowthBook) that aggregates those facts into a value.

Instrumentation specifications for metrics should be re-used or referenced across experiments targeting the same metric. The specification is a contract, since the fact table expects the raw experiment event data to look a certain way.

Please review docs.growthbook.io/app/metrics as it covers the vast majority of what you need to know about metrics in GrowthBook, including:

Types (mean, proportion, ratio, quantile, retention, daily participation)
Windows (e.g. conversion window, lookback window)
Winsorizing (capping)

Choosing type of metric

Some general patterns:

Metric	Suggested type in GrowthBook	Explanation
Daily Active Users (DAU)	Daily Participation	DAU is not an experimentation-friendly metric, but the normalized version works. Increasing the % of days that a user is active (for some definition of active) should lead to an increase in DAU.
Clickthrough rates	Ratio	Clicks/impressions fits the general pattern of number of successes / number of opportunities or attempts.
Task completion rate	Ratio	Completed tasks / started tasks fits the general pattern of number of successes / number of opportunities or attempts.
Latency	Quantile	Usually not interested in the average but rather median (P50) if distribution is skewed or upper percentiles like P90, P95, and P99.
Conversion rate (e.g. Account creation rate, Constructive activation rate)	Proportion	When the outcome for each user in the experiment is a binary success/failure, where a successful outcome can be: user registered for a permanent account newly registered user saved at least one edit that turned out to be constructive
Engagement rate	Daily Participation or Proportion	Use Daily Participation if each day the user visits they may or may not be considered engaged (depending on the actions they take that qualify them as engaged). Use Proportion if you are interested in whether the user was engaged at any point at all.

Choosing Value of Ratio type metrics

Ratio type metrics can be customized to be formatted as percentages (e.g. 34% instead of 0.34). On the Edit Metric screen: Show Advanced Settings > Display Settings > Format variation value as a percentage.

Ratio metrics let you customize the denominator, which is useful for metrics like clickthrough rates where the denominator is usually "number of impressions of the UI element" or "number of users who saw the UI element" – rather than "all users in the experiment."

Any column of a fact table may be used for Value, and depending on the data type of the column you will have different aggregation options. Here we focus on two Special options for Value: Unique Users and Count of Rows.

Comparison of Unique Users vs Count of Rows
Scenario	Row filter (example)	Value for Numerator & Denominator	Interpretation
Clickthrough rate	Numerator: `action = click` Denominator: `action = impression`	Unique Users	Out of all users who saw the UI element, what proportion clicked on it at least once?
Clickthrough rate		Count of Rows	Out of all the times the UI element was seen, what proportion of those times resulted in a click?
Task completion rate	Numerator: `action = success` Denominator: `action = task_init`	Unique Users	Out of all users who started on the task, what proportion completed it at least once?
Task completion rate		Count of Rows	Out of all the times the task was started, what proportion was the task completed?

Which one to pick depends on the behavior/phenomenon that we are interested in.

For clickthrough rate on the Donate link, users are probably not going to click on it more than once in a while. Once a user has donated once, it may be some time before they decide to do it again. In that case we're probably interested in the probability that a user clicks on the Donate link at all, so we would choose Unique Users.

For tasks which may be performed repeatedly by users (e.g. sharing a highlighted piece of text, editing an article based on a machine-generated suggestion), we are generally interested in the proportion of times that the task is completed, rather than what proportion of users completed the task at least once.

Creating a new fact table

While the Actions fact table can power many different metrics, it is possible to have a metric that requires some additional wrangling (e.g. extract and transform some data out of action_context) and thus a new fact table. Use the template below to get started, as it includes all the necessary components (columns that every fact table must return and some performance optimizations).

Whenever you create a new fact table or modify an existing fact table, the query must return some results in order to save changes.

Fact table template

Start with this query:

SELECT
  experiment.subject_id AS subject_id,
  timestamp,
  ... -- columns that you will use for your metric
FROM
  wmf_experiments.experiment_event_v1
WHERE
  timestamp BETWEEN from_iso8601_timestamp('{{ startDateISO }}') AND from_iso8601_timestamp('{{ endDateISO }}')
  AND experiment.enrolled LIKE '{{ experimentId }}'

This query:

Returns the required columns subject_id (the identifier we use for analysis of experiments) and timestamp
Is optimized for the wmf_experiments.experiment_event_v1 Iceberg table which is partitioned on the timestamp column

Why LIKE instead of more performant =? When the query is executed outside the context of a particular experiment's analysis, {{ experimentId }} is replaced with %, so the condition becomes AND experiment.enrolled LIKE '%'.

Join with Iceberg tables only

Joining experiment data with other event tables requires those event tables to also be Iceberg. It is not possible to, say, join with event.mediawiki_revision_tags_change (a Hive table) to check whether edits made by subjects of an experiment have been reverted or not. Likewise, it is not possible to join with event.ServerSideAccountCreation (a Hive table) to check whether the subject of an experiment created their own account or the account was auto-created when they visited a wiki for the first time while logged-in to SUL.

Boolean columns

If your fact table has a boolean column (e.g. to indicate status of a session or an event) and you need to filter on a true/false value.

Do not use the regular row filter because it produces a statement like WHERE (your_boolean_column IS TRUE) which is not valid for Presto. Instead pick SQL Expression and reference your_boolean_column directly, so the materialized SQL query becomes WHERE (your_boolean_column) and would be valid for Presto.

Guidelines and recommendations

Metrics should use instrumented events

Metrics in Test Kitchen should be instrumented and calculated from events produced by instruments. Test Kitchen metrics should not rely on external data sources such as Webrequest, MediaWiki History, and MariaDB analytics replicas.

Metric definitions should be agnostic of group/variation

There is just a single definition of a metric – you start with a fact table, apply some row filters, and perform some kind of calculation with the rows. At no point should the definition be conditional on which group the subject of an experiment is in.

This is not to be confused with treatment-exclusive metrics, such as a clickthrough rate on a brand new link or engagement rate with a brand new feature. In such cases, the definition is still agnostic, but only the treatment group (which has access to new link or feature) will produce data for calculation of the metric.

Metrics should be specific

Metrics can be similar and follow same general patterns, but they should represent a specific behavior or phenomenon. For example: clickthrough rates (CTRs) are generally a ratio type metric where the numerator is a count of click events and the denominator is a count of impression events. However, it does not make sense to have a generic CTR metric unless you do simply do not care what the user clicked on, just that they clicked on something. In that case, we would probably want to define a proportion type or daily participation type "Engagement rate" metric that looks for a variety of interactions (of which click is one).

Instead we define specific clickthrough rates (CTRs) for specific UI elements. A CTR on element_friendly_name = 'Donate Link' UI element is different than a CTR on element_friendly_name = 'Create Account Link' UI element.

The product_metrics/web/base and product_metrics/app/base schemas include properties element_id, element_friendly_name, funnel_name, and action_source which should be used to record specific information about the user's interaction. They should be as specific as possible.

Recommendations for `element_friendly_name`
Instead of…	Use…
"banner"	Specific type of banner "Email confirmation banner" or "Fundraising CentralNotice banner"
"link" or "button"	Specific link/button, such as: "Donate link" "Create account link" "Log in link" "View source link" "Appearance settings hide button"

Example: funnel metrics

Funnel metrics are best defined as ratio type metrics and should represent flow of users from one specific step to another specific step within a specific workflow/funnel. For example:

Funnel metric examples for Incident Reporting System
Metric	Numerator		Denominator
Metric	Row filter	Value	Row filter	Value
Proportion of Incident Reporting System users who reached step 4 after reaching step 3 of the emergency flow	Filters on the rows returned by the Actions fact table: `action = "view"` `funnel_name = "emergency_flow"` `action_subtype = "step_4"` Note: If there is a way to skip step 3 and go to step 4 from, say, step 2, then we would probably want to include `action_source = "step_3"` for consistency with the definition and the denominator.	Unique Users	Filters on the rows returned by the Actions fact table: `action = "view"` `funnel_name = "emergency_flow"` `action_subtype = "step_3"`	Unique Users
Proportion of emergency incident reports that reached step 4 after reaching step 3		Count of Rows		Count of Rows

Notably, both metrics are specific about representing the flow from step 3 to step 4 of the emergency flow. The first metric is about the number of users we have reached these steps. The second metric is about the number of times that these steps have been reached. See also the section Choosing Value of Ratio type metrics.

Metric definitions should try to re-use fact tables

When designing metrics we should try to avoid bespoke, single-use fact tables and instead try to re-use/share fact tables. Fewer fact tables also means fewer data assets to steward, and data lineage becomes easier to maintain. The best metrics are simple count metrics that can be calculated using the official Actions fact table (growthbook.wikimedia.org/fact-tables/ftb_6v4betpmluz7yff).

GrowthBook has a performance optimization where if multiple metrics use the same fact table then it runs the query for that fact table only once during experiment analysis and uses that set of results across all those metrics, so the more metrics re-use the same fact tables the faster and computationally cheaper it is to update experiment results.

Metrics generally target the default minimum detectable effect (MDE)

All metrics have a Target MDE which is the percentage change that you want to reliably detect before ending your experiment. This is used to estimate the "Days Left" for running experiments. By default this is 10%, which is neither too small nor too big.

Some metrics may be more difficult to lift (needle harder to move), so setting the target MDE to 5% or 2% may be more realistic. However, decreasing MDE means that the sample size must be larger to reach enough statistical power for a trustworthy experiment.

To learn more about selecting Target MDE:

Examples and case studies

Newly created account activation rate

For the Newly created account activation rate (definition in GrowthBook) metric:

The proportion of newly accounted accounts which saved an article (main namespace) edit in the first 24 hours after registration.

Unlike Constructive activation rate, this does not account for constructive-ness of saved edits, just that a newly created account saved an edit (in the first 24 hours).

This metric requires the experiment to produce two events:

An account_created event (example instrumentation) after a permanent account is created.
An edit_saved event, including page.namespace_id (example instrumentation) after an edit is successfully saved/published.

There is not a way to define this metric using the Actions table, because while we can make a Ratio type metric where the denominator is subjects with an account_created event, there is not a way to have a numerator that is subjects with both an account_created event and at least one edit_saved event (within article namespace) made within 24 hours of the account_created event. This metric needs its own fact table.

We created the fact table Newly created account article edits (definition in GrowthBook) to power this metric. The query for that fact table is:

WITH t_account_created AS (
  SELECT
    experiment.subject_id AS subject_id,
    timestamp AS account_created_timestamp -- when the account was created
  FROM
    wmf_experiments.experiment_event_v1
  WHERE
    timestamp BETWEEN from_iso8601_timestamp('{{ startDateISO }}') AND from_iso8601_timestamp('{{ endDateISO }}')
    AND experiment.enrolled LIKE '{{ experimentId }}'
    AND action = 'account_created'
), t_edit_saved AS (
  SELECT
    experiment.subject_id AS subject_id,
    timestamp AS edit_saved_timestamp -- when the edit was saved
  FROM
    wmf_experiments.experiment_event_v1
  WHERE
    timestamp BETWEEN from_iso8601_timestamp('{{ startDateISO }}') AND from_iso8601_timestamp('{{ endDateISO }}')
    AND experiment.enrolled LIKE '{{ experimentId }}'
    AND action = 'edit_saved'
    AND page.namespace_id = 0
)
SELECT
  tac.subject_id AS subject_id,
  tac.account_created_timestamp AS timestamp,
  COALESCE(
    -- edit was saved AFTER account was created
    (tes.edit_saved_timestamp > tac.account_created_timestamp)
    -- but DURING the first 24 hours since the account was created
    AND (tes.edit_saved_timestamp < tac.account_created_timestamp + INTERVAL '24' HOUR),
    false -- if no edit was saved at all
  ) AS edit_saved_in_first_24h
FROM t_account_created tac
LEFT JOIN t_edit_saved tes ON tac.subject_id = tes.subject_id

This fact table has 3 columns:

subject_id (string, required)
timestamp (date, required)
edit_saved_in_first_24h (boolean)

We can then define the metric as a Ratio metric with:

Component	Fact Table	Value	Row Filter
Numerator	Newly created account article edits	Unique Users	SQL Expression: `edit_saved_in_first_24h`
Denominator	Newly created account article edits	Unique Users	(none)

(Remembering to also go to Show Advanced Settings > Display Settings > Format variation value as a percentage.)

Using the usual column-based filtering like edit_saved_in_first_24h = TRUE yields an error in Presto, so the Row Filter should use the boolean column edit_saved_in_first_24h directly via SQL Expression.