Data Platform Engineering/Automated Traffic Incident Runbook

This page is a guide for Data Engineering folks who are dealing with a data issue potentially related to misclassified automated traffic.

Read this if...

...you receive an alert from Airflow's pageview_human_bot_daily DAG, telling you that the proportion of human traffic has suddenly increased.
...you are notified by an analyst or a Community member that the pageviews and unique devices metrics for a given set of wikis are seeing a suspicious spike since 3 weeks ago.

Tracking

Before continuing, file an automated traffic data incident ticket to track the response to this incident.

Responsibilities

Data Engineering is responsible for:

Detecting the incident, ideally via automated alerting.
Providing an initial impact assessment.
Ruling out infrastructure issues as the cause.
Pausing DAGs to allow investigation and backfill.
Mitigating the incident by implementing changes specified by RDS after investigation.
Performing a backfill, if requested by RDS.

Our partners at RDS are responsible for:

Investigating anomalous traffic to characterize it for mitigation.
Communicating the impact of the incident to internal and external audiences, as applicable.

Use the incident tracking ticket you filed above to coordinate on those actions.

Ruling out infrastructure issues

We're trying to discard the possibility that the suspicious behavior is caused by infrastructure or data processing issues.

Check Airflow history for the DAGs that generate the suspicious data. For instance, if the affected data is pageviews and unique devices, you can check:

refine_webrequest_hourly_text
webrequest_actor_metrics_hourly
webrequest_actor_metrics_rollup_hourly
webrequest_actor_label_hourly
pageview_actor_hourly

You can look for frequent job failures, sensor timeouts, big variations in the task processing times. If Airflow history is within normality, we can discard an infrastructure or processing issue.

Initial impact statement

Update the tracking ticket with an initial time range, rough estimated impact to metrics and affected projects.

You can do this by opening Turnilo's pageviews_daily datasource, and making sure that the suspicious spike follows some of these bot patterns:

Some wikis are affected, others aren't.
Some Countries are affected, some aren't.
Some UserAgents are affected, others aren't.
Both human and bot metrics either stay the same or go up.

Usually, when there's a bot traffic spike, part of it is properly classified by our system, which makes the bot metric go up; but also part of the traffic is not properly classified, which makes the human metric go up. If human pageviews go up, but bot pageviews go down (or vice-versa), it could mean a classification issue, or a data issue, not an automated traffic increase.

Look at time range(s), affected wikis, countries issuing the requests, and try to get a rough estimate of the magnitude of the increase in traffic.

Mitigating

RDS will be included in the tracking ticket through the above template, and will assign an analyst to help us find a pattern in the misclassified bot traffic so it can be isolated.

If we find a pattern that we can use to isolate the misclassified bot traffic, we should file a task to implement the pattern into our automated traffic detection pipeline, and test the results alongside the analyst point of contact.

In order to simplify mitigation, the most likely outcome of this step will be to make updates recommended by the analyst to our list of known-bot fingerprints.

Backfilling

After applying bot detection changes, RDS may request that we backfill the data. We have an agreed budget of 6 weeks to complete this operation, to control the disruption it may cause to other Data Engineering priorities.

File a ticket to track the backfill itself (example).
Identify which DAGs need to be rerun. Consider DAG dependencies: usually, if we rerun a DAG, we also have to rerun all of its dependencies, recursively.
Make sure all affected DAGs use ExternalTaskSensors to sense for their parents. When rerunning a DAG with ExternalTaskSensors, it waits for their parents to have finished before starting any processing that could result in corrupted data. This does not work well with HivePartitionSensors or other data sensors. Modify the affected DAGs if necessary.
Identify the exact date ranges each DAG needs to be rerun for. Note DAGs can be hourly, daily, weekly and monthly, and they will probably need different rerun start/stop times. Some DAGs are windowed, i.e. look 24h backwards, calculate parent DAGs rerun dates accordingly.
Consider making a backup copy of the original data at the top of the pipeline tree. For instance, if the affected metrics are pageviews and unique devices, you could backup webrequest data. This way, if the rerun fails for any reason, we would be able to recover the previous state of the data.
Use the rerun script (or improve it to better fit your needs!) to clear the affected Airflow dag_runs robustly and at a pace sustainable for the cluster. The order in which you specify the DAGs to rerun is important. The script only reruns DAGs for 1 granularity at a time. You should rerun hourly DAGs first, then daily, then weekly and then monthly. Make sure you only start rerunning the monthly ones when the corresponding source data for the whole month has been previously rerun. You can run the script from the Kubernetes Airflow command line.
Vet the data as the reruns are happening.
Keep WMF analysts informed. Leave weekly updates to #working-with-data on Slack (RDS is responsible for communicating to other groups.)