Jump to content

Wikidata Query Service/Runbook/High replication lag and query timeout

From Wikitech

Scope & intent

This runbook is a troubleshooting and escalation guide for Wikidata Platform engineers to respond to periods of elevated WDQS load and user-facing query failures.

It does not replace the standard SRE incident response process. Its purpose is to:

  • Quickly assess system impact
  • Distinguish SLO-impacting incidents from noisy traffic
  • Engage the right stakeholders early

The following alerts are covered by this runbook:

1. Trigger conditions

Use this runbook when any of the following occur:

⚠️ Note: A single transient spike may self-heal, but repeated alerts or correlation with user reports should trigger investigation.

2. Initial assessment

2.1 Check SLO / system health (Grafana)

Dashboard: Wikidata Query Service  Link

Focus on:

  • Max replication lag
    • Above SLO?
    • Trending upward or stabilizing?
  • Ratio of failed queries
    • Elevated compared to baseline?
    • Increasing over time?
  • Overall traffic
    • Compare against baseline (last X days / hours)

Decision point:

  • If max lag is breaching or trending up =>  treat as potential incident  and move ahead with point 4.
  • If SLOs are fine but failed queries are elevated => proceed with traffic analysis

3. Traffic analysis & attribution

3.1 Inspect live traffic

Dashboard: Webrequest sampled (live)  Link

Key views:

Look for:

  • Single or few user agents dominating traffic
  • High error rates (timeouts, 5xx) tied to specific agents
  • Patterns consistent with bots or scrapers

⚠️ The vast majority of WDQS traffic is programmatic. Bot traffic is not inherently bad.  Intervene only if it degrades user experience or system stability

3.2 Identify known bad actors

Check whether the user agent / TLS signature:

  • Was previously block-listed
  • Is a known commercial scraper

⚠️  Use TLS signatures where possible. These are more precise than UA strings

4. Decision & prioritization

4.1 If SLOs are impacted

  • Escalate immediately using standard incident response procedures
  • Notify #wikidata-platform-sre
  • Prepare mitigations (rate limiting, fleet restarts)

4.2 If SLOs are not impacted but user pain is real

  • Update the incident log document (internal) with
    • SLO status
    • Traffic patterns
    • Suspected offending actors
  • Apply targeted mitigations to reduce pressure

5. Mitigation steps if bad actors have been identified

5.1 Engage SRE / Security

Contact SREs via #mediawiki_security (IRC) and share:

Possible actions:

  • SRE applies  requestctl rules to
    • Rate-limit specific actors
    • Temporary block known abusers

5.2 Monitor effect

After mitigation:

  • Confirm ratio of failed queries decreases
  • Confirm max lag stabilizes or recovers
  • Watch for traffic normalization

6. Fleet restart (last-resort / cleanup)

Perform only if:

  • Load remains high
  • Failed queries persist
  • SREs recommend cleanup

Actions:

⚠️ Notes:Update #talk-to-wikidata-platform with:

  • Current status (SLO impacted or not)
  • Mitigation steps taken
  • Restarting the fleet is a standard cleanup action under heavy load
  • Should be done deliberately and communicated

7. Communication & reporting

7.1 Internal updates

  • Keep SREs in the loop during active mitigation

7.2 Post-event follow-up

  • Update the incident log document
  • File or update Phabricator tasks if gaps were identified
  • Triage with the Wikidata Platform Product Lead, and determine whether a follow up with the offending actor is required.

References

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Wikidata_Query_Service/Runbook/High_replication_lag_and_query_timeout&oldid=2453534"