Incidents/2025-06-30 eventgate-analytics has stopped producing events
document status: in-review
Summary
| Incident ID | 2025-06-30 eventgate-analytics has stopped producing events | Start | 2025-06-25 09:46:00 |
|---|---|---|---|
| Task | T398187 | End | 2025-06-30 11:41:00 |
| People paged | 0 | Responder count | Ben Tullis, Gabriele Modena, Joseph Allemandou, Sam Smith |
| Coordinators | Gabriele Modena | Affected metrics/SLOs | The affected services' SLOs lack SLIs that monitor traffic drops. |
| Impact | During the incident, no MediaWiki structured logging events were routed through eventgate-analytics. As a result, no data is available to internal consumers (e.g., Hive and Kafka users) for the duration of the incident. | ||
A mediawiki config change inadvertently disabled MediaWiki logging via EventBus The change was reverted and the event production rate is back to pre-incident levels. . This incident impacted all MediaWiki streams routed through eventgate-analytics:
api-gateway.requestmediawiki.api-requestmediawiki.cirrussearch-request'/^swift\.(.+\.)?upload-complete$/'
wdqs and wcqs streams were not affected by this incident.
Timeline
.png)
All times in UTC.
- 2025-06-25 09:46 OUTAGE BEGINS
- 2025-06-30 09:13 Martin Urbanec (Software Engineer - Growth) reports data loss for event.mediawiki_api_request. Gabriele Modena (Software Engineer - Data Platform Engineering) and Joseph Allemandou (Software Engineer - Data Platform Engineering) start an investigation. Data is missing for all eventgate-analytics streams.
- 2025-06-30 10:31 Ben Tullis (SRE - Data Platform Engineering) Gabriele Modena Joseph Allemandou Sam Smith (Software Engineer - Data Platform Engineering) triage. Sam Smith identifies a mediawiki configuration change that caused the incident in this SAL log https://sal.toolforge.org/log/yf97ppcB8tZ8Ohr0dNWN . Gabriele Modena creates a revert. Sam Smith coordinates an emergency deployment https://sal.toolforge.org/log/I-2UwJcBvg159pQrvGdF .
- 2025-06-30 11:41 OUTAGE ENDS
Detection
The EventBus MediaWiki extension publishes events to the Event Platform on certain state changes and user interactions. Those events are periodically collected into tables in the Data Lake.
event.mediawiki_api_request is one such table, containing logs for API requests that are often useful to MediaWiki developers troubleshooting errors or analyzing API usage patterns.
The issue was detected by Martin Urbanec as missing data in a hive table. No alert fired for this issue. Once we noticed that data was missing from Kafka, we investigated eventgate-analytics and EventBus producers. We realized that eventgate-analytics stopped producing events (event rate went to 0) on 2025-06-25 at 09:46UTC. Sam Smith identified a MediaWiki configuration change that correlated with the incident start time. We observed that the traffic drop in eventgate-analytics correlated with a traffic drop, for the same streams, in EventBus suggesting that eventgate-analytics was not losing messages, but rather MediaWiki stopped producing. Moreover, the drop was not sudden but smoothed over a time window suggesting that a queue was draining rather than a loss of connectivity.
.png)
After inspecting the config change, we determined that was the root cause and reverted.
Conclusions
Event production rate is back to pre-incident levels. We won't be able to backfill lost events for the following streams (and downstream hive datasets):
api-gateway.requestmediawiki.api-requestmediawiki.cirrussearch-request'/^swift\.(.+\.)?upload-complete$/'
wdqs and wcqs streams were not affected by this incident.
What went well?
- Once detected, the issue was root caused and resolved quickly.
What went poorly?
- Detection was manual. No alert fired.
- The issue has been ongoing for several days without being noticed.
Where did we get lucky?
- …
OPTIONAL: (Use bullet points) for example: user's error report was exceptionally detailed, incident occurred when the most people were online to assist, etc
Links to relevant documentation
- …
Add links to information that someone responding to this alert should have (runbook, plus supporting docs). If that documentation does not exist, add an action item to create it.
Actionables
- task T398187Add alerting to eventbus and eventgate for drastic changes in event rate production.
Scorecard
| Question | Answer
(yes/no) |
Notes | |
|---|---|---|---|
| People | Were the people responding to this incident sufficiently different than the previous five incidents? | ||
| Were the people who responded prepared enough to respond effectively | |||
| Were fewer than five people paged? | |||
| Were pages routed to the correct sub-team(s)? | |||
| Were pages routed to online (business hours) engineers? Answer “no” if engineers were paged after business hours. | |||
| Process | Was the "Incident status" section atop the Google Doc kept up-to-date during the incident? | ||
| Was a public wikimediastatus.net entry created? | |||
| Is there a phabricator task for the incident? | |||
| Are the documented action items assigned? | |||
| Is this incident sufficiently different from earlier incidents so as not to be a repeat occurrence? | |||
| Tooling | To the best of your knowledge was the open task queue free of any tasks that would have prevented this incident? Answer “no” if there are open tasks that would prevent this incident or make mitigation easier if implemented. | ||
| Were the people responding able to communicate effectively during the incident with the existing tooling? | |||
| Did existing monitoring notify the initial responders? | |||
| Were the engineering tools that were to be used during the incident, available and in service? | |||
| Were the steps taken to mitigate guided by an existing runbook? | |||
| Total score (count of all “yes” answers above) | |||