MediaWiki Event Enrichment/HTML Enrichment
mw-page-html-content-change-enrich
Why
MediaWiki page change streams (e.g. mediawiki.page_change) carry event notifications but do not include rendered HTML content. Consumers that need the current rendered HTML of a page would otherwise have to call the MediaWiki REST API themselves — and do so independently, for every event, from every service.
Rendered HTML is not static: from the moment a page change occurs, the HTML output can drift due to template re-renders, Parsoid updates, or other background processing. Calling the REST API as early as possible after the change event is therefore critical to capturing the most accurate snapshot of the page at the time of the change.
mw-page-html-content-change-enrich addresses both problems: it fetches rendered HTML once, as soon as a mediawiki.page_change event is received, and propagates it to the mediawiki.page_html_content_change stream. Any number of downstream consumers can then read from that stream without ever contacting the MediaWiki API directly.
In order to enable other uses cases, this pipeline is also retrieving the HTML of the parent revision, and storing a unified_diff inside the content.
Enrichment Job
mw-page-html-content-change-enrich is a PyFlink streaming application (part of the mediawiki-event-enrichment repository). It reads mediawiki.page_change events, enriches them with rendered HTML content via the MediaWiki REST API (Revisions), and writes the result to the mediawiki.page_html_content_change stream.
What gets enriched
Only events with a wikitext content model are enriched. Events for other content models (e.g. Wikidata JSON items) are passed through without enrichment.
The enrichment behaviour depends on the page_change_kind field:
page_change_kind |
HTML content fetched | HTML diff computed | Notes |
|---|---|---|---|
edit |
Yes | Yes | Latest revision + diff against parent revision |
create |
Yes | No | Latest revision only; no parent to diff against |
move |
Yes | Yes | Latest revision + diff; current rendering of old revisions |
undelete |
Yes | Yes | Latest revision + diff; current rendering of old revisions |
visibility_change |
Yes | Yes | Latest revision + diff; current rendering of old revisions |
delete |
No | No | Passed through as-is; consumers should propagate the delete |
When a diff is needed, both the current and parent revision HTML are fetched in parallel.
Error handling
Events that cannot be enriched (e.g. API errors, missing revisions, or enriched payloads exceeding the configured maximum size of 20 MB) are routed to a dead-letter queue (DLQ) error topic instead of being dropped silently. See the Kafka section for topic names.
Operational
The application is deployed in the mw-page-html-content-change-enrich namespace of the dse-k8s Kubernetes cluster (eqiad), using the flink-app Helm chart via the Flink Kubernetes Operator. Configuration lives in operations/deployment-charts under helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich/.
For general Kubernetes deployment procedures, see Kubernetes/Deployments.
Deploy / update
After merging a change to deployment-charts, log in to a deployment server and run:
ssh deployment.eqiad.wmnet
cd /srv/deployment-charts/helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich
helmfile -e dse-k8s-eqiad -i apply --context 5
Restart
To force a rolling restart of all pods:
helmfile -e dse-k8s-eqiad --state-values-set roll_restart=1 sync
Note: There is a known issue in Flink when restarting applications, sometimes Flink might think the application is already running and won't start a new one. If the application is in a bad state and restarting it doesn't solve the issue, please, check the logs, if they say something about "job already submitted", the solution could be to destroy and start the application again.
Destroy
To tear down the deployment completely:
helmfile -e dse-k8s-eqiad -i destroy
Kafka
The application reads from and writes to the Kafka Jumbo cluster.
| Role | Topic pattern |
|---|---|
| Source | eqiad.mediawiki.page_change.v1 / codfw.mediawiki.page_change.v1
|
| Sink (enriched output) | eqiad.mediawiki.page_html_content_change.* / codfw.mediawiki.page_html_content_change.*
|
| Error / DLQ | eqiad.mw_page_html_content_change_enrich.error / codfw.mw_page_html_content_change_enrich.error
|
Output messages are compressed with Snappy. The maximum enriched event size is 20 MB, matching the destination topic's max.message.bytes limit.
Monitoring
- HTML Edit Types dashboard — overview of enrichment throughput and edit type distribution
- Flink App dashboard — Flink cluster health, task manager status, and consumer lag (requires to select the right cluster and namespace)
Backfilling
If for any reason the application is down for a while, after a restart, it will try to catch-up with all messages. Given the issues with high latency responses from the REST API, it might be hard to process a lot of messages per second, but the application should be able to catch-up properly without changes.
If a big backfill is needed, only replicas should be increased. Given the current memory limits, probably the max number of allowed replicas is 23. (There's a max memory allowed per namespace of 150GB, and our replicas are configured to use 6GB + 500MB per TaskManager.
Alerting
Alerts are defined in operations/alerts under team-data-engineering/mw-page-html-content-change-enrich.yaml and fire only on the dse-k8s cluster.
- Severity
- Critical
- Condition
- No
flink_jobmanager_job_uptimemetric is present for the job for more than 5 minutes. - Meaning
- The Flink JobManager is not running. The enrichment pipeline is completely down.
- Actions
- Check the HTML Edit Types dashboard.
- Check Kubernetes pod status in the
mw-page-html-content-change-enrichnamespace. - Check logs to see if the application if failing for a reason.
- Redeploy if needed.
- Severity
- Critical
- Condition
- The Flink cluster has zero registered TaskManagers for more than 5 minutes.
- Meaning
- The Flink application is running but there are no workers. No events are being processed.
- Actions
- Check TaskManager pod logs and the HTML Edit Types dashboard.
- Check logs to see if the application is failing for any reason. Redeploy if needed.
- Check if there's an issue with Kubernetes, like not enough resources.
- Severity
- Warning
- Condition
- More than 2% of messages are routed to the DLQ error topic over a 10-minute window.
- Meaning
- A significant fraction of events cannot be enriched. This may indicate a major bug during the enrichment, MediaWiki API issues, malformed events, or oversized payloads.
- Actions
- Check the HTML Edit Types dashboard
- There is a widget with the number of messages sent to the DLQ, as well as the %
- Inspect the DLQ and output topics in Kafka
- Review Flink and Envoy logs.
- If the % goes above 2 only a bit, for a small amount of time, maybe there is something happening on a Wiki.
- Severity
- Critical
- Condition
- No messages are produced to the output topic in the last 30 minutes.
- Meaning
- The job has stalled or is exclusively routing events to the DLQ.
- Actions
- Check the HTML Edit Types dashboard
- Check whether Flink and Envoy are running
- Check Flink UI and whether there is traffic on the source topic or the DLQ.
- If everything looks good, maybe the source is failing or routing messages to another Kafka topic, like another version of the topic.
- Severity
- Warning
- Condition
- The maximum Kafka consumer lag across partitions exceeds 1200 records for more than 5 minutes.
- Meaning
- The job is falling behind its input rate. This can be triggered if the application is down or if it is working slower than expected.
- Actions
- Check the HTML Edit Types dashboard for resource bottlenecks (CPU, memory, network)
- Check if there was an outage, maybe the job is already recovering but it will need time.
- Check if Envoy latencies are bigger than usual, it might cause lag on the application.
- Consider increasing replicas (TaskManagers) to recover faster.