mw-page-html-feature-counts-change-enrich

This job consumes the output of mw-page-html-content-change-enrich. Rendered HTML and unified diffs are produced upstream; this pipeline only computes feature-count deltas.

Why

Downstream analytics and research often need a compact summary of what changed in the rendered HTML (sections, wikilinks, words, etc.) without storing or re-fetching full HTML. Running mwedittypes on every consumer would duplicate work and increase load.

mw-page-html-feature-counts-change-enrich reads mediawiki.page_html_content_change, reconstructs the parent revision’s HTML from the current HTML and the unified diff, classifies edits with SimpleEditTypes, and emits mediawiki.page_html_feature_counts_change events. The sink payload strips large HTML bodies so the stream stays small.

Enrichment Job

mw-page-html-feature-counts-change-enrich is a PyFlink job in the mediawiki-event-enrichment repository.
Entry point: html_feature_counts_enrichment/page_html_feature_counts_change.py.

What gets processed

Canary events are dropped before enrichment.
Only wikitext main-slot events are enriched. Events with non-wikitext content models are dropped.

For remaining events, feature counts are computed only when the change looks like a normal diffable edit:

Situation	Behaviour
`page_change_kind` is `create` or `delete`	Event is passed through without counts.
`rev_parent_id == 0` (no parent revision)	Event is passed through without counts.
Edit with HTML body, unified diff, and `content_language` set	Enriches event with `SimpleEditTypes`.

Counts are written under delta.revision.rendering.features (software metadata, language, and per–edit-type inserted/removed/modified tallies). Edit-type names from mwedittypes are normalized to the schema’s snake_case keys.

What goes to the error topic

On enrichment failure, the job raises an exception and routes the original event to the error topic. See the Kafka section for topic names. Typical reasons an event might end up in the error topic:

Missing or empty HTML or diff when enrichment was required (not one of the pass-through cases above).
Invalid unified diff (cannot reconstruct parent HTML).
Missing or empty content_language (required for mwedittypes).
SimpleEditTypes errors (e.g. unsuitable HTML).

Before successful emit, HTML/diff bodies are removed from the payload to keep messages small.

Execution model

Processing is synchronous (process_async_enabled_default: false) to avoid OOM from batching very large HTML messages. The stream is partitioned by (wiki_id, page_id) so ordering per page is preserved while scaling TaskManagers.

Operational

The application is deployed in the mw-page-html-feature-counts-change-enrich namespace of the dse-k8s Kubernetes cluster (eqiad), using the flink-app Helm chart via the Flink Kubernetes Operator. Configuration lives in operations/deployment-charts under helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich/. A staging release exists under mw-page-html-feature-counts-change-enrich-next/.

For general Kubernetes deployment procedures, see Kubernetes/Deployments.

Upgrades and Deployment

Make changes in the flink application if required in mediawiki-event-enrichment. Once a new docker image has been built and published, we can bump the image version in the relevant helmfile values.yaml file. Once merged, for the most part, the usual kubernetes helmfile deployment process can be followed. But, because this is a streaming Flink application, special care may need to be taken to handle the application restart.

After merging a change to deployment-charts, run the following commands:

## log in to a deployment server
ssh deployment.eqiad.wmnet

## cd into the flink app helm chart
cd /srv/deployment-charts/helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich

## check changes that will be applied
helmfile -e dse-k8s-eqiad diff

## deploy the application
helmfile -e dse-k8s-eqiad -i apply --context 5

Restart

To force a rolling restart of all pods:

helmfile -e dse-k8s-eqiad --state-values-set roll_restart=1 sync

There is a known issue in Flink when restarting applications, sometimes Flink might think the application is already running and won't start a new one. If the application is in a bad state and restarting it doesn't solve the issue, please check the logs, if it says something about "job already submitted", the solution could be to destroy and start the application again.

Destroy

To tear down the deployment completely:

helmfile -e dse-k8s-eqiad -i destroy

Kafka

The application reads from and writes to the Kafka Jumbo cluster. Production consumes both DC-prefixed source topics (eqiad/codfw); only one DC is active at a time, but both topic names are listed so failover does not require a config change.

Role	Topic pattern (examples)
Source	`eqiad.mediawiki.page_html_content_change.v1`, `codfw.mediawiki.page_html_content_change.v1`
Sink	`eqiad.mediawiki.page_html_feature_counts_change.v1` / `codfw.mediawiki.page_html_feature_counts_change.v1`
Error / DLQ	`eqiad.mw_page_html_feature_counts_change_enrich.error` / `codfw.mw_page_html_feature_counts_change_enrich.error`

Sink and error sinks use at-least-once delivery. Maximum request size is aligned with large HTML in error records (20 MB).

Development

Kafka UI

You can set up Kafka UI with Kafbat: javiermonton/kafka-ui-scripts. This helps easily look at existing topics, messages in topics, and some basic stats.

Monitoring

HTML Edit Types dashboard — throughput, error event counts, and overview metrics shared with the HTML pipeline.
Flink App dashboard — Flink cluster health, job/task manager status, consumer lag, etc. Select eqiad prometheus/k8s-dse datasource, mw-page-html-feature-counts-change-enrich namespace.
Logs go to Logstash. Select dse-k8s cluster and mw-page-html-feature-counts-change-enrich namespace.
More here (e.g. Flink UI): MediaWiki_Event_Enrichment#Monitoring

Backfilling

On restarts, the job catches up from the last Kafka offset. If the job restarts soon after, the catchup should not be a problem. If it has been down for a while (days), the catchup can take a long time and may fire HighKafkaConsumerLag alert. Increase TaskManager replicas temporarily to speed up for big backfills.

helmfile apply -e dse-k8s-eqiad --set app.taskManager.replicas=20

Alerting

Alerts are defined in operations/alerts under team-data-engineering/mw-page-html-feature-counts-change-enrich.yaml and fire only on the dse-k8s cluster.

MediawikiPageHtmlFeatureCountsChangeEnrichJobManagerNotRunning

Severity: Critical
Condition: No flink_jobmanager_job_uptime metric is present for the job for more than 5 minutes.
Meaning: The Flink JobManager is not running. The enrichment pipeline is completely down.
Actions

Check the HTML Edit Types dashboard.
Check Kubernetes pod status in the mw-page-html-feature-counts-change-enrich namespace.
Check logs to see if the application is failing for a reason.
Redeploy if needed.

MediawikiPageHtmlFeatureCountsChangeEnrichTaskManagerNotRunning

Severity: Critical
Condition: The Flink cluster has zero registered TaskManagers for more than 5 minutes.
Meaning: The Flink application is running but there are no workers. No events are being processed.
Actions

Check TaskManager pod logs and the HTML Edit Types dashboard.
Check logs to see if the application is failing for any reason. Redeploy if needed.
Check if there's an issue with Kubernetes, like not enough resources.

MediawikiPageHtmlFeatureCountsChangeEnrichHighErrorStreamRate

Severity: Warning
Condition: More than 0.05% of messages are routed to the error topic over a 10-minute window.
Meaning: A significant fraction of events cannot be enriched. This may indicate a major bug during the enrichment, malformed events, or oversized payloads.
Actions

Check the HTML Edit Types dashboard
- There is a widget with the number of messages sent to the error topic, as well as the %
Inspect the error and output topics in Kafka
Review Flink and Envoy logs.

MediawikiPageHtmlFeatureCountsChangeEnrichNoOutputMessages

Severity: Critical
Condition: No messages are produced to the output topic in the last 30 minutes.
Meaning: The job has stalled or is exclusively routing events to the error topic.
Actions

Check the HTML Edit Types dashboard
Check whether Flink and Envoy are running
Check Flink UI and whether there is traffic on the source or error topic.
If everything looks good, maybe the source is failing or routing messages to another Kafka topic, like another version of the topic.

MediawikiPageHtmlFeatureCountsChangeEnrichHighKafkaConsumerLag

Severity: Warning
Condition: The maximum Kafka consumer lag across partitions exceeds 1200 records for more than 5 minutes.
Meaning: The job is falling behind its input rate. This can be triggered if the application is down or if it is working slower than expected. Can fire during backfills.
Actions

Check the HTML Edit Types dashboard for resource bottlenecks (CPU, memory, network)
Check if there was an outage, maybe the job is already recovering but it will need time.
Consider increasing replicas (TaskManagers) to recover faster.