Wikidata Query Service/Runbook/Update not propagated to WDQS

Scope & Intent

This runbook is a troubleshooting guide for Wikidata Platform engineers to correct a situation where Blazegraph has become out of sync with the Wikidata source of truth because of events never making it to the streaming-updater pipeline in the first place.

Verify the problem

We currently find out about this situation via a bug report like T407702, in which a user reports that an item deleted from Wikidata is still featured in the results returned for a query in WDQS. Using Q1 as a placeholder in the following commands, here is what to do:

Check that the item has been deleted: https://www.wikidata.org/wiki/Q1 would say so
Check that the item is still present in https://query.wikidata.org/ : the following query returns TRUE if present as a subject or object and FALSE otherwise:

ASK {
  { wd:Q1 ?p ?o. }
  UNION
  { ?s ?p wd:Q1. }
}

Solution

From a stat or deployment host, run the reconcile_items.py script, using the command in the comment at the top of the source code. You'll need to run it twice, once for eqiad and once for codfw.

The script rereads the source of truth and pushes a reconciliation event to Kafka.

Validation

After a few minutes, the SPARQL query above should start returning FALSE.

You can also view the event stream and see the update directly in Kafka from a stat host:

$ kafkacat -C -b kafka-main1006.eqiad.wmnet -t eqiad.rdf-streaming-updater.reconcile

Finally, after a couple hours, you'll be able to query for events in the event.rdf_streaming_updater_reconcile table.