WMDE/Wikidata/PropertySuggester update
Occasionally, the data for the property suggester needs to be updated from the latest JSON dumps.
This process requires access to a Wikimedia deploy host (deployment.eqiad.wmnet).
Please note that occasionally this property suggester will be referred to as the 'legacy property suggester'. However as of 2024 December, the new property suggester has not yet been deployed.
One-time setup
Each update
- On
stat1010create the following script and run it (preferably viacronor in a screen / tmux session):
DUMP=/mnt/data/xmldatadumps/public/wikidatawiki/entities/latest-all.json.gz
DATE=$(readlink -f $DUMP | grep -Eo '20[0-9]+' | head -n 1)
echo Analyzing dump of $DATE
if [ -d "wikibase-property-suggester-scripts" ]; then
rm -rf wikibase-property-suggester-scripts
fi
git clone "https://gerrit.wikimedia.org/r/wikibase/property-suggester-scripts" wikibase-property-suggester-scripts
cd wikibase-property-suggester-scripts
https_proxy=http://webproxy.eqiad.wmnet:8080 virtualenv -p python3 venv
source venv/bin/activate
https_proxy=http://webproxy.eqiad.wmnet:8080 pip install .
export LC_ALL=en_US.UTF-8
python scripts/dumpconverter.py $DUMP > dumpconvert.csv
python scripts/analyzer.py dumpconvert.csv analyzed-out
export CHECKSUM=$(sha1sum analyzed-out | awk '{print $1}')
echo Checksum: $CHECKSUM
rm dumpconvert.csv
gzip -9 analyzed-out
mkdir -p "/srv/published/datasets/wmde-analytics-engineering/Wikidata/wbs_propertypairs/$DATE"
mv analyzed-out.gz "/srv/published/datasets/wmde-analytics-engineering/Wikidata/wbs_propertypairs/$DATE/analyzed-out.gz"
echo $CHECKSUM > "/srv/published/datasets/wmde-analytics-engineering/Wikidata/wbs_propertypairs/$DATE/checksum"
- Find the latest
wbs_propertypairson https://analytics.wikimedia.org/published/datasets/wmde-analytics-engineering/Wikidata/wbs_propertypairs/. We’ll use yyyymmdd as a placeholder for its name below. - Pull
analyzed-out.gzto your local machine, apply wbs_propertypairs-refine refine.py (README) and commit to the wbs_propertypairs repo with the commit messageAdd propertypairs from the yyyymmdd dump
. - Load it down to the deployment host with
https_proxy=http://webproxy.eqiad.wmnet:8080 wget 'https://github.com/wmde/wbs_propertypairs/raw/master/yyyymmdd/wbs_propertypairs.csv.gz'. - Unpack it:
gzip -d wbs_propertypairs.csv - Update the actual table:
mwscript-k8s --attach -- extensions/PropertySuggester/maintenance/UpdateTable.php --wiki wikidatawiki --file php://stdin < wbs_propertypairs.csv.- This will take ca. 10 minutes.
- It will first log (to your terminal) a bunch of “Deleting a batch” lines, then “X rows inserted” up to the total number of lines in the CSV file (which you can count with
wc -l wbs_propertypairs.csvbeforehand).
- Log your changes:
!log Updated the Wikidata property suggester with data from yyyymmdd's JSON dump