Nova Resource:Wikiqlever
See https://phabricator.wikimedia.org/T377655 for initial setup and configuration
Indexing
physikerwelt@qlever1:/srv/qlever$ qlever index
Command: index
echo '{ "languages-internal": [], "prefixes-external": [""], "locale": { "language": "en", "country": "US", "ignore-punctuation": true }, "ascii-prefixes-only": true, "num-triples-per-batch": 5000000 }' > wikidata.settings.json
docker run --rm -u $(id -u):$(id -g) -v /etc/localtime:/etc/localtime:ro -v $(pwd):/index -w /index --name qlever.index.wikidata --init --entrypoint bash adfreiburg/qlever -c 'ulimit -Sn 500000 && IndexBuilderMain -i wikidata -s wikidata.settings.json --vocabulary-type on-disk-compressed -f <(lbzcat -n 4 latest-all.ttl.bz2) -g - -F ttl -p true -f <(lbzcat -n 1 latest-lexemes.ttl.bz2) -g - -F ttl -p false -f <(cat dcatap.nt) -g - -F nt -p false --stxxl-memory 10G | tee wikidata.index-log.txt'
2026-01-08 11:15:01.717 - INFO: QLever IndexBuilder, compiled on Mon Dec 15 23:24:16 UTC 2025 using git hash 959c50
2026-01-08 11:15:01.732 - INFO: You specified "locale = en_US" and "ignore-punctuation = 1"
2026-01-08 11:15:01.733 - INFO: You specified "ascii-prefixes-only = true", which enables faster parsing for well-behaved TTL files
2026-01-08 11:15:01.733 - INFO: You specified "num-triples-per-batch = 5,000,000", choose a lower value if the index builder runs out of memory
2026-01-08 11:15:01.733 - INFO: By default, integers that cannot be represented by QLever will throw an exception
2026-01-08 11:15:01.733 - INFO: Processing triples from 3 input streams ...
2026-01-08 11:15:01.738 - INFO: Parsing input triples and creating partial vocabularies, one per batch ...
2026-01-08 11:16:06.650 - INFO: Triples parsed: 70,000,000 [average speed 1.1 M/s, last batch 1.2 M/s, fastest 1.2 M/s,
2026-01-08 11:50:54.827 - INFO: Triples parsed: 2,560,000,000 [average speed 1.2 M/s, last 2026-01-08 12:17:11.387 - INFO: Triples parsed: 4,400,000,000 [average speed 1.2 M/s, last batch 1.2 M/s, fastest 1.3 M/s, slowest 0.8 M/s]2026-01-08 16:19:50.124 - INFO: Triples parsed: 21,403,934,478 [average speed 1.2 M/s, last batch 1.3 M/s, fastest 1.6 M/s, slowest 0.8 M/s] 6 M/s, slowest 0.8 M/s]
2026-01-08 16:19:50.891 - INFO: Number of triples created (including QLever-internal ones): 33,360,170,410 [may contain duplicates]
2026-01-08 16:19:50.891 - INFO: Number of partial vocabularies created: 4,186
2026-01-08 16:19:50.891 - INFO: Merging partial vocabularies ...
2026-01-08 16:20:05.138 - WARN: Total vocabulary order violated for "◌𑇉"@mul and "◌𞥇"@en
2026-01-08 16:20:05.138 - WARN: Total vocabulary order violated for "◌𖿰"@mul and "◌𞥊"@en
2026-01-08 16:20:05.138 - WARN: Total vocabulary order violated for "◌𖿱"@mul and "◌𞥉"@en
2026-01-08 16:25:01.517 - WARN: Total vocabulary order violated for "Category:ﾟ"@en and "Category:？"@enstest 1.5 M/s, slowest 0.6 M/s]
2026-01-08 16:25:03.681 - WARN: Total vocabulary order violated for "Category:Khmer terms spelled with ◌៉"@en and "Category:Khmer terms spelled with ◌៌"@en
2026-01-08 16:25:10.350 - WARN: Total vocabulary order violated for "Category:帶「ः」的詞"@zh and "Category:帶「◌̃」的詞"@zhest 0.6 M/s]
2026-01-08 16:25:10.350 - WARN: Total vocabulary order violated for "Category:帶「◌៉」的高棉語詞"@zh and "Category:帶「◌៌」的高棉語詞"@zh
2026-01-08 16:25:10.350 - WARN: Total vocabulary order violated for "Category:帶◌៉的高棉語詞"@zh and "Category:帶◌៌的高棉語詞"@zh
2026-01-08 17:48:16.777 - INFO: Words merged: 3,766,094,741 [average speed 0.7 M/s, last batch 0.1 M/s, fastest 2.0 M/s, slowest 0.1 M/s]
2026-01-08 17:48:23.282 - INFO: Finished writing compressed internal vocabulary, size = 76 GB [uncompressed = 240.7 GB, ratio = 31%]
2026-01-08 17:48:23.330 - INFO: Number of words in external vocabulary: 3,766,094,741
2026-01-08 17:51:16.879 - INFO: Converting triples from local IDs to global IDs ...
2026-01-08 19:46:01.383 - INFO: Triples converted: 33,360,170,410 [average speed 4.8 M/s, last batch 6.8 M/s, fastest 25.9 M/s, slowest 0.2 M/s]
2026-01-08 19:49:10.897 - INFO: Creating permutations SPO and SOP ...
2026-01-08 20:23:57.338 - INFO: Triples sorted: 9,030,000,000 [average speed 4.3 M/s, last batch 2.1 M/s, fastest 28.9 M/s, slo
2026-01-08 20:24:52.021 - INFO: Triples sorted: 9,330,000,000 [average speed 4.4 M/s, last batch 20.9 M/s, fastest 28.9 M/s, slowest 0.1 M/s]
2026-01-08 21:05:05.424 - INFO: Number of inputs to `uniqueView`: 21,403,934,4786 M/s, last batch 8.1 M/s, fastest 28.9 M/s, slowest 0.1 M/s]
2026-01-08 21:05:05.424 - INFO: Number of unique elements: 21,026,885,878
2026-01-08 21:05:05.424 - INFO: Triples sorted: 21,026,885,878 [average speed 4.6 M/s, last batch 8.1 M/s, fastest 28.9 M/s, slowest 0.1 M/s]
2026-01-08 21:05:06.857 - INFO: Statistics for SPO: #relations = 2,314,099,525, #blocks = 449,033, #triples = 21,026,885,878
2026-01-08 21:05:06.861 - INFO: Statistics for SOP: #relations = 2,314,099,525, #blocks = 449,033, #triples = 21,026,885,878
2026-01-08 21:05:22.521 - INFO: Number of distinct patterns: 10,623,197
2026-01-08 21:05:22.521 - INFO: Number of subjects with pattern: 2,314,099,525 [all]
2026-01-08 21:05:22.521 - INFO: Total number of distinct subject-predicate pairs: 12,886,085,278
2026-01-08 21:05:22.521 - INFO: Average number of predicates per subject: 5.6
2026-01-08 21:05:22.526 - INFO: Average number of subjects per predicate: 207,991
2026-01-08 21:05:29.380 - INFO: Creating permutations OSP and OPS ...
2026-01-08 22:47:03.885 - INFO: Triples sorted: 21,026,885,878 [average speed 3.5 M/s, last batch 9.6 M/s, fastest 19.3 M/s, slowest 0.0 M/s]
2026-01-08 22:47:04.381 - INFO: Statistics for OSP: #relations = 3,808,600,035, #blocks = 588,349, #triples = 21,026,885,878
2026-01-08 22:47:04.381 - INFO: Statistics for OPS: #relations = 3,808,600,035, #blocks = 588,349, #triples = 21,026,885,878
2026-01-08 22:47:06.959 - INFO: Adding 2,314,099,525 triples to the POS and PSO permutation for the internal `ql:has-pattern` ...
2026-01-08 22:56:25.847 - INFO: Creating permutations PSO and POS ...
2026-01-08 23:39:57.727 - INFO: Number of inputs to `uniqueView`: 14,270,335,457 M/s, last batch 2.5 M/s, fastest 34.7 M/s, slowest 0.0 M/s]
2026-01-08 23:39:57.727 - INFO: Number of unique elements: 9,244,828,353
2026-01-08 23:39:57.727 - INFO: Triples sorted: 9,244,828,353 [average speed 3.5 M/s, last batch 2.5 M/s, fastest 34.7 M/s, slowest 0.0 M/s]
2026-01-08 23:39:58.279 - INFO: Statistics for PSO: #relations = 25,778, #blocks = 298,229, #triples = 9,244,828,353
2026-01-08 23:39:58.279 - INFO: Statistics for POS: #relations = 25,778, #blocks = 298,229, #triples = 9,244,828,353
2026-01-08 23:40:01.266 - INFO: Creating permutations PSO and POS ...
2026-01-09 01:34:58.022 - INFO: Triples sorted: 21,026,885,878 [average speed 3.0 M/s, last batch 7.1 M/s, fastest 25.9 M/s, slowest 0.0 M/s]
2026-01-09 01:34:58.755 - INFO: Statistics for PSO: #relations = 61,955, #blocks = 677,297, #triples = 21,026,885,878
2026-01-09 01:34:58.756 - INFO: Statistics for POS: #relations = 61,955, #blocks = 677,297, #triples = 21,026,885,878
2026-01-09 01:35:01.769 - INFO: Index build completed
physikerwelt@qlever1:/srv/qlever$ df -h
/dev/sdb 1007G 570G 387G 60% /srv
| Project Name | wikiqlever |
|---|---|
| Details, admins/members |
used openstack-browser |
| Monitoring |
Server admin log
2026-01-19
- 08:58 physikerwelt: running
2026-01-09
- 11:26 physikerwelt: service available from https://qlever-backend-demo1.wmcloud.org/ (backend) pointing to qlever1 port 7019 https://qlever-ui-demo1.wmcloud.org (frontend) pointing to qlever1 port 8176
- 11:25 physikerwelt: Adjusted the ports and started the qlever server
2026-01-07