Jump to content

User:Aisha Khatun/Wikidata Vertical Analysis

From Wikitech

Wikidata is an open knowledge base in the form of a graph accessible through SPARQL queries (among other things). The graph is formed using triples in the form - Subject, Predicate, Object. These components connect each other forming a huge interconnected web of data. Wikidata is growing super fast and it is time to think scaling. With this aim, this page shows some analysis on Wikidata to find out:

  • Amount of certain kinds of vertical data like labels, descriptions, scientific articles etc
  • How many queries ask for each of these vertical slices
  • Analysis of these queries

Phabricator tickets: T282790, T291190.

TL;DR

If blazegraph (wikidata's backend) were to fail, what can we remove temporarily from wikidata so that it can keep functioning? Some data points found across items in wikidata such as labels, descriptions, identifiers etc are possible candidates. Analysis done on these vertical data are described in the following sections.

"Number of days for Wikidata to recover" is the estimated number of days for Wikidata to get back to its current size if some amount of triples is removed from Wikidata. To clarify: Descriptions form ~20% of Wikidata triples. If we were to remove them, then given the rate at which Wikidata is growing, it would take ~500 days for Wikidata to jump back to its current size, despite removing the descriptions. See more about Wikidata growth rate below.

Distribution of triples
PredicatesNumber of Triples% of Total TriplesNumber of days for Wikidata to recover
altLabel102,593,8540.821.5
description2,471,378,66119.5518
external id1,140,577,5558.9239
label499,663,1743.9104
name78,785,7680.616
Distribution of queries that access vertical data (monthly)
PredicatesNumber of Queries% of Total QueriesQuery Time (hr)% of Total Query Time
altLabel29.3 M168275
description21.8 M12260118
external id55 M30545539
label88 M481009972
name15 M8182913

Vertical Data Analysis

Wikidata snapshot of 20210712 was used for this analysis.

Total triples

Before we begin, the total number of triples in this specific snapshot of wikidata is 12671768950, approximately 12.6 billion. The growth rate of triples is not constant, but considering the growth an approximate straight line, in grafana dashboard, Wikidata grows at a rate of 4.77 million triples per day. This rate was calculated from the number of triples at the start and end of a 90-day interval (11/3/21 to 6/6/21). During this period wikidata grew 3.38%!.
Following analyses are done assuming 4.77M triples per day growth where applicable, therefore take numbers as an approximate only. To repeat, the wikidata growth rate is not constant, the 4.77M per day growth is a wide approximation.

Description

The number of triples with the predicate schema:description is 2471378661, 19.5% of all triples.

DescriptionTriple CountTriple %Number of days for Wikidata to recover
English726090160.5715
Other Languages239876964518.93502.8
Total247137866119.5518

Additional Info

Some more information of descriptions.

Number of items that have a description 87048501
Average description per item 28.4
Maximum description count per item 258
Number of item with one description 9910091 (11% of items)
Number of item with more than one description 77138410 (88% of items)
Number of items that have a English description 72609016
Number of items that don't have English descriptions 14439485 (16.6%)

Therefore, 16.6% of all items that have a description don't have English descriptions. If we were to remove all non-English description, 16.6% items that had a description won't have a description anymore.

Distribution of descriptions per item

Top 10 number of descriptions per item
Description per ItemCountCount %Cummulative %
1991009111.3811.38
21084575012.4623.84
31393957916.0139.85
452218766.0045.85
531800513.6549.50
620617532.3751.87
714560361.6753.54
89387501.0854.62
99188641.0655.68
108866631.0256.70

Language distribution of descriptions

440 different language tags in descriptions. 50% of the descriptions are of 32 languages and 90% of the descriptions are of 94 languages.

Top language tags in descriptions
Language tagDescription countDescription %
nl754059653.05
en726090162.94
de617162922.50
ar459391991.86
fr428612551.73
es399893991.62
uk398598461.61
ast386428011.56
ca369014111.49
bn367509361.49

Extra distribution figures in Jupyter Notebook # Description ## Distribution of language tags

Label

The number of triples with the predicate rdfs:label is 499663174, 3.9% of all triples.

LabelTriple CountTriple %Number of days for Wikidata to recover
English797781290.616
Other Languages4198850453.388
Total4996631743.9104

Additional Info

Some more information of labels.

Number of items that have a label 93474062
Avgerage label per item 5.34
Maximum label count per item 446
Number of item with one label 20084825 (21% of items)
Number of item with more than one label 73389237 (78% of items)
Number of items that have a English label 79778129
Number of items that don't have English labels 13695933 (14.65%)

Therefore, 14.7% of all items that have a label don't have English labels. If we were to remove all non-English labels, 14.7% that had a label won't have a label anymore.

Distribution of labels per item

Top ten labels per item
Label per ItemCountCount %Cummulative %
12008482521.4921.49
24169750744.6166.10
31003089510.7376.83
449883615.3482.17
525680682.7584.92
618578911.9986.91
713668631.4688.37
815924801.7090.07
96831020.7390.80
107312730.7891.58

Language distribution of labels

476 different language tags in labels. 40% of the labels are of only 6 languages and 50% of the labels are of 12 languages.

Top language tags in labels
Language tagLabel countLabel %
en7977812915.97
nl5694066511.40
ast161063243.22
fr145949372.92
de143524352.87
es130051302.60
ga91621801.83
it90900371.82
bn85313921.71
pt79664951.59

More distribution figures in Jupyter Notebook # Labels ## Distribution of language tags

Other predicates like Label

Other predicates are skos:altLabel, schema:name. Note that there are no triples with the predicate skos:prefLabel.

Distribution of predicates
PredicateTriple CountTriple %Number of days for Wikidata to recover
shema:name787857680.6216.5
skos:altLabel1025938540.8121.5
rdfs:label4996631743.9104


Language distribution of other predicates
LabelTriple CountTriple %Number of days for Wikidata to recover
English schema:name137213240.113
Other Languages schema:name650644440.5113.6
English skos:altLabel91570380.072
Other Languages skos:altLabel650644440.7419.5
English rdfs:label797781290.616
Other Languages rdfs:label4198850453.388
Total English 1026564910.821.5
Total Other Language 5500139334.3115
Total 6526704245.15137

More distributions in Jupyter Notebook # altLabels and Jupyter Notebook # schema:name

External Identifier

Identifiers are properties, like P297. They are wikibase:propertyType wikibase:ExternalId , i.e they are of property type External ID. Example identifiers are UNBIS Thesaurus ID, BBK (library and bibliographic classification), Symptom Ontology ID, Bilibili bangumi ID etc. There are 6322 distinct external identifiers (as of 10 August, 2021).

These properties appear as /prop, meaning the object is a statement and holds more information. Or as /prop/direct (and /prop/direct-normalized) meaning the object a single URI or literal, doesn't hold more information than that.

Triples related to external identifiers*
Triple typeTriple Count% TripleNumber of days for Wikidata to recover
external identifiers as /prop1796793291.437
external identifiers as /prop/direct1794865501.437
external identifiers as /prop/direct-normalized636662170.513
triples of /prop statement7177454595.6150
Total11405775558.897239

*Note that triples that define the IDs themselves are not included here. Those are in the range of 0.009% of the entire dataset.

Top external IDs
IDID labelTriple Count% Triples with ID% Cummulative
P356DOI8147971619.2719.27
P698PubMed ID6392001015.1234.39
P2671Google Knowledge Graph ID221278985.2339.62
P3083SIMBAD ID163167113.8643.48
P646Freebase ID132743363.1446.62
P932PMCID127067763.0149.63
P1566GeoNames ID111150962.6352.26
P5875ResearchGate publication ID91573492.1754.43
P214VIAF ID81085571.9256.35
P496ORCID iD52228251.2457.59
P846GBIF taxon ID45733911.0858.67
P244Library of Congress authority ID38945770.9259.59
P227GND ID37630720.8960.48
P7859WorldCat Identities ID36675890.8761.35
P6179Dimensions Publication ID30805550.7362.08
P2326GNS Unique Feature ID29359760.6962.77
P5055IRMNG ID27171190.6463.41
P213ISNI26593100.6364.04
P235InChIKey25313750.6064.64
P234InChI25162440.6065.24

Around 19% of the triples with external IDs are triples related to P356 (DOI), 15% to P698 (PubMed ID). 64 (out of 6322) IDs form 80% of the triples having external IDs, 209 form 90%. See more with figures in Jupyter Notebook # External Identifiers


Query Analysis

WDQS external queries of 08/2021 was used for this analysis. All the following numbers were calculated for monthly data.

Note that:

  • Only the queries that contain direct mention of the predicates were considered. Generic open ended queries that happen to match the predicates were not considered here. For example, queries like ?sub ?pred ?obj or ?sub ?obj "label_string" were not counted, but queries like ?sub rdfs:label ?obj or ?sub rdfs:label "label_string" were taken into consideration.
  • The query counts and percentages are not mutually exclusive across vertical slices. Queries that contain rdfs:label, for example, more often than not also contain skos:altLabel. Such queries increase counts for both categories.

Total Queries

  • Total number of monthly queries: ~190M
  • Total monthly query execution time: ~14,000 hours
Total query time class distribution
Query Time ClassNumber of queries
less_10ms2,832,276
10ms_to_100ms132,966,668
100ms_to_1s42,001,720
1s_to_10s3,144,002
more_10s879,330

Description

  • Number of queries where schema:description occurs anywhere in the query (predicate/object/VALUES etc): 21,863,454
  • Number of queries where schema:description is the predicate (subset of the former): 21,862,863
  • Number of queries where schema:description is part of a more complex path: 4
  • Total number of queries with schema:description: 21,863,454, which is 12% of the monthly queries.
  • Queries with descriptions make up 2,600 hours or 18.65% of monthly query time.
Top user agents that use descriptions
User agentNumber of queries% of description queries% of all queries
searx/1.0.05,000,782222.7
UA#X3,532,030161.9
Python-urllib/3.63,233,15114.71.77
searx/0.18.02,313,343101.27
searx/1.0.0-unknown952,2434.30.52
Top user agents that use descriptions (by time)
User agentQuery time (hr)% time of description queries% time of all queries
searx/1.0.063824.54.5
Python-urllib/3.633512.82.4
searx/0.18.02569.81.8
UA-X2047.81.4
UA-X1566.01.1

Label

  • Number of queries where rdfs:label occurs anywhere in the query (predicate/object/VALUES etc): 42,883,256
  • Number of queries where rdfs:label is the predicate (subset of the former): 40,532,779
  • Number of queries where wikibase:label service is used to access labels: 72,936,044
  • Number of queries where rdfs:label is part of a more complex path: 2,537,238
  • Total number of queries with rdfs:label: 88,861,469, which is 48.8% of the monthly queries.
  • Queries with labels make up 10,000 hours or 72% of monthly query time.
Top user agents that use labels
User agentNumber of queries% of label queries% of all queries
UA-X7,817,6128.84.3
wikidataintegrator/0.8.46,999,8777.93.8
NERBot/0.06,118,2056.93.36
searx/1.0.05,097,9135.72.8
UA-X3,532,0303.91.9
Pywikibot/6.1.03,347,9773.71.8
Python-urllib/3.63,233,6893.61.77
UA-X2,947,9103.31.62
UA-X2,502,0712.81.37
searx/0.18.02,346,6492.61.29
WikidataQueryServiceR2,131,1862.41.17
Top user agents that use labels (by time)
User agentQuery time (hr)% time of label queries% time of all queries
UA-X167916.6312.04
UA-X8318.225.95
searx/1.0.06466.44.63
UA-X4714.673.38
Python-urllib/3.63353.322.4
NERBot/0.02912.882.08
UA-X2852.822.04
searx/0.18.02592.561.85
UA-X2001.991.44
UA-X1561.541.11
searx/1.0.0-unknown1101.090.79

altLabel

  • Number of queries where skos:altLabel occurs anywhere in the query (predicate/object/VALUES etc): 29,325,216
  • Number of queries where skos:altLabel is the predicate (subset of the former): 25,928,709
  • Number of queries where skos:altLabel is part of a more complex path: 2,470,100
  • Total number of queries with skos:altLabel: 29,325,216, which is 16% of the monthly queries.
  • Queries with altLabels make up 800 hours or 5% of monthly query time.
Top user agents that use altLabels
User agentNumber of queries% of altLabel queries% of all queries
Toolforge - mix-n-match20,215,0936811
Python-urllib/3.63,233,151111.77
UA-X919,67830.5
UA-X655,3852.20.36
UA-X388,5071.30.2
Top user agents that use altLabels (by time)
User agentQuery time (hr)% time of altLabel queries% time of all queries
Python-urllib/3.633540.52.4
Toolforge - mix-n-match14117.11
UA-X759.10.54
UA-X313.80.22
UA-X303.60.21


Name

  • Number of queries where schema:name occurs anywhere in the query (predicate/object/VALUES etc): 14,965,990
  • Number of queries where schema:name is the predicate (subset of the former): 14,964,300
  • Number of queries where schema:name is part of a more complex path: 1,327
  • Total number of queries with schema:name: 14,965,990, which is 8% of the monthly queries.
  • Queries with labels make up 1,800 hours or 13% of monthly query time.
Top user agents that use schema:name
User agentNumber of queries% of schema:name queries% of all queries
searx/1.0.05,001,390332.7
searx/0.18.02,313,343151.27
searx/1.0.0-unknown952,24360.52
searx/1.0.0-200-313a9847513,2203.40.28
WikidataIdTool/1.0507,3673.4.27
Top user agents that use schema:name (by time)
User agentQuery time (hr)% time of schema:name queries% time of all queries
searx/1.0.063834.94.5
searx/0.18.0256141.8
searx/1.0.0-unknown11060.8
searx/1.0.0-200-313a9847663.60.5
searx/1.0.0-211-968b2899512.80.3

External Identifiers

External Identifiers are those that have the wikibase:propertyType of wikibase:ExternalId. There are ~6500 such properties (as of Sep, 2021). Ids with the top usage in queries, occurring anywhere in the query from predicate, objects, to VALUES table etc, are shown below. Again, the counts are not mutually exclusive since the same query can host multiple of these properties.


Top External Ids used in queries (monthly)
Id P valueId nameQuery count% of all queries
P345IMDb ID163042538.967
P2013Facebook ID142335167.828
P2002Twitter username140401557.722
P212ISBN-13136462497.505
P2003Instagram username135187667.435
P218ISO 639-1 code135129207.432
P957ISBN-10134557997.400
P498ISO 4217 code134222127.382
P2397YouTube channel ID133896897.364
P434MusicBrainz artist ID133130177.322
P1651YouTube video ID132917167.310
P436MusicBrainz release group ID132886207.309
P435MusicBrainz work ID132593307.292
P966MusicBrainz label ID132548237.290
P846GBIF taxon ID66503333.658
P300ISO 3166-2 code65944793.627
P691NKCR AUT ID39927812.196
P214VIAF ID19204191.056
P698PubMed ID18430361.014
P2949WikiTree person ID16981420.934
  • Total number of queries with External Ids: 55,127,216, which is 30% of the monthly queries.
  • Queries with external ids make up 5,500 hours or 39% of monthly query time.
Top user agents that use external Ids (by counts)
User agentNumber of queries% of external Id queries% of all queries
wikidataintegrator/0.8.46993803123.8
Rust mediawiki API/0.2.76603929113.6
Hub6231886113.4
searx/1.0.0509732992.8
Googlebot/2.1261468341.43
Top user agents that use external Ids (by time)
User agentQuery time (hr)% time of external Id queries% time of all queries
UA-X130023.89.3
searx/1.0.064611.84.6
searx/0.18.02594.71.8
Needle/0.9.22053.71.4
UA-X2043.71.4

Combined

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=User:Aisha_Khatun/Wikidata_Vertical_Analysis&oldid=2270523"