Nova Resource:Wikidata-query

From Wikitech
Jump to navigation Jump to search
Project Name wikidata-query
Details,
admins/members
openstack-browser
Monitoring

Wikidata Query Service

Description

Wikidata Query service - see https://www.mediawiki.org/wiki/Wikidata_query_service

Purpose

The Wikidata Query Service (WDQS) provides a way to access Wikidata data, via a SPARQL API.

Anticipated traffic level

10-100 hits per day

Anticipated time span

months

Project status

currently running

Contact address

smalyshev@wikimedia.org

Willing to take contributors or not

willing

Subject area narrow or broad

broad


Puppetized instance setup

Blazegraph HA journal cluster Implementation log/notes

These notes use this as a guide: http://wiki.blazegraph.com/wiki/index.php/HAJournalServer

  • Each machine has a 60Gb /srv partition
  • The three machines share /data/project
  • The compressed journal is shared at /data/project/blazegraph/bigdata-wikidata-statements.jnl.gz

Prerequisites

Make /data/project/blazegraph, with group write permissions for project-wikidata-query. This is shared between all machines in the cluster.

$ sudo mkdir /data/project/blazegraph
$ sudo chgrp project-wikidata-query /data/project/blazegraph
$ sudo chmod g+rwx /data/project/blazegraph

Install Java:

$ sudo apt-get install openjdk-7-jdk

ZooKeeper

Get ZooKeeper:

$ cd /srv
$ sudo mkdir zookeeper
$ sudo wget http://apache.mirrors.tds.net/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz \
  -O /srv/zookeeper/zookeeper-3.4.6.tar.gz

Extract ZooKeeper and set some useful permissions:

$ cd zookeeper
$ sudo tar -xzf zookeeper-3.4.6.tar.gz
$ sudo chown -R root.project-wikidata-query /srv/zookeeper
$ sudo chmod -R g+rw /srv/zookeeper

Configure ZooKeeper:

$ cd /srv/zookeeper/zookeeper-3.4.6/conf
$ cp zoo_sample.cfg zoo.cfg

Change the data directory in zoo.cfg:

dataDir=/srv/zookeeper

Append to zoo.cfg:

initLimit=5
syncLimit=2
server.1=wdq-bg1:2888:3888
server.2=wdq-bg2:2888:3888
server.3=wdq-bg3:2888:3888

Set Zookeeper's myid (use 1, 2, or 3 depending on which server, wdq-bg1, wdg-bg2, or wdq-bg3):

$ echo 1 > /srv/zookeeper/myid

Start ZooKeeper:

$ cd /srv/zookeeper/zookeeper-3.4.6/bin
$ ./zkServer.sh start

Blazegraph

Create /srv/blazegraph, with group write permissions for project-wikidata-query

$ sudo mkdir /srv/blazegraph
$ sudo chgrp project-wikidata-query /srv/blazegraph
$ sudo chmod g+rwx /srv/blazegraph

Clone the Blazegraph 1.5.0 release to /srv/blazegraph/BIGDATA_RELEASE_1_5_0

$ git clone -b BIGDATA_RELEASE_1_5_0 --single-branch git://git.code.sf.net/p/bigdata/git \
  /srv/blazegraph/BIGDATA_RELEASE_1_5_0

Extract Journal to /srv/blazegraph/BIGDATA_RELEASE_1_5_0/bigdata.jnl

$ gunzip -c -k /data/project/blazegraph/bigdata-wikidata-statements.jnl.gz > \
  /srv/blazegraph/BIGDATA_RELEASE_1_5_0/bigdata.jnl

Install ant

$ sudo apt-get install ant

Test Blazegraph

This part is optional.

Tunnel port 9999

$ ssh -L 9999:localhost:9999 wdq-bg1

Test Blazegraph:

$ cd /srv/blazegraph/BIGDATA_RELEASE_1_5_0
$ ant start-blazegraph

"Use" the kb database via http://localhost:9999/bigdata/#namespaces.

Run a query through http://localhost:9999/bigdata/#query:

prefix wdq: <http://www.wikidata.org/entity/>
prefix wdo: <http://www.wikidata.org/ontology#>
prefix xs: <http://www.w3.org/2001/XMLSchema#>
select ?entity ?date WHERE {
  ?entity ?relatedTo ?dateS .
  ?dateS wdq:P569v ?dateV .
  ?dateV wdo:preferredCalendar wdq:Q1985727 .
  ?dateV wdo:time ?date .
  FILTER (?date > "1918-04-11"^^xs:date && ?date < "1918-06-11"^^xs:date)
}

Blazegraph HA journal cluster

For more details, follow along with these steps as a guide.

Create the deployment artifacts:

$ cd /srv/blazegraph/BIGDATA_RELEASE_1_5_0
$ ant deploy-artifact

Substitute some values in config.sh. You can also just append these to the end of the file, and they'll override any existing values already in config.sh.

/srv/blazegraph/BIGDATA_RELEASE_1_5_0/dist/bigdata/bin/config.sh:

export FEDNAME=wdq-bg
export FED_DIR=/srv/blazegraph
export LOGICAL_SERVICE_ID=HA-Replication-Cluster-1
export LOCATORS="jini://wdq-bg1/,jini://wdq-bg2/,jini://wdq-bg3/"
export ZK_SERVERS="wdq-bg1:2181,wdq-bg2:2181,wdq-bg3:2181"
export REPLICATION_FACTOR=3
export JAVA_OPTS="${JAVA_OPTS} -Xmx4g"

Change the write cache buffer count in HAJournal.config.

/srv/blazegraph/BIGDATA_RELEASE_1_5_0/dist/bigdata/var/config/jini/HAJournal.config:

// new NV(Options.WRITE_CACHE_BUFFER_COUNT,ConfigMath.getProperty("WRITE_CACHE_BUFFER_COUNT","2000")),
new NV(Options.WRITE_CACHE_BUFFER_COUNT,ConfigMath.getProperty("WRITE_CACHE_BUFFER_COUNT","6")), 

Switch to user blazegraph.

Start the Zookeeper:

$ cd /srv/zookeeper/zookeeper-3.4.6/bin
$ ./zkServer.sh start

Launch Blazegraph:

$ /srv/blazegraph/BIGDATA_RELEASE_1_5_0/dist/bigdata/bin/startHAServices

To be able to access all three servers locally, use ssh tunneling:

$ ssh wdq-bg1 -L 8081:localhost:8080
$ ssh wdq-bg2 -L 8082:localhost:8080
$ ssh wdq-bg3 -L 8083:localhost:8080

Now you can browse to localhost:8081 for wdq-bg1, localhost:8082 for wdq-bg2, and localhost:8083 for wdq-bg3.

DB test server

DB test server with big disk (160G) is db01. Blazegraph is set up to run under blazegraph user with home at /srv/blazegraph. Running (after su'ing to blazegraph user):

 
$ cd /srv/blazegraph
$ sh run.sh
Edit documentation

Server admin log

2021-02-02

  • 07:29 dcaro: large VM wcqs-beta-01 is exhausting the hosts disk space (cloudvirt-wdqs1001) (T273579)

2020-10-01

  • 18:31 andrewbogott: moving wcqs-beta-01 to cloudvirt-wdqs1001 so I can upgrade 1002 to Buster

2020-07-16

  • 18:58 andrewbogott: removing role::sdoc::cloud from sdcquery01 because that role seems to not exist

2020-07-08

  • 17:06 bd808: Added self (bd808) as projectadmin

2019-06-20

  • 14:12 andrewbogott: moving wdqs-test to a new cloudvirt

2019-01-31

  • 12:07 arturo: VM instances ldfclient-new, were stoppe... (more)