Jump to content

User:AndreaWest/WDQS Testing/Running Iguana

From Wikitech

The Iguana framework will be used to evaluate query compliance and performance on a local installation of a database. Changes were made to Dice-Group's codebase in order to support the desired metrics. This page explains the changes, as well as providing references to all the backing code. Also included are the steps to execute the tests, using a Stardog DB for the example, and details on how to extend both the metrics and the tests.

Testing Overview

The Iguana infrastructure was forked from the "main" branch of the Dice Group IGUANA repository. The tests are manually defined as discussed here.

Iguana is defined as a maven project. Java 11 (AdoptOpenJDK build 11.0.10+9) was used in the development and execution of these tests. The current (as of 13 June 2022) iguana-3.3.1.jar file can be directly downloaded from the iguana.corecontroller/src/test/resources/jar-test directory in the GitHub repository.

Alternately, to build a new version of Iguana, begin by installing maven. This is accomplished by downloading the maven archive file, unpacking it and then adding the bin folder with the "mvn" command to your PATH. In addition, the bash profile was modified to give additional memory to the Iguana JVM. The following changes were made to the .profile: 

# Setting up for IGUANA
export IGUANA_JVM=-Xmx8g
PATH="/appropriate_path_to_apache_maven-dir/bin:${PATH}"

To verify the maven installation and profile changes, execute "mvn -v". The output should be similar to: 

% mvn -v
Apache Maven 3.8.5 (3599d3414f046de2324203b78ddcf9b5e4388aa0)
Maven home: /my_path/apache-maven-current
Java version: 11.0.10, vendor: AdoptOpenJDK, runtime: /Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac o''s'' x", version: "10.16", arch: "x86_64", family: "mac"

To build Iguana, execute the following: 

git clone https://github.com/AndreaWesterinen/IGUANA.git
cd IGUANA
mvn install -pl iguana.corecontroller -am

The last instruction, above, indicates that the IGUANA CoreController should be built along with all its dependencies (i.e., the Commons and ResultProcessor modules). While building, the environment executes its defined tests. Since these tests check for valid function and error conditions, various errors will be reported. These are not a concern unless the overall build ends with "[ERROR]" messages in red.

If the build is successful, something similar to the following will be displayed: 

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Iguana Commons 3.3.1:
[INFO]
[INFO] Iguana Commons ..................................... SUCCESS [  8.206 s]
[INFO] Iguana ResultProcessor ............................. SUCCESS [  4.766 s]
[INFO] Iguanas Core Controller ............................ SUCCESS [02:17 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:30 min
[INFO] Finished at: 2022-06-02T23:45:02-04:00
[INFO] ------------------------------------------------------------------------

Note that, if needed, the individual Commons or ResultProcessor modules can be individually built using the "mvn install" command above, but replacing iguana.corecontroller with either iguana.commons or iguana.resultprocessor. The "-am" argument is not needed for iguana.commons, since that module has no dependencies.

The resulting jar file (iguana-3.3.1.jar) is located in the iguana.corecontroller/target directory.

Code Modifications

Modifications were made to the original Iguana codebase and are found in the forked repository. The details are as follows:

  • pom.xml files in the root directory, corecontroller, commons and resultprocessor were updated:
    • The Iguana version references were changed to 3.3.1 (from 3.3.0)
    • The log4j version was rev'ed to the latest (2.17.2) and junit to the latest v4 version (4.13.2)
    • Jena was updated to version 4.2.0 since that was referenced in the (original) IGUANA repository's pull requests
      • Moving to Jena 4.5.0 was attempted but the ResultProcessor module failed to build with the errors
        • metaTest(org.aksw.iguana.rp.storage.impl.TriplestoreStorageTest) Time elapsed: 0.252 sec <<< ERROR!
          • java.lang.NoSuchMethodError: 'org.apache.jena.update.UpdateProcessor org.apache.jena.update.UpdateExecutionFactory.createRemote(org.apache.jena.update.UpdateRequest, java.lang.String, org.apache.http.client.HttpClient)'
        • dataTest(org.aksw.iguana.rp.storage.impl.TriplestoreStorageTest) Time elapsed: 0.005 sec <<< ERROR!
          • java.lang.NoSuchMethodError: 'org.apache.jena.update.UpdateProcessor org.apache.jena.update.UpdateExecutionFactory.createRemote(org.apache.jena.update.UpdateRequest, java.lang.String, org.apache.http.client.HttpClient)'
        • Further debugging and updates were not attempted as they were not needed for the tests
  • iguana-schema.json files in the iguana.corecontroller/src/main/resources and schema directories were updated:
    • Tabbing and indentations were made consistent for easier readability
    • An extraneous "allOf" reference was removed from the "then" clause for CLIInputFileWorker
    • Removed "additionalProperties" references of "type": "undefined" since this is invalid syntax
      • "additionalProperties" can be defined generically with a boolean or specifying valid types of properties
      • But JSON "additional" extensions were not needed for this work
    • Note that the two .json files are identical and so updates were made in both locations
  • iguana.owl file in the schema directory was updated:
    • To correct where multiple domains were referenced in a property definition
      • These should reference a UNION of the classes, or else the meaning is that an instance must be BOTH classes
      • For example, both a Worker and a Task, which are disjoint concepts
    • To change references to a 'double' datatype (which is not valid for OWL) to xsd:decimal
    • To standardize the indentation for readability
    • To define new properties - meanQueryTime, penalizedMeanQueryTime, geometricMeanQueryTime, penalizedGeometricMeanQueryTime, minQueryTime, maxQueryTime
  • Tests (iguana.commons' ScriptExecutorTest.java, and iguana.corecontroller's CLIProcessManagerTest.java and CLIWorkersTests.java) were updated to adjust for the locations of utilities on MAC OS and to correct grammar
  • A similar change was made to iguana.corecontroller's CLIProcessManager.java (to support MAC OS) and the line counting algorithm slightly modified for improved readability

The modifications noted above are relatively minor. Changes of substance were made to the metrics code and tests in iguana.resultprocessor's iguana.resultprocessor metrics/impl (code) and test/resources/nt (test results) directories. The changes involved:

  • Reporting floating point property values as BigDecimals, instead of 'doubles', for the real number property values
    • This was done since BigDecimals are translated by Jena to xsd:decimal values, which are valid in OWL
    • Test results (.nt files) also had to be modified to report decimal (vs double) values
  • Modifying the processing for the QPS metric to additionally generate mean, geometric mean, and min/max query time values
    • The code for QPS metrics was modified since it output metric information by query for a worker and then generalized across all workers for a dataset (a task)
    • Test results (.nt files) also had to be modified to report the new metrics
    • Note that new metrics are calculated as java 'double's, then converted to BigDecimal, converted to floats, and converted back to BigDecimal in order to avoid floating point arithmetic issues such as the number "150" being represented as "149.99999997"
    • Also note that the geometric mean is calculated using logarithmic sums (versus multiplication) to avoid increasingly large multiplication products as the number of iterations of a test increases
  • Correcting metric descriptions and grammar and spelling errors

In addition, rather significant changes were made to iguana.corecontoller's source and test files (along with a small change to iguana.resultprocessor) in order to support sending SPARQL INSERT/DELETE DATA requests as part of workload testing. This code was only partially implemented in the original IGUANA repository. The following describes the changes:

  • CoreController's SPARQLLanguageProcessor.java in the lang/impl directory
    • Added descriptive properties for UPDATE queries (iprop:insert and iprop:delete booleans)
    • Added code to create the appropriate triples describing either a SPARQL UPDATE or query
  • ResultProcessor's Vocab.java in the vocab directory
    • Added constants for the iprop:insert and iprop:delete predicates
  • CoreController's InstancesQueryHandler.java and DelimInstancesQueryHandler.java in the query/impl directory
    • Set the this.queryFiles property in the generateUPDATE method, which was causing null pointer exceptions in the code
    • Modified the method name, generateUpdatesPerLine, to generateUpdatePerLine (singular), to correspond to the generateQueryPerLine method
  • CoreController's HttpWorker.java in the worker/impl directory
    • Modified code to checked for a null value for abortCurrentRequestFuture, before using the variable
    • Sent UPDATE requests to the updateEndpoint (versus the query endpoint)
    • Updated response checking for an UPDATE request since there is no content when an INSERT/DELETE is successful
      • Previously, code was generating null pointers
      • Now it just checks for HTTP success or failure and records the execution time
  • CoreController's UPDATEWorker.java in the worker/impl directory
    • Modified to implement HttpGetWorker (versus HttpPostWorker) to simplify debugging and not affect other POST functionality
    • Added method, buildRequest, specific to UPDATE, with appropriate logic from both the HttpGet/PostWorkers' code
    • Changed getNextQuery to issue sequential UPDATEs vs random
      • Otherwise, INSERTs/DELETEs would occur out of order - deleting triples that did not exist, or inserting triples after they should have been removed
  • CoreController Test's UPDATEWorkerTest.java in the test/.../worker directory
    • Removed testing using the "DISTRIBUTED" timing strategy since it led to errors in tests and was not necessary for Wikidata evaluations

Please note that due to limited development time, testing of the SPARQL UPDATE changes focused on issuing INSERT/DELETE requests sequentially. Problems may occur if not operating with these test characteristics. For example, as noted above, using the "distributed" timing strategy is NOT possible with this code base.

It should also be noted that the F1MeasureMetric.java and corresponding f1test.nt file were NOT examined. This metric was not reviewed because it was not needed for Wikidata testing and its status is unclear (it is not discussed in the IGUANA documentation).

Executing a Sample Benchmark Test

The following is a modified form of the Stress Test documented in the Iguana Tutorial. It is run on a local Stardog triple store.

The Iguana jar and all data are present in the jar-test directory of the revised Iguana repository.

To execute a set of tests (either with the provided iguana .jar file or one that was locally built), move the datasets directory (maintaining the directory name and contents), as well as the .jar, .yaml and .txt files, into a new directory and:

  • Start the triple store server, in this case, Stardog, using a command such as stardog-admin server start --bind 127.0.0.1
  • Validate that the connections configuration defined in benchmark-testsuite.yaml references the correct endpoint, user name and password
  • In benchmark-testsuite.yaml, modify the number of workers, test execution time, etc. as appropriate for your test
  • Execute java -jar iguana-3.3.1.jar benchmark-testsuite.yaml

Note that the expected results are also captured in the jar-test directory as iguana-results.ttl (F1 metrics are not calculated). An example of the results output is provided at the end of the next section on this page.

Executing the sample (or any) test initially outputs a warning of the form: 

"2022-06-06 11:51:24,975 [main] WARN [com.networknt.schema.JsonMetaSchema] - 
<Unknown keyword then - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword>

This is due to a known problem in the networknt JSON schema validator.

In addition, executing the tests may intermittently output errors of the form: 

2022-06-06 11:51:41,336 [main] ERROR [org.aksw.iguana.cc.worker.AbstractWorker] - <Timeout Executor could be shutdown orderly. Terminating.>

These are related to attempting to end a worker/thread due to the "experiment" time being exceeded. If the worker shutdown does not conclude within the 'awaitTimeout' (=3000), likely due to processing a query, Iguana issues a shutdownNow() request to forcibly terminate it.

Below is an example of the output using the provided .jar, .yaml and queries.txt files. There are a few items to note:

  • The prescriptHook creates the database and loads the data set (as specified in the benchmark-testsuite.yaml file, line 13)
  • 3 worker threads (as specified in the .yaml file, line 23) are started
  • Queries specified in the queries.txt file define the "query mix"
  • The mixes run for 3 seconds (as specified in the .yaml file, line 19)
 % java -jar iguana-3.3.1.jar benchmark-testsuite.yaml
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2022-06-06 11:51:24,975 [main] WARN [com.networknt.schema.JsonMetaSchema] - <Unknown keyword then - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword>
2022-06-06 11:51:25,811 [main] INFO [org.reflections.Reflections] - <Reflections took 739 ms to scan 1 urls, producing 3625 keys and 17762 values >
2022-06-06 11:51:25,811 [main] INFO [org.reflections.Reflections] - <Reflections took 739 ms to scan 1 urls, producing 3625 keys and 17762 values >
2022-06-06 11:51:25,827 [main] INFO [org.aksw.iguana.commons.reflect.ShorthandMapper] - <Found 26 annotated classes>
2022-06-06 11:51:25,827 [main] INFO [org.aksw.iguana.commons.reflect.ShorthandMapper] - <Annotated Classes : [class org.aksw.iguana.cc.worker.impl.CLIInputWorker, class org.aksw.iguana.cc.worker.impl.CLIInputPrefixWorker, class org.aksw.iguana.rp.metrics.impl.AvgQPSMetric, class org.aksw.iguana.cc.lang.impl.ThrowawayLanguageProcessor, class org.aksw.iguana.rp.metrics.impl.QMPHMetric, class org.aksw.iguana.rp.storage.impl.TriplestoreStorage, class org.aksw.iguana.cc.query.impl.PatternQueryHandler, class org.aksw.iguana.cc.worker.impl.UPDATEWorker, class org.aksw.iguana.cc.lang.impl.RDFLanguageProcessor, class org.aksw.iguana.cc.tasks.impl.Stresstest, class org.aksw.iguana.cc.worker.impl.HttpGetWorker, class org.aksw.iguana.cc.query.impl.InstancesQueryHandler, class org.aksw.iguana.cc.worker.impl.HttpPostWorker, class org.aksw.iguana.cc.worker.impl.CLIWorker, class org.aksw.iguana.rp.metrics.impl.NoQPHMetric, class org.aksw.iguana.cc.worker.impl.SPARQLWorker, class org.aksw.iguana.rp.metrics.impl.NoQMetric, class org.aksw.iguana.cc.lang.impl.SPARQLLanguageProcessor, class org.aksw.iguana.cc.query.impl.DelimInstancesQueryHandler, class org.aksw.iguana.cc.worker.impl.CLIInputFileWorker, class org.aksw.iguana.rp.storage.impl.NTFileStorage, class org.aksw.iguana.rp.metrics.impl.F1MeasureMetric, class org.aksw.iguana.rp.metrics.impl.EachQueryMetric, class org.aksw.iguana.rp.storage.impl.RDFFileStorage, class org.aksw.iguana.rp.metrics.impl.QPSMetric, class org.aksw.iguana.cc.worker.impl.MultipleCLIInputWorker]>
2022-06-06 11:51:26,043 [main] INFO [org.aksw.iguana.rp.controller.RPController] - <Storages : {RDFFileStorage}>
2022-06-06 11:51:26,043 [main] INFO [org.aksw.iguana.rp.controller.RPController] - <GlobalMetrics : {QPS, EachQuery, QMPH, AvgQPS, NoQ, NoQPH}>
2022-06-06 11:51:26,044 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Executing preScriptHook>
2022-06-06 11:51:26,048 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Finished preScriptHook>
Bulk loading data to new database iguana-tests.
Loaded 13,114 triples to iguana-tests from 1 file(s) in 00:00:00.412 @ 31.8K triples/sec.
Successfully created database 'iguana-tests'.

2022-06-06 11:51:28,144 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Executing Task [1: ekaw-2012, stardog, Stresstest]>
2022-06-06 11:51:28,177 [main] INFO [org.aksw.iguana.cc.query.impl.InstancesQueryHandler] - <[QueryHandler: {org.aksw.iguana.cc.query.impl.InstancesQueryHandler}] Queries will now be instantiated>
2022-06-06 11:51:28,181 [main] INFO [org.aksw.iguana.cc.query.impl.InstancesQueryHandler] - <[QueryHandler: {org.aksw.iguana.cc.query.impl.InstancesQueryHandler}] Finished instantiation of queries>
2022-06-06 11:51:28,240 [main] INFO [org.aksw.iguana.rp.experiment.ExperimentManager] - <Got start flag for experiment task ID 1654530686/1/1>
2022-06-06 11:51:28,245 [main] INFO [org.aksw.iguana.rp.experiment.ExperimentManager] - <Will start experiment task with ID 1654530686/1/1 now.>
2022-06-06 11:51:28,245 [main] INFO [org.aksw.iguana.cc.tasks.impl.Stresstest] - <Task with ID {1654530686/1/1} will be executed now>
2022-06-06 11:51:28,246 [main] INFO [org.aksw.iguana.cc.tasks.impl.Stresstest] - <[TaskID: {1654530686/1/1}]All {3} workers have been started>
2022-06-06 11:51:28,246 [pool-8-thread-2] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Starting Worker[{SPARQLWorker} : {1}].>
2022-06-06 11:51:28,246 [pool-8-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Starting Worker[{SPARQLWorker} : {2}].>
2022-06-06 11:51:28,246 [pool-8-thread-1] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Starting Worker[{SPARQLWorker} : {0}].>
2022-06-06 11:51:28,678 [pool-8-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 1.0 queryMixes>
2022-06-06 11:51:28,831 [pool-2-thread-1] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 1.0 queryMixes>
2022-06-06 11:51:28,833 [pool-4-thread-1] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 1.0 queryMixes>
2022-06-06 11:51:28,879 [pool-8-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 2.0 queryMixes>
...

There are two data sets. So, after 10 seconds, all the workers are ended, the first part of the "experiment" ends, and its metrics are tallied. Then, the postscriptHook is executed (as specified in the .yaml file, line 14), and the second part of the "experiment" begins with the execution of the prescriptHook to load the new data set. This is reported as follows: 

...
2022-06-06 11:51:41,168 [pool-8-thread-2] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 100.0 queryMixes>
2022-06-06 11:51:41,256 [pool-8-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 101.0 queryMixes>
2022-06-06 11:51:41,290 [pool-8-thread-2] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 101.0 queryMixes>
2022-06-06 11:51:41,336 [main] ERROR [org.aksw.iguana.cc.worker.AbstractWorker] - <Timeout Executor could be shutdown orderly. Terminating.>
2022-06-06 11:51:41,338 [pool-8-thread-2] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Stopping Worker[{SPARQLWorker} : {1}].>
2022-06-06 11:51:41,373 [pool-8-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 102.0 queryMixes>
2022-06-06 11:51:41,479 [pool-8-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 103.0 queryMixes>
...
2022-06-06 11:51:44,133 [pool-8-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 128.0 queryMixes>
2022-06-06 11:51:44,238 [pool-8-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 129.0 queryMixes>
2022-06-06 11:51:44,338 [main] ERROR [org.aksw.iguana.cc.worker.AbstractWorker] - <Timeout Executor could be shutdown orderly. Terminating.>
2022-06-06 11:51:44,340 [pool-8-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Stopping Worker[{SPARQLWorker} : {2}].>
2022-06-06 11:51:47,345 [main] ERROR [org.aksw.iguana.cc.worker.AbstractWorker] - <Timeout Executor could be shutdown orderly. Terminating.>
2022-06-06 11:51:47,346 [main] INFO [org.aksw.iguana.cc.tasks.impl.Stresstest] - <[TaskID: {1654530686/1/1}] Will shutdown and await termination in 5s.>
2022-06-06 11:51:47,346 [main] INFO [org.aksw.iguana.cc.tasks.impl.Stresstest] - <[TaskID: {1654530686/1/1}] Task completed. Thread finished status true>
2022-06-06 11:51:47,347 [main] INFO [org.aksw.iguana.rp.experiment.ExperimentManager] - <Got end Flag for experiment task ID 1654530686/1/1>
2022-06-06 11:51:47,355 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Executing postScriptHook /Users/andreaw/Hold/stardog-current/bin/stardog-admin db drop iguana-tests>
Successfully deleted database 'iguana-tests'.
2022-06-06 11:51:49,264 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Finished postScriptHook>
2022-06-06 11:51:49,264 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Executing preScriptHook>
2022-06-06 11:51:49,264 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Finished preScriptHook>
Bulk loading data to new database iguana-tests.
Loaded 32,022 triples to iguana-tests from 1 file(s) in 00:00:00.461 @ 69.5K triples/sec.
Successfully created database 'iguana-tests'.

2022-06-06 11:51:51,406 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Executing Task [1: iswc-2010, stardog, Stresstest]>
2022-06-06 11:51:51,408 [main] INFO [org.aksw.iguana.cc.query.impl.InstancesQueryHandler] - <[QueryHandler: {org.aksw.iguana.cc.query.impl.InstancesQueryHandler}] Queries will now be instantiated>
2022-06-06 11:51:51,409 [main] INFO [org.aksw.iguana.cc.query.impl.InstancesQueryHandler] - <[QueryHandler: {org.aksw.iguana.cc.query.impl.InstancesQueryHandler}] Finished instantiation of queries>
2022-06-06 11:51:51,411 [main] INFO [org.aksw.iguana.rp.experiment.ExperimentManager] - <Got start flag for experiment task ID 1654530686/2/1>
2022-06-06 11:51:51,412 [main] INFO [org.aksw.iguana.rp.experiment.ExperimentManager] - <Will start experiment task with ID 1654530686/2/1 now.>
2022-06-06 11:51:51,412 [main] INFO [org.aksw.iguana.cc.tasks.impl.Stresstest] - <Task with ID {1654530686/2/1} will be executed now>
2022-06-06 11:51:51,412 [main] INFO [org.aksw.iguana.cc.tasks.impl.Stresstest] - <[TaskID: {1654530686/2/1}]All {3} workers have been started>
2022-06-06 11:51:51,413 [pool-15-thread-2] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Starting Worker[{SPARQLWorker} : {1}].>
2022-06-06 11:51:51,413 [pool-15-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Starting Worker[{SPARQLWorker} : {2}].>
2022-06-06 11:51:51,413 [pool-15-thread-1] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Starting Worker[{SPARQLWorker} : {0}].>
2022-06-06 11:51:51,736 [pool-15-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 1.0 queryMixes>
2022-06-06 11:51:51,876 [pool-9-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 1.0 queryMixes>
2022-06-06 11:51:51,877 [pool-11-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 1.0 queryMixes>
2022-06-06 11:51:52,052 [pool-15-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 2.0 queryMixes>
2022-06-06 11:51:52,082 [pool-15-thread-1] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 2.0 queryMixes>
2022-06-06 11:51:52,082 [pool-15-thread-2] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 2.0 queryMixes>
...

The test executions conclude with the following output: 

...
2022-06-06 11:52:07,139 [pool-15-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 57.0 queryMixes>
2022-06-06 11:52:07,386 [pool-15-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Worker executed 58.0 queryMixes>
2022-06-06 11:52:07,480 [main] ERROR [org.aksw.iguana.cc.worker.AbstractWorker] - <Timeout Executor could be shutdown orderly. Terminating.>
2022-06-06 11:52:07,482 [pool-15-thread-3] INFO [org.aksw.iguana.cc.worker.AbstractWorker] - <Stopping Worker[{SPARQLWorker} : {2}].>
2022-06-06 11:52:10,486 [main] ERROR [org.aksw.iguana.cc.worker.AbstractWorker] - <Timeout Executor could be shutdown orderly. Terminating.>
2022-06-06 11:52:10,486 [main] INFO [org.aksw.iguana.cc.tasks.impl.Stresstest] - <[TaskID: {1654530686/2/1}] Will shutdown and await termination in 5s.>
2022-06-06 11:52:10,486 [main] INFO [org.aksw.iguana.cc.tasks.impl.Stresstest] - <[TaskID: {1654530686/2/1}] Task completed. Thread finished status true>
2022-06-06 11:52:10,487 [main] INFO [org.aksw.iguana.rp.experiment.ExperimentManager] - <Got end Flag for experiment task ID 1654530686/2/1>
2022-06-06 11:52:10,491 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Executing postScriptHook /Users/andreaw/Hold/stardog-current/bin/stardog-admin db drop iguana-tests>
Successfully deleted database 'iguana-tests'.
2022-06-06 11:52:12,362 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Finished postScriptHook>
2022-06-06 11:52:12,429 [main] INFO [org.aksw.iguana.cc.config.IguanaConfig] - <Finished benchmark>
2022-06-06 11:52:12,429 [main] INFO [org.aksw.iguana.cc.controller.MainController] - <Stopping Iguana>

How to Change or Extend the Tests and Infrastructure

Modifications to Iguana are fairly straightforward and most occur in the iguana.resultprocessor module, related to metrics. The commit history for the repository can be reviewed to understand how the original code base was modified.

Originally, a calculation of standard deviation of the mean query time was to be added. Due to a need to focus on the actual test definitions, this was set as a lower priority. In addition, the current metrics provide some indication of the variability of the times by reporting the minimum, maximum and geometric means. If standard deviation is needed in the future, there are at least three options for supplying it. Due to the need to retain results data (potentially) over 10s of minutes, the latter, two approaches may be less memory and time intensive.

  • Design 1 - Capture the individual time values of successful queries in QPSMetric.java's putResults method in a new double[] (array) in the tmp array
    • When outputting the results, in QPSMetric.java's addToModel, after calculating the mean, iterate through the double array to get the differences from the mean to calculate the standard deviation
    • Save the resulting metric using m.add(), similar to the other metrics
  • Design 2 - Record the individual time values of successful queries in QPSMetric.java's putResults method in separate OutputStreams for each query + each worker, and an additional stream for each query across all workers
    • When outputting the results, before completing the QPSMetric.java's qpsClose method, iterate through the streams to determine the individual means and calculate the standard deviations (or store and reuse the means that were calculated earlier)
    • Add the new standard deviation triples to the model, "m", before calling the sendData method
    • The instances to be added are defined using the IRIs, http://iguana-benchmark.eu/resource/<experiment_id>/<dataset_number>/<database_number>/<worker_number>/<sparql_query_number> for query + worker results, and http://iguana-benchmark.eu/resource/<experiment_id>/<data_set_number>/<database_number>/<sparql_query_number> for query results across all workers
      • Note that the data set and database numbers start from 1, and the worker numbers start at 0
      • SPARQL query numbers correspond to the row numbers in the file where the queries are defined, starting with line 1 which is query number 0
  • Design 3 - Specify that the EachQuery metric is to be captured when running the tests, and after the tests are complete, parse the details from the "results" file
    • The EachQuery metric causes the individual query details to be written to the results file
    • Standard deviation and other statistics can be easily calculated and added to the results file or output separately

The results for EachQuery are written as shown below. Note that: 

<http://iguana-benchmark.eu/resource/1654530686/2/1/2/sparql2/4>
        iprop:code        "1"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:queryID     <http://iguana-benchmark.eu/resource/534030630/sparql2> ;
        iprop:resultSize  "16584"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:run         "4"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:success     true ;
        iprop:time        81.587 .

<http://iguana-benchmark.eu/resource/1654530686/1/1/0/sparql1/9>
        iprop:code        "1"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:queryID     <http://iguana-benchmark.eu/resource/534030630/sparql1> ;
        iprop:resultSize  "10"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:run         "9"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:success     true ;
        iprop:time        2.426 .

...

<http://iguana-benchmark.eu/resource/534030630/sparql2>
        a                   iont:Query ;
        <http://www.w3.org/2000/01/rdf-schema#ID> "2" ;
        <http://www.w3.org/2000/01/rdf-schema#label>
                "SELECT * {?s <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?o}" ;
        iprop:aggregations  false ;
        iprop:filter        false ;
        iprop:groupBy       false ;
        iprop:having        false ;
        iprop:offset        false ;
        iprop:optional      false ;
        iprop:orderBy       false ;
        iprop:triples       "1"^^<http://www.w3.org/2001/XMLSchema#int> ;
        iprop:union         false ;
        <http://www.w3.org/2002/07/owl#sameAs>
                lsqr:q-FerRSCgDSQqzdH2gjCHMs1Ik5kgmosurrDgShTEjZIU .

...

<http://iguana-benchmark.eu/resource/1654530686/2/1/1/sparql2>
        a                             iont:ExecutedQuery ;
        iprop:QPS                     12.786930252936761 ;
        iprop:failed                  "0"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:geometricMeanQueryTime  78.15816 ;
        iprop:maxQueryTime            90.865 ;
        iprop:meanQueryTime           78.20485294117647 ;
        iprop:minQueryTime            74.415 ;
        iprop:penalizedGeometricMeanQueryTime 78.15816 ;
        iprop:penalizedMeanQueryTime  78.20485294117647 ;
        iprop:penalizedQPS            12.786930252936761 ;
        iprop:queryExecution          <http://iguana-benchmark.eu/resource/1654530686/2/1/1/sparql2/33> , <http://iguana-benchmark.eu/resource/1654530686/2/1/1/sparql2/30> , ...> ;
        iprop:queryID                 <http://iguana-benchmark.eu/resource/534030630/sparql2> ;
        iprop:resultSize              "16584"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:succeeded               "34"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:timeOuts                "0"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:totalTime               2658.965 ;
        iprop:unknownException        "0"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:wrongCodes              "0"^^<http://www.w3.org/2001/XMLSchema#long> .

...

<http://iguana-benchmark.eu/resource/1654530686/2/1/sparql2>
        a                             iont:ExecutedQuery ;
        iprop:QPS                     12.761089727682062 ;
        iprop:failed                  "0"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:geometricMeanQueryTime  78.28266 ;
        iprop:maxQueryTime            90.865 ;
        iprop:meanQueryTime           78.36321359223301 ;
        iprop:minQueryTime            74.271 ;
        iprop:penalizedGeometricMeanQueryTime
                78.28266 ;
        iprop:penalizedMeanQueryTime  78.36321359223301 ;
        iprop:penalizedQPS            12.761089727682062 ;
        iprop:queryID                 <http://iguana-benchmark.eu/resource/534030630/sparql2> ;
        iprop:resultSize              "16584"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:succeeded               "103"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:timeOuts                "0"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:totalTime               8071.411 ;
        iprop:unknownException        "0"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:wrongCodes              "0"^^<http://www.w3.org/2001/XMLSchema#long> .

...

<http://iguana-benchmark.eu/resource/1654530686/2/1>
        a                      iont:Task , ires:Stresstest ;
        <http://www.w3.org/2000/01/rdf-schema#endDate>
                "2022-06-06T15:52:10.486Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        <http://www.w3.org/2000/01/rdf-schema#startDate>
                "2022-06-06T15:51:51.412Z"^^<http://www.w3.org/2001/XMLSchema#dateTime> ;
        iprop:AvgQPS           183.5456989374586 ;
        iprop:NoQ              "308"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:NoQPH            113065.54472951844 ;
        iprop:QMPH             37688.514909839476 ;
        iprop:connection       ires:stardog ;
        iprop:metric           ires:AvgQPS , ires:NoQPH , ires:NoQ , ires:EachQuery , ires:QPS , ires:QMPH ;
        iprop:noOfWorkers      "3"^^<http://www.w3.org/2001/XMLSchema#long> ;
        iprop:penalizedAvgQPS  183.5456989374586 ;
        iprop:query            <http://iguana-benchmark.eu/resource/1654530686/2/1/sparql1> , <http://iguana-benchmark.eu/resource/1654530686/2/1/sparql0> , ... ;
        iprop:timeLimit        "10000.0"^^<http://www.w3.org/2001/XMLSchema#double> ;
        iprop:workerResult     <http://iguana-benchmark.eu/resource/1654530686/2/1/2> , <http://iguana-benchmark.eu/resource/1654530686/2/1/1> , ... .

Executing the Wikidata Benchmark Tests

For details on using both TFT and Iguana to evaluate the Wikidata alternatives, see this page.

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=User:AndreaWest/WDQS_Testing/Running_Iguana&oldid=1991617"