Difference between revisions of "EXist-db testing"

(Why the eXist-db XQSuite is unsuitable)
Line 1: Line 1:
 
This page will explore how to add (autmated) tests to the XQuery code of art-decor.
 
This page will explore how to add (autmated) tests to the XQuery code of art-decor.
  
== Why the eXist-db XQSuite is unsuitable ==
+
== Background information ==
 +
 
 +
Testing an eXist XQuery application like art-decor can (and probably must) be done on two levels:
 +
 
 +
# Testing function libraries (XQuery modules): the inside of the application
 +
# Testing XQuery scripts: The outside, the interface of, the application
 +
 
 +
=== General notes  ===
 +
 
 +
* Testing of any kind is done best using some kind of testing framework: An application in which you can ''declare'' the tests (usually in some XML dialect), run them and finally generates a report.
 +
* In an elaborate  system, like art-decor, simple tests are not enough. You'll have to do things like, for instance, setting up a some art-decor reference project, because what comes out of functions and scripts depends on the state of the database. A test framework should be able to set this up.
 +
* Comparing a result when its XML is not trivial. To be complete, a test framework must be able to compare a piece of XML with one or more of the following:
 +
** A literal piece of XML
 +
** An XML Schema (or RelaxNG schema)
 +
** A Schematron schema
 +
** Some XPath expressions
 +
 
 +
 
 +
=== Testing function libraries ===
 +
 
 +
Testing function libraries means calling the functions with several different inputs and compare the output. Some properties of this:
 +
 
 +
* In- and output parameters can be ''any'' data type. This includes, besides the usual atomic types like strings and numbers,  sequences, elements, attributes, documents, maps, arrays, etc. art-decor uses at least elements.
 +
* eXist has a native test suite called XQSuite, but its utterly incomplete and totally insufficient. See below.
 +
* There is no directly applicable test framework for eXist to the best of my knowledge.
 +
* A promising candidate is [https://github.com/xspec/xspec/wiki XSpec]. This is able to test XQuery modules (experimentally) but it runs outside eXist only.
 +
 
 +
==== Why the eXist-db XQSuite is unsuitable ====
  
 
eXist offers a standard testing framework for testing XQuery function modules, [http://exist-db.org/exist/apps/doc/xqsuite.xml XQSuite]. It seems however totally unsuitable for anything but simple testing scenarios.  
 
eXist offers a standard testing framework for testing XQuery function modules, [http://exist-db.org/exist/apps/doc/xqsuite.xml XQSuite]. It seems however totally unsuitable for anything but simple testing scenarios.  

Revision as of 11:55, 15 October 2018

This page will explore how to add (autmated) tests to the XQuery code of art-decor.

Background information

Testing an eXist XQuery application like art-decor can (and probably must) be done on two levels:

  1. Testing function libraries (XQuery modules): the inside of the application
  2. Testing XQuery scripts: The outside, the interface of, the application

General notes

  • Testing of any kind is done best using some kind of testing framework: An application in which you can declare the tests (usually in some XML dialect), run them and finally generates a report.
  • In an elaborate system, like art-decor, simple tests are not enough. You'll have to do things like, for instance, setting up a some art-decor reference project, because what comes out of functions and scripts depends on the state of the database. A test framework should be able to set this up.
  • Comparing a result when its XML is not trivial. To be complete, a test framework must be able to compare a piece of XML with one or more of the following:
    • A literal piece of XML
    • An XML Schema (or RelaxNG schema)
    • A Schematron schema
    • Some XPath expressions


Testing function libraries

Testing function libraries means calling the functions with several different inputs and compare the output. Some properties of this:

  • In- and output parameters can be any data type. This includes, besides the usual atomic types like strings and numbers, sequences, elements, attributes, documents, maps, arrays, etc. art-decor uses at least elements.
  • eXist has a native test suite called XQSuite, but its utterly incomplete and totally insufficient. See below.
  • There is no directly applicable test framework for eXist to the best of my knowledge.
  • A promising candidate is XSpec. This is able to test XQuery modules (experimentally) but it runs outside eXist only.

Why the eXist-db XQSuite is unsuitable

eXist offers a standard testing framework for testing XQuery function modules, XQSuite. It seems however totally unsuitable for anything but simple testing scenarios.

XQSuite works by adding annotations to the function declaration, for example:

 declare 
   %test:arg("a", 3) %test:assertEquals(27)
   %test:arg("a", 4) %test:assertEquals(64)
 function tt:testthis(
   $a as xs:integer
 ) as xs:integer
 {
   $a * $a * $a
 };

XQsuite doesn't fit the bill because:

  • It does not support passing parameters other than atomic types (strings, integers, etc.). You can't pass elements, attributes or documents as function parameters.
  • The same is true for the function's result type. You can't test it when it's not an atomic type.
  • You can't add any documentations or annotations to a test or a bunch of tests. This means that when you get an error you have to search the tests where it came from and infer from that what's wrong.
  • You can't use other functions, variables, etc. in setting up the test or testing the result. Everything passed in and tested against must be literal.
  • When your function is more than trivial (and most are) the list of test annotations will be rather long and clutter the code
  • According to the documentation you can do a test setup and tear-down, but I couldn't get that working. Anyway, that will only work for testing a single function. But what if you want to do an extensive time-consuming setup/tear-down (e.g. setting up a reference art-decor-project) once for a whole group of tests?