EXist-db testing

This page will explore how to add (automated) tests to the XQuery code of art-decor.

Background information

Testing an eXist XQuery application like art-decor can (and probably must) be done on two levels:

  1. Testing function libraries (XQuery modules): the inside of the application
  2. Testing XQuery scripts: The outside, the interface of, the application

General notes

  • Testing of any kind is done best using some kind of testing framework: An application in which you can declare the tests (usually in some XML dialect), run them and finally generates a report.
  • You can, in the absence of a framework, of course write out the tests in XQuery, but... who'll test the test? And you'll need some central functionality around for stuff like reporting. So usually this approach ends up being a framework after all.
  • In an elaborate system, like art-decor, simple tests are not enough. You'll have to do things like, for instance, setting up a some art-decor reference project, because what comes out of functions and scripts depends on the state of the database. A test framework should be able to set this up.
  • Comparing a result when its XML is not trivial. To be complete, a test framework must be able to compare a piece of XML with one or more of the following:
    • A literal piece of XML
    • An XML Schema (or RelaxNG schema)
    • A Schematron schema
    • Some XPath expressions
  • But whatever you decide to use, test suits are software as well and take effort to create and maintain! Don't underestimate this.

Testing function libraries

Testing function libraries means calling the functions with several different inputs and compare the output. Some properties of this:

  • In- and output parameters can be any data type. This includes, besides the usual atomic types like strings and numbers: sequences, elements, attributes, documents, maps, arrays, etc. Art-decor uses at least elements.
  • eXist has a native test suite called XQSuite, but its utterly incomplete and totally insufficient. See below.
  • There is no directly applicable test framework for eXist to the best of my knowledge.
  • A promising candidate is XSpec. This is able to test XQuery modules (experimentally) but it runs outside eXist only, so no deal.

Testing XQuery scripts

Although not completely the same, testing XQuery scripts for art-decor comes down to interface testing: Firing REST calls at the database and checking the returned result.

  • There are several REST test tools, like SoapUI and Postman.
  • SoapUI seems the most promising one because it offers the most XML testing functionality (XPath and schema).
    • It has free version which might be sufficient (we can tell only by doing it in practice). The non-free version costs ~€600/year.
  • There is not much enthusiasm within Nictiz for adding yet another tool to the toolchain.

Why the eXist-db XQSuite is unsuitable

eXist offers a standard testing framework for testing XQuery function modules, XQSuite. It seems however totally unsuitable for anything but simple testing scenarios.

XQSuite works by adding annotations to the function declaration, for example:

 declare 
   %test:arg("a", 3) %test:assertEquals(27)
   %test:arg("a", 4) %test:assertEquals(64)
 function tt:testthis(
   $a as xs:integer
 ) as xs:integer
 {
   $a * $a * $a
 };

XQsuite doesn't fit the bill because:

  • It does not support passing parameters other than atomic types (strings, integers, etc.). You can't pass elements, attributes or documents as function parameters.
  • The same is true for the function's result type. You can't test it when it's not an atomic type.
  • You can't add any documentations or annotations to a test or a bunch of tests. This means that when you get an error you have to search the tests where it came from and infer from that what's wrong.
  • You can't use other functions, variables, etc. in setting up the test or testing the result. Everything passed in and tested against must be literal.
  • When your function is more than trivial (and most are) the list of test annotations will be rather long and clutter the code
  • According to the documentation you can do a test setup and tear-down, but I couldn't get that working. Anyway, that will only work for testing a single function. But what if you want to do an extensive time-consuming setup/tear-down (e.g. setting up a reference art-decor-project) once for a whole group of tests?