Introduction

The XLT plug-in for Jenkins allows to configure certain criteria that must be met by the generated report. This useful feature would also make sense for difference or trend reports. This way, you would be able to check for criteria based on how certain values evolved over time or how much they differ compared to a given baseline.
Last but not least, this feature should not be restricted to Jenkins users but work in any environment supported by XLT.

We therefore came to the conclusion to provide a stand-alone tool for this purpose that is able to handle all kind of reports generated by XLT.

Approach

To keep things simple, we decided to use XPath expressions that are evaluated on the parsed XML document to define whether a given criterion is met.
These expressions must evaluate to a boolean result (e.g. count(/some/element) > 0), otherwise it is seen as an error (same as for invalid XPath expressions).

All criteria that should be validated by the tool are defined in a single file – the criteria definition file – which is pure JSON and looks like this:

{
  "criteria": [
    {
      "id": "MaxRequestRuntime",
      "enabled": true,
      "condition": "count(//request/max/relativeDifference[number()>10])=0",
      "message": "Maximum request runtime exceeded 10%" 
    },
    {
      "id": "P95RequestRuntime",
      "enabled": true,
      "condition": "not(//request/percentiles/p95/absoluteDifference[number()>1000])",
      "message": "Request runtime (P95) increased by more than 1 second" 
    },
    {
      "id": "NoRequestErrors",
      "enabled": true,
      "condition": "not(//request/errors/*[number() > 0]",
      "message": "Errorneous requests encountered" 
    }
  ]
}

Each criterion listed in the array criteria is an object with following properties:

  • id – The ID of the criterion (required; non-blank and unique)
  • condition – The criterion’s condition as XPath expression (required; non-blank)
  • enabled – Whether or not the criterion is enabled (optional; defaults to true)
  • message – Failure message (optional)

The output generated by the tool is pure JSON as well and contains the following:

  • used/defined criteria
  • validation status (and failure/error message) of each criterion
  • summary per document
  • name of validated document (as the number of validated documents is potentially greater than 1)

Usage

To use the tool, open a terminal window and execute the following command sequence:

cd <XLT>/bin
./check_criteria.sh -c /path/to/criteria.json /path/to/report.xml

Windows users have to use the appropriate .cmd file located in the same directory.

The tool is able to process more than one XML file in one pass. Simply specify the path of each file as additional argument:

cd <XLT>/bin
./check_criteria.sh -c /path/to/criteria.json /path/to/report1.xml /path/to/report2.xml ... 

Per default, the output will be written to standard-out but the tool also offers an option to write it to a file of your choice.

cd <XLT>/bin
./check_criteria.sh -c /path/to/criteria.json -o /path/to/validation_output.json /path/to/report.xml

Sample output:

{
  "criteria": [
    {
      "id": "MaxRequestRuntime",
      "enabled": true,
      "condition": "count(//request/max/relativeDifference[number()>10])=0",
      "message": "Maximum request runtime exceeded 10%" 
    },
    {
      "id": "P95RequestRuntime",
      "enabled": true,
      "condition": "not(//request/percentiles/p95/absoluteDifference[number()>1000])",
      "message": "Request runtime (P95) increased by more than 1 second" 
    },
    {
      "id": "NoRequestErrors",
      "enabled": true,
      "condition": "not(//request/errors/*[number() > 0]",
      "message": "Errorneous requests encountered" 
    }
  ],
  "checks": [
    {
      "document": "foo/bar/bumm",
      "total": {
        "passed": 1,
        "failed": 1,
        "skipped": 0,
        "error": 1
      },
      "details": {
        "MaxRequestRuntime": { "status": "failed", "message": "Maximum request runtime exceeded 10%"  },
        "P95RequestRuntime": { "status": "passed" },
        "NoRequestErrors": { "status": "error", "message": "Invalid XPath expression" }
      }
    }
  ]
}

Status Codes

Similar as the other tools shipped with XLT, the criteria validation tool returns one of the following status codes:

  • 0 on success
  • 1 if one of its input files or the criteria definition file could not be parsed
  • 2 if a required argument is missing or some of the given arguments are invalid
  • 3 if a criterion is not met or could not be evaluated (caused an error)