Introduction

About this Document

This document describes how to establish continuous load testing with the XLT plug-in for the Jenkins continuous integration system. To work effectively with this plug-in, a basic understanding of XLT, load testing with XLT and Jenkins may be helpful.

Continuous Load Testing

Load testing is usually done as one of the last steps in a project, often only a few weeks before release. This is somewhat problematic since serious performance issues sometimes require architectural changes. If these changes are not tested thoroughly, the risk of undetected bugs increases tremendously.

To spot performance problems early, ideally while the project is still under development, load testing should take place on a regular basis. If you are willing to invest time up front to establish continuous load testing, it will surely pay out later on. XLT provides the tools to make continuous load testing an engineering habit.

Jenkins

Jenkins is a widely used continuous integration system. It helps you set up and control automated build and deployment processes, which can be run regularly. Jenkins can be extended by plug-ins. The XLT plug-in for Jenkins brings load testing and continuous integration together. In Jenkins terms, an XLT load test project is a build project, while an individual load test run is a build.

XLT Plug-In Features

Easy integration of XLT load tests – The plug-in makes it easy for you to run a load test. You simply configure some basic settings via the Jenkins UI and the plug-in will then take care of carrying out the necessary steps with the parameters you have provided. You don’t have to deal with XLT tools directly, that is you don’t need to write shell scripts to drive the load test. Furthermore, the plug-in automatically stores the results, creates load test reports, and updates trend reports for you.

Automatic evaluation of the load test results – Don’t think you’re all set if you just run a load test on a regular basis. You also need to check if the test results are actually acceptable. Doing this manually after each test run doesn’t really fit into the work processes of continuous development; it should therefore be automated as well. To this end, you can define success criteria. For example, you may verify that no errors occurred, that the maximum request runtime was below 10 seconds, and that page X took less than 2 seconds to load. The plug-in tests these criteria after each load test. If a criterion has not been fulfilled, the build will be marked as unstable. This should encourage you to have a closer look at the results and find out what went wrong.

Visualization of the performance trend – To show you when the application behavior has changed and to which extend, the plug-in can visualize a long-term trend across multiple builds. Simply define the values that you would like to watch over time, for example the response time of page X, and the plug-in will then visualize the trend in overview charts on the project page. For an in-depth evaluation of all data, the plug-in can also generate two different types of trend reports which contain all the details.

The plug-in enriches the standard Jenkins project and build pages. On a build’s page, the corresponding load test report is available as well as a list of any success criterion that may have been violated during this build. The project page shows the trend charts and provides links to access the trend reports.

Installation Instructions

First, check the version of Jenkins you are using. The XLT plug-in requires v1.642.3 or later.

To install the XLT plug-in into Jenkins, simply copy the plug-in file <xlt>/tools/xlt-jenkins-plugin-X.X.X/xlt-jenkins-plugin.hpi to the plugins sub-directory of your Jenkins installation. Restart Jenkins for the plug-in to be picked up.

Note that you will also need an XLT installation on the Jenkins machine. This installation acts as a template and will be copied to a temporary location for each load test. To accelerate this step, remove any sub-directory from the installation folder that is not necessary for load testing, such as doc, samples, or tools.

Setting Up an Automated Load Test

Develop an XLT Load Test Project

A regular XLT load test project will be the foundation for your automated load tests, so simply design and implement a load test suite for your application as usual. This is completely independent of Jenkins, there is nothing special to do or to consider.

In case you use Maven or Ivy as dependency management tool, you might take a look at this How-To on how to integrate XLT into your project as dependency.

When the code is complete, prepare the test configurations for which you want to run automatic load tests as separate properties files (for example, smoketest.properties and nightly.properties). Note that you can also do this and other fine-tuning later on.

Now, test your load test suite and configurations thoroughly. Everything should work as expected when being run manually.

Create a Load Test Build Project

An automated load test suite requires a separate build project to be set up. The reason is that the test suite itself needs to be retrieved from a version control system and the code needs to be compiled. This is best accomplished in an own build project, so let’s create one now. In Jenkins, create the load test project and give it a meaningful name and description. Choose “free-style project” as the project type. Now the project needs to be configured.

Step 1 – Connect to your version control system

Every time a load test is to be executed, the latest version of the test suite will be retrieved from a version control system first. So select the adapter for your favourite version control system and configure access credentials and the path to your test suite.

Step 2 – Compile your test suite

Now configure the way how your test suite is compiled. Typically, you would do this via Ant or Maven. Whatever tool you choose, make sure an appropriate build file (build.xml for Ant, pom.xml for Maven) is available in the root of your test suite project. Add the appropriate build step to your project and enter the build target that compiles your test suite.

Step 3 – Configure your load test

Now add the XLT plug-in as a new build step to your project and configure it as follows:

  • Specify the full path to the XLT installation template on the Jenkins machine.
  • Enter the name of the properties file with the load test configuration settings (for example, smoketest.properties). This file will be searched for in the config folder of your test suite.
  • In the agent controller section, check the radio button “Use a single embedded agent controller”.
  • Go with the defaults for all other settings for now.

Please note that with the default configuration the load test will be driven from the Jenkins machine, so the configured load profile should not be too demanding. See below for an overview of what else can be configured.

To generate higher load it might be not enough to run one XLT agent controller from the Jenkins machine. Dedicated load generating machines are recommended. When configuring the XLT plug-in the agent controller section offers several options one of them being “Start agent machines in Amazon’s EC2”. This allows to automatically start Amazon machine images (AMI) in Amazon’s Elastic Compute Cloud (EC2) that can be used to generate the load. Typically, you will use one of the machine images with XLT already installed. For the current list of AMI IDs, see the Xceptance Homepage.

When using the option “Start agent machines in Amazon’s EC2” the XLT master controller has to wait for the remote agent controller EC2 machines to come up. This can take several minutes. This is why you should adjust the XLT property com.xceptance.xlt.mastercontroller.initialResponseTimeout to a value that is useful for the described scenario.

Step 4 – Other project settings

Configure any other project setting you might find useful. For example, specify for how long/for how many builds you want to keep the load test results. You may also want to configure email notification (as a post-build step) to get a message whenever the build was not successful.

Once you’re done with the configuration, start your build project manually to verify that everything works as expected. If all went well, your build should show SUCCESS. If the load test could not be executed for whatever reason (wrong configuration, etc.), the build will be marked as FAILED. If the load test ran to completion, but the configured success criteria were violated, the build will be marked as UNSTABLE.

Define a Trigger for the Load Test

To run your load test automatically, you need to define the event that should trigger the load test build project. You basically have two options:

  1. You can run the load test once another build project completed successfully, for example after your application has been built and deployed. To this end, configure your load test project to depend on the application’s main build project.
  2. You might also run your load test periodically, for example every night, as an independent or stand-alone project. In this case, you would need to configure a time pattern that defines the times when the project will be executed by Jenkins.

Advanced Configuration

Many other details of the XLT plug-in functionality can be configured. This includes:

  • Agent Controllers
  • Report Options
  • Trend Charts and Success Criteria
  • Chart Options

Each configuration value provides an extensive help text, so make sure to click the Help icon next to the value if in doubt what can be configured there.

Please note that the XLT plug-in can be added to a project/job not only once, but multiple times. This allows to run multiple tests with the same test suite in a row, for example a short smoke test, followed by a longer performance test. In this case, each XLT build step has to be configured individually.

A Note on the ‘stepId’ Parameter

As stated above, a job might perform multiple load tests in one build. Each of these load tests creates a result that belongs to a certain result set which is used as input not only for chart generation but also for creating the summary, trend and difference reports.

As it makes no sense to merge all load test results into a single result set, the question was: How can we identify one load test across several builds of the same job? Well, by using an identifier that is distinct for all load test steps in the same job: the step identifier or stepId.

Build Steps in a Job Build Steps in a Job

Success Criteria

The XLT Jenkins plug-in offers two ways to define success criteria that must be met by the results of the test in order to have a successful build/run. In both cases, success criteria are specified as XPath expressions that will be applied on an XML file generated by XLT when creating a test or difference report.

Success Criteria for Test Reports

Test reports are generated automatically by XLT when the load test has finished and its results have been saved to disk. Success criteria for test reports can be specified in the text area named ‘Configuration’ in section ‘Validation/Plot Configuration’. Here you can define which values should be displayed on the various Trend Charts, along with a condition (as XPath predicate) that must be satisfied. For details, please see the corresponding help text.

Success Criteria for Difference Reports

In contrast to Test Reports, Difference Reports will only be generated when the checkbox ‘Create a diff report’ is checked or – if using the xlt pipeline step – when the parameter ‘diffReport’ is given. The difference report option has an optional parameter where you can specify the path to the criteria file that should be used for criteria validation as described in this How-To.

Pipelines

Starting with Jenkins 2.0, build jobs can also be modeled as pipelines which allows greater flexibility and usability.

Since XLT 4.11.0, the Jenkins XLT plug-in supports this new concept entirely. It provides a custom pipeline step called xlt that can be invoked in any pipeline script in the same way as any other Jenkins pipeline step.

Parameters and their values can be obtained easily by using Jenkins’ Snippet Generator. You can use the UI to configure the plug-in as usual and get the corresponding pipeline script snippet upon hitting the ‘Generate Pipeline Script’ button.

Furthermore, the xlt pipeline step returns a result object that can be queried for details if desired. The result object has the following fields:

  • runFailed – (boolean) Indicates whether the build status is FAILED
  • conditionFailed – (boolean) Indicates whether a criterion is not met
  • conditionError – (boolean) Indicates whether a criterion caused an error
  • conditionCritical – (boolean) Indicates whether the build is marked as critical according to the given ‘markCritical’ option
  • conditionMessage – (string) Summary message of all failed and erroneous criteria
  • reportUrl – (string) XLT Report URL
  • diffReportUrl – (string) XLT Difference Report URL
  • testFailures – (list) Information about failed tests. Each object in the list has the following fields:
    • testCaseName – (string) Name of the test
    • actionName – (string) Name of the action where the error occurred
    • message – (string) Error message
  • criteriaFailures – (list) Information about criteria whose condition is not met. Each object in this list has the following fields:
    • id – (string) The criterion’s ID
    • condition – (string) The criterion’s condition
    • message – (string) Failure/error message
  • criteriaErrors – (list) Information about criteria whose evaluation caused an error. The objects have the same fields as criteriaFailures.

Example:

def r = xlt stepId: 'any-step-id', xltTemplateDir: '/path/to/xlt'
echo "Run failed: ${r.runFailed} | Report URL: ${r.reportUrl}"

Feel free to take a look at Jenkins’ Pipeline Reference page which lists all available pipeline steps, along with their documentation, including names, types and description of their parameters and return values.