Legacy Documentation

This is the old XLT documentation which is here for reference purposes only. You can find the latest version at xltdoc.xceptance.com.

Basic Concepts

When you perform web-based application tests, the possible paths from page to page define the web page flow. The web page flow can be depicted as a directed graph with the vertices representing the web pages and the transitions representing the actions to get from one page to another. Typically, a test scenario covers a certain part of the page flow only, such as a specific path through the application.

XLT provides a programming paradigm that makes use of a three-level architecture (transactions, actions, and requests). These levels are illustrated in the following sections.

Transaction

A transaction is the execution of exactly one test case or test scenario. To perform the scenario, the page flow is modeled in code. The test scenario is implemented as a test case which itself executes a sequence of one or more actions.

Action

An action can be defined as one irreducible step within a test case. Thus, an action interacts with the current page and – as a result – loads the next page. The resulting page is associated with this action and becomes the current page for the next action in the test scenario. Generally, an action triggers one or more requests.

Request

This level is equivalent to the HTTP request level used in web browsers or in any other application that relies on HTTP communication. You don’t have to deal with requests directly because they are automatically generated by the underlying HtmlUnit framework when you perform actions on HTML elements.

Validation

As you’re testing the functionality of applications or pieces of software, you have to check the correctness of all responses. It is strongly recommended that you handle all potential situations and use validations as often as possible. It’s better to have too many checks rather than too few! They can’t do any harm and will increase your confidence that your application works correctly. Thus, make sure you insert as much validation as necessary to detect any abnormal application behavior of the software being tested.

Pre-validation

Each action should have a pre-validation section that checks whether or not all of the required data is available to interact with that page and allow for the advance to the next one. From the end user’s point of view, you simply look for the information on the page that you need to continue your web experience, such as a form to fill in or a link to click. In case the required information can’t be found, an exception is thrown. If you run a load test, XLT will catch this exception and log all relevant information. This lets you evaluate the results after running the test and narrow down error conditions.

Post-validation

Post-validations work similarly to pre-validations. They are used to validate the result of the interaction and ensure that the data matches the expectations.

Example

The following example illustrates a very simple scenario to help you understand the terminology. It is based on the demo application shipped with XLT. Imagine a typical user searching the shop for products. The user will possibly:

  • Open the shop’s home page,
  • Search for a keyword but without results,
  • Search for another keyword, this time with some results
  • Select one of the shown products and open the product detail page

Test Case And Actions Test Case and Actions

In this example, the test scenario is modeled as test case TSearch. A single execution of this test case is a single transaction. OpenHomepage, Search and ViewProductDetails are the actions of this page flow to go from one page to the next. Validations after each page transition ensure you arrived on the right page with the right content.

Available Programming Interfaces

Overview

XLT offers different approaches for writing test cases in Java. Several programming interfaces are available when using the XLT framework. Built on each other, they represent different abstraction levels.

You can extend your test suite using the XLT Scripting API. It lets you start with Java code that has automatically been generated from the recorded Script Developer test cases, and features a high-level command scripting API with a very intuitive syntax. You may also directly access the WebDriver API underlying the XLT Scripting API to write more advanced tests.

XltDriver, which is also part of the XLT framework, is a WebDriver implementation extending the HtmlUnitDriver. Both in turn are built on the HtmlUnit API. HtmlUnit is a headless browser offering a low-level API that lets you have full control when creating web tests.

To serve as the main framework when creating HtmlUnit-based tests, XLT provides the XLT Action API that structures the code in action classes and test case classes.

XLT Framework API XLT Framework API

XLT Test Cases are JUnit4 Tests

Except for script test cases exclusively springing from Script Developer, XLT test cases use a Java test case class with one test() method, regardless of the chosen approach for test writing.

The test() method of each test case class has a @Test annotation (see TSearch code example line 12). XLT builds upon JUnit4 principles and its annotations to implement and tag test cases. This way, each XLT test is in fact a JUnit test enabling XLT tests to be executed just like any other unit test in the IDE or within an existing build process. The sole difference between XLT and standard JUnit4 tests is that XLT tests can only take one active test method per test class. That means that, although there can be an arbitrary number of methods within a class, only one method is permitted to be annotated with @Test. However, this limitation rather serves the purpose of simplification than leading to actual restrictions.

Implementing the test case as a JUnit4 test also lets you use standard JUnit assertion to validate the page, mainly when you create test cases using the HtmlUnit API.

XLT Scripting API

When exporting a Script Developer test case to Java (see Export test cases to Java), you may choose the resulting code to be based on the XLT Scripting API. This API is an easy-to-use programming interface with a simple syntax deriving from the commands available in Script Developer.

You can write test cases from scratch using the XLT Scripting API. However, the most common way is to record a test case with Script Developer and, after exporting it to Java, extend it with the XLT Scripting API.

The following screenshot shows how the test case TSearch introduced in the section above may look like if it would have been recorded using Script Developer. Note that this test case uses validation rather poorly to keep the example short and simple; a real test should have more validations to ensure correct page display.

TSearch Demo Script Test Case TSearch Demo Script Test Case

The automatically generated Java code of the TSearch test case after the export is shown below:

/**
 * <p>Simulates storefront search.</p>
 */
public class TSearch extends AbstractWebDriverScriptTestCase
{
    /**
     * Constructor.
     */
    public TSearch()
    {
        super(new XltDriver(true), "http://localhost:8080/");
    }

    /**
     * Executes the test.
     *
     * @throws Throwable if anything went wrong
     */
    @Test
    public void test() throws Throwable
    {
        // Open the homepage and delete cookies (module call)
        final OpenHomepage _openHomepage = new OpenHomepage();
        _openHomepage.execute();

        //
        // ~~~ Search ~~~
        //
        startAction("Search");
        // Store a search phrase that gives results
        store(resolve("${searchTerm_hits}"), "searchTerm");
        // Execute the search (module call)
		final Search _search = new Search();
        _search.execute(resolve("${searchTerm_hits}"));

        // Validate the entered search phrase is still visible in the input
        assertText("id=searchTextValue", resolve("${searchTerm_hits}"));
        // Validate presence of the search results page headline
        assertElementPresent("id=titleSearchText");
        // Validate the headline contains the search phrase
        assertText("id=titleSearchText",
              resolve("glob:*Your results for your search: '${searchTerm_hits}'*"));
		// validate result counter
        assertText("id=totalProductCount", resolve("${resultProductCount}"));
        //
        // ~~~ ViewProduct ~~~
        //
        startAction("ViewProduct");
        // Assert presence of one of the product thumbnails
        assertElementPresent("id=product0");
        // Store the name of the first product
        storeText("css=#product0 .pInfo .pName", "productName");
        // Click the product link to open the product detail page
        clickAndWait("css=#product0 img");
        // Validate it's the correct product detail page
        assertText("css=#titleProductName", resolve("${productName}"));
    }

Note that the code is structured in blocks embodying the actions. Each action starts with a startAction() command. The other commands are very similar to the commands available in Script Developer. Each Script Developer command has an appropriate counterpart in the XLT Scripting API. The generated test case class extends the abstract class AbstractWebDriverScriptTestCase; it inherits the methods that represent the scripting commands to interact with the page and perform validations.

See package com.xceptance.xlt.api.engine.scripting for more information about the available scripting commands of the XLT Scripting API.

Script Modules

You may have noticed that a Search action in TSearch appears twice, first time with no hits and then again with hits. To prevent implementing the search
related steps twice Script Developer allows to extract parts of the script as a module – a convenient feature that, in the present example, can be easily applied to the Search action.

When exporting the script to Java, XLT generates a class for each of the modules that extends AbstractWebDriverScriptModule and extracts the code to reuse it in the test case. Using the execute() method, the module can be called in the test case after creating an instance of the module class. It also possible to pass parameter to a module.

Modules can embed submodules, that is they can be called by test cases or by other modules. That’s why the module class needs two different constructors, one that takes a test case and another one that takes a module as parameter.

WebDriver API

WebDriver is a tool for automated tests of web applications. It was originally introduced by Google and features a simple and efficient API permitting control of real web browsers, such as Firefox, Internet Explorer, Safari, and the HtmlUnit headless browser.

The XLT framework incorporates the WebDriver API so that external test cases using the WebDriver API can generally run in the XLT framework as well. XLT-based WebDriver tests can’t be executed with a stand-alone WebDriver because the XLT-WebDriver API integrates the concept of action names.

When being exported from Script Developer, the generated Java code originates from the XLT Scripting API, which in turn is built upon WebDriver. When you look at the TSearch example, you may notice that its strictly linear approach is limited. Using the lower-level WebDriver API in combination with the XLT scripting API is a good way to overcome possible limitations.

When you run the TSearch example as functional or load test, your test scenario actually turns out to be somewhat unrealistic since we open the product details page always for the same product. Randomising the ViewProductDetails action would allow you to browse to different products and select the posters randomly.

The exported code uses an XltDriver to simulate users. Additionally, the WebDriver API lets you easily switch to other WebDrivers to imitate real-world browsers, like Chrome, FirefoxDriver, or Internet Explorer. The XltDriver extends the HtmlUnitDriver implementation of the WebDriver API.

The following code is an example of how to use the WebDriver API to introduce random factors. The WebDriver functionality is refactored into a method allowing it to be reused for random product selection. The example also contains a JUnit4 assertion and shows how you can use standard Java functionality.

The refactored code described above could look like this:

/**
 * <p>Simulates storefront search.</p>
 */
public class TSearch extends AbstractWebDriverScriptTestCase
{
    /**
     * Constructor.
     */
    public TSearch()
    {
        super(new XltDriver(true), "http://localhost:8080/");
    }

   /**
    * This method randomly picks one of the web elements that match the given xpath
    * It uses the WebDriver API and a JUnit assertion
    */
    private WebElement findWebElementsAndPickOne(String xpath)
    {
        // get all elements that match the given expression
        final List<WebElement> articleLinks = getWebDriver().findElements(By.xpath(xpath));

        //Make sure there is at least one element; This is a pure JUnit assertion
        Assert.assertFalse("No elements found", articleLinks.isEmpty());

        // grab one of the elements randomly
        WebElement element = articleLinks.get(XltRandom.nextInt(articleLinks.size()));

        //return the element
        return element;
    }

    /**
     * Executes the test.
     *
     * @throws Throwable if anything went wrong
     */
    @Test
    public void test() throws Throwable
    {
        // Open the homepage and delete cookies (module call)
        final OpenHomepage _openHomepage = new OpenHomepage();
        _openHomepage.execute();

        //
        // ~~~ Search ~~~
        //
        startAction("Search");
        // Store a search phrase that gives results
        store(resolve("${searchTerm_hits}"), "searchTerm");
        // Execute the search (module call)
		final Search _search = new Search();
        _search.execute(resolve("${searchTerm_hits}"));

        // Validate the entered search phrase is still visible in the input
        assertText("id=searchTextValue", resolve("${searchTerm_hits}"));
        // Validate presence of the search results page headline
        assertElementPresent("id=titleSearchText");
        // Validate the headline contains the search phrase
        assertText("id=titleSearchText",
              resolve("glob:*Your results for your search: '${searchTerm_hits}'*"));
		// validate result counter
        assertText("id=totalProductCount", resolve("${resultProductCount}"));
        //
        // ~~~ ViewProduct ~~~
        //
        startAction("ViewProduct");
        //Find all product links by xpath, pick one randomly and click the link
        findWebElementsAndPickOne("id('productOverview')/div/ul[@class='thumbnails']/li/div/div/a").click();
    }

See file <XLT>/doc/xlt-<version>-javadoc.zip for a full documentation of the XLT framework, including information on the WebDriver support.

XLT Action API

Besides the components described above, the XLT framework supports one more approach for modeling test scenarios in Java code. The test cases and action classes can be implemented in Java from scratch or they may be generated automatically while being exported from Script Developer. This approach is based on special action classes and makes use of the XLT Action API.

To test web-based applications, the XLT Action API relies on the HtmlUnit framework, which also includes the Mozilla Rhino engine for JavaScript support. Web tests are executed by a low-level web browser simulation. Basically, this means that – as in real browsers – a web page is translated into a DOM (Document Object Model) tree. Analysis, validation, and any other access is performed afterwards on this DOM tree construction.

The XLT Action API provides a programming paradigm to translate a test scenario into a unit test. The test scenario is implemented as a test case class which itself executes a sequence of one or more actions.

Test Case and Action Classes

All test case classes should inherit the abstract class AbstractTestCase which supplies some basic features, like logging the test results to disk and easy access to properties.

As a test case models a transaction and as transactions rely on actions, defining the appropriate actions is the first step.

All actions must inherit the abstract class AbstractAction which forces you to implement the three methods execute(), preValidate(), and postValidate(). As mentioned earlier, the preValidate() and postValidate() methods perform validations before and after the execution of that action itself. Therefore, the call sequence of an action generated by the XLT framework is always:

  1. preValidate()
  2. execute()
  3. postValidate()

This call sequence will be executed exactly once when the instance method run() of an action is called.

Note that the XLT Action API forces you to implement the validation methods and that is the whole purpose of testing: validating data. Therefore, implementing the abstract validation methods in a non-trivial way (that is not leaving them empty) is strongly recommended. Otherwise, you will sacrifice test quality.

Each of the three methods may throw an exception which always indicates a problem. To check if an action can be executed safely, the abstract class AbstractAction provides a method called preValidateSafe(). This method internally calls preValidate() and catches any thrown exception. If no exception is thrown, preValidateSafe() returns true; otherwise it returns false. This helps you determine if the prerequisites are fulfilled to continue the page flow in a certain direction. A simple example is the flow through a catalog with nested categories. As you don’t know the nesting level up-front when you create a dynamic and random test, it might be necessary to call preValidateSafe() before trying to go to the next level of categories.

Note that AbstractAction doesn’t offer any web support. Therefore, any web-based test should inherit the abstract class AbstractHtmlPageAction, which is a specialization of AbstractAction and which does offer support for web testing.

Validation

Assertion

JUnit provides the concept of assertions and XLT uses this concept for all validations. Since XLT doesn’t change JUnit in any way, you can use assertions just as you’re used from JUnit.

Pre-validation

XLT offers two ways of using the preValidate() method. Any exception on the direct path stops the test with an error message. In case you just want to check whether or not a requirement is fulfilled, you can call the preValidate in a safe way (by using preValidateSafe()) so that any exception is caught and no error is reported. Should you accidentally cause a Java exception different to AssertionException, such as NullPointerException or IndexOutOfBoundException, XLT issues a warning because the code might contain a problem from a programming point of view. Errors from the application under test should always come up as assertion failures.

Post-validation

The postValidate() method works similarly to preValidate(). It is used to validate the page just loaded in execute() and ensures that the data matches the expectations. The full set of JUnit assertions is available.

You can’t explicitly call the postValidate() method; the framework does so instead. Additionally, error messages can’t be suppressed. If a page has different outcomes based on random data or states, you have to explicitly handle that in your validation code.

Validators

We strongly encourage you to write individual validation classes for easy reuse. As soon as a certain check has to be done more than once, it is suited for a validator implementation. This simplifies the maintenance of tests and makes them less error-prone because copy-paste causes typical programming errors.

Some common validation routines are already covered by default validators, such as an HTTP response code, HTML end tag, and HTTP content length validation. See package com.xceptance.xlt.api.validators in the API documentation for more information on this topic.

Example

Let’s imagine a poster search test case again to illustrate the XLT Action API. The most important action would be to “search”, that is to fill in the search phrase and then click “Go”, “Search”, or something similar that loads a list of results. The preconditions are the existence of a search input field and of an appropriate button labeled Search or Go. The execute() method should fill in the search phrase and click the button.

After the new page has been loaded, the result should be validated. This validation consists of general validation, performed by validators, and action-specific validation.

The resulting implementation of the search action would then look like this:

/**
 * Enter the given search phrase in the site's search bar and submit the form.
 */
public class Search extends AbstractHtmlPageAction
{
    /**
     * Search phrase.
     */
    private final String phrase;

    /**
     * Search form.
     */
    private HtmlForm searchForm;

    /**
     * Search option ({@link SearchOption#HITS} or {@link SearchOption#NO_HITS} ).
     */
    private final SearchOption searchOption;

    /**
     * Constructor
     *
     * @param previousAction
     *            The previously performed action
     * @param phrase
     *            The search phrase
     * @param option
     *            The search option that defines if we expect a hit or a no-hit
     */
    public Search(final AbstractHtmlPageAction previousAction,
                  final String phrase,
                  final SearchOption option)
    {
        super(previousAction, null);
        this.phrase = phrase;
        searchOption = option;
    }

    /**
     * Validation prior to execution.
     * @throws Exception
     *             if some of the required input elements couldn't be found.
     */
    @Override
    public void preValidate() throws Exception
    {
        // Get the current page.
        final HtmlPage page = getPreviousAction().getHtmlPage();
        Assert.assertNotNull("Failed to get page from previous action.", page);

        // Check that the search form is available
        Assert.assertTrue("Search form not found.", HtmlPageUtils.isElementPresent(page, "id('search')"));

        // Remember the search form
        searchForm = HtmlPageUtils.findSingleHtmlElementByID(page, "search");
    }

    /**
     * Executes the search. Primarily this includes the input of the search
     * phrase and a click on the proper search button.
     * @throws Exception
     *             if some of the inputs have become invalid or setting the
     *             value attribute of the search input field has failed.
     */
    @Override
    protected void execute() throws Exception
    {
        // Fill the search form with the given phrase
        HtmlPageUtils.setInputValue(searchForm, "searchText", phrase);

        // Submit the search
        loadPageByFormSubmit(searchForm);
    }

    /**
     * Validation after search has become complete.
     * @throws Exception
     *             if no search result block element could be found
     */
    @Override
    protected void postValidate() throws Exception
    {
        // Get the result of the action
        final HtmlPage page = getHtmlPage();

        // Basic checks - see action 'Homepage' for some more details how and when to use these validators
        HttpResponseCodeValidator.getInstance().validate(page);
        ContentLengthValidator.getInstance().validate(page);
        HtmlEndTagValidator.getInstance().validate(page);

        HeaderValidator.getInstance().validate(page);

        // Check that the desired option result was achieved.
        switch (searchOption)
        {
            case HITS:
                Assert.assertNotNull("Expected at least one hit for '" + phrase + "'.",
                                     HtmlPageUtils.findSingleHtmlElementByID(page, "productOverview"));
                break;

            case NO_HITS:
                Assert.assertFalse("Search phrase '" + phrase + "' should result in no hits.",
                                   HtmlPageUtils.isElementPresent(page, "productOverview"));
                break;

            default:
                Assert.fail("Unknown search option.");
                break;
        }
    }
}

Note that the constructor of this class has two parameters. One of them is the search phrase the action has to know about. The other parameter is the previously performed action. To enable a flow, all the actions that will be used in page flows need to provide a constructor with a parameter representing the previous action. Without passing the previous action, each action would be stand-alone and behave as if you had just opened a new web browser. Normally, only the start action does so.

You’ll notice that the postValidate() method uses some of the predefined validators. XLT also offers a StandardValidator performing the most common validations in one go. This includes:

  • HTTP response code validation,
  • HTML end tag validation,
  • content length validation, and
  • XHTML validation.

Having the search action at hand, the implementation of a test case using this action is almost done. A very simple test case would be a repeated search for some phrases. These phrases can be stored in a data file and obtained using the XLT data provider mechanism:

public class TSearch extends AbstractTestCase
{
    // Container that holds all the search phrases
    private static DataProvider phrases = null;

    @Before
    public void initialize() throws Exception
    {
        // Data container already initialized?
        if(phrases != null) return;
        // No. Go for it.
        phrases = DataProvider.getInstance(getProperty("searchphrases.filename", "phrases.txt"));
    }

    @Test
    public void search() throws Throwable
    {
        // Start on Homepage.
        Startpage start = new Startpage();
        start.run();

        for (int i = 0; i < XltRandom.nextInt(10); i++)
        {
            // Take a random search phrase.
            String searchPhrase = phrases.getRandomRow(false);

            // Search.
            Search search = new Search(start,searchPhrase);
            search.run();
        }
    }
}

The example above also demonstrates the use of the XltRandom class offering some convenient randomization features. See the package com.xceptance.xlt.api.util for additional functionality that may help implementing tests.

Each execution of the search action requires an appropriate search phrase obtained from a DataProvider object. This class offers a generic mechanism to handle and provide test data that is stored in a text file. The location of the text file is specified as relative path to the directory <testsuite>/config/data and passed as parameter to the static method getInstance(). When the class is instantiated, all data is kept in memory, allowing easy and fast access. XLT is shipped with a predefined set of data files containing email addresses, first and last names, street and city names, and so on. This data can be acquired from the GeneralDataProvider class that uses the appropriate text files located in directory <testsuite>/config/data with <testsuite> referring to your test project directory.

Last but not least, the present example illustrates how you can use JUnit4 annotations in the standard manner.