XLT 4.2.11

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.11. Registered customers can view changes and the current roadmap in the XLT Information Center.

Framework

Endless loop with JQuery 1.9.x and 1.10.x (#1995)

Sometimes XLT got stuck in an endless loop when loading a page which uses JQuery 1.9.x or 1.10.×. Fixed.

Load Test Environment

Master downloads results even though the test is still running (#1996)

If agent controllers become (temporarily) unavailable during a test run because of network problems, the master controller might leave the status display loop (when in auto mode) and start downloading the results even though the test is still running. This happened even when there were other, still reachable agent controllers. Fixed.

Report Generator

Speed up calculation of the Concurrent Users chart (#2000)

Calculating the data set for the Concurrent Users charts was a bit slow, especially for larger result sets. The respective code has been optimized.

Script Developer

No possibility to close Script Developer on tiling window managers (#2008)

Certain window managers do not decorate application windows with a window frame, so the usual close button is not always available. In this case, Script Developer could not be closed. Now there is the new Quit menu item, which can also be triggered via the keyboard shortcut Ctrl+Q.

XLT 4.2.10

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.10. Registered customers can view changes and the current roadmap in the XLT Information Center.

Framework

Long delays when executing test cases with Firefox 23 (#1977)

When executing test cases via FirefoxDriver, strange delays of approximately 10 seconds could be observed. This happened beginning with the newest Firefox 23, and only when FirefoxDriver was instructed to execute custom JavaScript code via JavascriptExecutor#executeScript(...). Seems to be a bug in FirefoxDriver. See here for more information. Note that other drivers were not affected.

Since small pieces of JavaScript code are executed under the hood when interpreting many of the XLT script commands, this effect could be observed especially for script test cases. However, test cases that use the plain WebDriver API were affected as well if they use JavascriptExecutor#executeScript(...).

To fix this issue, the WebDriver libraries have been updated to version 2.35.0. Please keep in mind to update any driver server executable you might have installed locally (chromedriver, IEDriverServer.exe, etc.) to match this new WebDriver version.

InterruptedIOException when downloading static content (#1980)

Sporadically, XLT failed to download static content and wrote “java.io.InterruptedIOException: Connection has been shut down” messages to the logs. This was caused by an issue in the internal connection management (and not by a network or server problem). Fixed now.

Javascript URLs evaluated in wrong context (#1981)

The src attribute of an iframe element may also be specified as JavaScript URL javascript:someJSExp. Those URLs are special constructs as they can be used to set the content of a frame. Unfortunately, the javascript expressions were always evaluated in the context of the surrounding page instead of the frame page which obviously lead to different results. Fixed.

Loading default frame content may overwrite any already created content (#1983)

Sometimes frames may have no or an empty src attribute, in which case the frame is explicitly initialized with an empty page (“about:blank”). Usually, the actual frame content will later be generated/loaded via some JavaScript code. Due to a timing issue, it could happen that the frame content was created before the page was initialized with the empty page. So in the end the frame content was lost. Fixed.

XLT 4.2.9

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.9. Registered customers can view changes and the current roadmap in the XLT Information Center.

Script Developer

Support Latest Firefox (#1929)

Script Developer supports Firefox version 4 up to 22 now.

Script Developer reported a loading problem if a value was “false” (#1808)

When Script Developer encountered commands with a command value of “false” while loading a script, it reported a “No value provided” error and denied loading the script. Fixed now.

StoreEval could not resolve certain variables (#1819)

When replaying storeEval commands in Script Developer, JavaScript expressions referring to custom variables/properties defined in the top-level scope (aka “window”), such as “$” or “window.$”, were always evaluated to undefined. Fixed.

Load Test Environment

Allow agent controller to be bound to a specific host (#1878)

Sometimes it might be necessary to force the agent controller to listen on a specific network interface instead of all available interfaces. To accomplish this, it’s now possible to configure the agent controller’s host – either by name or by IP – in the corresponding properties file <xlt>/config/agentcontroller.properties:

com.xceptance.xlt.agentcontroller.host = myhost.domain.com

Report Generator

Data processor threads not stopped, causing a memory leak (#1916)

When creating reports from the master controller’s main menu, the master controller simply calls the report generator. Unfortunately, the report generator did not clean up properly. In particular, the data processor threads for both regular and external data were not stopped. Since these threads reference a lot of data, this effectively caused a memory leak. When creating multiple reports in a row, the master controller might easily go out of memory. Fixed now.

Note that neither the stand-alone report generator, nor the master controller in auto mode was affected, as they quit immediately right after creating the report.

Result Browser

Result browser showed error message for requests that have no response content (#1920)

In the result browser, the tab Response Content can be used to view the data sent as the response body. However, in case the response does not have a response body, the result browser displayed a loading error. Now empty response bodies are shown correctly as an empty page.

Framework

Variable references not resolved (#1881)

The test suite configuration files support the ${...} notation in property values, so values can be defined based on other properties. Unfortunately, such variable references where not resolved properly when the method XltProperties#getPropertiesForKey() was called. Fixed now.

Exception when reading compressed, but empty responses (#1921)

Some types of HTTP responses (such as 204) do not have a response body. If, however, a Content-Encoding header was part of the response (indicating gzip- or deflate-compressed content), the framework tried to interpret the (empty) content accordingly which led to an EOFException. Now the content encoding is ignored for empty responses.

XLT 4.2.8

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.8. Registered customers can view changes and the current roadmap in the XLT Information Center.

Script Developer

Mouse events dispatched by SD don’t use correct target element coordinates (#1802)

When replaying commands that mimic some mouse activity such as click or mouseover, Script Developer dispatches appropriate mouse events as specified by DOM Level 2 Events.
However, it did not pass the correct target element’s coordinates to the event which might cause event handlers to work incorrectly. This is fixed now.

Load Test Environment

Allow to specify result target directory (#1804)

Usually, XLT downloads load test results into a directory whose name is automatically generated by using the current time-stamp. Unfortunately, this mechanism may lead to complications when XLT is run in highly automated processes.

To overcome this issue, the result target directory can now be specified by using the new option -o on the master controller startup, where relative directory names will be resolved by using the configured test results root directory as base.

Please notice that this option gets ignored when the master controller is started in interactive or sequential mode.

XLT 4.2.7

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.7. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

URLs could not be deleted from base URL drop-down list (#1795)

Now obsolete entries in the base URL drop-down list can be deleted by selecting the respective entry with the mouse or cursor keys and pressing the Delete key.

XPath locator recorded instead of link locator (#1796)

When recording scripts, Script Developer might generate an xpath locator for links instead of the more appropriate link locator. This happened only if the link text contained special characters such as “>”. Fixed.

Load Test Environment

Custom samplers should not stop working in case of an exception (#1768)

If a custom value sampler threw a RuntimeException, the custom sampler was taken out of service for the rest of the load test. This behavior has been changed. Now the sampler continues running even in case of exceptions.

Wrong handling of test comments (#1780)

After downloading the test results in the Master Controller, the user can optionally specify a test comment. However, when leaving the comment empty, the comment entered for a previous download was used instead of just the empty comment. Fixed.

Master controller should return a non-zero exit code in case of errors (#1783)

When using XLT in automated environments, the master controller’s exit code is the only indicator whether a load test was or was not run successfully. The master controller already returns a non-zero exit code under certain error conditions, but there were more conditions which were not covered so far, for example if the load test was not started at all because not all configured agent controllers could be contacted. Fixed now.

Framework

Authorization header not sent automatically (#1787)

If access to a web site is protected by Basic Authentication, the server challenges the client (with a 401 response for the first request), which in turn repeats the request with the Authorization header set. For every following request, regular browsers send this header automatically, however, HtmlUnit did not, so the server needed to send out a challenge for each and every request. Fixed now.

XLT 4.2.6

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.6. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Incorrect encoding of doubles quotes in script files (#1777)

Sometimes quote characters must be used in script commands, for example in XPath expressions such as //div[class=“foo”]@. Script Developer supports both single and double quotes. When using double quotes, Script Developer automatically encodes them before storing the script to an XML file as the double quote character has special meaning in XML.

Unfortunately, in Script Developer 4.2.5 the double quotes were mistakenly encoded as &quote; (instead of &quot;). Because of the wrong encoding, the affected scripts could not be loaded again as the XML parser does not know the &quote; entity.

Even though the encoding issue is fixed now, the affected files still contain the wrongly encoded double quotes and need to be fixed manually. To this end, open your test suite in Script Developer. It will report all scripts which cannot be loaded. Open each of the listed scripts in a text editor, search for quote; and replace it with quot;. After saving all the modified scripts, reload the test suite in Script Developer. It should now be able to successfully load all scripts.

We apologize for any inconvenience this issue may have caused.

XLT 4.2.5

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.5. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Support Latest Firefox (#1750)

Script Developer supports Firefox version 4 up to 19 now.

Steps to create a new test suite not obvious (#1710)

Previously, creating a new test suite and importing an existing one was done via the same directory selection dialog. In the first case, the user had to create a new directory in the dialog before selecting it. This was not obvious. Now the two operations are separated from each other and it is much clearer to the user what to do in each case.

Generated classes for a disabled script test case did not compile (#1740)

When creating the wrapper classes for disabled script test cases or when exporting disabled script test cases to Java code, the generated classes did not compile because of a missing space character. Fixed.

Script Developer hung after executing an assertText command (#1741)

Under rare circumstances, Script Developer hung after executing an assertText command. The command was marked as failed, but the failure message box did not appear and the test case was still in state “executing”. This problem is fixed now.

Could not drop a module at the end of a script / into an empty script (#1747)

Module calls can be added to a test script by simply dragging the respective module from the package tree and dropping it on the test script at the location where the module is to be called. However, this did neither work for empty scripts nor if the module was dropped at the end of a script. Fixed now.

Redefining module parameters mixed up the parameter values of called modules (#1749)

When redefining the set of input parameters of a module, for example deleting one or more parameters, it happened that the parameters passed to sub-modules got mixed up. This is fixed now.

Wrong path recorded when dealing with a file input (#1753)

When recording an interaction with an input element of type “file”, Script Developer mistakenly prefixed the recorded file path with “glob:” (on Windows only), for example: “glob:d:\dummy.txt”. When replaying the test script, the path was not set as such a file did not exist, of course. Fixed.

Wrong base URL used when replaying a script after recording another script (#1754)

After recording a new script test case, its base URL was also used when replaying a different script test case later on, even though the base URL field on the top displayed the correct base URL.

Framework

WebDriver updated to latest available version (#1759)

The bundled WebDriver API and its implementations have been updated to the latest available version 2.31.0.

XLT 4.2.4

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.4. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Test execution cannot be cancelled in case of unresolved variables (#1654)

Before a test script is replayed, Script Developer checks that any variable in the script could be resolved. Any unresolved variable is then reported to the user. Afterwards Script Developer did execute the script test case, even if there were unresolved variables. Now the user is asked whether or not the test case should be run in this case.

Replay speed value not persistent (#1720)

When changing the replay speed, the current value was not stored to the project settings. So when reopening the project in Script Developer, the replay speed value was not restored properly. Fixed.

Reordering script editor tabs (#1652)

Multiple script editor tabs can be open in Script Developer at the same time. These tabs can be reordered now. Use this feature to regroup the open tabs to your liking.

createCookie command uses base URL as fallback (#1727)

The createCookie command needs domain/path information when creating a cookie. If this information is not explicitly specified, the command extracts it from the current document’s location attribute. However, if the current document has no location, e.g. it is in its initial state (empty tab/window), the test case’s base URL, if defined, is used now. This behavior is in line with the deleteCookie command which already uses this fallback.

Exporting modified test scripts (#1729)

Modified, but not yet saved test scripts need to be saved now before they can be exported to Java code.

Support Firefox 18 (#1721)

The Script Developer supports Firefox version 4 up to 18 now.

Load Test Report

Navigating from the charts back to the statistics table (#1713)

On the timer pages in the load test report, such as the Requests page, there are Back to table links now, near each chart tab. Use these links to quickly navigate back to the corresponding entry in the statistics table at the top of the page.

Network bandwidth values are 0 (#1712)

In the load test report, the Requests section also provides a table for the network bandwidth used per request. However, some values (1/s, 1/min, 1/h, 1/d) were 0 in almost all cases for both inbound and outbound traffic. Fixed now.

Nested exceptions do not appear in the load test report (#1724)

In order to save disk space, XLT removes the irrelevant parts (JUnit framework stuff, etc.) from exception stack traces before storing the traces to the results files. However, the stack trace of any nested exception was also cut off during this process. Nested exceptions are preserved now.

Load Test Environment

Sub-optimal agent utilization in phases of lower load (#1622)

XLT distributes test users evenly to all available agents. When all of these users are running, the CPU load on the agents is comparable. However, if only a subset of these users run, for example during the ramp-up period, the CPU load on the agents could diverge significantly. So it happened that one agent was very busy while another agent was idling at the same time. Now the CPU load on the agents is comparable also in phases of lower load.

Note that no agent was ever overloaded. It simply had to do its full share of work earlier than other agents.

Problems when processing external data (#1732)

XLT may process externally gathered data and include it in the load test report. When processing such data, the report generator might print “Queue full” error messages, and as a consequence parts of or all external data was missing in the report. This happened if the data reader thread was faster than the data processor thread. Fixed now.

Framework

regexpi matching strategy in script test cases (#1675)

Script test cases support different text matching strategies when checking for text, such as the glob or the regexp strategy. Script Developer also knows the regexpi strategy (for case-insensitive matching of regular expression patterns), however, the XLT framework did not implement this strategy yet. Fixed.

CSS import rules with media type ignored (#1703)

XLT did not recognize CSS import rules if the rule specified a certain media type, for example:

@import "foo.css";              /* OK */
@import "bar.css" screen;       /* ignored */

This is fixed now.

Resources referenced from link elements not loaded (#1717)

Resources referenced by <link> elements were downloaded only if these <link> elements were present at the page right from the beginning. However, they were not loaded if the <link> elements were added to the page by JavaScript code later on. Fixed.

Images referenced from included style sheets not loaded (#1718)

In the image download mode onDemand, images referenced from style sheets need to be downloaded only if the respective style rule applies to an element on the page. However, only top-level style sheets were examined for such style rules, but not style sheets included via a CSS import rule. Fixed.

WebDriver updated to latest available version (#1708)

The bundled WebDriver API and its implementations have been updated to the latest available version 2.29.0.

XLT 4.2.3

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.3. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Reports

Open result browsers right from the load test report (#1617)

If an error occurred during the load test, the error entry in the load test report may also contain the name of the corresponding result browser directory. Now these result browsers can be made directly accessible right from the load test report. This greatly speeds up error analysis as the result browsers are just one click away.

In order to let the report generator create links from the load report to the result browsers, set the property com.xceptance.xlt.reportgenerator.linkToResultBrowsers in <XLT>/config/reportgenerator.properties to true. By default, the report generator calculates the path from the report to the result browsers based on the results directory and the report directory. The computed path will be a relative path if possible and an absolute path otherwise (on Windows, if report and results are on different drives).

Sometimes the relative path approach is not suitable, for example if you send the report to your team members, but not the results. In this case, the results must be made available somewhere on the net. The report generator needs to know about this location to properly generate the links. To this end, you configure a results base URI, for example http://myhost/results. The URI is a base URI as it is common for the results of all of your load tests. The report generator automatically appends the name of the results directory (for example 20121106-111751) to this URI when generating the links to the result browsers, so the resulting link might look like this: http://myhost/results/20121106-111751/ac01_00/TSearch/126/output/1352194484275/index.html

By using a base URI you do not need to reconfigure the report generator when creating the report for another load test, unless you choose the publish the results at a totally different location. To configure the base URI, set the property com.xceptance.xlt.reportgenerator.resultsBaseUri in <XLT>/config/reportgenerator.properties to the appropriate value.

In any case, make sure that the results are (or will be) published at the target location and will always stay there. Otherwise, the viewers of the report would experience broken links.

Order of reports in a trend report (#1642)

The trend report generator creates a trend report from individual load test report directories given on the command line. The input reports can be specified either explicitly by listing them all one by one or by using wildcard characters such as “*”. In order to guarantee a deterministic order in which the individual reports are processed, the trend report generator sorted the input reports by directory name.

This approach had two major drawbacks. First, to make the sorting by name work, the report directory names had to be always timestamps (or similar), so custom names were not possible effectively. To solve this, the default sorting is now based on the test start date/time as stored in the report. Second, sometimes the default order may not be appropriate. Now the user can disable the default sorting by specifying the new command line switch -nosorting, in which case the order of reports in the trend report will be exactly as given on the command line. Note that when using wildcard characters together with -nosorting, the order of reports may be non-deterministic.

Ramp-up period missing in the load test report (#1647)

In the Overview section of the load test report the value for the ramp-up period that is part of the load profile table was sometimes or always empty even when a ramp-up period was explicitly configured. The formerly missing value will be displayed in the report again.

List result browser directories only if they exist (#1638)

In case of a transaction error, XLT dumps a result browser to disk, but only if the dump mode is set to always or onError. However, the Errors table in the load test report did always list such result browser directory entries, even if the dump mode was set to never. From now on result browser directories will be listed in the report only if they indeed exist on disk.

Load Test Environment

Test case specific ramp-up start value not recognized (#1640)

When defining both a default and a test-case-specific ramp-up start value for a user count load profile, the specific value was not used, but the default one. Now test case specific settings take precedence over configured default settings again, and the property rampUpInitialValue takes precedence over the old and deprecated property rampUpInitialUsers.

Please note that this may cause different behavior in already existing load test configurations if both properties rampUpInitialValue and rampUpInitialUsers are in use.

Overshooting arrival rate if the load function increases abruptly (#1646)

When using load functions where a period with a small arrival rate is immediately followed by a period with a much larger arrival rate, it happened that a great number of users were started at the same time, thereby overshooting the intended arrival rate for a short time. This is not possible any longer.

Script Developer

Cannot type into element found using dom locator (#1645)

The dom locator can be used to locate elements by evaluating expressions like document.getElementById('myID'). However, attempts to type into an element that was found by such a locator failed. This problem is solved now.

Script Developer keeps old reference to reloaded script (#1685)

Under certain circumstances it happened that the reference to an included module script will not be updated when the module script is reloaded. As a side effect the module script could be opened in two different editor tabs since it existed in two different states internally. When a script is reloaded all of its references will be correctly updated now.

Recorded base URL is test-specific but handled like globally defined (#1688)

In Script Developer, base URLs can be defined on test script level as well as on project level. In the first case, the configured base URL takes effect only when replaying the corresponding test script. In the latter case, the configured base URL takes effect when replaying any test script, regardless whether or not the test script defines its own base URL.

When recording a new test script the Script Developer computes the base URL from the start URL and sets it automatically at that test script. Unfortunately, this base URL was handled as it would have been defined on project level instead of test script level. As a consequence, this base URL did override the individual base URLs of all other test cases, so replaying the other test scripts failed as the base URL of the newly recorded test script was still active. This issue has been solved and the recorded base URL is correctly handled from now on.

Valid DOM expressions reported as invalid (#1690)

Commands may also use DOM expressions to locate elements. If a command with such a DOM locator was opened in the Edit Details dialog, but no element could be found via this element locator (on the current browser page), the element locator was always reported as invalid even if it was perfectly valid. Fixed.

Class filter include/exclude patterns did not take effect (#1691)

The Script Developer provides the possibility to configure include/exclude patterns that are used to filter out certain element locator expressions during record. This is mainly used to prevent element locators to be recorded that use generated IDs, names, or classes, as these may change with every page reload/revisit. However, the include/exclude patterns defined for an element’s class attribute did not take effect, so the recorded XPath might still use the unwanted classes. This is fixed now.

Export to Java broken when using storeEval commands (#1655)

Since the JavaScript code embedded in a storeEval command’s target value might contain characters like double quotes and backslashes, it has to be handled accordingly by escaping such characters correctly which was not the case yet. Hence, exporting scripts that contain such commands either resulted in broken code or the execution of the exported scripts and their JUnit wrapper classes failed. This has been fixed.

In-place editing of modules broken (#1656)

Script Developer allows to edit the commands of a module not only when the module itself is opened in an own editor tab, but also in the context of the calling test script. In the latter case, however, when inserting a new command or action into this module using the context menu, the new command/action was not inserted into the module but inserted into the surrounding test script right after the module call. This has been fixed. Furthermore, it is now possible to change the order of the module’s commands/actions and to perform copy/cut & paste and drag & drop operations, right from the test script tab.

Test data variable reported as undefined (#1665)

Sometimes test data variables used in a module were mistakenly reported as undefined although they could successfully be resolved during test case execution. This has been fixed.

Latest Firefox supported (#1644)

The Script Developer supports Firefox version 4 up to 17 now.

Module parameter values lost (#1524)

After the extraction of a new module with parameters and after modifying this module, e.g. by inserting a new command, any module parameter value previously defined in the calling script were lost. This annoying issue is solved.

Framework

AbstractResponseProcessor should adjust content length header (#1641)

Implementations of the ResponseProcessor interface can be used to modify a response before it is passed to HtmlUnit. The class AbstractResponseProcessor is a helper class to simplify creating own response processor implementations. However, if the processor added or removed content from the response body, the class AbstractResponseProcessor did not adjust the Content-Length response header accordingly. This caused the validator classes ContentLengthValidator and StandardValidator that are part of the API to throw assertion errors as the new content length did not match the header. Fixed.

Upgrade WebDriver to latest available version (#1663)

The bundled WebDriver API and its implementations have been updated to the latest available version 2.26.0.

XLT 4.2.2

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.2. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Framework

Referrer request header not set (#1620)

When loading static content, the Referer HTTP request header was not set. However, some applications check for the presence of this header even for static content and might return error responses if it is missing. Now the header will be set for all requests.

Exception when reading compressed, but empty responses (#1623)

If a server delivered a response which is flagged as compressed (for example: Content-Encoding: gzip), but has an empty response body (Content-Length: 0), an EOFException was thrown. Fixed now.

Image not re-downloaded and onload handler not called after src change (#1624)

Suppose, you encounter the following HTML:

<img src="someImage.jpg" onclick="this.setAttribute('src', 'someOtherImage.jpg')" onload="...." />

When you click on this image, then its src attribute changes which in turn should cause the onload handler to be called again. Unfortunately, this was not the case and has been fixed.

Note, that calling the image onload handler triggers the image download automatically.

Aborting long-running transactions did not work reliably (#1628)

Transactions that exceed a certain run time are aborted by the XLT framework (if so configured). However, this did not work reliably. Now and then transactions kept on running even though the timeout expired long ago. Fixed.

XltWebDriver-based test cases leaked threads when aborted (#1631)

When a transaction of an XltWebDriver-based test case was aborted because the transaction run time limit was exceeded, quitting the underlying XltWebDriver failed, and as a consequence, any of its helper threads (JS, download) kept on running. Fixed.

CSSParser library updated (#1621)

The CSSParser library was updated to v0.9.7. This new version has better support for more recent CSS features.

Reports

Concurrent Users chart showed too many users (#1614)

The Concurrent Users chart for a certain transaction might show a higher user number than configured in the respective load profile. This happened if the name of that transaction happened to be a prefix of another transaction’s name (for example TFoo and TFooBar). In this case, the chart for TFoo showed the sum of the users of TFoo and TFooBar. Fixed now.

Implausible GC CPU usage values (#1626)

It did happen that the CPU usage values for garbage collection shown in the CPU usage chart on the Agents page were much higher than 100%. Fixed now.

Load Test Environment

Test users might keep on running beyond the load test period (#1630)

Now and then it happened that a user thread did not finish after the measurement (+shutdown) period was over. This way the load test did not come to an end and needed to be aborted manually. Fixed now.

Agent quits after the load test despite of hanging user threads (#1632)

Previously, an XLT agent process did wait until each of its user threads had finished execution before it finished itself. However, if a user thread hung for whatever reason, the agent would not finish and so the load test would not come to an end. This was especially annoying when running the master controller in non-interactive mode, e.g. in automated test environments. Now the agent process terminates after the longest-running test user thread is expected to finish, no matter whether or not there are still hanging threads.

Users will be aborted after the measurement period if no shutdown period is configured (#1636)

If a shutdown period is configured, a user might continue to run after the measurement period in order to finish its current iteration. Afterwards it will stop voluntarily. If the user is not able to finish its iteration in time, it will be aborted at the end of the shutdown period. However, if no shutdown period was configured for a user, the user was allowed to finish its iteration, no matter how long this may take! This behavior was somewhat unexpected and has been changed such that the user is aborted immediately after the measurement period. This corresponds to a shutdown period of length 0.

Script Developer

Do not report variables created by a store command as undefined (#1629)

Before a test case is replayed in the Script Developer, all commands are inspected for test data variables (such as ${foo}). Each of the found variables is checked whether it has been properly defined. If no definition could be found, the variable will be reported as undefined. This is to assist the user to detect potential pitfalls such as spelling errors.

Previously, only the test case’s data file (*_data.xml) was consulted to decide whether or not a variable is defined. However, since XLT 4.2.0 variables cannot only be defined in test data files, but also at runtime via a store command. These runtime variables were always reported as undefined as they could of course not be found in the data file. Very irritating. Now they are only reported as undefined if they are used before a store command defines them.

XLT 4.2.1

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.2.1. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Recursive resolution of variable names (#1590)

Test data variables are resolved recursively when the value of a test data variable contains a reference to another test data variable, e.g. foo ${bar} baz. This mechanism has been enhanced to support nested expressions, too. So, it is now possible to specify an expression like foo ${foo_${bar}} which is resolved recursively starting at the innermost expression.

For example, let bar and foo_baz be two test data variables whose values are specified as baz and bum, respectively. Then, the expression foo ${foo_${bar}} will be resolved to foo ${foo_baz} which in turn is resolved to foo bum.

Macro execution not working properly (#1599)

Commands whose target or value contains macro statements that are immediately preceded by an @ sign (for example: “@${RANDOM.String(3)}”), were not correctly replayed since these macro statements were not executed. This was because the @ sign had been incorrectly treated as escape character. Fixed now.

Load Test Environment

Not all configured users started in case of multiple agents (#1603)

When using multiple agents and applying the user count load model for a test, not all of the configured users were started. For example, the master controller reported that only 5 out of 10 configured users were actually running. This is fixed now.

Incorrect distribution of test users (#1604, #1606)

Before a load test starts, the test users are distributed equally to all of the available agents. However, they were distributed regardless of their type. So it could happen that one agent got 10 users of a rather resource-intensive test case, while another agent got also 10 users, but with a less demanding test case. This might lead to an uneven resource utilization on the agent machines. Now the agents would get 5 users of either test case.

Furthermore, if there were agent controllers of different weights configured, it might happen that in rare cases some users were distributed to agents having a lower weight than one of the other agents. This has been fixed as well.

Machine weight ignored when using the arrival rate load model (#1610)

In order to use machines of different power together in a load test cluster, the machines may be assigned a different weight, or load factor. Machines with a higher weight will get more users than machines with a lower weight. However, when using the arrival rate load model, not the number of assigned users is relevant, but the arrival rate. Since the arrival rate was equal for all agent machines, all machines effectively received the same load. Now the individual users are controlled such that in summary machines with a higher weight contribute a greater share of invocations to the total number of invocations per hour, as dictated by the arrival rate.

Arrival rate not adjusted often enough when using variable load profiles (#1611)

With variable arrival rates, the arrival rate was adjusted only after the current period was over. However, if the current arrival rate was rather low, the corresponding period was rather long. So it took some time until the new arrival rate had taken effect. Now the arrival rate is adjusted every second.

Incorrect interpretation of incomplete load functions (#1601)

Beginning with XLT 4.2.0, the shape of a load profile can also be defined by a so-called load function. The load function may be incomplete, i.e. it does not need to specify a start value (for time 0) or an end value (for the end of the measurement period), as there are reasonable defaults. For example, the following load function is totally valid:

com.xceptance.xlt.loadtests.TAuthor.users = 30s/5

It is expected (and documented) that in this case an additional time/value pair 0s/1 is added at the beginning and another pair <measurement period>/5 is added at the end. As a consequence, the load parameter is expected to be ramped up from 1 to 5 in a period of 30 seconds. However, the load parameter stayed 0 during this ramp-up period and then went up to 5 immediately. Fixed now.

Please note that when you specify your load function as x/1 (where x is greater than zero), the additional pair 0s/1 will be inserted as well, but in this case there is of course no ramp-up as the target value is already reached.

Unavailable agent controllers not reported immediately (#1602)

In case some of the configured agent controllers are not available, the user was not notified about this issue until the master controller failed to upload the configured test suite to the unreachable agent controller. The time that elapsed so far was unnecessarily long. To provide a faster feedback to the user, an additional agent controller connectivity check will now be done right in the beginning.

Nevertheless, it might take up to 15 seconds to detect whether an agent controller is available or not since this time depends on the operating system’s default socket connection timeout. However, this value can be changed by specifying an appropriate value for the property com.xceptance.xlt.mastercontroller.connectTimeout in file <xlt>/config/mastercontroller.properties.

Non-existent test properties file reported (#1565)

In general, test-specific settings such as the load profile are stored in a test properties file whose name needs to be specified as the value of the property com.xceptance.xlt.testPropertiesFile in order to be effective. However, the user was not notified if this test properties file does not exist or is not readable. Now the load test will be aborted with an appropriate error message.

OutOfMemoryError when uploading large test suites (#1609)

Since XLT 4.2.0, the master controller uploads a test suite to all connected agent controllers in parallel. This could cause the master controller to go out of memory as the whole test suite was buffered in memory before being uploaded. Now the test suite is not buffered any longer, but streamed right from the disk to the agent controllers.

Reports

More precise CPU charts (#1592)

All charts that show CPU utilization used integral values for plotting. Especially when the CPU usage is rather low (for example, see the CPU usage graph for the Garbage Collector), the graph might jump from 0 to 1 and back. However, since all CPU utilization data is collected as fixed-point numbers anyway, the report generator has been enhanced to make use of the provided precision. It creates more fine-grained CPU charts now.

Missing values in the load profile table in the test report (#1612)

The Users value for a test case might be missing in the load profile table in the load test report. This happened only if another test case used the same value. Fixed now.

Framework

Proper handling of CSS property opacity (#1600)

Mozilla’s web browser Firefox uses a floating point number between 0 and 1 to denote the opacity of an element, as specified by CSS3. This number can also be specified without the leading zero, e.g. .55. But when the value is requested using JavaScript it is always returned with the leading zero.

When simulating Firefox via HtmlUnit, the opacity value was returned just as it was given which may have subtle side effects. One of these side effects was that jQuery’s browser detection got confused. It checks if the opacity value for a certain element that was previously set to .55, is exactly 0.55 on return. Since this check failed on HtmlUnit, jQuery used methods specific to the Internet Explorer as this browser uses the non-standard CSS properties -filter and -ms-filter.

This issue has been reported to the HtmlUnit team and is fixed now.

XLT 4.2.0

This section lists and documents all new features, improvements, and important fixes of Xceptance LoadTest 4.2.0. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Release Overview

XLT 4.2.0 delivers a large set of new and improved functionality. This chapter will give you a quick overview about what you can expect to make your upgrade decision easier.

The Script Developer has seen a lot of enhancements. Most noticeably is the support of variables now. You can store data from the web page, run calculations in JavaScript and store the result, and reuse all later. There is now the notion of global and local test data. So any data that applies to multiple test cases at the same time can be managed more easily now. Besides the already mentioned store commands, the Script Developer also permits assertion and wait commands for visibility, select and check boxes, as well as value attributes of an element.

Further usability improvements permit easier switching between projects and the execution of a single command in a test case for faster development. Test cases can now be run directly from the package tree without the need to open them first. The test data management has been completely reworked to simplify and speed up editing of data.

The load test engine supports a fully variable load test profile now. You can define any load pattern you want, such as a 24 hour test with peaks around noon and early afternoon, low traffic during night hours, slow ramp up in the morning and multiple load changes over the course of the day. Additionally the ramp up of arrival rates is now supported as well.

To speed up things and fully utilize the available connectivity, the upload to and the download from agents has been fully parallelized. Optionally you can adjust the desired concurrency. Also the timeout against agents can now be set.

The documentation layout as well as the report design have been redesigned to improve usability, make them visually more pleasant, and unify their appearance. Third party libraries used to deliver enhanced functionality have been upgraded to cover the latest browsers as well as UI trends.

Please see the detailed release notes below for more information about new features, enhancements, and defect fixes.

Script Developer

Global test data (#811)

Certain test data (such as user credentials or product data) are often the same for many or all test cases in a test suite. In order to minimize the effort of maintaining this data for each test case separately, it is now possible to manage it as global test data in a central location. By default, all test cases share that global test data, but it is possible to overwrite the value of a certain global setting by redefining that setting for this test case.

The Script Developer provides ways to manage both the global test data and test case specific data. To edit the global test data, click XLT Script Developer > Manage Global Test Data and define your global key/value mappings. They are stored in a file named global_testdata.properties located in the root directory of your test suite.

Macro for random numbers (#1467)

In order to support not only random strings, but also random numbers in script test cases, a new macro RANDOM.Number has been introduced. It can be used in two ways:

  • ${RANDOM.Number(max)}: returns an integer between 0 (inclusive) and max (exclusive)
  • ${RANDOM.Number(min,max)}: returns an integer between min (inclusive) and max (inclusive)

New script commands (#811, #1107, #1498)

In script test cases, it might be necessary to remember some text or an attribute value for later use. The following new commands have been implemented to cover this:

  • store: stores the given text in a variable
  • storeEval: stores the result of the given JavaScript expression in a variable
  • storeText: stores the text content of the given element in a variable
  • storeXpathCount: stores the number of matches of the given XPath expression in a variable

The commands to perform assertions or to wait for certain conditions have been extended as well:

  • <type>Visible: checks the visibility of an element
  • <type>SelectedId, <type>SelectedIndex, <type>SelectedLabel, and <type>SelectedValue: checks the selection status of select boxes
  • <type>Checked: checks the checked status of check boxes or radio buttons
  • <type>Value: checks the value attribute of an element

where <type> can be assert, assertNot, waitFor, or waitForNot.

For more information about the new commands, please refer to the command reference in the User Manual.

Ignore script test cases during batch execution (#1497)

Sometimes it might be useful to temporarily exclude test cases from the batched test case execution, for example if these test cases are known to fail at the moment. You can now mark a test case as enabled/disabled by choosing Enable/Disable from the test case context menu. This simplifies running all valid test cases at once, because you do not have to unselect test cases first.

Escaping of $ and @ (#1469)

In script test cases, text that looks like an expression can be now used as literal text by escaping it:

  • ${foobar} is escaped by $${foobar} and
  • (){paramExp}@ is escaped by (){paramEx} respectively.

When recording assertions or wait commands that use text or attribute values of elements, escaping is done automatically if required.

Renaming and reordering of module parameters (#1134, #1142)

Sometimes the name or the order of parameters of a script module has to be adjusted due to a specification change or just to get a better naming. The Script Developer aids such changes now. Just open the Edit Details dialog from the module context menu and reorder/rename the parameters as required. After closing the dialog, all scripts that call this module will be updated accordingly.

Free form test data management (#1191)

Previously, the Manage Test Data dialog was a command-centric view for displaying and managing the data mappings. The dialog has been completely redesigned in order to deliver a data-centric approach. All data can be managed independently of their usage in commands now.

Replay of a single command (#1330)

You can now execute a single command within a test case or module by pressing the shortcut key X or clicking Execute Command in the command context menu.

Switching of test projects (#1421)

Switching from one test project to another is much more comfortable now. Just import all your test projects once by clicking the Create/Import button and selecting the test project root directory. This way, the project select box next to the button gets populated. Afterwards, you can easily switch between projects by simply choosing another project from the select box.

You can also remove test projects from the select box, if you do not need them any longer. Just select the project you want to remove while the drop-down is shown and press the Delete key. Please note, that this won’t delete the test project itself, it will only remove it from the project list.

Configurable implicit wait timeout (#1474)

An Implicit Wait Timeout can be configured now in the Settings dialog. An implicit wait timeout is the time the Script Developer will wait for an element to appear or disappear before an error is reported. Once the element is present or gone, the Script Developer will continue the execution immediately.

This timeout is extremely useful if your web application is very dynamic and uses asynchronous JavaScript to build and modify the page. Normally, you would use a waitFor command to wait for the element to appear before you can interact with it. When an implicit wait timeout is defined, most of the waitFor commands can be omitted, making your test cases shorter and easier to maintain.

Run test case directly from package tree (#1496)

It is possible now to execute a single test case directly from the package tree. It is no longer required to open the test case first. Just choose Run from the test case context menu.

Latest Firefox supported (#1518, #1555)

The Script Developer supports Firefox version 4 to 14 now.

Load Test Environment

Variable Load Profiles (#359)

So far, XLT supported constant user and constant arrival rate profiles. Additionally you could add a ramp-up phase to these profiles. To extend the capabilities of XLT, we added the support of fully variable load profiles.

You have full control over the load parameters (arrival rate or user count) at any given time during the test. This mean you cannot only increase and keep the load steady, but increased and decreased it freely. The variable profile comes in handy when you want to combine different load levels within one test run. For example, a test where phases with regular load alternate with peaks of much higher load, or another test that models the load profile of a typical day.

For more information see the load test section in the user manual.

Ramp-up of arrival rates

The ramp-up behavior of arrival-rate-based tests has been improved. Previously, only the number of runnable users could be ramped up. Depending on the number of users necessary to fulfill a certain arrival rate and the total number of users, it might happened that the full load was put on the system under test even though the ramp-up period was not over yet.

Now, for arrival-rate-based tests, the ramp-up parameters are applied directly to the arrival rate, not the number of runnable users, which remains fixed.

In the following sample configuration, the TAuthor scenario is run exactly 100 times per hour using at most 10 concurrent users, but in the beginning the arrival rate is steadily ramped up from 1 up to 100 over a period of 5 minutes.

com.xceptance.xlt.loadtests.TAuthor.arrivalRate = 100
com.xceptance.xlt.loadtests.TAuthor.users = 10
com.xceptance.xlt.loadtests.TAuthor.rampUpPeriod = 5m

Note that if no arrival rate is defined for a test scenario, the ramp-up parameters are applied to the user count instead. Imagine this configuration:

com.xceptance.xlt.loadtests.TAuthor.users = 10
com.xceptance.xlt.loadtests.TAuthor.rampUpPeriod = 5m

This way, the user count will be ramped up from 1 to 10 over a period of 5 minutes.

Since the ramp-up parameters are applicable for both user count and arrival rate now, the ramp-up parameter rampUpInitialUsers should be replaced with the new and more general parameter rampUpInitialValue in your existing test projects. The old parameter is still evaluated (when ramping up the user count), but has been deprecated.

Master controller provides details on agent startup failure (#414)

If an agent could not be started successfully, for instance due to a JVM configuration error, the master controller will report this on the console now.

Parallel communication with remote agent controllers (#591)

Now any communication between the master controller and all its remote agent controllers is done in parallel to speed up managing larger clusters of load machines. This also includes uploading the test suite and downloading the test results. Since uploading/downloading stresses the network connection a lot more than simple control commands, the degree of concurrency can be configured. There are two new properties available in mastercontroller.properties:

com.xceptance.xlt.mastercontroller.maxParallelUploads = 4
com.xceptance.xlt.mastercontroller.maxParallelDownloads = 8

In the given example, the master controller will run at most 4 uploads and 8 downloads concurrently. Configuring different values for upload and download is useful only for network connections with an asymmetric inbound/outbound bandwidth. By default, no limits are applied.

Console status output decoupled from status retrieval (#667)

Retrieving the current status from all participating agent controllers can be time-consuming, especially for very large clusters of load machines and it might lead to information delays. Now the master controller requests the status information in the background and updates the console live report using the status it has retrieved so far.

Connect/read timeout for master/agent controller connections configurable (#1170)

In order to get faster feedback in case of master/agent controller connection failures, the socket connect/read timeouts can be configured now. Two new properties have been introduced which are located in the mastercontroller.properties file:

com.xceptance.xlt.mastercontroller.connectTimeout = 5000
com.xceptance.xlt.mastercontroller.readTimeout = 10000

If the given value is negative or zero, no timeout will be set. The value is the number of milliseconds before a timeout event occurs.

Summary line in console live report (#1370)

If multiple scenarios are run at the same time during a load test, the master controller prints an additional summary line that displays the total number of users, iterations, events, and errors as well as the overall progress.

Time period syntax support for custom samplers (#1486)

As already possible for other settings, the time period definitions in the custom sampler configuration can be set using easier readable formats. For example, the value 12345 given in com.xceptance.xlt.customSamplers.1.interval = 12345 can also be specified as 3h25m45s, 3:25:45 or simply as 12345s.

Reports

New report design

The reports have been redesigned to provide a more modern and consistent user experience. The colors have been adjusted to be more pleasing. All third party libraries such as jQuery have been updated to more recent versions.

You can still apply your own design at any time by adjusting the delivered CSS and image data.

Request merge/split rules with negated conditions (#1473)

During report generation, requests can be grouped into different buckets by specifying merge and split rules. Until now, the rules could only select requests that do match a certain pattern.

However, if a bucket should contain all requests that do not match a pattern, a sequence of rules were required that filtered out the unwanted requests. To simplify the definition of rules, the rule syntax has been extended to also support an exclude pattern for almost all pattern types. All requests that match the exclude pattern will not be selected. For example, to create a bucket for all non-JavaScript resources, you would setup a rule like that:

com.xceptance.xlt.reportgenerator.requestMergeRules.1.newName = {n:0} NonJS
com.xceptance.xlt.reportgenerator.requestMergeRules.1.namePattern = .+
com.xceptance.xlt.reportgenerator.requestMergeRules.1.contentTypePattern.exclude = javascript
com.xceptance.xlt.reportgenerator.requestMergeRules.1.stopOnMatch = false

Please note that an include pattern as well as an exclude pattern can be specified for a pattern type at the same time. In this case, a request is selected if and only if it matches the include pattern, but does not match the exclude pattern.

Collapsed stack traces (#688)

To make the Error section shorter and easier to read, only the error or exception message is shown by default now. The corresponding stack trace is still available and can be expanded for further analysis.

Correct number of active users (#157)

Previously, the number of concurrent users shown on the Active Users chart could be higher than the real value. This happened only if the transaction run times were very short (< 1 sec). This problem is fixed now.

You might see a slight difference in the active users charts when comparing reports created by XLT 4.2.0 or above with reports created by an earlier version of XLT.

Custom time zone for reports (#1536)

By default, the load test report will be generated based on the default time zone of the machine creating the report. However, sometimes the report should rather be based on a different time zone, such as UTC. There is a new command-line parameter available to specify the time zone to be used:

create_report.sh -timezone UTC ../results/20120625-211541

Error markers drawn at the end of a transaction/action/request (#1534)

When a transaction/action/request failed, an error marker is drawn in the respective chart. Previously, the marker was placed at the time the transaction/action/request started. Now, the marker is set at the time the transaction/action/request finished. This illustrates the exact time of a failure better and aftereffects such as transitive errors (request → action or action → transaction) can be correlated easier.

Framework

3rd-party libraries updated (#1208, #1363, #1480)

All used 3rd-party libraries have been updated to the newest available version. Most notably, the libraries HtmlUnit v2.9 (see Release Notes for more information) and WebDriver/Selenium v2.25.0.

Script exceptions carry the script call stack (#1311)

Now, the stack trace of a ScriptException does not longer show the Java call stack, but the script call stack, including the line number information.

Note that the line numbers shown in the Script Developer and the line numbers of the XML script file differ from each other.

You can configure what line numbering should be used when generating line number information. There is a new property com.xceptance.xlt.scripting.lineNumberType available. Use the value scriptdeveloper (default) if you prefer analyzing script errors with the Script Developer, or file, if you want to locate the corresponding line in a text editor.

ETag HTTP header supported (#1382)

The simulation of the caching/download behavior of real browsers has been improved by adding the ETag HTTP header evaluation. When validating a cached but potentially outdated resource, the framework sends the ETag value it has received for the resource back to the server. This allows the server to respond with a 304 Not-Modified message instead of delivering the resource once again.

Utility method to evaluate relative XPath expressions (#1435)

The utility class com.xceptance.xlt.api.util.HtmlPageUtils provides an additional version of the overloaded method isElementPresent(). It accepts an com.gargoylesoftware.htmlunit.html.HtmlElement instance as additional parameter, which is used as the context node when evaluating relative XPath expressions.

Support of empty option value in utility method (#1563)

Since option elements may have an empty string as value attribute, the utility method select(HtmlForm, String, String) in class com.xceptance.xlt.api.util.HtmlPageUtils has been adjusted because it did not allow the empty string as option value parameter.

Test data set files in the class path (#1470)

When running data-driven tests, the XLT framework searches several directories for a potential test data set file. Now, for Java-based test cases, the framework also searches the class path for test data set files. This way, test data set files can be located next to their test cases.

Action run times measured for all web drivers (#1537)

Now the XLT framework measures the run time of actions when using any web driver, not just the XltDriver.

Additional script commands for Java-bases test cases (#1107)

For advanced test scenarios, script test cases can be exported to Java and be enhanced with loops or conditionals. Previously, to implement condition checks, it was necessary to use the lower-level WebDriver API. Now, Java-based test cases can make use of a variety of new script commands. This greatly simplify specifying conditions and does not force the test designer to mix different APIs. These are:

  • isChecked
  • isElementPresent
  • isSelectedId
  • isSelectedIndex
  • isSelectedLabel
  • isSelectedValue
  • isTextPresent
  • isVisible
  • hasText
  • hasTitle

All these commands return a boolean value, so they can be used in any conditional expression.

Since the Script Developer does not support conditionals in test cases, these commands are not available there.

Miscellaneous

Support of dependency management systems (#1531)

In order to better support test projects that use dependency management systems like Maven or Ivy, we have made a public repository available that hosts all published XLT releases together with their corresponding POM files. This makes updating the XLT version used in test projects much easier. See below for some examples how to configure your test project to use the Xceptance repository.

Integration into Maven
<repositories>
    <!-- Declare Xceptance repository that hosts the XLT artifacts -->
    <repository>
        <id>xceptance-releases</id>
        <url>https://lab.xceptance.de/nexus/content/repositories/releases/</url>
    </repository>
</repositories>
...
<dependencies>
    ...
    <!-- Declare XLT dependency -->
    <dependency>
        <groupId>com.xceptance</groupId>
        <artifactId>xlt</artifactId>
        <version>4.2.0</version>
    </dependency>
    ...
</dependencies>
Integration into Ivy
ivysettings.xml
<ivysettings>
    <settings defaultResolver="resolver-chain" />
    <resolvers>
        <chain name="resolver-chain">
            <filesystem name="local">
                <artifact pattern="${ivy.default.ivy.user.dir}/.ivy/local/[artifact]-[revision](-[classifier]).[ext]" />
            </filesystem>
            <!-- Declare Xceptance repository that hosts the XLT artifacts -->
            <ibiblio name="xceptance-releases" root="https://lab.xceptance.de/nexus/content/repositories/releases" m2compatible="true"/>
        </chain>
    </resolvers>
    <conflict-managers>
        <compatible-cm/>
    </conflict-managers>
</ivysettings>
ivy.xml
<dependencies>
    ...
    <!-- Declare XLT dependency -->
    <dependency org="com.xceptance" name="xlt" rev="4.2.0" />
    ...
</dependencies>

When configuring your test project to use a newer version of XLT, do not forget to update XLT on your load machines as well. The version you have used to develop your test scripts must match the execution version of your load test environment.

Incompatible Changes

Ruby no longer supported as scripting language (#1490)

Ruby is not supported any longer as a scripting language for test cases. This means beginning with XLT 4.2.0 you can neither create new test cases in Ruby, nor execute existing Ruby test cases.

Firefox 3.6 no longer supported (#1491)

Because the official support for Firefox 3.6 ended on April 24th, 2012, the Script Developer will not longer support this version. Firefox 4 or higher is now required to run the Script Developer.