XLT 4.5.8

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.5.8. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Support of Firefox 42 (#2538)

Script Developer has been made compatible with Firefox 42. It now supports Firefox 31/ESR to 42.

Framework

Extended the scope of dynamic test data (#2531)

It is possible to customize the Java wrapper classes for XML script test cases to run additional prepare scripts before or cleanup scripts after the actual test script. In such a scenario, any dynamic test data (that is data created with store... commands) was available in the current script only, but not in the following scripts. Now all scripts executed from the same wrapper class share the same scope for dynamic test data and thus can access data created in earlier stages. This makes certain tasks, especially cleanup tasks, a lot easier.

Note that exported script test cases have always had a shared scope for test data so this change also unifies the behavior of XML-based and Java-based script test cases.

XLT 4.5.7

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.5.7. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Support of Firefox 41 (#2524)

Script Developer has been made compatible with Firefox 41. It now supports Firefox 31 ESR to 41.

Note that beginning with Firefox 40, Mozilla introduced a code-signing mechanism to prevent abuse by malicious extensions and to improve security and privacy. This means that each Firefox extension needs to be signed by Mozilla, otherwise it can be installed only after clicking away a warning message (Firefox 40 to 42) or it won’t install at all (Firefox 43+). To accommodate this, Script Developer is signed now as well.

In this course, Script Developer has also got a new extension ID. However, this new ID will make it impossible for Firefox to auto-update Script Developer to version 4.5.7. You will need to do this manually. Please uninstall the currently installed version first and afterwards install the new version by simply dragging and dropping the file <xlt>/tools/xlt-scriptdeveloper-4.5.7.xpi onto Firefox. Note that this manual process has to be done only once. Beginning with version 4.5.8, auto-update will work as usual again.

Empty module class generated for Java modules (#2505)

An XML script test case may call custom Java modules. When exporting this script test case to Java (Scripting API), the code to call this custom Java module will be generated correctly. However, same as for regular script modules, Script Developer created an (empty) module class for custom modules as well. This class is not needed at all as the custom module is already available as Java code.

Removing a project not possible on Mac (#2520)

To remove a test project from Script Developer, you have to click the project drop-down list, hover over the project in question, and press the Delete key (or any of its equivalents on Macs without a full-size keyboard). However, on a Mac nothing happened and the project was still there. Fixed now.

Load Testing

More flexible test suite directory layout (#2516)

Previously, the load test agent searched for compiled classes and libraries in <testsuite>/classes and <testsuite>/lib only. However, some build tools (e.g. Maven) or IDEs (e.g. Eclipse) use other output directories by default. In order to relieve the user from having to reconfigure the default directories, the agent will now look for classes and libraries in all of the “usual suspects”:

Classes

  • <testsuite>/classes (XLT default)
  • <testsuite>/target/classes (Maven)
  • <testsuite>/target/test-classes (Maven)
  • <testsuite>/bin (Eclipse)

Libraries

  • <testsuite>/lib (XLT default)
  • <testsuite>/target/dependency (Maven)

Empty host names when listing AWS instances (#2513)

ec2_admin can list the agent controller URLs for all/selected AWS instances. However, instances of certain types (for example m4.xlarge) may not always have a public host name assigned, so the host part in the URLs was empty. Now the public IP address will be used as a fallback.

Framework

Empty result browser if taking screenshots fails (#2502)

When taking screenshots, the browser may fail with an exception, for instance, if the configured window size is too small. The framework does not abort the test run in such cases, but as a side effect the generated result browser was completely empty. Now the result browser shows the list of executed actions again, and if a screenshot is not available it will be replaced with an empty placeholder image.

Broken result browser (#2509)

In case the value of a request/response header or request parameter is null or missing, the result browser failed to initialize and was not usable. Fixed now.

XLT 4.5.6

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.5.6. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Incomplete script migration (#2484)

When (re-)loading a test suite, Script Developer checks if the scripts need to be migrated to a newer script version. If so and the user confirms, all the scripts will be rewritten and the version number in the scripts will be updated. However, this did not happen for Java modules. So when reloading an already migrated suite, Script Developer prompted for migration once again as the previous migration was incomplete.

Load Testing

Custom value graphs showed averaged values only (#2477)

In the load test report, the custom value charts represented all the values measured for a certain second as a simple average value only, but not as a min/max value. This way, spikes were hardly visible and the shown value range in the charts did not match the minimum/maximum values in the data table. Fixed.

Client-performance data didn’t show up in Graphite (#2472)

When running client-performance tests via Firefox with script test cases that have been exported to Java, client-performance data was gathered successfully, but wasn’t reported to Graphite.

Unzipping a just uploaded test suite archive failed (#2494)

It could happen that an agent controller failed to unzip a freshly uploaded test suite if there are multiple XLT installations on the same machine and the agent controllers are all active and in use. In such a scenario, one agent controller could overwrite/delete the test suite archive of another agent controller as all agent controllers used the same file name in the tmp directory.

Master controller stuck during upload (#2483)

In case an agent controller had problems installing a test suite after an upload operation, the processing status was not updated correctly so the master controller waited forever for the agent controller to finish.

Upload and download may fail (#2487)

The communication between master and agent controllers is protected by a password. If the configured password contains non-ASCII characters, then parts of the communication (upload and download, that is) could fail depending on the platform’s default character encoding. Fixed now.

Framework

Retry a command if the target element is reported as stale (#2489)

Sometimes parts of an HTML page are updated asynchronously by removing an element and afterwards adding a similar element, but with updated content. In this situation, a test case could have successfully looked up the old element, but may no longer access it/interact with it because it has become “stale” in the meantime (i.e. it was replaced by the new element). A test case designer has to deal with this situation. The usual approach is to retry the lookup and the operation.

For XML script test cases, this issue is now handled transparently by the XLT framework. By default, a command that failed because of a stale element will now be retried once, but you can reconfigure the number of retries as follows:

com.xceptance.xlt.scripting.commandRetries = 1

In case the command still fails after the last retry, the test case will fail.

Line numbers in script stack traces incorrect (#2466)

In case an XML script test case failed when being replayed via the XLT framework, the corresponding script stack trace contains the line number of the offending command. However, this line number could be smaller than the command’s actual line number as shown in Script Developer. This happened if inline comments were used in the script or if there have been disabled module calls. Now the line numbers are correct again.

Handshake failed for some servers (#2478)

In case a server is more restrictive regarding the supported encryption protocols, XLT may fail to establish a secure connection. The reason is that the JRE limits the set of enabled protocols to SSLv3 and TLSv1 by default. So if the server supports TLSv1.1 or TLSv1.2 only, the handshake will fail. Luckily, the JRE supports a wider range of protocols, namely SSLv2Hello, SSLv3, TLSv1, TLSv1.1, and TLSv1.2. They just need to be enabled. To reconfigure the set of enabled protocols, use the following setting:

com.xceptance.xlt.ssl.protocols = SSLv3, TLSv1, TLSv1.1, TLSv1.2

XLT 4.5.5

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.5.5. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Support of Firefox 39 (#2463)

Script Developer has been made compatible with Firefox 39. It now supports Firefox 31 ESR to 39.

Delete domain cookies as well (#2458)

When executing the command deleteAllVisibleCookies, Script Developer does now not only delete the cookies set for the current host, but also those set for the current domain.

Speed slider not working on OS X (#2421/2423)

When using Script Developer on Macs, the replay speed slider could not be dragged. Also, the ‘+’ key to increase the replay speed did not work. Fixed.

Load Testing

Lost iterations when using the arrival rate model (#2287)

If the system under test responds slowly, the available number of virtual users may not be sufficient to maintain the configured arrival rate. This is normal. In this case, the behavior of the arrival rate load model turns into that of the user count load model. However, the finally achieved number of transactions was not as high as when using the user count load model directly. Fixed now.

Framework

Delete visible cookies only (#2458)

Previously, XltDriver.manage().deleteAllCookies() deleted all cookies, but does now delete the cookies for the current host/domain only. This change was necessary to be compatible with the Web drivers for native browsers. Likewise, when replaying the script command deleteAllVisibleCookies only the cookies set for the current host/domain are deleted.

XLT 4.5.4

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.5.4. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Support of Firefox 38 (#2444)

Script Developer has been made compatible with Firefox 38. It now supports Firefox 31 ESR to 38.

Different results when executing a text matching condition and testing it (#2424)

The condition of assert... or waitFor... commands can be tested against the current browser page in the Edit Command dialog. Just press the Evaluate button in the Try It Out section. Obviously, the evaluation result is supposed to be the same as when really executing the command during test case replay. However, for text matching conditions this was not always the case. Fixed.

Selected text trimmed when recording text assertion (#2446)

When recording a text assertion using the appropriate item in the context menu and the user has selected some text, this text was recorded with leading and trailing whitespace trimmed. This could be problematic if the user wants to explicitly check for the presence of whitespace. For instance, with the selected search text " bar " trimmed, the command would have accepted test texts such as “foo bar baz”, but also “foobarbaz”, which was not intended. Fixed.

Load Testing

Load profile table showed wrong total ramp-up period value (#2418)

If a load profile contains a test case that defines a ramp-up period and another one that does not, the total ramp-up period value shown in the footer row of the load profile table in the test report was wrong. It was, for instance, -01:-01:-01...00:00:30, but should have been 00:00:00...00:00:30 instead. Fixed.

Framework

Check for transaction termination requests whenever a new action begins (#2200)

At the end of the shutdown period, any still running user thread is requested to abort its current transaction gracefully. The same happens to transactions that exceed the transaction runtime limit. A user thread has to regularly check whether such a termination request is pending and respond accordingly. XLT performs such a check automatically at various places in the framework code, and now it does so also when starting a new action. This way, test cases that do not involve much XLT code (such as plain WebDriver tests that use a FirefoxDriver) are now able to respond to termination requests more quickly.

Headless Firefox tests occasionally failed at startup (#2429)

When running tests with a real Firefox in headless mode, it could happen that the tests immediately failed at browser start-up with an error message like this one: Error: cannot open display :40. This is fixed now.

Newlines or tabs in class attributes prevented element lookup by class name (#2430)

HtmlUnit / XltDriver supports finding elements by CSS selectors, for instance, by class name. If the class attribute of the targeted element contains tabs or newlines (instead of regular spaces) to separate multiple class names from each other, the lookup failed. Fixed.

XLT 4.5.3

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.5.3. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Support of Firefox 37 (#2397)

Script Developer has been made compatible with Firefox 37. It now supports Firefox 31 ESR to 37.

Permit to save currently invalid locators (#2369)

It is perfectly valid for locator expressions to contain variables. However, if a variable could not be resolved, the command editor dialog complained with an error message and refused to save the command. This was a little bit too restrictive, as variables defined by store commands receive their value exactly at the moment of their execution and are therefore always undefined during edit time. Now Script Developer warns you if it encounters an undefined variable, but lets you save the command.

Character escaping rules for Java properties files not applied (#2388)

Script Developer uses Java properties files for test data defined globally or at package level. Unfortunately, Script Developer wrote the data unmodified to these files, ignoring any character escaping rules defined for Java properties files. This could cause problems when replaying tests with the XLT framework.

Imagine a test data item holding a Windows-like path c:\temp. When replaying the test with the XLT framework, the Java code converts the \t to a tab character. If the data item had correctly been c:\\temp, all would have been fine.

This has been fixed and Script Developer now applies the same character escaping rules as Java when reading and writing properties files.

Data sets file not handled properly when deleting or refactoring a script (#2392)

A script file TFoo.xml may be accompanied by a data file TFoo_data.xml, but also by a data sets file TFoo_datasets.xml (or other variants, such as TFoo_datasets.csv). When renaming/moving/copying/deleting the script, the data file was renamed/moved/copied/deleted accordingly, but the data sets file was not (or at least not consistently). Fixed.

Load Testing

Process did not terminate when agent controller failed to start (#2384)

It might happen that an agent controller cannot start successfully, for instance when the configured port is already in use. In such cases, the Java process did not terminate and needed to be killed manually. Now the process terminates immediately in such a case and returns an appropriate error code.

Report updated metrics to Graphite before agent shutdown (#2391)

All updated metrics are sent to the configured Graphite host periodically as a batch. However, if a load test has finished somewhere in the middle between two reporting times, the data gathered since the last reporting time was lost. Now the agent reports any pending data right before it shuts down. The time-stamp attached to the sent data will always be the next regular reporting time, regardless of the actual amount of time passed since the last reporting time.

Test Reports

Total CPU usage was always 0 with Java 8 (#2386)

The load test report does not only show the CPU usage of an agent process, but also the total CPU usage of the agent machine. However, the latter value was always reported as 0 when the test has been run using a Java 8 runtime. Fixed.

Optionally extend the value range of trend charts to include 0 (#2400)

The trend report uses charts to visualize the trend graphically. Currently, the charts show the effective value range only (min to max). As a consequence, even steady trends may appear fluctuating heavily, and only when looking at the runtime axis will you see that the differences are actually small. To make it easier for you to assess the significance of deviations at a glance, the charts’ value range may now be optionally extended to include the value 0. This way, larger deviations can still be recognized easily, while smaller deviations don’t stand out anymore.

Whether the value range should start at 0 can be configured in <xlt>/config/trendreportgenerator.properties using a new property:

com.xceptance.xlt.trendreportgenerator.charts.autoRangeIncludesZero = true

Framework

Typing into file inputs failed for native Web Drivers (#2387)

Uploading a file is usually done by typing the file name into a file input element and submitting the form afterwards. While typing into a file input with the type script command worked well in Script Developer and XltDriver, the Web Drivers for native browsers threw an exception. Fixed.

Script Doc Generator

Mark a disabled module in test steps (#2375)

In Script Developer, disabled module calls are shown crossed out. In the generated script documentation of a test case, a disabled module call will now be shown crossed out, too.

XLT 4.5.2

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.5.2. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Framework

Selenium updated (#2382)

Selenium has been updated to v2.45.0. This version fixes an issue in Selenium when using FirefoxDriver in combination with Firefox 35 or higher. The result of a JavaScript expression passed to FirefoxDriver for execution was always undefined. See Firefox 35: Passing arguments to executeScript isn’t working. Consequently, a script test case containing any of the *Eval commands (assertEval, storeEval, etc.) failed when being replayed with FirefoxDriver.

XLT 4.5.1

This section lists and documents all improvements and important fixes of Xceptance LoadTest 4.5.1. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Script Developer

Support of Firefox 36 (#2371)

Script Developer has been made compatible with Firefox 36. It now supports Firefox 31 ESR to 36.

Unresponsive Script message when migrating a test suite (#2357)

When opening an existing test suite, Script Developer may need to migrate all the scripts of that test suite. For larger suites, this migration can take some time and Firefox might complain about an “unresponsive script” (note: Script Developer is implemented in JavaScript). Script Developer has been modified to avoid this message, even when migrating large test suites. The migration process has been tuned as well. Additionally, a newly introduced progress bar shows the state of the migration or loading of the test suite now.

Test suite mix up (#2318)

Previously, it was possible to switch to another test suite while the current test suite was still being loaded. This could have caused a mixed-up package tree with scripts from both test suites. Now all essential UI controls are locked while loading a test suite.

Variables created by storeElementCount command reported as undefined (#2349)

Before replaying a test case, all used test data placeholders are verified. A message is shown in case some of them could not be resolved. Variables defined by store* commands inside the script don’t count as undefined test data placeholders. Unfortunately, the command storeElementCount was skipped and all variables defined by this command were reported as undefined. Fixed.

Empty test data definitions reported as undefined (#2368)

Test data items with an empty string as value were reported as undefined. Fixed.

Test data names not allowed to start with underscore (#2344)

Previously, names of test data items couldn’t start with an underscore character. This restriction was not really needed. Now you can use the underscore character to prefix your test data items.

Values of renamed test data items aren’t shown in script editor (#2340)

The script editor normally shows the actual value of a test data item instead of its name when hovering over it. This sometimes didn’t work when the test data item was just renamed. Fixed.

Module test data not available to called modules (#2359)

A script module can define its own test data items. These test data items are supposed to be available for all modules called by this module. Unfortunately, this did not work correctly for the entire call chain. Fixed.

Non-element nodes returned by XPath element locators should be ignored (#2331)

Element locators should, as the name suggests, locate element nodes only. However, certain expressions in XPath element locators may result in text or number nodes being returned, for instance text() or count() functions. Any such node will be filtered out from the result set now.

Load Testing

ec2_admin sporadically failed to tag freshly created instances (#2350)

Sometimes ec2_admin failed to assign a name tag to a machine instance that was just created. It complained that the instance didn’t exist. AWS instances are created in the background and may not be available instantly. Therefore, ec2_admin will now wait for an instance to become available before tagging it.

ec2_admin did not always return with an error code in case of errors (#2364)

With the new non-interactive mode, the ec2_admin tool can also be used in scripts now. Therefore it is essential that the tool indicates any error condition by returning an appropriate return code. This was not always the case. Fixed.

Cannot run traditional scenarios and client-performance test scenarios together (#2352)

In a load test, test cases flagged as clientPerformanceTest in their load profile configuration are deployed to only those agent controllers that are flagged as clientPerformance in the agent controller configuration. All other test cases are deployed to the remaining agent controllers. However, when mixing regular and client-performance tests, an exception was thrown and the load test aborted.

Framework

Sanitize the name of a metric before sending it to Graphite (#2360)

Graphite does not define what characters are allowed for naming a metric. Even if a metric was received and stored on disk successfully, it doesn’t automatically mean that the metric can be queried. Therefore we have restricted the range of characters to letters, numbers, the underscore character, and the dash. Any other character is replaced with an underscore before the metric is sent to Graphite.

Status code missing in client-performance timer data (#2351)

When measuring the client performance, the collected timer data is stored in a CSV file. However, sometimes the status code field for requests was empty and the load report generator had problems when parsing this data. Now the value 0 is set in case a status code was unavailable.

XLT Plug-In for Jenkins

Agent controller connect timeout configurable in the UI (#2360)

The XLT plug-in can be configured to start load agents in Amazon’s compute cloud. It can take a while until the agent controllers are up and running. Therefore the plug-in has to wait for the agent controllers to become available. This time period can now be configured in the XLT section of the load test job’s configuration page.

Known Issues

If a script test case contains any *Eval commands (assertEval, storeEval, etc.) and this test case is executed in Firefox 35+ via the FirefoxDriver, the test case will likely fail since the result of the JavaScript expression passed to this command will always be undefined. The reason is a known bug in the WebDriver plug-in for Firefox. Unfortunately, a new Selenium release with a bug fix is not available yet. See Firefox 35: Passing arguments to executeScript isn’t working.

XLT 4.5.0

This section lists and documents all new features and improvements of Xceptance LoadTest 4.5.0. Registered customers can see an overview of changes and view the current roadmap in the XLT Information Center.

Release Overview

XLT 4.5.0 comes with many new and improved functionalities.

Script Developer has been enhanced to let you maintain the documentation of your script library directly in Script Developer. To this end, all description/comment fields have been enlarged and support the MarkDown syntax for rich-text descriptions. From all the descriptions in your scripts, an extensive HTML documentation of your script library can be generated, ready to be shared with your team and your clients. The newly introduced in-line comments enable you to document a sequence of commands and visually structure your test scripts at the same time.

Load testing is more fun now. XLT load agents can send performance metrics to Graphite, a data collection and graphing tool. This allows you to watch the most important performance results in real time while a load test is still running. The load test report including all performance details has been improved as well. Most notably, the runtime charts now visualize the distribution of values and show how this distribution varies over time. Also, generating the load test report is much faster now on multi-core machines. Users of Amazon’s EC2 cloud will be glad to hear that the ec2_admin tool can now also be used in scripted processes, making it possible to fully automate the starting and stopping of Amazon machines.

The scripting capabilities in Script Developer and the XLT framework have been extended. There are new commands to operate an application’s context menu. If you need to switch browser windows, you can use wild cards or regular expressions to locate the target window. Package-level test data, which was previously accessible to script test cases only, can now be used from script modules as well.

As usual for a major release, the technology stack the XLT framework is built upon has been updated and provides latest WebDriver support. Last but not least, the XLT Jenkins plug-in supports build nodes now and may automatically start and stop agent machines in Amazon’s compute cloud.

See below for a more detailed introduction to the most important features and improvements. The new release contains various other enhancements and fixes, so we strongly recommend to upgrade your XLT installation as soon as possible.

Script Developer

Script Documentation

As always, good documentation is key, especially when working with large test suites. It helps you avoid redundant code and it facilitates the learning progress of new team members. We encourage you to write elaborate test case and module descriptions. That’s why we extended the documentation possibilities in Script Developer:

  • Comments fields have been turned into multi-line text areas, so they can hold longer texts.
  • Description/comment text areas now support the MarkDown syntax (GitHub flavor) for rich-text descriptions. The formatted text can later be viewed in the Information panel.
  • The parameters of modules can now be documented in dedicated fields.
  • Test cases have an additional ID field. Use this field to maintain an alternative, maybe externally defined name or ID.

From all the descriptions and meta data in the scripts an extensive HTML documentation of your script library can be generated. This gives you a good overview on all available test cases and all already existing modules, which form the building blocks for your new test cases. For each script, the documentation contains:

  • name and description,
  • description of parameters (for modules only),
  • basic steps (automatically excerpted from the code),
  • any test data used, and
  • any variables created.

To build the HTML documentation, run the script documentation generator as follows:

cd <xlt>/bin
create_scriptdoc.sh <path_to_testsuite>

By default, the documentation for your test suite will be generated to a newly created scriptdoc sub-directory in your test suite. To specify a target directory other than the default directory, use the -o <output_dir> command line option.

The HTML output can be customized if needed. See the page templates in <xlt>/config/scriptdoc/templates and the accompanying CSS stylesheets in <xlt>/config/scriptdoc/resources/css.

The script documentation generator uses FreeMarker as the template engine. See their website for more information about the template language.

In-line Comments

Script Developer supports in-line comments now. Use such comments to document the purpose of a sequence of commands and to visually structure the code at the same time. No need to misuse action markers for this purpose anymore!

New and Extended Commands

Script Developer supports new commands:

  • contextMenu – simulates clicking the right mouse button on the specified element
  • contextMenuAt – simulates clicking the right mouse button on the specified element at the given offset coordinates

The selectWindow command has been extended to accept wild cards or regular expressions in window locators. This makes it easier to select a certain browser window if the window’s name/title contains variable parts.

See the User Manual for more details and usage examples.

Package-Level Test Data for Modules

Script Developer allows to define test data not only on script level and globally, but also on package level. Previously, only test cases could access package-level test data. It is now available for modules as well.

Usability Improvements

From now on, Script Developer maintains an always empty last line in a script editor tab. This line makes it easy to insert commands, module calls, comments, and actions at the end of the script.

Supported Firefox Versions

Script Developer has been made compatible with the latest available Firefox version (as of January 2015), while outdated versions are not supported any longer. Thus, Script Developer runs on Firefox 31 ESR up to 35 now.

Load Testing

Real-Time Reporting via Graphite

While a load test is running, the master controller only shows basic information about the status. In interactive mode, it allows to get intermediate results, so you can download them and generate a report to see how the test is going. In automated environments, however, you would have to wait until the test run is finished before you can actually do so.

Wouldn’t it be great if you could watch the results in real time and see how the performance varies over time while the test is still running? To this end, we have added the support of Graphite, a well-known data collection and graphing tool. During a load test, XLT pushes selected metrics to Graphite. Using Graphite’s graphing capabilities or another graphing/dashboard tool on top of Graphite, you can watch the most important performance data instantly:

Load Testing Dashboard

See below for the XLT settings needed to enable and configure real-time reporting:

xlt.reporting.enabled = true
xlt.reporting.interval = 15
xlt.reporting.metricNamePrefix = xlt.MyProject.
xlt.reporting.graphite.host = my.grahite.machine
xlt.reporting.graphite.port = 2003

If enabled, XLT reports the following metrics to Graphite:

  • runtime statistics (in total and per name), errors (in total and per name) and counts (in total) for
    • transactions,
    • actions,
    • requests, and
    • custom timers,
  • arrival rates (in total and per transaction name),
  • bytes/sent received (in total and per request name),
  • event count (in total), and
  • agent metrics such as the total CPU usage and heap usage.

Since not all performance details are sent to Graphite, you will only get a first impression of the application’s behavior. For a detailed analysis, the load test report is still the tool of choice.

Graphite is not bundled with XLT. You need to install, configure, and run it yourself.

Load Test Report

Visualization of Value Clustering

Quite often, the measured values are not equally distributed, but form certain clusters. To make this clustering visible, charts show the values not only as line graphs, but also as dots now. While the line graph perfectly visualizes the value range at a certain point in time, the dotted representation naturally builds the clusters. Areas with no or only a few dots will be lighter, areas with a higher concentration of dots will be darker.

New Chart

We believe that the enhanced charts will give you new insights into the behavior of your web application, especially with respect to the distribution of the values and how this distribution varies over time. While the former aspect can also be seen in the histogram charts, the latter hasn’t been visible until now.

Other Improvements
  • Runtime charts now provide a small histogram next to the runtime graphs. This histogram lets you quickly assess the distribution of values without having to consult the separate histogram chart first.
  • The transaction charts have been extended to show not only the error count, but also a (smoothed) error rate. Use the error rate graph to get the percentage of the transactions that failed at a certain time.
  • The Network chart for a request has been replaced by the Response Size chart that shows whether and how much the size of the entire HTTP response varies.
  • The Events section includes an overview table, much like the one that already exists for errors. It summarizes the total number/percentage per event type.

Load Test Report Generator

Faster Report Creation on Multi-CPU Machines

The load test report generator makes use of all available CPUs now when generating charts, when reading, parsing and pre-processing data records, and while generating the HTML report pages. Especially result sets with many or very sophisticated request merge rules will be processed much faster.

In case you want to limit the number of CPUs used during report generation, set the property com.xceptance.xlt.reportgenerator.threads in <xlt>/config/reportgenerator.properties to an appropriate value.

New Ways to Specify the Report Calculation Period

The load test report generator allows you to create a report for the total time of the test as well as for a shorter period. This is useful to exclude the ramp-up period from the report or to “zoom into” an especially interesting period, for instance. In former versions of XLT, this could be done by specifying an absolute date/time value as the in/out time, e.g.:

create_report -from 20141210-140000 -to 20141210-150000 ...

The in/out times can now also be given as time offsets relative to the beginning (+x) or the ending (-x) of the test. In the following example, we omit both the first 15 minutes and the last 10 minutes of the test:

create_report -from +15m -to -10m ...

Compared to the absolute notation, this is much shorter and doesn’t require you to calculate the in/out times based on the test’s start/end time.

Another common use case is to create the report for a certain duration. Again, you don’t need to calculate the out time any longer since the duration can now be specified directly. For example:

create_report -from 20141210-140000 -l 1h ...
create_report -from +15m -l 1h ...

In the first example, you will get a report for exactly the time between 14:00:00 and 15:00:00. In the second example, you will also get a report for exactly an hour, but this time the report starts 15 minutes after the test has been started, no matter when that was.

To specify time periods, you may not only use the “1h15m10s” syntax as in the examples above, but also the “1:15:10” syntax.

ec2_admin

ec2_admin, our tool to manage Amazon EC2 machine instances, has been extended with a non-interactive mode where all required parameters have to be passed on to the tool’s command line. This allows you to automate the management of Amazon machines.

  • To start machines: ec2_admin.sh run <region> <instance-type> <ami-id> <instance-count> <tag>
  • To stop machines: ec2_admin.sh terminate <region> <tag>

But starting and stopping machines is only half the story. The master controller also needs to know about the freshly started agent machines so it can use them for a load test. To this end, ec2_admin emits the corresponding agent machine configuration right after the machines were started. By default, it prints this configuration to the console. Alternatively, the configuration can also be written to a certain file via the -o option. We recommend the latter approach since the master controller can read the agent configuration directly from this file later on.

For a fully automated load testing process with Amazon machines, use a sequence of commands (in a script) similar to the following one:

ec2_admin.sh  run  eu-central-1  c3.2xlarge  ami-de5dcdb6  5  Posters  -o agents.properties
mastercontroller.sh  -auto  -report  -pf agents.properties
ec2_admin.sh  terminate  eu-central-1  Posters

Note how ec2_admin writes the agent machine configuration to the file agents.properties which in turn is passed on to the master controller as input. Be aware, though, that it may take a while until the agent controllers are up and running on the freshly started machines. To stop the master controller from complaining too early about unreachable agent controllers, you should configure an appropriate waiting time in mastercontroller.properties, one minute, for example:

com.xceptance.xlt.mastercontroller.initialResponseTimeout = 60000

Framework

New Script Commands and Package-Level Test Data

As mentioned above, Script Developer introduces some new commands and the ability to use package-level test data for modules. The XLT framework has been extended to support these new features as well when replaying script test cases.

XLT Plug-In for Jenkins

Support for AWS EC2 machines

Typically, larger load tests require a couple of dedicated load agent machines. On-demand machines as offered by Amazon’s EC2 service are a perfect choice for this purpose. That’s why the XLT plug-in also supports Amazon machines now as an alternative to using a fixed set of local machines. All you need to do is to configure the region where the machines will be located, the number of machines to use, the machine type, and the machine image. The plug-in will automatically start these machines before the load test, configure the master controller to use these machines for the test, and terminate them once the test has finished.

Running on Build Nodes

Jenkins supports the distributed execution of build jobs on dedicated slave machines known as “build nodes”. The XLT plug-in has been extended to successfully run on such build nodes as well.