Automating Eclipse Jubula Tests with Jenkins

Eclipse Jubula is a pretty new addition to the Eclipse universe. It’s a functional UI testing tool that allows you to specify and run tests. The tests are not code-based but can be assembled from available building blocks or recorded in the UI.

While evaluating tools for doing functional tests for an Eclipse Plugin, I wanted to check out how easy it would be to integrate it into our Continuous Integration builds with Jenkins (the CI server formerly known as Hudson). It turned out there were a few pitfalls, but in the end it was pretty straightforward – hopefully this article will save you some time.

To get started, download and install Jubula from the download page. After installation (simply run setup.sh, and install it for example into /home/jenkins/jubula), you’ll have to prepare your target RCP or Eclipse Runtime by extracting rcp-support.zip from the Jubula installation to the RCP Application/Eclipse directory. This contains a plugin that needs to be available in the AUT (Application Under Test) for Jubula to be able to run the tests.

Pitfall: The provided Jubula installation package for Linux includes a 32-bit JRE and SWT in a 32-bit version that is used to run Jubula and its tools. The CI server being a 64-bit machine I had to install quite a few 32-bit shared libraries to get Jubula to run. To find out what libraries are missing, starting Jubula locally on the CI server with the -debug -console -consoleLog options helped. On the CI build server you might get away without this step if all you run locally is the autagent (see below).

Specifying a Test

Jubula comes with quite a good User Manual. For an introduction on how to create your first Jubula test, I’d recommend the excellent series of posts by Joachim.

This post assumes that you have a test project name MyTest with a test suite named AboutDialogTest and AUT configuration named Eclipse_Helios.

Pitfall: Make sure your test specifies either German (Germany) or English (United States) as the keyboard layout (see here). These are the only two keyboard layouts provided out-of-the-box by Jubula at the moment.

Test Executor

Running Jubula from the command line is done using the Test Executor (testexec). It’s command line options are documented here.

If, for example, you’re  testing an Eclipse Plugin, your testexec command line would look something like this:

$JUBULA_HOME/jubula/testexec -project MyTest -version 1.0 -testsuite AboutDialogTest 
-server localhost -port 60000 -autconfig Eclipse_Helios 
-datadir $WORKSPACE/testdata -resultdir $WORKSPACE/testresults
-data /home/jenkins/.jubula/workspace 
-language en_US -dbscheme "Default Embedded (H2)"
-dbuser sa -dbpw ""

This is using the default embedded database scheme. In a production environment, you’ll typically use a real database that is available to both testers specifying tests and the build server. Available DB schemes are configured in Jubula under Window/Preferences/Test/Database Connections. The -data parameter points to the workspace directory where the database configuration is stored. So you’ll have to either edit this locally on the server, or copy the workspace over to the server to somewhere Jenkins can see it.

The test executor can be configured to either

• Start a new AUT instance as shown above with the -autconfig parameter
• Connect to a running AUT instance that has been started using the -autid parameter

You can also pass all these options via a configuration file instead of command line parameters should you prefer.

AUT Agent

Before running the Test Executor, you’ll have to start the AUT Agent (Jubula daemon providing the bridge between the test executor and the Application Under Test) first.

The AUT agent can be started with

$JUBULA_HOME/server/autagent -p 60000 -v

• -p: Port
• -v: Verbose mode

As this this an UI test, the agent is started under X11. I haven’t yet explored the possibilities of running the AUT without having an active X11 session. Our CI server has an active X11 session all the time for driving an Extreme Feedback Panel anyways.

Pitfall: The AUT agent seems to accept only a single connection at a time, the test executor can’t connect to it while Jubula is still connected to the same AUT Agent.

Configuring Jenkins

1.) Create a new Job

2.) Add a step for creating necessary directories, and a step to run testexec

Note: The shell commands are just provided as an example. In a real job you would most probably replace this with something like an Ant script to prepare the environment, auto-deploy your plugin or the whole RCP application, run the the tests and then check the return code from testexec. Edit: This has been documented here

3.) Run the job

4.) Check out the results

The generated HTML report will look something like this:

Possible Job Improvements

• Auto-deploy plugin and/or RCP runtime
• Connect to a real database instead of the embedded Jubula instance
• This is simply a matter of configuring a new database in the workspace pointed to by the -data parameter (see above), and setting the -dbscheme parameter to the name of the DB configuration.
• Show screenshots taken by Jubula upon test failures directly in the test report
• This is currently only possible inside Jubula’s test result perspective. Would be nice to have the screenshot embedded into the HTML reports.