«Back
Headless Testing with Vaadin TestBench

Photo by Mary Margret

Vaadin TestBench 3.1.0 is hot out of the build agent’s oven and with it comes the wonderful new possibility of fully featured headless testing! The headless testing capability is brought to TestBench by PhantomJS, a full fledged WebKit browser running without any graphical UI. PhantomJS is open source and has packages for Mac OS X, Windows and Linux.

Running your test suite headlessly means that you won’t have any windows popping up on your development machine and you can even run directly on your CI server or build agent without having to set up a graphical environment and/or testing grid. The PhantomJS driver is also generally much faster than the Firefox driver.

Decapitating Your Tests

Running your tests headlessly is super easy. First you need to install PhantomJS as described on their site. Basically download it, unzip and put the executable in a directory on the system’s executable search PATH (e.g. C:\windows\system32 or /usr/local/bin), where it automatically can be found by TestBench. Then, instead of instantiating a FirefoxDriver or a ChromeDriver, etc. instantiate a PhantomJSDriver. It really is that easy!

Everything that is supported by the other browser drivers is supported by PhantomJS. This means that you can also perform screenshot comparison testing completely headlessly!

An Example Project

An example project with headless TestBench tests can be found at https://github.com/vaadin/testbench-headless-example. The project is based on our Address book tutorial, which has been modified to be more TestBench friendly by adding IDs to the components, as well as making the randomly generated data deterministic. The TestBench dependency was, of course, added as well.

The example contains only one test found in the src/test/java/com/vaadin/tutorial/addressbook/NewEntryIT.java file. The test adds a new entry in the address book and verifies that it is found when searching. The test was developed using the Firefox driver and, once finished, switched to the PhantomJS driver. I recommend you use a browser like Firefox or Chrome while developing your tests, as it can be very useful to see what actually happens while you are fine tuning your tests.

Running the test in the example project is as easy as cloning the repository (git clone git://github.com/zch/headless-testbench.git) and running mvn verify on the command line. Note that you will need to have PhantomJS installed, as described above, for the tests to run correctly. Try to switch the new PhantomJSDriver(DesiredCapabilities.phantomjs()) to something else, like new FirefoxDriver() to see every step of the test as it executes and perhaps compare the execution times.

Another thing to try is to make the test fail. You can do this either by changing the address book UI, e.g. removing the contacts table or something similar or changing the test. Let’s change the name that’s entered in our test case. By changing the FIRST_NAME constant from “Sven” to e.g. “Bob” we can make the screenshot comparison fail. Go ahead and change the name and run mvn verify again. Instead of the normal

Tests run: 1, Failures:0, Errors: 0, Skipped: 0
    .
    .
    [INFO] --------------------------------------------------------------
    [INFO] BUILD SUCCESS
    [INFO] --------------------------------------------------------------

you'll now see the message

Tests run: 1, Failures:1, Errors: 0, Skipped: 0
    .
    .
    [INFO] --------------------------------------------------------------
    [INFO] BUILD FAILURE
    [INFO] --------------------------------------------------------------

Check the target/testbench/errors folder, there is now an error report here named search-result.html, which looks like this:

A failure report

Clicking on a highlighted area will show what the reference image looks like in that area, while clicking anywhere else will toggle between the reference image and the newly grabbed screenshot. You did notice that the screenshot was grabbed without any window popping up, right?

Headless testing brings many new possibilities to Vaadin TestBench. How will you make use of this awesome new feature? Tell us in the comments!

Happy testing!

Jonatan is a Senior Vaadin Expert currently working on helping Vaadin customers build the best UIs in the world by providing awesome tools and services that streamline the development process.

Comments
Thanks for working on this. We are a windows shop, so having to deal with UI tests in the past have been a major pain for us in getting them to run in a CI server environment. Running our tests headless in our CI environment can't get any easier now. Thanks again.

Posted on 6/17/13 4:45 PM.

Top
You should say that it is not working with push enable because phantomjs does not yet support WebSocket. Version 2 should.

Posted on 11/15/13 9:34 PM.

Top
We're pleased to announce the public release of Tori – the modern and sleek discussion forum for Liferay portal . We've kept Tori under the blanket while developing it and
There is a point in every enterprise web application project when the scalability of the application under development is wanted to be measured. There are several reasons for why scalability tests
Vaadin TestBench is a browser automation tool, suited for creating UI level tests for Vaadin applications. It is based on Selenium 2, which means that all features of Selenium 2/WebDriver are also
Servlet 3.0 is awesome, so is CDI . They work well and are a joy to set up. Even adding the Vaadin Navigator to the mix isn’t an issue, since you can use the CDIViewProvider to maintain the
A bit over a year ago we released Vaadin 7 renewing the framework inside out. Then came push support with 7.1 . Now what? Let’s take a look into the future. First off, we do not see
Showing 1 - 5 of 100 results.
of 20

See all blog entries
Subscribe to this blog