Vaadin TestBench 3.1.0 is hot out of the build agent’s oven and with it comes the wonderful new possibility of fully featured headless testing! The headless testing capability is brought to TestBench by PhantomJS, a full fledged WebKit browser running without any graphical UI. PhantomJS is open source and has packages for Mac OS X, Windows and Linux.
Running your test suite headlessly means that you won’t have any windows popping up on your development machine and you can even run directly on your CI server or build agent without having to set up a graphical environment and/or testing grid. The PhantomJS driver is also generally much faster than the Firefox driver.
Decapitating Your Tests
Running your tests headlessly is super easy. First you need to install PhantomJS as described on their site. Basically download it, unzip and put the executable in a directory on the system’s executable search PATH (e.g. C:\windows\system32 or /usr/local/bin), where it automatically can be found by TestBench. Then, instead of instantiating a FirefoxDriver or a ChromeDriver, etc. instantiate a PhantomJSDriver. It really is that easy!
Everything that is supported by the other browser drivers is supported by PhantomJS. This means that you can also perform screenshot comparison testing completely headlessly!
An Example Project
An example project with headless TestBench tests can be found at https://github.com/vaadin/testbench-headless-example. The project is based on our Address book tutorial, which has been modified to be more TestBench friendly by adding IDs to the components, as well as making the randomly generated data deterministic. The TestBench dependency was, of course, added as well.
The example contains only one test found in the src/test/java/com/vaadin/tutorial/addressbook/NewEntryIT.java file. The test adds a new entry in the address book and verifies that it is found when searching. The test was developed using the Firefox driver and, once finished, switched to the PhantomJS driver. I recommend you use a browser like Firefox or Chrome while developing your tests, as it can be very useful to see what actually happens while you are fine tuning your tests.
Running the test in the example project is as easy as cloning the repository (git clone git://github.com/zch/headless-testbench.git) and running mvn verify on the command line. Note that you will need to have PhantomJS installed, as described above, for the tests to run correctly. Try to switch the new PhantomJSDriver(DesiredCapabilities.phantomjs()) to something else, like new FirefoxDriver() to see every step of the test as it executes and perhaps compare the execution times.
Another thing to try is to make the test fail. You can do this either by changing the address book UI, e.g. removing the contacts table or something similar or changing the test. Let’s change the name that’s entered in our test case. By changing the FIRST_NAME constant from “Sven” to e.g. “Bob” we can make the screenshot comparison fail. Go ahead and change the name and run mvn verify again. Instead of the normal
Tests run: 1, Failures:0, Errors: 0, Skipped: 0
.
.
[INFO] --------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] --------------------------------------------------------------
you'll now see the message
Tests run: 1, Failures:1, Errors: 0, Skipped: 0
.
.
[INFO] --------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] --------------------------------------------------------------
Check the target/testbench/errors folder, there is now an error report here named search-result.html, which looks like this:
Clicking on a highlighted area will show what the reference image looks like in that area, while clicking anywhere else will toggle between the reference image and the newly grabbed screenshot. You did notice that the screenshot was grabbed without any window popping up, right?
Headless testing brings many new possibilities to Vaadin TestBench. How will you make use of this awesome new feature? Tell us in the comments!
Happy testing!
Jonatan is a Senior Vaadin Expert currently working on helping Vaadin customers build the best UIs in the world by providing awesome tools and services that streamline the development process.