Testing is one of the cornerstones of modern software development. Extending throughout the development process, testing is the thread that binds the product to the requirements. In agile and other iterative development processes, with ever shorter release cycles and continuous integration, the automation of integration, regression, endurance, and acceptance testing is paramount. Further, UI automation may be needed for integration purposes, such as for assistive technologies. The special nature of web applications creates many unique requirements for both testing and UI automation.
Vaadin TestBench allows controlling the browser from Java code, as illustrated in Controlling the Browser with Testbench. It can open a new browser window to start the application, interact with the UI components, for example, by clicking them, and then get the HTML element values.
Before going further into feature details, you may want to try out Vaadin TestBench yourself. You just need to create a new Vaadin project either with the Eclipse plugin or the Maven archetype. Both create a simple application stub that includes TestBench test cases for testing the UI. You also need to install an evaluation license. For instructions, jump to "Quick Start" and, after trying it out, come back.
Vaadin TestBench can work as the centerpiece of the software development process, for testing the application at all modular levels and in all the phases of the development cycle:
Automated acceptance tests
End-to-end integration tests
Let us look at each of these topics separately.
Any methodological software development, agile or not, is preceded by specification of requirements, which define what the software should actually do. Acceptance tests ensure that the product conforms to the requirements. In agile development, their automation allows continuous tracking of progress towards iteration goals, as well as detecting regressions. The importance of requirements is emphasized in test-driven development (TDD), where tests are written before actual code. In "Behaviour-Driven Development", we show how to use Vaadin TestBench for behaviour-driven development (BDD), a form of TDD that concentrates on the formal behavioural specification of requirements.
Unit testing is applied to the smallest scale of software components; in Vaadin applications these are typically individual UI components or view classes. You may also want to generate many different kinds of inputs for the application and check that they produce the desired outputs. For complex composites, such as views, you can use the Page Object Pattern described in "The Page Object Pattern". The pattern simplifies and modularizes testing by separating low-level details from the more abstract UI logic. In addition to serving the purpose of unit tests, it creates an abstraction layer for higher-level tests, such as acceptance and end-to-end tests.
Integration tests ensure that software units work together at different levels of modularization. At the broadest level, end-to-end tests extend through the entire application lifecycle from start to finish, going through many or all user stories. The aim is not just to verify the functional requirements for user interaction, but also that data integrity is maintained. For example, in a messaging application, a user would log in, both send and receive messages, and finally log out. Such test workflows could include configuration, registration, interaction between users, administrative tasks, deletion of user accounts, and so forth.
In regression testing, you want to ensure that only intended changes occur in the behaviour after modifying the code. There are two lines of defence against such regressions. The primary source of regression tests are the acceptance, unit, and integration tests that validate that the displayed values in the UI’s HTML representation are logically correct. Yet, they are not sufficient for detecting visual regressions, for example, because of invalid UI rendering or theme problems. Comparing screenshots to reference images forms a more sensitive layer to detect regressions, at the expense of losing robustness for changes in layout and themeing. The costs of the tradeoff can be minimized by careful application of screenshot comparison only at critical points and by making the analysis of such regressions as easy as possible. As described in "Taking and Comparing Screenshots", Vaadin TestBench automatically highlights differences in screenshots and allows masking irrelevant areas from image comparison.
You can develop such tests along with your application code, for example with JUnit, which is a widely used Java unit testing framework. You can run the tests as many times as you want in your workstation or in a distributed grid setup.
The main features of Vaadin TestBench are:
Control a browser from Java
Generate component selectors in debug window
Validate UI state by assertions and screen capture comparison
Screen capture comparison with difference highlighting
Distributed test grid for running tests
Integration with unit testing
Test with browsers on mobile devices
Execution of tests can be distributed over a grid of test nodes, which speeds up testing. The grid nodes can run different operating systems and have different browsers installed. In a minimal setup, such as for developing the tests, you can use Vaadin TestBench on just a single computer.
Vaadin TestBench is based on the Selenium web browser automation library, especially Selenium WebDriver, which allows you to control browsers straight from Java code.
Selenium is augmented with Vaadin-specific extensions, such as:
Proper handling of Ajax-based communications of Vaadin
A high-level, statically typed element query API for Vaadin components
Performance testing of Vaadin applications
Screen capture comparison
Finding HTML elements by a Vaadin selector
The TestBench library includes WebDriver, which provides API to control a browser like a user would. This API can be used to build tests, for example, with JUnit. It also includes the grid hub and node servers, which you can use to run tests in a grid configuration.
Vaadin TestBench Library provides the central control logic for:
Executing tests with the WebDriver
Additional support for testing Vaadin-based applications
Comparing screen captures with reference images
Distributed testing with grid node and hub services
Requirements for developing and running tests are:
Java JDK 1.6 or newer
Browsers installed on test nodes as supported by Selenium WebDriver
Mozilla Firefox (ESR version recommended)
Mobile browsers: Android, iPhone
A build system, such as Ant or Maven, to automate execution of tests during build process (recommended)
Note that running tests on an Extended Support Release (ESR) version of Firefox is recommended because of the frequent release cycle of Firefox, which often cause tests to fail. Download an ESR release of Firefox from http://www.mozilla.org/en-US/firefox/organizations/all.html. Install it alongside your normal Firefox install (do not overwrite).
For Mac OS X, note the issue mentioned in "Running Firefox Tests on Mac OS X".
Continuous integration means automatic compilation and testing of applications frequently, typically at least daily, but ideally every time when code changes are committed to the source repository. This practice allows catching integration problems early and finding the changes that first caused them to occur.
You can make unit tests with Vaadin TestBench just like you would do any other Java unit tests, so they work seamlessly with continuous integration systems. Vaadin TestBench is tested to work with at least TeamCity and Hudson/Jenkins build management and continuous integration servers, which all have special support for the JUnit unit testing framework.
Continuous Integration Workflow illustrates a typical development setup. Both changes to application and test sources are checked in into a source repository, from where the CIS server checks them out, compiles, and deploys the web application to a server. Then, it runs the tests and collects the results.
You can download Vaadin TestBench from Vaadin Directory and try it out for a free 30-day trial period, after which you are required to acquire the needed licenses. You can purchase licenses from the Directory. A license for Vaadin TestBench is also included in the Vaadin Pro Account subscription.