Vaadin 8 TestBench

Vaadin TestBench can work as the centerpiece of the software development process, for testing the application at all modular levels and in all the phases of the development cycle:

Let us look at each of these topics separately.

Any methodological software development, agile or not, is preceded by specification of requirements, which define what the software should actually do. Acceptance tests ensure that the product conforms to the requirements. In agile development, their automation allows continuous tracking of progress towards iteration goals, as well as detecting regressions. The importance of requirements is emphasized in test-driven development (TDD), where tests are written before actual code. In "Behaviour-Driven Development", we show how to use Vaadin TestBench for behaviour-driven development (BDD), a form of TDD that concentrates on the formal behavioural specification of requirements.

Unit testing is applied to the smallest scale of software components; in Vaadin applications these are typically individual UI components or view classes. You may also want to generate many different kinds of inputs for the application and check that they produce the desired outputs. For complex composites, such as views, you can use the Page Object Pattern described in "The Page Object Pattern". The pattern simplifies and modularizes testing by separating low-level details from the more abstract UI logic. In addition to serving the purpose of unit tests, it creates an abstraction layer for higher-level tests, such as acceptance and end-to-end tests.

Integration tests ensure that software units work together at different levels of modularization. At the broadest level, end-to-end tests extend through the entire application lifecycle from start to finish, going through many or all user stories. The aim is not just to verify the functional requirements for user interaction, but also that data integrity is maintained. For example, in a messaging application, a user would log in, both send and receive messages, and finally log out. Such test workflows could include configuration, registration, interaction between users, administrative tasks, deletion of user accounts, and so forth.

In regression testing, you want to ensure that only intended changes occur in the behaviour after modifying the code. There are two lines of defence against such regressions. The primary source of regression tests are the acceptance, unit, and integration tests that validate that the displayed values in the UI’s HTML representation are logically correct. Yet, they are not sufficient for detecting visual regressions, for example, because of invalid UI rendering or theme problems. Comparing screenshots to reference images forms a more sensitive layer to detect regressions, at the expense of losing robustness for changes in layout and themeing. The costs of the tradeoff can be minimized by careful application of screenshot comparison only at critical points and by making the analysis of such regressions as easy as possible. As described in "Taking and Comparing Screenshots", Vaadin TestBench automatically highlights differences in screenshots and allows masking irrelevant areas from image comparison.

You can develop such tests along with your application code, for example with JUnit, which is a widely used Java unit testing framework. You can run the tests as many times as you want in your workstation or in a distributed grid setup.

The main features of Vaadin TestBench are:

Execution of tests can be distributed over a grid of test nodes, which speeds up testing. The grid nodes can run different operating systems and have different browsers installed. In a minimal setup, such as for developing the tests, you can use Vaadin TestBench on just a single computer.

Vaadin TestBench is based on the Selenium web browser automation library, especially Selenium WebDriver, which allows you to control browsers straight from Java code.

Selenium is augmented with Vaadin-specific extensions, such as:

The TestBench library includes WebDriver, which provides API to control a browser like a user would. This API can be used to build tests, for example, with JUnit. It also includes the grid hub and node servers, which you can use to run tests in a grid configuration.

Vaadin TestBench Library provides the central control logic for:

Requirements for developing and running tests are:

Note that running tests on an Extended Support Release (ESR) version of Firefox is recommended because of the frequent release cycle of Firefox, which often cause tests to fail. Download an ESR release of Firefox from https://www.mozilla.org/en-US/firefox/organizations/all.html. Install it alongside your normal Firefox install (do not overwrite).

For Mac OS X, note the issue mentioned in "Running Firefox Tests on Mac OS X".

Continuous integration means automatic compilation and testing of applications frequently, typically at least daily, but ideally every time when code changes are committed to the source repository. This practice allows catching integration problems early and finding the changes that first caused them to occur.

You can make unit tests with Vaadin TestBench just like you would do any other Java unit tests, so they work seamlessly with continuous integration systems. Vaadin TestBench is tested to work with at least TeamCity and Hudson/Jenkins build management and continuous integration servers, which all have special support for the JUnit unit testing framework.

Continuous Integration Workflow illustrates a typical development setup. Both changes to application and test sources are checked in into a source repository, from where the CIS server checks them out, compiles, and deploys the web application to a server. Then, it runs the tests and collects the results.

Vaadin TestBench is a commercial product and part of the Pro subscription. A free 14-day trial is available.