Documentation

Documentation versions (currently viewingVaadin 23)
New Acceleration Kits: Observability Kit, SSO Kit, and Swing Kit. Read the blog post.

Testing Vaadin Applications Using TestBench

TestBench allows you to create UI unit and end-to-end tests for your Vaadin applications.
Note
Commercial feature

A commercial Vaadin subscription is required to use TestBench in your project.

TestBench includes special support for other Vaadin products, making testing easy and robust compared to generic web testing solutions, such as Selenium.

TestBench provides two approaches – UI unit testing and end-to-end testing – to test your application, with different advantages. Although Selenium has certain limitations compared to TestBench end-to-end testing, you can also use it for testing your applications.

UI Unit Testing
UI unit testing removes the need to run both the browser and the servlet container, to test your Vaadin-based applications faster.
End-to-End Testing
End-to-end testing simulates a user of your application, performs the tasks specified using Java code, and verifies that the expected actions take place in the application.
Testing with Selenium
How to set up testing automation for a Vaadin application using Selenium.

Comparison

UI unit tests allow you to write user interface tests that don’t need a browser or a servlet container to run. This testing approach speeds up your test runs, making the results less flaky compared to end-to-end tests. End-to-end tests are closer to a real-world user experience and can simulate user actions that are impossible with UI unit tests.

Because UI unit tests are faster and more stable, they are suitable for development workflows where you need to write many tests, perhaps with every commit, and you want to run those tests often. This is the case, for example, with the Test-Driven Development (TDD) approach.

End-to-end tests are more suitable for testing the critical parts of your application, such as the login and the checkout process, to simulate your users' real-life experience with your application.

Advantages of UI Unit Testing

Advantages of UI unit testing over end-to-end testing:

  • Fast: Browser-less tests are typically 100× faster than end-to-end tests and run in 5–60 milliseconds, depending on their complexity.

  • Reliable: No arbitrary sleeps are needed, as the test is executed on the server side and it can wait until the request is fully processed. No random failures because of incompatibility between the Selenium drivers used in end-to-end testing and the browser.

  • Headless: The tests run headless, as there is no browser. No need to set up the screen in your CI environment.

  • Robust: The test runs in the same JVM as the server-side components. If the server-side bootstrap fails and throws an exception, the test method fails with the same exception. No need to go hunting for exceptions in a log located somewhere on a CI server.

  • No need to write Page Objects: You are already on the server, and you have access to the actual Java components that are already providing you with high-level APIs, exactly as Page Objects do.

  • Full access to the database: You are on the server side, so you can access the database from your tests in the same way your business logic accesses the database. You can run a bunch of SQL statements to restore the database to a known state before every test. Even better, you can run the test in a transaction, then roll back after the test to perform a fast database revert to a known state.

With this technique, you can run 600 UI tests in 7 seconds, as opposed to 1 to 2 hours with the end-to-end testing approach. Because of the speed, you can let the UI tests run after every commit via your continuous integration server.

Bypassing the browser and talking to the Vaadin server API directly eliminates the need to start the servlet container. You can add the server JARs to the testing classpath and call the Vaadin server API, which, in turn, invokes your server logic.

Advantages of End-to-End Testing

Though UI unit testing is faster and less flaky, end-to-end testing is still sometimes needed.

Specifically, end-to-end testing is more suitable for testing the critical parts of your application, such as the login and the checkout process, to simulate your users' real-life experience with your application.

Moreover, end-to-end testing enables you to test the client-side functionality of your application, such as JavaScript code or the parts of the UI that are implemented using the Polymer or Lit libraries.

These tests are impossible with the UI unit testing approach because UI unit tests run on the server side without initializing the browser.

Differences between TestBench and Selenium

TestBench is itself based on Selenium. However, TestBench is specifically designed for testing Vaadin applications. Compared to Selenium, TestBench is designed to:

  1. ease the handling of Vaadin components. Unlike Selenium, which operates on the DOM of the web page and provides basic methods to interact with elements like click() or sendKeys(), TestBench provides helper methods to interact with Vaadin components, like select() or setValue(). Moreover, TestBench provides a more convenient way to interact with elements in the shadow DOM of Vaadin components.

  2. take the nature of client-server communication in Vaadin applications into account. During the course of a test, TestBench automatically suspends client-side execution if the server is busy and resume it when the server is ready. This makes it possible to write tests in TestBench without the need to explicitly set wait timeouts for various events like page load or waiting for long-running server calls to finish.

  3. an easy-to-use API for conducting visual regression tests (screenshot comparison).

  4. enable parallel testing of Vaadin applications.