Tests are developed using the Selenium WebDriver, which is augmented with Vaadin TestBench API features useful for testing Vaadin applications.

Perhaps the easiest way to start developing tests is to use the Recorder to create a JUnit test stub, which is described in the next section. The main purpose of the recorder is to help identify the HTML DOM paths of the user interface elements that you want to interact with and use for assertions. Once you get the hang of coding tests, you should be able to do it without using the Recorder. Working with debug IDs and using a browser debugger, such as Firebug, is usually the easiest way to find out the DOM paths. You can also use the Recorder just to find the paths, and copy and paste them directly to your source code without going through the export hassle.

While this section describes the development of JUnit tests, Vaadin TestBench and the WebDriver are in no way specific to JUnit and you can use any test execution framework, or just regular Java applications, to develop TestBench tests.

Let us assume that you recorded a simple application, as described earlier, and exported it as a JUnit stub. You can add it to a project in a suitable package. You may want to keep your test classes in a separate source tree in your application project, or in an altogether separate project, so that you do not have to include them in the web application WAR. Having them in the same project may be nicer for version control purposes.

You need to perform at least the following routine tasks:

  • Rename the package
  • Rename the class
  • Check the base URL
  • Clean up unnecessary code

A JUnit stub will look somewhat as follows:

package com.example.tests;

import java.util.regex.Pattern;
import java.util.concurrent.TimeUnit;
import org.junit.*;
import static org.junit.Assert.*;
import static org.hamcrest.CoreMatchers.*;
import org.openqa.selenium.*;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.support.ui.Select;
import com.vaadin.testbench.By;
import com.vaadin.testbench.TestBench;
import com.vaadin.testbench.TestBenchTestCase;

public class Testcase1 extends TestBenchTestCase {
    private WebDriver driver;
    private String baseUrl;
    private StringBuffer verificationErrors = new StringBuffer();
    ...

The verificationErrors is used to collect some errors in some recorded commands, but can be removed if such commands are not used. You can also use it to collect non-fatal errors, for example screenshot comparison errors, and only fail on logic errors.

The Selenium WebDriver API provides a number of different selectors for finding HTML DOM elements. The available selectors are defined as static methods in the org.openqa.selenium.By class. They create and return a By instance, which you can use for the findElement() method in WebDriver.

The ID, CSS class, and Vaadin selectors are described below. For others, we refer to the Selenium WebDriver API documentation.

Selecting elements by their HTML element id attribute is usually the easiest way to select elements. It requires that you use debug IDs, as described in Section 20.3, “Preparing an Application for Testing”. The debug ID is used as is for the id attribute of the top element of the component. Selecting is done by the By.id() selector.

For example, in the SimpleCalculatorITCase.java example we use the debug ID as follows to click on the calculator buttons:

@Test
public void testOnePlusTwo() throws Exception {
    openCalculator();

    // Click the buttons in the user interface
    getDriver().findElement(By.id("button_1")).click();
    getDriver().findElement(By.id("button_+")).click();
    getDriver().findElement(By.id("button_2")).click();
    getDriver().findElement(By.id("button_=")).click();

    // Get the result label value
    assertEquals("3.0", getDriver().findElement(
            By.id("display")).getText());
}

The ID selectors are used extensively in the TestBench examples.

In addition to the Selenium selectors, Vaadin TestBench provides a Vaadin selector, which allows pointing to a Vaadin component by its layout path. The JUnit test cases saved from the Recorder use Vaadin selectors by default.

You can create a Vaadin selector with the By.vaadin() method. You need to use the Vaadin By, defined in the com.vaadin.testbench package, which extends the Selenium By.

The other way is to use the findElementByVaadinSelector() method in the TestBenchCommands interface. It returns the WebElement object.

A Vaadin selector begins with an application identifier. It is the path to application without any slashes or other special characters. For example, /book-examples/tobetested would be bookexamplestobetested. After the identifier, comes two colons "::", followed by a slash-delimited component path to the component to be selected. The elements in the component path are client-side classes of the Vaadin user interfacer components. For example, the server-side VerticalLayout component has VVerticalLayout client-side counterpart. All path elements except the leaves are component containers, usually layouts. The exact contained component is identified by its index in brackets.

A reference to a debug ID is given with a PID_S suffix to the debug ID.

For example:

// Get the button's element.
// Use the debug ID given with setDebugId().
WebElement button = driver.findElement(By.vaadin(
    "bookexamplestobetested::PID_Smain.button"));

// Get the caption text
assertEquals("Push Me!", button.getText());

// And click it
button.click();

// Get the Label's element by full path
WebElement label = driver.findElement(By.vaadin(
    "bookexamplestobetested::/VVerticalLayout[0]/"+
    "ChildComponentContainer[1]/VLabel[0]"));

// Make the assertion
assertEquals("Thanks!", label.getText());

Executing JUnit tests with Vaadin TestBench under Maven requires installing the TestBench library in the local Maven repository and defining it as a dependency in any POM that needs to execute TestBench tests.

A complete example of a Maven test setup is given in the example/maven folder in the installation package. Please see the README file in the folder for further instructions.

Vaadin TestBench uses Selenium WebDriver to execute tests in a browser. The WebDriver instance is created with the static createDriver() method in the TestBench class. It takes the driver as the parameter and returns it after registering it. The test cases must extend the TestBenchTestCase class, which manages the TestBench-specific features.

The basic way is to create the driver in a method annotated with the JUnit @Before annotation and close it in a method annotated with @After.

public class AdvancedTest extends TestBenchTestCase {
    private WebDriver driver;

    @Before
    public void setUp() throws Exception {
        ...
        driver = TestBench.createDriver(new FirefoxDriver());
    }
    ...
    @After
    public void tearDown() throws Exception {
        driver.quit();
    }
}

This creates the driver for each test you have in the test class, causing a new browser instance to be opened and closed. If you want to keep the browser open between the test, you can use @BeforeClass and @AfterClass methods to create and quit the driver. In that case, the methods as well as the driver instance have to be static.

public class AdvancedTest extends TestBenchTestCase {
    static private WebDriver driver;

    @BeforeClass
    static public void createDriver() throws Exception {
        driver = TestBench.createDriver(new FirefoxDriver());
    }
    ...
    @AfterClass
    static public void tearDown() throws Exception {
        driver.quit();
    }
}

Please see the API documentation of the WebDriver interface for a complete list of supported drivers, that is, classes implementing the interface.

Both the Internet Explorer and Chrome require a special driver, as was noted in Section 20.2.7, “Installing Browser Drivers”. The driver executable must be included in the operating system PATH or be given with a driver-specific system property in Java with: System.setProperty(prop, key)).

  • Chrome: webdriver.chrome.driver
  • IE: webdriver.ie.driver

If you use the Firefox 10.x ESR version, which is recommended because of test stability, you need to the binary when creating the driver as follows:

FirefoxBinary binary =
    new FirefoxBinary(new File("/path/to/firefox_ESR_10"));
driver = TestBench.createDriver(
    new FirefoxDriver(binary, new FirefoxProfile()));

A typical test case does the following:

  1. Open the URL
  2. Navigate to desired state
    1. Find a HTML element (WebElement) for navigation
    2. Use click() and other commands to interact with the element
    3. Repeat with different elements until desired state is reached
  3. Find a HTML element (WebElement) to check
  4. Get and assert the value of the HTML element
  5. Get a screenshot

The WebDriver allows finding HTML elements in a page in various ways, for example, with XPath expressions. The access methods are defined statically in the By class.

These tasks are realized in the following test code:

@Test
public void testCase1() throws Exception {
    driver.get(baseUrl + "/book-examples/tobetested");
    
    // Get the button's element.
    // (Actually the caption element inside the button.)
    // Use the debug ID given with setDebugId().
    WebElement button = driver.findElement(By.xpath(
        "//div[@id='main.button']/span/span"));
    
    // Get the caption text
    assertEquals("Push Me!", button.getText());
    
    // And click it. It's OK to click the caption element.
    button.click();
    
    // Get the Label's element.
    // Use the automatically generated ID.
    WebElement label = driver.findElement(By.xpath(
        "//div[@id='myapp-949693921']" +
        "/div/div[2]/div/div[2]/div/div"));

    // Make the assertion
    assertEquals("Thanks!", label.getText());
}

You can also use URI fragments in the URL to open the application at a specific state. For information about URI fragments, see Section 12.10, “URI Fragment and History Management with UriFragmentUtility.

You should use the JUnit assertion commands. They are static methods defined in the org.junit.Assert class, which you can import (for example) with:

import static org.junit.Assert.assertEquals;

Please see the Selenium API documentation for a complete reference of the element search methods in the WebDriver and By classes and for the interaction commands in the WebElement class.

TestBench has a collection of its own commands, defined in the TestBenchCommands interface. You can get a command object that you can use by calling testBench(driver) in a test case.

It is not just that it works, but also how long it takes. Profiling test execution times consistently is not trivial, as a test environment can have different kinds of latency and interference. For example in a distributed setup, timings taken on the test server would include the latencies between the test server, the grid hub, a grid node running the browser, and the web server running the application. In such a setup, you could also expect interference between multiple test nodes, which all might make requests to a shared application server and possibly also share virtual machine resources.

Furthermore, in Vaadin applications, there are two sides which need to be profiled: the server-side, on which the application logic is executed, and the client-side, where it is rendered in the browser. Vaadin TestBench includes methods for measuring execution time both on the server-side and the client-side.

The TestBenchCommands interface offers the following methods for profiling test execution time:

totalTimeSpentServicingRequests()

Returns the total time (in milliseconds) spent servicing requests in the application on the server-side. The timer starts when you first navigate to the application and hence start a new session. The time passes only when servicing requests for the particular session. The timer is shared in the servlet session, so if you have, for example, multiple portlets in the same application (session), their execution times will be included in the same total.

Notice that if you are also interested in the client-side performance for the last request, you must call the timeSpentRenderingLastRequest() before calling this method. This is due to the fact that this method makes an extra server request, which will cause an empty response to be rendered.

timeSpentServicingLastRequest()

Returns the time (in milliseconds) spent servicing the last request in the application on the server-side. Notice that not all user interaction through the WebDriver cause server requests.

As with the total above, if you are also interested in the client-side performance for the last request, you must call the timeSpentRenderingLastRequest() before calling this method.

totalTimeSpentRendering()

Returns the total time (in milliseconds) spent rendering the user interface of the application on the client-side, that is, in the browser. This time only passes when the browser is rendering after interacting with it through the WebDriver. The timer is shared in the servlet session, so if you have, for example, multiple portlets in the same application (session), their execution times will be included in the same total.

timeSpentRenderingLastRequest()

Returns the time (in milliseconds) spent rendering user interface of the application after the last server request. Notice that not all user interaction through the WebDriver cause server requests.

If you also call the timeSpentServicingLastRequest() or totalTimeSpentServicingRequests(), you should do so before calling this method. The methods cause a server request, which will zero the rendering time measured by this method.

Generally, only interaction with fields in the immediate mode cause server requests. This includes button clicks. Some components, such as Table, also cause requests otherwise, such as when loading data while scrolling. Some interaction could cause multiple requests, such as when images are loaded from the server as the result of user interaction.

The following example is given in the VerifyExecutionTimeITCase.java file under the TestBench examples.

@Test
public void verifyServerExecutionTime() throws Exception {
    openCalculator();

    // Get start time on the server-side
    long currentSessionTime = testBench(getDriver())
            .totalTimeSpentServicingRequests();

    // Interact with the application
    calculateOnePlusTwo();

    // Calculate the passed processing time on the serve-side
    long timeSpentByServerForSimpleCalculation = testBench()
        .totalTimeSpentServicingRequests() - currentSessionTime;

    // Report the timing
    System.out.println("Calculating 1+2 took about "
            + timeSpentByServerForSimpleCalculation
            + "ms in servlets service method.");

    // Fail if the processing time was critically long
    if (timeSpentByServerForSimpleCalculation > 30) {
        fail("Simple calculation shouldn't take "
             + timeSpentByServerForSimpleCalculation + "ms!");
    }

    // Do the same with rendering time
    long totalTimeSpentRendering =
            testBench().totalTimeSpentRendering();
    System.out.println("Rendering UI took " +
            totalTimeSpentRendering + "ms");
    if (timeSpentByServerForSimpleCalculation > 400) {
        fail("Rendering UI shouldn't take "
             + timeSpentByServerForSimpleCalculation + "ms!");
    }

    // A regular assertion on the UI state
    assertEquals("3.0", getDriver().findElement(
                        By.id("display")).getText());
}