Documentation

Documentation versions (currently viewingVaadin 23)
New Acceleration Kits: Observability Kit, SSO Kit, and Swing Kit. Read the blog post.

Making Tests Reliable

Different types of issues can cause problems in your tests:

  • Tests aren’t understood by other developers/testers and are disabled or broken

  • Changes in the application cause tests to fail

  • Problems in the testing environment cause tests to fail

You need to take these into account or you quickly end up with a test suite where a few tests always fail. As experienced testers can tell you, this test suite is as good as having no tests at all, because a test suite which is always "a bit red" isn’t taken seriously by any developer.

Tip
You should make sure that your test suite is run on a regular basis. Having a manually triggered test suite which is run only after a lot of changes have been made to the application makes maintenance difficult. The best approach is to run the test suite on every change.

Creating Readable Tests

As with any code, it’s important to write tests so that the reader understands the intent. When each test contains high-level, meaningful calls, the reader immediately grasps what’s being tested. If they want to know more details about some part of the test, they can then dig into that part. If the test is full of low-level details about how you locate the parts of the application you want to interact with, it becomes completely overwhelming to try to decode what the test is actually trying to verify.

By using page/view objects you can abstract away the low-level details about how the view is built and what exact components are used. You can also use Behavior-Driven-Development (BDD) to describe your test scenarios using normal English sentences.

Guarding against Application Changes

If your application never changes, you can test it manually once and you know that it works. However, usually, your application is developed forward and you need to maintain the tests as the application evolves.

As long as you abstract away the details from the tests to page/view objects, you only need to take care that your page/view objects are built in a robust way.

You should take all the steps necessary to avoid depending on the HTML DOM structure. If you depend on finding a <div> inside a <span> or anything similar, you have to update the page/view object for every small detail that changes in the application.

Similarly, you should avoid depending on strings targeted for humans in your application. Although it’s often tempting to find the button with the text "Save", you run into unnecessary problems when somebody decides to change the text to "Store", or to internationalize the application.

Define Ids for the Components

For most cases, it makes sense to define ids for all the elements you want to interact with inside your page/view object. The ids are created to be able to identify a given element and there is typically no reason to change them when the application evolves.

When using templates, you also don’t need to worry about global ids and ids colliding with each other, as the id of a given element only needs to be unique inside the shadow root, that is, the template. For layouts and components outside templates (and inside a single template), you should take care that you don’t use the same id in multiple places.

Tip
Use ids which describe the action that occurs when the button is pressed, not, for example, where in the hierarchy the button is. If your id is tied to the hierarchy, you indirectly depend on the hierarchy and lose many benefits of using ids.

Dealing with Test Environment Problems

When dealing with browser-based tests, especially with older browser such as IE11, you need to consider that the environment isn’t always as stable as you would want it to be. Ideally, the test would fire up the browser, execute the actions and close the browser nicely – always. In practice, there is potential for network problems (especially when using a cloud-based browser provider). There can also be browser problems causing randomness, or even browser crashes (yes, this is about you, IE11).

When the point of failure is outside your control, for example a temporary network failure, your options are very limited. To deal with all kinds of unexpected randomness, in the network or the browsers, TestBench offers a RetryRule, which is a way to automatically run the test again to see if the temporary problem has disappeared.

RetryRule is used as a JUnit 4 @Rule, with an parameter describing the maximum number of times the test should be run. For example:

public class RandomFailureTest extends TestBenchTestCase {

    // Run the test max two times
    @Rule
    public RetryRule rule = new RetryRule(2);

    @Test
    public void doStuff() {
      ...
    }

}

If the test passes on the first try, it isn’t rerun. Only if the first try fails does it try again until either the test passes or the maximum number of attempts has been reached.

Note
RetryRule affects all the test methods in the class and also child classes.
Note
The default value of maxAttempts is 1, meaning that test is run only once. You can change the value of maxAttempts globally using the Java system property -Dcom.vaadin.testbench.Parameters.maxAttempts=2.
Note
Use RetryRule when you are sure that the test fails because of problems with the WebDriver, but not your application. Using RetryRule without caution may hide random problems happening in your application.

872ED8E3-65E6-4926-A5AB-E1CEA77B9A69