Documentation

Documentation versions (currently viewingVaadin 23)
Check out the new styling guides

Making Tests Reliable

There are different types of issues that can cause problems in your tests:

  • Tests are not understood by other developers/testers and are disabled or accidentally broken

  • Changes in the application cause tests to fail

  • Problems in the testing environment cause tests to fail

You need to take these into account or you will quickly end up with a test suite where a few tests always fail. As experienced testers can tell you, this test suite is as good as having no tests at all, because a test suite which is always "a bit red" is not taken seriously by any developer.

Tip
You should make sure that your test suite is run on a regular basis. Having a manually triggered test suite which is run only after a lot of changes have been made to the application makes maintenance extremely difficult. The best approach is to run the test suite on every change.

Creating Readable Tests

Just as with code, it is important to write tests so that the reader understands the intent. When each test contains high-level, meaningful calls, the reader will immediately grasp what is being tested. If they want to know more details about some part of the test, they can then dig into that part. If the test is full of low-level details about how you locate the parts of the application you want to interact with, it becomes completely overwhelming to try to decode what the test is actually trying to verify.

By using page/view objects you can abstract away the low-level details about how the view is built and what exact components are used. You can also use BDD to describe your test scenarios using normal English sentences.

Guarding Against Application Changes

If your application never changes, you can test it manually just once and you will know that it works properly. However, in most cases, your application will be developed forward and you need to maintain the tests as the application evolves.

As long as you abstract away the details from the tests to page/view objects, you only need to take care that your page/view objects are built in a robust way.

You should take all the steps necessary to avoid depending on the HTML DOM structure. If you depend on finding a <div> inside a <span> or anything similar, you will have to update the page/view object for every small detail that changes in the application.

Similarly, you should avoid depending on strings targeted for humans in your application. Although it is often tempting to find the button with the text "Save", you will run into unnecessary problems when somebody decides to change the text to "Store", or to internationalize the application.

Define Ids for the Components

For most cases, it makes sense to define ids for all the elements you want to interact with inside your page/view object. The ids are created just to be able to identify a given element and there is typically no reason to change them when the application evolves.

When using templates, you also do not need to worry about global ids and ids colliding with each other, as the id of a given element only needs to be unique inside the shadow root, that is, the template. For layouts and components outside templates (and inside a single template), you should take care that you do not use the same id in multiple places.

Tip
Use ids which describe the action that occurs when the button is pressed, not, for example, where in the hierarchy the button is. If your id is tied to the hierarchy, you will indirectly depend on the hierarchy and lose many benefits of using ids.

Dealing With Test Environment Problems

When dealing with browser-based tests, especially with older browser such as IE11, you need to take into account that the environment is not always as stable as you would want it to be. Ideally, the test would fire up the browser, execute the actions and terminate the browser nicely – always. In practice, there is potential for network problems (especially when using a cloud-based browser provider). There can also be browser problems causing randomness, or even browser crashes (yes, this is about you, IE11).

When the point of failure is outside your control, for example a temporary network failure, your options are very limited. To deal with all kinds of unexpected randomness, in the network or the browsers, TestBench offers a RetryRule, which is simply a way to automatically run the test again to see if the temporary problem has disappeared.

RetryRule is used as a JUnit 4 @Rule, with an parameter describing the maximum number of times the test should be run. For example:

public class RandomFailureTest extends TestBenchTestCase {

    // Run the test max two times
    @Rule
    public RetryRule rule = new RetryRule(2);

    @Test
    public void doStuff() {
      ...
    }

}

If the test passes on the first attempt, it will not be rerun. Only if the first attempt fails will it try again until either the test passes or the maximum number of attempts has been reached.

Note
RetryRule affects all the test methods in the class and also child classes.
Note
The default value of maxAttempts is 1, meaning that test is run only once. You can change the value of maxAttempts globally using the Java system property -Dcom.vaadin.testbench.Parameters.maxAttempts=2.
Note
Use RetryRule when you are sure that the test fails because of problems with the Web Driver, but not your application. Using RetryRule without caution may hide random problems happening in your application.

872ED8E3-65E6-4926-A5AB-E1CEA77B9A69