Making Tests Reliable

Creating Readable Tests
Guarding Against Application Changes
Dealing with Test Environment Problems

There are different types of problems which can cause problems in your tests:

Tests are not understood by other developers/testers and are disabled or accidentally broken
Changes in the application causes tests to fail
Problems in the testing environment causes tests to fail

You need to take these into account or will quickly end up with a test suite where a few test always fail. As experienced testers can tell you, this test suite is as good as having no tests at all, because a test suite which is always "a bit red" is not taken seriously by any developer.

Tip	You should make sure that your test suite is run on a regular basis. Having a manually triggered test suite which is run only after a lot of changes has been done to the application makes maintenance extermely difficult. The best approach is if you can run the test suite on every change.

Creating Readable Tests

Just as with code, it is important to write tests so that the reader understands the intent. When each test contains high level, meaningful calls, the reader will immediately grasp what is being tested. When/if she wants to know more details about some part of the test, she can then dig into that part. If the test is full of low level details about how you locate the parts of the application you want to interact with, it becomes completely overwhelming to try to decode what the test is actually trying to verify.

By using page/view objects you can abstract away the low level details about how the view is built and what exact components are used. You can also use BDD to describe your test scenarios using normal English sentences.

Guarding Against Application Changes

If your application never changes, you can test it manually just once and you will know that it works properly. In most cases though, your application will be developed forward and you need to maintain the tests when the application evolves.

As long as you abstract away the details from the tests to page/view objects, you only need to take care that your page/view objects are built in a robust way.

You should avoid by all means necessary to depend on the HTML DOM structure. If you depend on finding a <div> inside a <span> or anything similar, you will have to update the page/view object for every small detail that changes in the application.

Similarly, you should avoid depending on strings targeted for humans in your application. While it is in many cases tempting to find the button with the text "Save", you will run into unnecessary problems when somebody decides to change the text to "Store", or decides to internationalize the application.

Define Ids for the Components

For most cases, it makes sense to define ids for all the elements you want to interact with inside your page/view object. The ids are only created to be able to identify a given element and there is typically no reason to change them when the application evolves.

When using templates, you also do not need to worry about global ids and ids colliding with each other, as the id of a given element only needs to be unique inside the shadow root, i.e. the template. For layouts and components outside templates (an inside a single template), you should take care that you do not use the same id in multiple places.

Tip	Use `ids` which describe the action which will occur when pressing the button, not `ids` describing e.g. where in the hierarchy the button is. If your `id` is tied to the hierarchy, you will indirectly depend on the hierarchy and lose many benefits of using `ids`.

Dealing with Test Environment Problems

When dealing with browser based tests, and especially older browser such as IE11, you need to take into account that the environment is not always as stable as you would want it to be. Ideally the test would fire up the browser, execute the actions and terminate the browser nicely. Always. In practice, there is potential to have network problems (especially when using a cloud based browser provider), there can be browser problems causing randomness or even browser crashes (yes, this is about you IE11).

When the point of failure is outside your control, e.g. a temporary network failure, your options are very limited. To deal with all kinds of unexpected randomness, in the network or the browsers, TestBench offers a RetryRule, which is simply a way to automatically run the test again to see if the temporary problem has disappeared.

RetryRule is used as a JUnit 4 @Rule, with an parameter describing the maximum number of times the test should be run, e.g:

public class RandomFailureTest extends TestBenchTestCase {

    // Run the test max two times
    @Rule
    public RetryRule rule = new RetryRule(2);

    @Test
    public void doStuff() {
      ...
    }

}

If the test passes on the first attempt, it will not be re-run. Only if the first attempt fails, it will try again until either the test passes or the maximum number of attempst has been reached.

Note	RetryRule affects all the test methods in the class and also child classes.

Note	The default value of maxAttempts is 1, meaning that test is run only once. You can change the value of maxAttempts globally using the Java system property: `-Dcom.vaadin.testbench.Parameters.maxAttempts=2`.

Note	Use RetryRule when you are sure that the test fails because of the problems with the Web Driver, but not your application. Using RetryRule without cautions may hide random problems happening in your application.

D47AD6C3-C848-4ACC-92CC-8E36D6F20A91

Docs