Testbench Headless Tests

Flow
1

I want to create some basic headless Login View E2E tests with TestBench for Vaadin24 according the tutorial in https://vaadin.com/docs/latest/tutorial/e2e-testing-with-testbench#test-the-login-view. Nevertheless, the Firefox window is always shown, whether with firefoxOptions.addArguments("-headless"); or without. I use selenium firefox driver 4.14 and latest geckodriver 0.33.
With the headless argument, the window is show but without any html content, only the menu, navigation elements etc. are shown, but no content. Without the headless argument, everything is shown as expected. I’ve tried the https://github.com/vaadin/testbench-demo as well with same result: if I enable headless mode in AbstractIT.java, in this case the Chrome window is shown, no headless mode is available.

How can I enable true headless mode for E2E tests ?

2

This works for me ColorPicker/src/test/java/org/vaadin/addons/tatu/FirefoxTest.java at master · TatuLund/ColorPicker · GitHub

3

Hi @yummy-rhino thank you for your respnse. Unfortunately, the method options.setHeadless(true);is removed from the API in the version, Testbanch/Selenium for Vaadin 24 is using. Due to https://www.selenium.dev/blog/2023/headless-is-going-away/ one should use options.addArguments("-headless") but this doesn’t work.

4

I’m using the latest of everything (including chrome 119) and ChromeOptions::addArguments(“headless”) works in our docker container - which normally have no UI :sweat_smile:

5

firefoxOptions.addArguments(“-headless”)
is the single dash intended or a typo?

6

Looking at firefox --help output it should have two dashes

--headless         Run without a GUI.
7

Sorry, but this doesn’t work, neither with a clean skeleton project from vaadin starter. To reproduce the behaviour:

  1. go to start.vaadin.com and load the pre-defined Hello World Project with VaadinFlow 24, Java17, Maven + Spring Boot (3.1.5)

  2. this skeleton project already contains an E2E test MainViewIT.java.

  3. start the Application.java, then start the test (either in your IDE or via mvn clean integration-test -DMainViewIT -Pit => browser windows (Chrome per default) is opened and you see the view provided by the application, test succeeds :white_check_mark:

  4. edit the MainViewIT.javaand switch to headless mode by adding the following in in method public void open as mentioned in the Vaadin docs https://vaadin.com/docs/latest/testing/end-to-end/ci-server

ChromeOptions options = new ChromeOptions();
options.addArguments("--headless", "--disable-gpu");
setDriver(new ChromeDriver(options));
  1. run the test again :zap:

On my machine ( MacStudio with Apple Silicon (Apple M1 Max, MacOS Ventura 13.5.1 (22G90), Java 17 (Eclipse Temurin 17.0.8 via SDKMAN), Maven 3.9.4) I’ve got the following

  • browser windows is opened
  • no content within (as compared to the main view shown in non-headless mode)
  • test succeeds

The same behavoiur with firefox.
There are no webdrivers (neither geckodriver nor chromedrive) in my path.

8

What if you drop the "--disable-gpu" flag?

9

Same result. As I can see, selenium is using a chromedriver version 119.0.6045.105 automatically placed in ~/.cache/selenium

10

To my understanding, that shouldn’t be a problem. Modern Chrome versions come with the headless functionality built-in.

11

can you reproduce the issue with my step-by-step-list ?

12

Don’t have the same hardware as you.

13

Hopefully that should not depend on the type of the silicon :slightly_smiling_face:

14

Wouldn’t be the first time.

15

(I’m a Windows user myself)

16

Can you reproduce it with your setup ?

17

Seems to work ok for me.

18

really … :see_no_evil: … I try to reproduce it with another, older, Mac machine … I assume its OS specific.

19

My colleague has reproduced the issue on windows, same behaviour as on my MacOS machine.

20

I suppose the next logical step would be to create a ticket at Sign in to GitHub · GitHub