TestBench / Selenium Failing on Linux

peter.211 · January 20, 2025, 6:57pm

I have a reproducible issue where a test using Selenium works correctly when using Selenium natively. However, when using TestBench the same test fails.

I have uploaded a simple project to https://github.com/dolukhanov/vaadin-testbench-selenium-issue to demonstrate the issue.

Essentially, running the following code works fine on Amazon Linux:

@SpringBootTest
public class GooglePageTestPass
{

    private WebDriver driver;

    @BeforeEach
    public void setup()
    {
        // Headless
        ChromeOptions options = new ChromeOptions();
        options.addArguments( "--headless" );
        driver = new ChromeDriver( options );
    }

    @Test
    public void testGooglePageLoad()
    {
        // Navigate to Google
        driver.get( "https://www.google.com" );

        // Verify the title contains "Google"
        assertTrue( driver.getTitle().contains( "Google" ), "Page title should contain 'Google'" );

        // Verify the current URL is Google
        assertTrue( driver.getCurrentUrl().contains( "google.com" ), "URL should contain 'google.com'" );
    }

    @AfterEach
    public void tearDown()
    {
        if ( driver != null )
        {
            driver.quit();
        }
    }
}

However, if I make this class extend BrowserTestBase and use @BrowserTest instead of @Test, the test fails.

Code:

@SpringBootTest
public class GooglePageTestFail extends BrowserTestBase
{

    private WebDriver driver;

    @BeforeEach
    public void setup()
    {
        // Headless
        ChromeOptions options = new ChromeOptions();
        options.addArguments( "--headless" );
        driver = new ChromeDriver( options );
    }

    //    @Test
    @BrowserTest
    public void testGooglePageLoad()
    {
        // Navigate to Google
        driver.get( "https://www.google.com" );

        // Verify the title contains "Google"
        assertTrue( driver.getTitle().contains( "Google" ), "Page title should contain 'Google'" );

        // Verify the current URL is Google
        assertTrue( driver.getCurrentUrl().contains( "google.com" ), "URL should contain 'google.com'" );
    }

    @AfterEach
    public void tearDown()
    {
        if ( driver != null )
        {
            driver.quit();
        }
    }
}

Here are the passing logs:

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.485 s -- in com.example.seleniumtest.GooglePageTestPass
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  7.090 s
[INFO] Finished at: 2025-01-20T18:48:01Z
[INFO] ------------------------------------------------------------------------

Here are the failing logs:

2025-01-20T18:48:10.992Z  INFO 377940 --- [Pool-1-worker-2] c.v.testbench.browser.BrowserExtension   : Did not find a configuration to run locally, on Sauce Labs or on other test grid. Falling back to running locally on Chrome.
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.629 s <<< FAILURE! -- in com.example.seleniumtest.GooglePageTestFail
[ERROR] com.example.seleniumtest.GooglePageTestFail.testGooglePageLoad()[1] -- Time elapsed: 1.332 s <<< ERROR!
org.openqa.selenium.SessionNotCreatedException: 
Could not start a new session. Response code 500. Message: session not created: Chrome failed to start: exited normally.
  (session not created: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/local/bin/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) 
Host info: host: 'ip-172-31-29-253.us-west-2.compute.internal', ip: '172.31.29.253'
Build info: version: '4.25.0', revision: '8a8aea2337'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.1.119-129.201.amzn2023.x86_64', java.version: '21.0.5'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--test-type , --disable-search-engine-cho...], binary: /usr/local/bin/chrome, extensions: []}}]}]
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:114)
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:75)
	at org.openqa.selenium.remote.ProtocolHandshake.createSession(ProtocolHandshake.java:61)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:162)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:216)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:174)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:545)
	at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:245)
	at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:174)
	at org.openqa.selenium.chromium.ChromiumDriver.<init>(ChromiumDriver.java:114)
	at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:88)
	at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:83)
	at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:72)
	at com.vaadin.testbench.parallel.setup.LocalDriver.createDriver(LocalDriver.java:87)
	at com.vaadin.testbench.parallel.setup.SetupDriver.setupLocalDriver(SetupDriver.java:101)
	at com.vaadin.testbench.parallel.setup.SetupDriver.setupLocalDriver(SetupDriver.java:116)
	at com.vaadin.testbench.browser.BrowserExtension.createDriver(BrowserExtension.java:244)
	at com.vaadin.testbench.browser.BrowserExtension.setupDriver(BrowserExtension.java:210)
	at com.vaadin.testbench.browser.BrowserExtension.beforeEach(BrowserExtension.java:200)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   GooglePageTestFail.testGooglePageLoad()[1] » SessionNotCreated Could not start a new session. Response code 500. Message: session not created: Chrome failed to start: exited normally.
  (session not created: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/local/bin/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.) 
Host info: host: 'ip-172-31-29-253.us-west-2.compute.internal', ip: '172.31.29.253'
Build info: version: '4.25.0', revision: '8a8aea2337'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '6.1.119-129.201.amzn2023.x86_64', java.version: '21.0.5'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [null, newSession {capabilities=[Capabilities {browserName: chrome, goog:chromeOptions: {args: [--test-type , --disable-search-engine-cho...], binary: /usr/local/bin/chrome, extensions: []}}]}]
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

it would appear c.v.testbench.browser.BrowserExtension is doing something unusual to launch the ChromeDriver / Chrome to test - I haven’t been able to find any documentation on any settings I may need to use to get this to work.

marcoc_753 · January 20, 2025, 7:59pm

By default, Testbench does not start the browser in headless mode.
Try to set the -Dcom.vaadin.testbench.Parameters.headless=true system parameter.

Another option is to make you test class implement DriverSupplier and provide your own WebDriver instance.

BTW, if you extend BrowserTestBase you do not need to set up the driver on your own. The extension will do it for you, and you can access the driver calling getDriver()

peter.211 · January 20, 2025, 8:18pm

Debugging further - it appears when you extend BrowserTestBase - there is some initialization happening with the constructor of that class.

Specifically, it will retrieve the Parameters.getHeadless() value to determine what to pass to ChromeOptions before initializing a new ChromeDriver.

The only solution I could get to work is adding a static block at the top of each test:

static
    {
        Parameters.setHeadless( true );
    }

Is there a more elegant solution to achieve this?

peter.211 · January 20, 2025, 8:43pm

Thank you!

I was initially trying setDriver - but as the test was failing before that code, I simplified it as much as possible.

Extending DriverSupplier has fixed the issue for me.

knoobie · January 21, 2025, 6:04am

That’s a really weird default for a test tool that is suppose to run in CI :( this costed me literally days to debug in the past.

ollit.1 · January 21, 2025, 6:58am

I’m guessing it’s because Testbench predates headless browsers by a good margin and nobody thought to change that. Worth a ticket, IMO.

Tatu2 · January 21, 2025, 1:09pm

Check here ChromeOptions for headless mode

github.com

TatuLund/bean-table/blob/v24/src/test/java/org/vaadin/tatu/AbstractViewTest.java#L108


      
                  rootUrl += Arrays.stream(parameters)
                          .collect(Collectors.joining("&"));
              }
          
              return rootUrl;
          }
          
          @Before
          public void setup() throws Exception {
              ChromeOptions options = new ChromeOptions();
              options.addArguments("--headless=new");
              setDriver(TestBench.createDriver(new ChromeDriver(options)));
              getDriver().get(getURL(route));
          
              // We do screenshot testing, adjust settings to ensure less flakiness
              Parameters.setScreenshotComparisonTolerance(0.1);
              Parameters.setScreenshotComparisonCursorDetection(true);
              testBench().resizeViewPortTo(1280, 900);
              Parameters.setMaxScreenshotRetries(3);
              Parameters.setScreenshotRetryDelay(1000);

If you use Linux you may want to do some other settings as well

See example here my settings when I am running tests in GitHub actions / Ubuntu

github.com

TatuLund/vaadin-create23/blob/master/vaadincreate-ui/src/test/java/org/vaadin/tatu/vaadincreate/AbstractViewTest.java#L135


      
              }
          
              return rootUrl;
          }
          
          @Before
          public void setup() {
              ChromeOptions options = new ChromeOptions();
              options.addArguments("--headless=new");
              options.addArguments("--window-size=1280,900");
              if (Boolean.getBoolean("ghActions")) {
                  options.addArguments("--no-sandbox");
                  options.addArguments("--disable-dev-shm-usage");
                  options.addArguments("--disable-gpu");
                  options.addArguments("--disable-extensions");
              }
              setDriver(TestBench.createDriver(new ChromeDriver(options)));
              getDriver().get(getURL(urlFragment));
          
              // We do screenshot testing, adjust settings to ensure less flakiness
              Parameters.setScreenshotComparisonTolerance(0.1);

peter.211 · January 21, 2025, 3:58pm

Thank you. This is what I had initially - but without DriverSupplier - TestBench was initializing ChromeDriver before the tests without headless mode and failing.