Browserless and springless tests in 25.2

Flow
1

Is this a bug or did I miss something? I tried to follow instructions in Browserless testing in non-Spring Vaadin projects

In skeleton-starter-flow commit 545870138

diff --git a/pom.xml b/pom.xml
index 5cb5bcf..55afbc3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -15,8 +15,8 @@
         <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
         <failOnMissingWebXml>false</failOnMissingWebXml>
 
-        <vaadin.version>25.1.7</vaadin.version>
-        <jetty.version>12.1.2</jetty.version>
+        <vaadin.version>25.2.0-beta2</vaadin.version>
+        <jetty.version>12.1.10</jetty.version>
     </properties>
 
     <repositories>
@@ -108,6 +108,12 @@
             <artifactId>slf4j-simple</artifactId>
         </dependency>
 
+       <dependency>
+           <groupId>com.vaadin</groupId>
+           <artifactId>browserless-test-junit6</artifactId>
+           <scope>test</scope>
+       </dependency>
+
         <dependency>
             <groupId>com.vaadin</groupId>
             <artifactId>vaadin-testbench-junit6</artifactId>
diff --git a/src/test/java/org/vaadin/example/Test2.java b/src/test/java/org/vaadin/example/Test2.java
new file mode 100644
index 0000000..91ccf48
--- /dev/null
+++ b/src/test/java/org/vaadin/example/Test2.java
@@ -0,0 +1,18 @@
+package org.vaadin.example;
+
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+import org.junit.jupiter.api.Test;
+
+import com.vaadin.browserless.BrowserlessTest;
+import com.vaadin.flow.component.textfield.TextField;
+
+public class Test2 extends BrowserlessTest {
+       @Test
+       public void test2a() {
+               navigate(MainView.class);
+               var tf = find(TextField.class).single();
+               assertNotNull(tf);
+
+       }
+}

Running “mvn clean test” gives me

com.vaadin.browserless.BrowserlessTestSetupException: Test Vaadin environment is not initialized correctly. This may happen when the test is extending the wrong base class for the testing engine in use. Current test class is expected to run with JUnit 6.
        at com.vaadin.browserless.BaseBrowserlessTest.verifyAndGetUI(BaseBrowserlessTest.java:457)
        at com.vaadin.browserless.BaseBrowserlessTest.navigate(BaseBrowserlessTest.java:153)
        at org.vaadin.example.Test2.test2a(Test2.java:13)

Edit: I should of course be reading Browserless testing in non-Spring Vaadin projects but that seems to have essentially same instructions.

2

I cannot reproduce it. The code you posted works fine for me

3

I tried on another computer and created a new user. Still the same problem.

(Removed most ot the Download-lines)

Script started on 2026-06-15 06:52:06+03:00 [TERM="xterm" TTY="/dev/pts/12" COLUMNS="210" LINES="59"]
root@asdf:~# adduser test123 --disabled-password
Muutetaan käyttäjän test123 tietoja
Syötä uusi arvo tai paina ENTER jättääksesi oletuksen
        Koko nimi []: 
        Huonenumero []: 
        Työpuhelin []: 
        Kotipuhelin []: 
        Muu []: 
Is the information correct? [Y/n] y
root@asdf:~# su - test123
test123@asdf:~$ git clone --depth 1 https://github.com/vaadin/skeleton-starter-flow
Cloning into 'skeleton-starter-flow'...
remote: Enumerating objects: 42, done.
remote: Counting objects: 100% (42/42), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 42 (delta 1), reused 33 (delta 1), pack-reused 0 (from 0)
Receiving objects: 100% (42/42), 80.38 KiB | 1.41 MiB/s, done.
Resolving deltas: 100% (1/1), done.
test123@asdf:~$ gicd skeleton-starter-flow/
test123@asdf:~/skeleton-starter-flow$ git checkout 5458545870138
Note: switching to '545870138'.
 
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
 
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
 
  git switch -c <new-branch-name>
 
Or undo this operation with:
 
  git switch -
 
Turn off this advice by setting config variable advice.detachedHead to false
 
HEAD is now at 5458701 chore: Update Vaadin 25.1.7 (#1002)
test123@asdf:~/skeleton-starter-flow$ cat >browserless.diff
diff --git a/pom.xml b/pom.xml
index 5cb5bcf..55afbc3 100644
--- a/pom.xml
+++ b/pom.xml
@@ -15,8 +15,8 @@
         <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
         <failOnMissingWebXml>false</failOnMissingWebXml>
 
-        <vaadin.version>25.1.7</vaadin.version>
-        <jetty.version>12.1.2</jetty.version>
+        <vaadin.version>25.2.0-beta2</vaadin.version>
+        <jetty.version>12.1.10</jetty.version>
     </properties>
 
     <repositories>
@@ -108,6 +108,12 @@
             <artifactId>slf4j-simple</artifactId>
         </dependency>
 
+       <dependency>
+           <groupId>com.vaadin</groupId>
+           <artifactId>browserless-test-junit6</artifactId>
+           <scope>test</scope>
+       </dependency>
+
         <dependency>
             <groupId>com.vaadin</groupId>
             <artifactId>vaadin-testbench-junit6</artifactId>
diff --git a/src/test/java/org/vaadin/example/Test2.java b/src/test/java/org/vaadin/example/Test2.java
new file mode 100644
index 0000000..91ccf48
--- /dev/null
+++ b/src/test/java/org/vaadin/example/Test2.java
@@ -0,0 +1,18 @@
+package org.vaadin.example;
+
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+import org.junit.jupiter.api.Test;
+
+import com.vaadin.browserless.BrowserlessTest;
+import com.vaadin.flow.component.textfield.TextField;
+
+public class Test2 extends BrowserlessTest {
+       @Test
+       public void test2a() {
+               navigate(MainView.class);
+               var tf = find(TextField.class).single();
+               assertNotNull(tf);
+
+       }
+}
test123@asdf:~/skeleton-starter-flow$ git apply browserless.diff
test123@asdf:~/skeleton-starter-flow$ mvn clean test
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by com.google.common.util.concurrent.AbstractFuture$UnsafeAtomicHelper (file:/usr/share/maven/lib/guava.jar)
WARNING: Please consider reporting this to the maintainers of class com.google.common.util.concurrent.AbstractFuture$UnsafeAtomicHelper
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/com/vaadin/vaadin-bom/25.2.0-beta2/vaadin-bom-25.2.0-beta2.pom
...
Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-bom/2.0.17/slf4j-bom-2.0.17.pom (7.3 kB at 166 kB/s)
[INFO] 
[INFO] ----------------------< com.example:project-base >----------------------
[INFO] Building Project base for Vaadin 1.0-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ war ]---------------------------------
Downloading from central: https://repo.maven.apache.org/maven2/com/vaadin/vaadin-maven-plugin/25.2.0-beta2/vaadin-maven-plugin-25.2.0-beta2.pom
...
Downloaded from central: https://repo.maven.apache.org/maven2/com/vaadin/vaadin-upload-testbench/25.2.0-beta2/vaadin-upload-testbench-25.2.0-beta2.jar (10 kB at 737 B/s)
[INFO] 
[INFO] --- clean:3.2.0:clean (default-clean) @ project-base ---
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/3.3.4/maven-shared-utils-3.3.4.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/3.3.4/maven-shared-utils-3.3.4.pom (5.8 kB at 177 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-components/34/maven-shared-components-34.pom
Downloading from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.6/commons-io-2.6.jar
Downloaded from central: https://repo.maven.apache.org/maven2/commons-io/commons-io/2.6/commons-io-2.6.jar (215 kB at 3.8 MB/s)
[INFO] 
[INFO] --- resources:3.3.1:resources (default-resources) @ project-base ---
Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.26/plexus-interpolation-1.26.pom (2.7 kB at 86 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.5.1/plexus-utils-3.5.1.jar (269 kB at 2.0 MB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar (587 kB at 3.7 MB/s)
[INFO] Copying 1 resource from src/main/resources to target/classes
[INFO] 
[INFO] --- vaadin:25.2.0-beta2:prepare-frontend (default) @ project-base ---
Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.35/slf4j-api-1.7.35.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.35/slf4j-api-1.7.35.pom (3.8 kB at 15 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/slf4j/slf4j-parent/1.7.35/slf4j-parent-1.7.35.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/jboss/forge/roaster/roaster-jdt/2.31.0.Final/roaster-jdt-2.31.0.Final.jar (12 MB at 3.7 MB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/com/vaadin/vaadin-prod-bundle/25.2.0-beta2/vaadin-prod-bundle-25.2.0-beta2.jar (9.7 MB at 2.4 MB/s)
[INFO] ClassFinder initialized: 79 urls, 5168 classes scanned, 32 annotation types cached, 984 subtype relationships cached, took 227ms
[INFO] Visited 116 classes. Took 36 ms.
[INFO] Copying frontend resources from jar files ...
[INFO] Visited 89 resources. Took 30 ms.
[INFO] 
[INFO] --- compiler:3.14.1:compile (default-compile) @ project-base ---
Downloading from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/9.8/asm-9.8.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/ow2/asm/asm/9.8/asm-9.8.pom (2.4 kB at 53 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/ow2/ow2/1.5.1/ow2-1.5.1.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-xml/3.0.1/plexus-xml-3.0.1.jar (94 kB at 842 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/4.0.2/plexus-utils-4.0.2.jar (193 kB at 1.7 MB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/com/thoughtworks/qdox/qdox/2.2.0/qdox-2.2.0.jar (353 kB at 3.0 MB/s)
[INFO] Recompiling the module because of changed dependency.
[INFO] Compiling 3 source files with javac [debug release 21] to target/classes
[INFO] 
[INFO] --- resources:3.3.1:testResources (default-testResources) @ project-base ---
[INFO] skip non existing resourceDirectory /home/test123/skeleton-starter-flow/src/test/resources
[INFO] 
[INFO] --- compiler:3.14.1:testCompile (default-testCompile) @ project-base ---
[INFO] Recompiling the module because of changed dependency.
[INFO] Compiling 2 source files with javac [debug release 21] to target/test-classes
[WARNING] /home/test123/skeleton-starter-flow/src/test/java/org/vaadin/example/MainViewIT.java:[37,31] first() in com.vaadin.testbench.ElementQuery has been deprecated and marked for removal
[WARNING] /home/test123/skeleton-starter-flow/src/test/java/org/vaadin/example/MainViewIT.java:[44,54] first() in com.vaadin.testbench.ElementQuery has been deprecated and marked for removal
[WARNING] /home/test123/skeleton-starter-flow/src/test/java/org/vaadin/example/MainViewIT.java:[52,54] first() in com.vaadin.testbench.ElementQuery has been deprecated and marked for removal
[WARNING] /home/test123/skeleton-starter-flow/src/test/java/org/vaadin/example/MainViewIT.java:[55,57] first() in com.vaadin.testbench.ElementQuery has been deprecated and marked for removal
[WARNING] /home/test123/skeleton-starter-flow/src/test/java/org/vaadin/example/MainViewIT.java:[61,63] first() in com.vaadin.testbench.ElementQuery has been deprecated and marked for removal
[WARNING] /home/test123/skeleton-starter-flow/src/test/java/org/vaadin/example/MainViewIT.java:[63,54] first() in com.vaadin.testbench.ElementQuery has been deprecated and marked for removal
[WARNING] /home/test123/skeleton-starter-flow/src/test/java/org/vaadin/example/MainViewIT.java:[66,57] first() in com.vaadin.testbench.ElementQuery has been deprecated and marked for removal
[WARNING] /home/test123/skeleton-starter-flow/src/test/java/org/vaadin/example/MainViewIT.java:[72,63] first() in com.vaadin.testbench.ElementQuery has been deprecated and marked for removal
[WARNING] /home/test123/skeleton-starter-flow/src/test/java/org/vaadin/example/MainViewIT.java:[76,57] first() in com.vaadin.testbench.ElementQuery has been deprecated and marked for removal
[INFO] 
[INFO] --- surefire:2.17:test (default-test) @ project-base ---
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.9/maven-plugin-api-2.0.9.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.9/maven-plugin-api-2.0.9.pom (1.5 kB at 40 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/maven/2.0.9/maven-2.0.9.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-annotations/3.2/maven-plugin-annotations-3.2.jar (15 kB at 58 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-api/2.17/surefire-api-2.17.jar (147 kB at 542 kB/s)
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.1/commons-lang3-3.1.jar (316 kB at 1.1 MB/s)
[INFO] Surefire report directory: /home/test123/skeleton-starter-flow/target/surefire-reports
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit3/2.17/surefire-junit3-2.17.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit3/2.17/surefire-junit3-2.17.pom (1.7 kB at 13 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-providers/2.17/surefire-providers-2.17.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-providers/2.17/surefire-providers-2.17.pom (2.4 kB at 74 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit3/2.17/surefire-junit3-2.17.jar
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit3/2.17/surefire-junit3-2.17.jar (26 kB at 218 kB/s)
 
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.vaadin.example.Test2
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec <<< FAILURE! - in org.vaadin.example.Test2
org.vaadin.example.Test2.test2a()  Time elapsed: 0.008 sec  <<< FAILURE!
com.vaadin.browserless.BrowserlessTestSetupException: Test Vaadin environment is not initialized correctly. This may happen when the test is extending the wrong base class for the testing engine in use. Current test class is expected to run with JUnit 6.
        at com.vaadin.browserless.BaseBrowserlessTest.verifyAndGetUI(BaseBrowserlessTest.java:457)
        at com.vaadin.browserless.BaseBrowserlessTest.navigate(BaseBrowserlessTest.java:153)
        at org.vaadin.example.Test2.test2a(Test2.java:13)
 
 
Results :
 
Failed tests: 
  org.vaadin.example.Test2#test2a BrowserlessTestSetupException Test Vaadin envi...
 
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:04 min
[INFO] Finished at: 2026-06-15T06:56:31+03:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project project-base: There are test failures.
[ERROR] 
[ERROR] Please refer to /home/test123/skeleton-starter-flow/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
test123@asdf:~/skeleton-starter-flow$ java -version
openjdk version "25.0.3" 2026-04-21
OpenJDK Runtime Environment (build 25.0.3+9-2-deb13u1-Debian)
OpenJDK 64-Bit Server VM (build 25.0.3+9-2-deb13u1-Debian, mixed mode, sharing)
test123@asdf:~/skeleton-starter-flow$ 
lopetus
root@asdf:~# 
exit
 
Script done on 2026-06-15 06:57:17+03:00 [COMMAND_EXIT_CODE="0"]
4

The maven surefire plugin might be too old. Can you try with the latest version. Probably the version is not pinned in the pom file. If so the repo needs un update

5

Adding maven-surefire-plugin version helped.

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.6.0-M1</version>
</plugin>

I don’t know where the default surefire version of 2.17 comes from. It’s not in output of mvn dependency:tree -D scope=test and there is no surefire mentioned anyhwere in skeleton-starter-flow. Could that just be default for maven 3.9.9? (Debian trixie)

Perhaps the browserless tests should declare minimum supported surefire version somewhere.

6

That version is most likely from the implicit maven super pom, present in the maven installation folder.

IIRC, surefire greather than 2.22 is a requirement fir junit 5 to work properly

7

Claude blames Debian

Short version: that 2.17 is not coming from Maven 3.9.9 — it’s Debian’s Maven, and dependency:tree will never show it because surefire is a build plugin, not a dependency.

I checked the actual source. Stock Maven 3.9.9’s lifecycle bindings (maven-core/.../META-INF/plexus/default-bindings.xml) bind maven-surefire-plugin:3.2.5:test for every packaging. So vanilla 3.9.9 would give you 3.2.5, which handles JUnit Platform fine and you’d never have noticed anything. The 2.17 is Debian repackaging: their maven package patches default-bindings.xml to point the lifecycle defaults at whatever plugin versions Debian itself ships, and on your trixie box that surefire is 2.17. Plain mvn uses that patched maven-core, so the build silently picks 2.17 even though it appears nowhere in your POM or the dependency tree.

You can confirm it on your own machine without guessing:

unzip -p /usr/share/maven/lib/maven-core-*.jar META-INF/plexus/default-bindings.xml | grep surefire

You should see maven-surefire-plugin:2.17:test rather than 3.2.5. mvn -X test 2>&1 | grep -i surefire will show the same version being resolved at the test phase.