Issue with Server Push and HtmlUnit

Hi vaadins!

I am using HtmlUnit (net.sourceforge.htmlunit) for parsing websites. I noticed an issue that my Tomcat is complaining about not finding the class 'org.eclipse.jetty.websocket.server.WebSocketServerFactory' after I added the htmlunit dependency in my pom-file and therefore Server Push is not working anymore.

Thanks in advance!

java.lang.RuntimeException: Atmosphere init failed

    at com.vaadin.server.communication.PushRequestHandler.initAtmosphere(PushRequestHandler.java:228)

    at com.vaadin.server.communication.PushRequestHandler.<init>(PushRequestHandler.java:88)

    at com.vaadin.server.VaadinServletService.createRequestHandlers(VaadinServletService.java:54)

    at com.vaadin.server.VaadinService.init(VaadinService.java:192)

    at com.vaadin.server.VaadinServlet.createServletService(VaadinServlet.java:332)

    at com.vaadin.server.VaadinServlet.init(VaadinServlet.java:227)

    at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1183)

    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1099)

    at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:779)

    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:133)

    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)

    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)

    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)

    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)

    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784)

    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)

    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)

    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

    at java.lang.Thread.run(Thread.java:745)

Caused by: javax.servlet.ServletException: java.lang.NoClassDefFoundError: org/eclipse/jetty/websocket/server/WebSocketServerFactory

    at org.atmosphere.cpr.AtmosphereFramework.init(AtmosphereFramework.java:912)

    at org.atmosphere.cpr.AtmosphereFramework.init(AtmosphereFramework.java:780)

    at com.vaadin.server.communication.PushRequestHandler.initAtmosphere(PushRequestHandler.java:220)

    ... 25 more

Caused by: java.lang.NoClassDefFoundError: org/eclipse/jetty/websocket/server/WebSocketServerFactory

    at java.lang.Class.getDeclaredConstructors0(Native Method)

    at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)

    at java.lang.Class.getConstructor0(Class.java:3075)

    at java.lang.Class.getDeclaredConstructor(Class.java:2178)

    at org.atmosphere.cpr.DefaultAsyncSupportResolver.newCometSupport(DefaultAsyncSupportResolver.java:232)

    at org.atmosphere.cpr.DefaultAsyncSupportResolver.resolveWebSocket(DefaultAsyncSupportResolver.java:297)

    at org.atmosphere.cpr.DefaultAsyncSupportResolver.resolve(DefaultAsyncSupportResolver.java:283)

    at org.atmosphere.cpr.AtmosphereFramework.autoDetectContainer(AtmosphereFramework.java:1870)

    at org.atmosphere.cpr.AtmosphereFramework.init(AtmosphereFramework.java:888)

    ... 27 more

Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.websocket.server.WebSocketServerFactory

    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1275)

    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1109)

    ... 36 more

This is a wild guess. But since HtmlUnit has something to do with WebSockets, and possibly using some library for that, there might be conflicting dependencies. Errors like "java.lang.ClassNotFoundException: org.eclipse.jetty.websocket.server.WebSocketServerFactory" are usually caused by duplicates, i.e. class loader cannot resolve which one to use. Hence inspecting dependency tree and eliminating those dependencies in some suitable way could be an answer. Another approach could be to test Push with different transport mode, like long polling and check if that works.
 

Thanks for the quick response! I guess that is more a Maven problem. I also tried different transport modes, but still complaining about that missing class.
Do you know how to resolve those conflicting Maven dependencies?

sirilicious: Thanks for the quick response! I guess that is more a Maven problem. I also tried different transport modes, but still complaining about that missing class.
Do you know how to resolve those conflicting Maven dependencies?

Yes, that was what I was referring to in the earlier comment. Start by checking "mvn dependency:tree". What you need to do is highly dependent on your findings. You can either use "provided" scope on those libraries you need only compile time. You can use exclude to those libraries that brought with other package. E.g. try exclude WebSockets library from HtmlUnit dependency for example.

Thanks again for your assistance. I tried every proposed approach, but I was not successful.
When excluding the WebSockets library from the HtmlUnit dependency I experience ClassNotFound exceptions at runtime - frankly because Vaadin uses a different Websocket library than HtmlUnit - the same issue results from modifiing the scope to "provided".

I am feeling a little bit stucked here. Is the only viable solution to run my HtmlUnit functionality in a separate JVM?

Have you tried to exclude whole gwt-dev and refactor your imports accordingly?

Hint, mvn dependency:tree can give you important info about what is being included.

Ok, I finally got it - by upgrading to Vaadin 8. :) Thanks again.

htmlunit depends on a lots of stuff from Jetty:

|    |    |    |    |    |    \--- org.eclipse.jetty.websocket:websocket-client:9.2.15.v20160210 -> 9.3.14.v20161028
|    |    |    |    |    |         +--- org.eclipse.jetty:jetty-util:9.3.14.v20161028
|    |    |    |    |    |         +--- org.eclipse.jetty:jetty-io:9.3.14.v20161028
|    |    |    |    |    |         |    \--- org.eclipse.jetty:jetty-util:9.3.14.v20161028
|    |    |    |    |    |         \--- org.eclipse.jetty.websocket:websocket-common:9.3.14.v20161028
|    |    |    |    |    |              +--- org.eclipse.jetty.websocket:websocket-api:9.3.14.v20161028
|    |    |    |    |    |              +--- org.eclipse.jetty:jetty-util:9.3.14.v20161028
|    |    |    |    |    |              \--- org.eclipse.jetty:jetty-io:9.3.14.v20161028 (*)

Having those in WAR in Tomcat is almost always a bad idea. Apparently it confused and tricked Atmosphere into believing that it's running on Jetty; Atmosphere then tried to use Jetty native Comet Push implementation and failed.
Upgrading to Vaadin 8 probably upgraded Atmosphere to newer version as well; the detection routines could have been changed since then (probably it now by default relies for JSR356 WebSockets in Servlets) and thus it may work.
However, having htmlunit in a war is probably a bad idea since htmlunit is actually a client-side code, used for testing of the webapps.

Tatu Lund: Errors like "java.lang.ClassNotFoundException: org.eclipse.jetty.websocket.server.WebSocketServerFactory" are usually caused by duplicates, i.e. class loader cannot resolve which one to use.
 

Please allow me to disagree here: if a class is present in multiple jars, the classloader will simply load the class from a jar which is earlier in a classpath. While error-prone and an anti-pattern, it should never cause ClassNotFoundException.