Vaadin 7 Application intermittently hangs indefinitely: Server Push / Atmos

General
1

Hello everyone, I am trying to troubleshoot some strange issues we are seeing with our Vaadin 7.3.2 application.

While loading data to the screen, the red busy indicator will appear indefinitely, with the data never making it back to the screen.

Vaadin 7.3.2, Java 8, Tomcat 7, Transport.WEBSOCKET.

This has been happening frequently and we have traced the timing back to some strange errors we see in the logs.

[code]
2014-11-26 09:57:02,589 CST INFO - java.net.SocketException: Error when closing push connection
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at org.apache.coyote.http11.upgrade.UpgradeBioProcessor.write(UpgradeBioProcessor.java:61)
at org.apache.coyote.http11.upgrade.UpgradeOutbound.write(UpgradeOutbound.java:44)
at org.apache.catalina.websocket.WsOutbound.doWriteBytes(WsOutbound.java:477)
at org.apache.catalina.websocket.WsOutbound.doWriteText(WsOutbound.java:529)
at org.apache.catalina.websocket.WsOutbound.doFlush(WsOutbound.java:267)
at org.apache.catalina.websocket.WsOutbound.close(WsOutbound.java:349)
at org.atmosphere.container.version.TomcatWebSocket.close(TomcatWebSocket.java:84)
at org.atmosphere.cpr.AtmosphereResponse.close(AtmosphereResponse.java:836)
at org.atmosphere.cpr.AtmosphereResourceImpl.cancel(AtmosphereResourceImpl.java:720)
at org.atmosphere.cpr.AtmosphereResourceImpl.close(AtmosphereResourceImpl.java:785)
at com.vaadin.server.communication.AtmospherePushConnection.disconnect(AtmospherePushConnection.java:293)
at com.vaadin.server.communication.PushHandler.disconnect(PushHandler.java:430)
at com.vaadin.server.communication.PushHandler.access$000(PushHandler.java:56)
at com.vaadin.server.communication.PushHandler$1.onStateChange(PushHandler.java:65)
at org.atmosphere.cpr.AsynchronousProcessor.invokeAtmosphereHandler(AsynchronousProcessor.java:46 [8)]

 at org.atmosphere.cpr.AsynchronousProcessor.completeLifecycle(AsynchronousProcessor.java:420) 
 at org.atmosphere.cpr.AsynchronousProcessor.timedout(AsynchronousProcessor.java:381) 
 at org.atmosphere.cpr.AsynchronousProcessor$AsynchronousProcessorHook.timedOut(AsynchronousProcessor.java:545) 
 at org.atmosphere.websocket.DefaultWebSocketProcessor.close(DefaultWebSocketProcessor.java:493) 
 at org.atmosphere.container.TomcatWebSocketHandler.onClose(TomcatWebSocketHandler.java:80) 
 at org.apache.catalina.websocket.StreamInbound.doOnClose(StreamInbound.java:226) 
 at org.apache.catalina.websocket.WsOutbound.writeTextMessage(WsOutbound.java:229) 
 at org.atmosphere.container.version.TomcatWebSocket.write(TomcatWebSocket.java:53) 
 at org.atmosphere.websocket.WebSocket.write(WebSocket.java:217) 
 at org.atmosphere.websocket.WebSocket.write(WebSocket.java:185) 
 at org.atmosphere.websocket.WebSocket.write(WebSocket.java:42) 
 at org.atmosphere.cpr.AtmosphereResponse$2.write(AtmosphereResponse.java:503) 
 at org.atmosphere.handler.AbstractReflectorAtmosphereHandler.onStateChange(AbstractReflectorAtmosphereHandler.java:141) 
 at com.vaadin.server.communication.PushHandler$1.onStateChange(PushHandler.java:63) 
 at org.atmosphere.cpr.DefaultBroadcaster.invokeOnStateChange(DefaultBroadcaster.java:101 [8)]

 at org.atmosphere.cpr.DefaultBroadcaster.prepareInvokeOnStateChange(DefaultBroadcaster.java:103 [8)]

 at org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(DefaultBroadcaster.java:879) 
 at org.atmosphere.cpr.DefaultBroadcaster$3.run(DefaultBroadcaster.java:520) 
 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) 
 at java.util.concurrent.FutureTask.run(Unknown Source) 
 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
 at java.lang.Thread.run(Unknown Source)

[/code][code]
2014-11-26 11:58:45,829 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    2014-11-26 11:58:45,829 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    2014-11-26 11:58:45,830 CST WARN [null]

  • java.lang.AssertionError: Failed invoking AtmosphereFramework.doCometSupport()
    at com.vaadin.ui.ConnectorTracker.cleanConnectorMap(ConnectorTracker.java:289)
    at com.vaadin.server.communication.UidlWriter.write(UidlWriter.java:305)
    at com.vaadin.server.communication.AtmospherePushConnection.push(AtmospherePushConnection.java:155)
    at com.vaadin.server.communication.AtmospherePushConnection.push(AtmospherePushConnection.java:133)
    at com.vaadin.ui.UI.push(UI.java:1501)
    at com.vaadin.server.VaadinSession.unlock(VaadinSession.java:1026)
    at com.vaadin.server.VaadinService.requestEnd(VaadinService.java:1349)
    at com.vaadin.server.communication.PushHandler.callWithUi(PushHandler.java:291)
    at com.vaadin.server.communication.PushHandler.access$200(PushHandler.java:56)
    at com.vaadin.server.communication.PushHandler$1.onRequest(PushHandler.java:76)
    at org.atmosphere.cpr.AsynchronousProcessor.action(AsynchronousProcessor.java:174)
    at org.atmosphere.cpr.AsynchronousProcessor.suspended(AsynchronousProcessor.java:95)
    at org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:154)
    at org.atmosphere.container.Tomcat7Servlet30SupportWithWebSocket.service(Tomcat7Servlet30SupportWithWebSocket.java:62)
    at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1802)
    at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:432)
    at org.atmosphere.websocket.DefaultWebSocketProcessor$2.run(DefaultWebSocketProcessor.java:285)
    at org.atmosphere.util.VoidExecutorService.execute(VoidExecutorService.java:101)
    at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:280)
    at org.atmosphere.websocket.DefaultWebSocketProcessor.invokeWebSocketProtocol(DefaultWebSocketProcessor.java:303)
    at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatchReader(DefaultWebSocketProcessor.java:647)
    at org.atmosphere.websocket.DefaultWebSocketProcessor.invokeWebSocketProtocol(DefaultWebSocketProcessor.java:404)
    at org.atmosphere.container.TomcatWebSocketHandler.onTextData(TomcatWebSocketHandler.java:100)
    at org.apache.catalina.websocket.StreamInbound.doOnTextData(StreamInbound.java:191)
    at org.apache.catalina.websocket.StreamInbound.onData(StreamInbound.java:139)
    at org.apache.coyote.http11.upgrade.UpgradeProcessor.upgradeDispatch(UpgradeProcessor.java:8 [8)]

    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)
    2014-11-26 11:58:45,830 CST WARN [null]

  • java.lang.AssertionError: The connector hierarchy is corrupted. Check for missing calls to super.setParent(), super.attach() and super.detach() and that all custom component containers call child.setParent(this) when a child is added and child.setParent(null) when the child is no longer used. See previous log messages for details. Status 500 Message Server Error
    11:57:23 AM
    [/code][code]
    2014-11-26 11:47:15,366 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    2014-11-26 11:47:15,366 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    2014-11-26 11:47:15,366 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    2014-11-26 11:47:15,366 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    2014-11-26 11:47:15,366 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    2014-11-26 11:47:15,367 CST WARN [null]

  • java.lang.AssertionError: Failed invoking AtmosphereFramework.doCometSupport()
    at com.vaadin.ui.ConnectorTracker.cleanConnectorMap(ConnectorTracker.java:289)
    at com.vaadin.server.communication.UidlWriter.write(UidlWriter.java:305)
    at com.vaadin.server.communication.AtmospherePushConnection.push(AtmospherePushConnection.java:155)
    at com.vaadin.server.communication.AtmospherePushConnection.push(AtmospherePushConnection.java:133)
    at com.vaadin.ui.UI.push(UI.java:1501)
    at com.vaadin.server.VaadinSession.unlock(VaadinSession.java:1026)
    at com.vaadin.server.VaadinService.requestEnd(VaadinService.java:1349)
    at com.vaadin.server.communication.PushHandler.callWithUi(PushHandler.java:291)
    at com.vaadin.server.communication.PushHandler.access$200(PushHandler.java:56)
    at com.vaadin.server.communication.PushHandler$1.onRequest(PushHandler.java:76)
    at org.atmosphere.cpr.AsynchronousProcessor.action(AsynchronousProcessor.java:174)
    at org.atmosphere.cpr.AsynchronousProcessor.suspended(AsynchronousProcessor.java:95)
    at org.atmosphere.container.TomcatWebSocketUtil.doService(TomcatWebSocketUtil.java:154)
    at org.atmosphere.container.Tomcat7Servlet30SupportWithWebSocket.service(Tomcat7Servlet30SupportWithWebSocket.java:62)
    at org.atmosphere.cpr.AtmosphereFramework.doCometSupport(AtmosphereFramework.java:1802)
    at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:432)
    at org.atmosphere.websocket.DefaultWebSocketProcessor$2.run(DefaultWebSocketProcessor.java:285)
    at org.atmosphere.util.VoidExecutorService.execute(VoidExecutorService.java:101)
    at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatch(DefaultWebSocketProcessor.java:280)
    at org.atmosphere.websocket.DefaultWebSocketProcessor.invokeWebSocketProtocol(DefaultWebSocketProcessor.java:303)
    at org.atmosphere.websocket.DefaultWebSocketProcessor.dispatchReader(DefaultWebSocketProcessor.java:647)
    at org.atmosphere.websocket.DefaultWebSocketProcessor.invokeWebSocketProtocol(DefaultWebSocketProcessor.java:404)
    at org.atmosphere.container.TomcatWebSocketHandler.onTextData(TomcatWebSocketHandler.java:100)
    at org.apache.catalina.websocket.StreamInbound.doOnTextData(StreamInbound.java:191)
    at org.apache.catalina.websocket.StreamInbound.onData(StreamInbound.java:139)
    at org.apache.coyote.http11.upgrade.UpgradeProcessor.upgradeDispatch(UpgradeProcessor.java:88)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)
    2014-11-26 11:47:15,368 CST WARN [null]

  • java.lang.AssertionError: The connector hierarchy is corrupted. Check for missing calls to super.setParent(), super.attach() and super.detach() and that all custom component containers call child.setParent(this) when a child is added and child.setParent(null) when the child is no longer used. See previous log messages for details. Status 500 Message Server Error
    2014-11-26 11:47:31,568 CST INFO [AsyncAdvancedTable:run() AsyncAdvancedTable:execute() JobConsoleGridComponent:apply(EpiqFilter[2]
    ) JobTemplateController:find(EpiqFilter[2]
    ) JobTemplateController:performFind(EpiqFilter[2]
    ) JobTemplateController:getContainerFromCollection(Collection)] - Time: 1
    2014-11-26 11:47:31,623 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    2014-11-26 11:47:31,623 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    2014-11-26 11:47:31,623 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    2014-11-26 11:47:31,623 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.[size=4]
    6 11:58:45,829 CST WARN [null]

  • {0} claims that {1} is its parent, but the parent does not acknowledge the parenthood.
    [/size]
    [/code]Do you guys have any idea why this is happening, or what we can do to troubleshoot this issue further?

2

Here is a screen shot of jVisualVM showing all of the Atmopshere threads are parked.
17314.png

3


EDIT:
Disregard
 below, this did not fix the problem. We are still having this problem.

Hi, it appears we fixed this issue by adding the following @WebInitParam to our @WebServlet definition.

@WebInitParam
(
    name = "org.atmosphere.cpr.asyncSupport", 
    value = "org.atmosphere.container.Tomcat7AsyncSupportWithWebSocket"
)

What we think is happening is Atmosphere is trying to use the Jetty Async Library rather than the one for Tomcat.

4

Just to follow up with this, we spoke too soon, the issue has not been resolved and is still happening in our QA environment.

I have also posted on the atmosphere forums here:


https://groups.google.com/forum/#!topic/atmosphere-framework/LYHBWO7BxTY

5

Hi!

I encountered a similar issue on an application [1]
after the @Push (based on Transport.WEBSOCKET) was added. After some trial and error, we ended up changing the transport mechanism to Transport.STREAMING and the application has been running ever since.

[1]
Java 7, Tomcat 7, VAADIN 7.3.5

6

Hello Daniel,

I tried alternative Transport modes, namely Transport.LONG_POLLING and Transport.STREAMING, but both seemed to have issues with loading the data to the screen.

More specifically, we have various Master-Detail Relationships represented as Tables on the screen, and when using these other Transport modes, the dependant Tables seem to intermittently not load the data when they should.

This always works in Transport.WEBSOCKET, except for the intermittent hang that we see.

Thank you for the interest in the issue and the suggestion :slight_smile:

7

Ok, we experienced the same type of hangups again with Transport.STREAMING. :frowning:

Has a ticket been opened regarding this issue? I read in the referenced topic on the Atmosphere framework forums that it could be related to the usage of the framework by VAADIN.

D.

8

Hi Daniel, here is the link to the ticket opened in response to the issue:
http://dev.vaadin.com/ticket/15330#ticket

9

cboyd,

I am getting the same issue in my testing push. Java 1.8u25, tomcat 8 and virtually every browser. cboyd, have you found a solid fix for this yet?

Thanks,
Lucas

10

Hi Lucas, we have seen a number of different factors influence this.

We think it’s related to at least 2 oustanding Vaadin bugs, one of which is referenced in this thread, as well as a number of other factors.

Out of all the errors and strange things we encountered, they all lead to the same result: The web socket closes on the cient.

We mostly saw this issue on VMs, so as part of our configuration and troubleshooting attempts to solve this issue, we allocated more resources to the VM.
We upgraded to Tomcat 8, Vaadin 7.3.7, made sure we are using NIO, and set (in the Vaadin Servlet Configuration) closeIdleSessions = false.

With these changes, we are no longer seeing the issue.