Thread Hang / Deadlock with Vaadin 7 on WAS 8.5

Vaadin 6 / 7 / 8
1

Hi All,

We are having a major issue with one of our applications, which has been built using Vaadin 7.1.5 running on WAS 8.5.

we have hit a deadlock where one of the holders has Introscope objects and the other is in the vaadin framework. Intriguingly, we have now discovered work through the load balancer to a server with Introscope (APM 9) can lead to deadlocks whereas those going directly on to the server does not although there are timeouts and long responses.


Java Stack:

at sun/misc/Unsafe.park(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:186(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197(Compiled Code))
at java/util/concurrent/locks/ReentrantLock$NonfairSync.lock(ReentrantLock.java:214(Compiled Code))
at java/util/concurrent/locks/ReentrantLock.lock(ReentrantLock.java:290(Compiled Code))
at com/vaadin/server/VaadinSession.lock(VaadinSession.java:890(Compiled Code))
at com/vaadin/server/VaadinSession.accessSynchronously(VaadinSession.java:1213)
at com/vaadin/server/VaadinService.fireSessionDestroy(VaadinService.java:437)
at com/vaadin/server/VaadinSession.valueUnbound(VaadinSession.java:261)
at com/ibm/ws/session/http/HttpSessionObserver.sessionDestroyed(HttpSessionObserver.java:193)
at com/ibm/ws/session/SessionEventDispatcher.sessionDestroyed(SessionEventDispatcher.java:160)
at com/ibm/ws/session/StoreCallback.sessionInvalidated(StoreCallback.java:126)
at com/ibm/ws/session/store/memory/MemorySession.invalidate(MemorySession.java:232)
at com/ibm/ws/session/http/HttpSessionImpl.invalidate(HttpSessionImpl.java:303)
at com/ibm/ws/session/SessionData.invalidate(SessionData.java:247)
at com/ibm/ws/session/SessionContext.doSecurityCheck(SessionContext.java:567)
at com/ibm/ws/session/SessionContext.getIHttpSession(SessionContext.java:507)
at com/ibm/ws/session/SessionContext.getIHttpSession(SessionContext.java:426(Compiled Code))
at com/ibm/ws/webcontainer/srt/SRTRequestContext.getSession(SRTRequestContext.java:104(Compiled Code))
at com/ibm/ws/webcontainer/srt/SRTServletRequest.getSession(SRTServletRequest.java:2157(Compiled Code))
at com/wily/introscope/agent/servlethelper/ServletRequestHelper.getSession(ServletRequestHelper.java:286)
at com/wily/introscope/agent/servlethelper/ServletRequestHelper.get(ServletRequestHelper.java:513)
at com/wily/introscope/agent/trace/hc2/HttpServletLazyMapProvider.get(HttpServletLazyMapProvider.java:63)
at com/wily/util/adt/LazyMap.get(LazyMap.java:182(Compiled Code))
at com/wily/introscope/agent/trace/hc2/HttpServletLazyMap.putAllParameters(HttpServletLazyMap.java:191)
at com/wily/introscope/agent/trace/hc2/HttpServletTracer.IInvocationDataParameterCallback_addParameters(HttpServletTracer.java:782)
at com/wily/introscope/agent/trace/InvocationData.addParameterCallbacks(InvocationData.java:3486)
at com/wily/introscope/agent/feature/VirtualStackFeatureHelper.fillComponentParams(VirtualStackFeatureHelper.java:157)
at com/wily/introscope/agent/feature/VirtualStackFeatureHelper.getTransanctionSnapShotFromInvocationData(VirtualStackFeatureHelper.java:235)
at com/wily/introscope/agent/feature/VirtualStackFeatureHelper.getTransactionSnapshotFromStack(VirtualStackFeatureHelper.java:75)
at com/wily/introscope/agent/feature/StallFeatureBase.reportStall(StallFeatureBase.java:50)
at com/wily/introscope/agent/feature/StallCheckBehaviorCorrect.ITimestampedRunnable_execute(StallCheckBehaviorCorrect.java:61(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:972(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:491(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat.access$2(IntervalHeartbeat.java:445(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:668)
at java/lang/Thread.run(Thread.java:781)


and this is the blocked/deadlocked one:-

at com/ibm/ws/session/http/HttpSessionImpl.getAttribute(HttpSessionImpl.java:189(Compiled Code))
at com/ibm/ws/session/SessionData.getSessionValue(SessionData.java:307(Compiled Code))
at com/ibm/ws/session/SessionData.getAttribute(SessionData.java:163(Compiled Code))
at com/ibm/ws/session/HttpSessionFacade.getAttribute(HttpSessionFacade.java:139(Compiled Code))
at com/vaadin/server/WrappedHttpSession.getAttribute(WrappedHttpSession.java:54(Compiled Code))
at com/vaadin/server/VaadinService.lockSession(VaadinService.java:596)
at com/vaadin/server/VaadinService.findOrCreateVaadinSession(VaadinService.java:625)
at com/vaadin/server/VaadinService.findVaadinSession(VaadinService.java:495)
at com/vaadin/server/VaadinService.handleRequest(VaadinService.java:1365)
at com/vaadin/server/VaadinServlet.service(VaadinServlet.java:238)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668)
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1230)
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:779)
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:478)
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at com/tnt/vcm/application/security/LoginAuthorisationFilter.doFilter(LoginAuthorisationFilter.java:142)
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:960)
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1064)
at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:909)
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:200)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:283)
at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com/ibm/io/async/AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1862(Compiled Code))

So, basically we need to know what the root cause of the deadlock and thread hang ? Anyone came across similar issue and can advise please ?

Thanks

2

Hi All,

I’ve added further logs that might help in the investigation:

"The deadlock involves 2 locks:

  1. a com/ibm/ws/session/store/memory/MemorySession object to access the
    http session.

  2. a java/util/concurrent/locks/ReentrantLock$NonfairSync object that seems
    to be a concurrent lock to access VaadinSession (vaadin framework), which
    subscribed to IBM memory session as an observer.

So the deadlock is between Vaadin and IBM, wily Agent just happened to be
one user that also needs to access the http session.

Engineering says that Wily made this deadlock occur more often because of
our accessing http session, but is not causing the problem. This is proved by the last
thread dump on 7/31. This deadlock doesn’t involve Wily at all, it’s
between 2 “WebContainer” threads, and none of them’s stack trace has wily either.

1LKDEADLOCK Deadlock detected !!!
NULL ---------------------
NULL
2LKDEADLOCKTHR Thread “WebContainer : 0” (0x0E905E00)
3LKDEADLOCKWTR is waiting for:
4LKDEADLOCKOBJ
java/util/concurrent/locks/ReentrantLock$NonfairSync@0xBFEFACC0
3LKDEADLOCKOWN which is owned by:
2LKDEADLOCKTHR Thread “WebContainer : 3” (0x100E4000)
3LKDEADLOCKWTR which is waiting for:
4LKDEADLOCKMON sys_mon_t:0xA58D4D00 infl_mon_t: 0xA58D4D3C:
4LKDEADLOCKOBJ com/ibm/ws/session/store/memory/MemorySession@0xBFF03360
3LKDEADLOCKOWN which is owned by:
2LKDEADLOCKTHR Thread “WebContainer : 0” (0x0E905E00)[quote=Gulshan Babajee]
Hi All,

We are having a major issue with one of our applications, which has been built using Vaadin 7.1.5 running on WAS 8.5.

we have hit a deadlock where one of the holders has Introscope objects and the other is in the vaadin framework. Intriguingly, we have now discovered work through the load balancer to a server with Introscope (APM 9) can lead to deadlocks whereas those going directly on to the server does not although there are timeouts and long responses.


Java Stack:

at sun/misc/Unsafe.park(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:186(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197(Compiled Code))
at java/util/concurrent/locks/ReentrantLock$NonfairSync.lock(ReentrantLock.java:214(Compiled Code))
at java/util/concurrent/locks/ReentrantLock.lock(ReentrantLock.java:290(Compiled Code))
at com/vaadin/server/VaadinSession.lock(VaadinSession.java:890(Compiled Code))
at com/vaadin/server/VaadinSession.accessSynchronously(VaadinSession.java:1213)
at com/vaadin/server/VaadinService.fireSessionDestroy(VaadinService.java:437)
at com/vaadin/server/VaadinSession.valueUnbound(VaadinSession.java:261)
at com/ibm/ws/session/http/HttpSessionObserver.sessionDestroyed(HttpSessionObserver.java:193)
at com/ibm/ws/session/SessionEventDispatcher.sessionDestroyed(SessionEventDispatcher.java:160)
at com/ibm/ws/session/StoreCallback.sessionInvalidated(StoreCallback.java:126)
at com/ibm/ws/session/store/memory/MemorySession.invalidate(MemorySession.java:232)
at com/ibm/ws/session/http/HttpSessionImpl.invalidate(HttpSessionImpl.java:303)
at com/ibm/ws/session/SessionData.invalidate(SessionData.java:247)
at com/ibm/ws/session/SessionContext.doSecurityCheck(SessionContext.java:567)
at com/ibm/ws/session/SessionContext.getIHttpSession(SessionContext.java:507)
at com/ibm/ws/session/SessionContext.getIHttpSession(SessionContext.java:426(Compiled Code))
at com/ibm/ws/webcontainer/srt/SRTRequestContext.getSession(SRTRequestContext.java:104(Compiled Code))
at com/ibm/ws/webcontainer/srt/SRTServletRequest.getSession(SRTServletRequest.java:2157(Compiled Code))
at com/wily/introscope/agent/servlethelper/ServletRequestHelper.getSession(ServletRequestHelper.java:286)
at com/wily/introscope/agent/servlethelper/ServletRequestHelper.get(ServletRequestHelper.java:513)
at com/wily/introscope/agent/trace/hc2/HttpServletLazyMapProvider.get(HttpServletLazyMapProvider.java:63)
at com/wily/util/adt/LazyMap.get(LazyMap.java:182(Compiled Code))
at com/wily/introscope/agent/trace/hc2/HttpServletLazyMap.putAllParameters(HttpServletLazyMap.java:191)
at com/wily/introscope/agent/trace/hc2/HttpServletTracer.IInvocationDataParameterCallback_addParameters(HttpServletTracer.java:782)
at com/wily/introscope/agent/trace/InvocationData.addParameterCallbacks(InvocationData.java:3486)
at com/wily/introscope/agent/feature/VirtualStackFeatureHelper.fillComponentParams(VirtualStackFeatureHelper.java:157)
at com/wily/introscope/agent/feature/VirtualStackFeatureHelper.getTransanctionSnapShotFromInvocationData(VirtualStackFeatureHelper.java:235)
at com/wily/introscope/agent/feature/VirtualStackFeatureHelper.getTransactionSnapshotFromStack(VirtualStackFeatureHelper.java:75)
at com/wily/introscope/agent/feature/StallFeatureBase.reportStall(StallFeatureBase.java:50)
at com/wily/introscope/agent/feature/StallCheckBehaviorCorrect.ITimestampedRunnable_execute(StallCheckBehaviorCorrect.java:61(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat$BehaviorNode.execute(IntervalHeartbeat.java:972(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat.executeNextBehaviorAndCalculateSleepTime(IntervalHeartbeat.java:491(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat.access$2(IntervalHeartbeat.java:445(Compiled Code))
at com/wily/util/heartbeat/IntervalHeartbeat$HeartbeatRunnable.run(IntervalHeartbeat.java:668)
at java/lang/Thread.run(Thread.java:781)


and this is the blocked/deadlocked one:-

at com/ibm/ws/session/http/HttpSessionImpl.getAttribute(HttpSessionImpl.java:189(Compiled Code))
at com/ibm/ws/session/SessionData.getSessionValue(SessionData.java:307(Compiled Code))
at com/ibm/ws/session/SessionData.getAttribute(SessionData.java:163(Compiled Code))
at com/ibm/ws/session/HttpSessionFacade.getAttribute(HttpSessionFacade.java:139(Compiled Code))
at com/vaadin/server/WrappedHttpSession.getAttribute(WrappedHttpSession.java:54(Compiled Code))
at com/vaadin/server/VaadinService.lockSession(VaadinService.java:596)
at com/vaadin/server/VaadinService.findOrCreateVaadinSession(VaadinService.java:625)
at com/vaadin/server/VaadinService.findVaadinSession(VaadinService.java:495)
at com/vaadin/server/VaadinService.handleRequest(VaadinService.java:1365)
at com/vaadin/server/VaadinServlet.service(VaadinServlet.java:238)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668)
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1230)
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:779)
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:478)
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at com/tnt/vcm/application/security/LoginAuthorisationFilter.doFilter(LoginAuthorisationFilter.java:142)
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:960)
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1064)
at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:909)
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:200)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:312)
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:283)
at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com/ibm/io/async/AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1862(Compiled Code))

So, basically we need to know what the root cause of the deadlock and thread hang ? Anyone came across similar issue and can advise please ?

Thanks
[/quote]

3

We are seeing the same thing with a Websphere portal 7. Did you ever find a solution for this issue?

4

yes, there’s a bug in vaadin 7.1.15. So we were provided a patch release with the fix. I guess the fix will be available in the latest release 7.2.x as well.

Hope this helps.

5

Hi,
#14452
 is apparently the issue at hand.

6

Gulshan, Johannes, thank you both for responding.

After reading #14452, it seems it hasn’t been included in a new release. I’ll check the source of the latest version and if it isn’t I’ll try to extend VaadinService to include the change and see how it goes.

Again, thank you.

7

Yes, it’s yet to be fixed. I’ll try to get something done about it. Also, as a reminder, if any interested party has access to Bugfix Priority, prioritizing the ticket is a good way of getting it fixed ASAP.

8

Hi,

We are suffering a very similar issue with Vaadin 7.4.1 and Websphere 8.5.5. We checked the VaadinService class and it seems that the fix was included as described here (
#14452
) but I’m not sure if it works right in our case. Has somebody had the same problem?

We launch an action from our application and after some time we get a HTTP 500 error on the website and the following errors in the Websphere log.


[24/10/16 09:39:37:273 BST]
000000d3 ThreadMonitor W WSVR0605W: Thread “WebContainer : 13” (00000227) has been active for 616114 milliseconds and may be hung. There is/are 3 thread(s) in total in the server that may be hung.

at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:182)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:822)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:853)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1189)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:197)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:273)
at com.vaadin.server.VaadinService.lockSession(VaadinService.java:623)
at com.vaadin.server.VaadinService.findOrCreateVaadinSession(VaadinService.java:659)
at com.vaadin.server.VaadinService.findVaadinSession(VaadinService.java:524)
at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1402)
at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:350)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1071)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:914)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:283)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture$1.run(AsyncChannelFuture.java:205)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1864)


[24/10/16 10:07:59:963 BST]
00000227 ThreadMonitor W WSVR0606W: Thread “WebContainer : 13” (00000227) was previously reported to be hung but has completed. It was active for approximately 2318805 milliseconds. There is/are 0 thread(s) in total in the server that still may be hung.

Thanks in advance.

Regards,
Antonio.

9

I’m seeing similar hung threads on WebSphere Portal 8.5 (running on WAS 8.5).

We’re also having apparently random “invalid JSON” messages when an ajax request returns an empty response, which I think is related to the session being closed and/or the UI being invalidated before it should be.

Any ideas how this kind of thing could be resolved?

Thanks.

10

Hi Hector,

In our case we found that we were executing very heavy database queries so we changed them and indexed the database in a different way. I’m not sure if it helps you, sorry.

Regards,
Antonio.

11

Thank you, Antonio.

We don’t use a database but we use a lot of web services. I’m not quite sure what could cause the deadlock, if it’s the framework or something in WebSphere Portal.

I’m having problems with this since the deadlocks have been reported by IBM support while trying to replicate sessions.

"As you can see the javacore is full with deadlocks when trying to replicate sessions, the responsible for this lock is the Vaadin code:
com/vaadin/server/VaadinSession.writeObject

I have done some investigation and the customers were able to solve the problem after contacting the Vaadin Support and receiving a fix, but you need to check with them if this is your case or if you just need to change your code."

12

Hi everyone,
I’ve got same problem with WebSphere Portal 8 + Was 8.0.0.9 + Vaadin 7.6.3.
Using “IBM thread dump analyzer” i can see some thread reporting problem below. Any Suggestion?

at sun/misc/Unsafe.park(Native Method)
at java/util/concurrent/locks/LockSupport.park(LockSupport.java:182(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:822(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:853(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1189(Compiled Code))
at java/util/concurrent/locks/ReentrantLock$NonfairSync.lock(ReentrantLock.java:197(Compiled Code))
at java/util/concurrent/locks/ReentrantLock.lock(ReentrantLock.java:273(Compiled Code))
at com/vaadin/server/VaadinService.lockSession(VaadinService.java:626)
at com/vaadin/server/VaadinService.findOrCreateVaadinSession(VaadinService.java:662)
at com/vaadin/server/VaadinService.findVaadinSession(VaadinService.java:527)
at com/vaadin/server/VaadinService.handleRequest(VaadinService.java:1403)
at com/vaadin/server/VaadinPortlet.handleRequest(VaadinPortlet.java:527)
at com/vaadin/server/VaadinPortlet.serveResource(VaadinPortlet.java:609)
at com/ibm/ws/portletcontainer/invoker/impl/PortletFilterChainImpl.doFilter(PortletFilterChainImpl.java:153(Compiled Code))
at com/ibm/wps/engine/el/init/AttributeCopyFilter.doFilter(AttributeCopyFilter.java:185(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletFilterChainImpl.doFilter(PortletFilterChainImpl.java:146(Compiled Code))
at com/ibm/wps/resolver/iwidget/filter/IWidgetPortletFilter.doFilter(IWidgetPortletFilter.java:81(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletFilterChainImpl.doFilter(PortletFilterChainImpl.java:146(Compiled Code))
at com/ibm/wps/pe/pc/waspc/plm/GlobalPortletLoadMonitoringFilter.doFilter(GlobalPortletLoadMonitoringFilter.java:182(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletFilterChainImpl.doFilter(PortletFilterChainImpl.java:146(Compiled Code))
at com/ibm/wps/pe/pc/waspc/filter/impl/GlobalPortletFilter.doFilter(GlobalPortletFilter.java:192(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletFilterChainImpl.doFilter(PortletFilterChainImpl.java:146(Compiled Code))
at com/ibm/wps/pcm/scoping/filter/PCMScopingFilter.doFilter(PCMScopingFilter.java:114(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletFilterChainImpl.doFilter(PortletFilterChainImpl.java:146(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletServlet.doDispatch(PortletServlet.java:545(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletServlet.doDispatch(PortletServlet.java:332(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletServletCollaboratorChainImpl.doCollaborator(PortletServletCollaboratorChainImpl.java:82(Compiled Code))
at com/ibm/isclite/container/collaborator/PortletServletCollaborator.doDispatch(PortletServletCollaborator.java:143(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletServletCollaboratorChainImpl.doCollaborator(PortletServletCollaboratorChainImpl.java:74(Compiled Code))
at com/ibm/ws/portletcontainer/rrd/RRDServerPortletServletCollaborator.doDispatch(RRDServerPortletServletCollaborator.java:60(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletServletCollaboratorChainImpl.doCollaborator(PortletServletCollaboratorChainImpl.java:74(Compiled Code))
at com/ibm/ws/portletcontainer/cache/CacheCollaborator.doDispatch(CacheCollaborator.java:74(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletServletCollaboratorChainImpl.doCollaborator(PortletServletCollaboratorChainImpl.java:74(Compiled Code))
at com/ibm/wps/pe/pc/waspc/core/impl/PortletServletCollaboratorImpl.doDispatch(PortletServletCollaboratorImpl.java:121(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletServletCollaboratorChainImpl.doCollaborator(PortletServletCollaboratorChainImpl.java:74(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletServlet.dispatch(PortletServlet.java:208(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletServlet.service(PortletServlet.java:165(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:668(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1230(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:779(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:478(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178(Compiled Code))
at com/ibm/wsspi/webcontainer/servlet/GenericServletWrapper.handleRequest(GenericServletWrapper.java:122(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:79(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:960(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1064(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebAppRequestDispatcher.dispatch(WebAppRequestDispatcher.java:1385(Compiled Code))
at com/ibm/ws/webcontainer/webapp/WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:194(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletInvokerImpl.invoke(PortletInvokerImpl.java:212(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletInvokerCollaboratorChainImpl.doCollaboratorServeResource(PortletInvokerCollaboratorChainImpl.java:98(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletInvokerCollaboratorChainImpl.doCollaborator(PortletInvokerCollaboratorChainImpl.java:54(Compiled Code))
at com/ibm/ws/portletcontainer/cache/PortletInvokerCacheCollaborator.doServeResource(PortletInvokerCacheCollaborator.java:75(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletInvokerCollaboratorChainImpl.doCollaboratorServeResource(PortletInvokerCollaboratorChainImpl.java:92(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletInvokerCollaboratorChainImpl.doCollaborator(PortletInvokerCollaboratorChainImpl.java:54(Compiled Code))
at com/ibm/ws/portletcontainer/ext/PortletInvokerPerformanceCollaborator.invoke(PortletInvokerPerformanceCollaborator.java:313(Compiled Code))
at com/ibm/ws/portletcontainer/ext/PortletInvokerPerformanceCollaborator.doInvoke(PortletInvokerPerformanceCollaborator.java:101(Compiled Code))
at com/ibm/ws/portletcontainer/ext/PortletInvokerPerformanceCollaborator.invokePMI(PortletInvokerPerformanceCollaborator.java:163(Compiled Code))
at com/ibm/ws/portletcontainer/ext/PortletInvokerPerformanceCollaborator.doInvoke(PortletInvokerPerformanceCollaborator.java:91(Compiled Code))
at com/ibm/ws/portletcontainer/ext/PortletInvokerPerformanceCollaborator.doServeResource(PortletInvokerPerformanceCollaborator.java:349(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletInvokerCollaboratorChainImpl.doCollaboratorServeResource(PortletInvokerCollaboratorChainImpl.java:92(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletInvokerCollaboratorChainImpl.doCollaborator(PortletInvokerCollaboratorChainImpl.java:54(Compiled Code))
at com/ibm/ws/portletcontainer/invoker/impl/PortletInvokerImpl.serveResource(PortletInvokerImpl.java:135(Compiled Code))
at com/ibm/ws/portletcontainer/PortletContainerImpl.doServeResource(PortletContainerImpl.java:447(Compiled Code))
at com/ibm/ws/portletcontainer/ResourcePortletContainerInvokerCollaboratorChainImpl.doCollaborator(ResourcePortletContainerInvokerCollaboratorChainImpl.java:62(Compiled Code))
at com/ibm/ws/portletcontainer/cache/CacheInvokerCollaborator.doServeResource(CacheInvokerCollaborator.java:101(Compiled Code))
at com/ibm/ws/portletcontainer/ResourcePortletContainerInvokerCollaboratorChainImpl.doCollaborator(ResourcePortletContainerInvokerCollaboratorChainImpl.java:53(Compiled Code))
at com/ibm/ws/portletcontainer/PortletContainerImpl.servePortletResource(PortletContainerImpl.java:418(Compiled Code))
at com/ibm/ws/portletcontainer/pcinvoker/PortletInvokerImpl$4.run(PortletInvokerImpl.java:185(Compiled Code))
at java/security/AccessController.doPrivileged(AccessController.java:327(Compiled Code))
at com/ibm/ws/portletcontainer/pcinvoker/PortletInvokerImpl.invokeServeResource(PortletInvokerImpl.java:181(Compiled Code))
at com/ibm/wps/pe/pc/waspc/core/impl/PortletInvokerImpl$2.invoke(PortletInvokerImpl.java:107(Compiled Code))
at com/ibm/wps/pe/pc/waspc/core/impl/PortletInvokerImpl.invoke(PortletInvokerImpl.java:182(Compiled Code))
at com/ibm/wps/pe/pc/waspc/core/impl/PortletInvokerImpl.invokeServeResource(PortletInvokerImpl.java:104(Compiled Code))
at com/ibm/wps/pe/pc/waspc/PortletContainerImpl.servePortletResource(PortletContainerImpl.java:201(Compiled Code))
at com/ibm/wps/pe/pc/PortletContainerImpl.servePortletResource(PortletContainerImpl.java:137(Compiled Code))
at com/ibm/wps/engine/extension/render/ResourceServingRenderer.render(ResourceServingRenderer.java:128(Compiled Code))
at com/ibm/wps/engine/extension/render/ResourceServingRenderer.render(ResourceServingRenderer.java:85(Compiled Code))
at com/ibm/wps/engine/phases/WPResourceRenderPhase.processRendering(WPResourceRenderPhase.java:162(Compiled Code))
at com/ibm/wps/engine/phases/WPResourceRenderPhase.processRendering(WPResourceRenderPhase.java:108(Compiled Code))
at com/ibm/wps/engine/phases/WPBaseRenderPhase.execute(WPBaseRenderPhase.java:250(Compiled Code))
at com/ibm/wps/state/phases/AbstractRenderPhase.next(AbstractRenderPhase.java:106(Compiled Code))
at com/ibm/wps/engine/phases/WPAbstractRenderPhase.next(WPAbstractRenderPhase.java:100(Compiled Code))
at com/ibm/wps/engine/phases/WPResourceRenderPhase.next(WPResourceRenderPhase.java:242(Compiled Code))
at com/ibm/wps/engine/Servlet.callPortal(Servlet.java:315(Compiled Code))
at com/ibm/wps/engine/Servlet.doGet(Servlet.java:686(Compiled Code))
at com/ibm/wps/engine/Servlet.doPost(Servlet.java:852(Compiled Code))
at javax/servlet/http/HttpServlet.service(HttpServlet.java:595(Compiled Code))
at com/ibm/wps/engine/Servlet.doFilter(Servlet.java:411(Compiled Code))
at com/ibm/wps/resolver/servlet/ContentHandlerCleanup.doFilter(ContentHandlerCleanup.java:780(Compiled Code))
at com/ibm/wps/resolver/servlet/AbstractFilter.doFilter(AbstractFilter.java:96(Compiled Code))
at com/ibm/wps/engine/Servlet.service(Servlet.java:1388(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1230(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:779(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:478(Compiled Code))
at com/ibm/ws/webcontainer/servlet/ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:97(Compiled Code))
at com/ibm/wps/engine/ExtensionFilter$ExtendedFilterChain.doFilter(ExtensionFilter.java:110(Compiled Code))
at com/ibm/wps/engine/ExtensionFilter.doFilter(ExtensionFilter.java:197(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/wps/engine/SupportedLocalesFilter.doFilter(SupportedLocalesFilter.java:113(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/wps/engine/ExtendedLocaleFilter.doFilter(ExtendedLocaleFilter.java:113(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/wps/resolver/friendly/servlet/FriendlySelectionFilter.doFilter(FriendlySelectionFilter.java:191(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/wps/project/filter/ProjectIdFilter.doFilterWithoutProjectID(ProjectIdFilter.java:405(Compiled Code))
at com/ibm/wps/project/filter/ProjectIdFilter.doFilter(ProjectIdFilter.java:319(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/wps/services/preview/PreviewFilterImpl.doFilter(PreviewFilterImpl.java:351(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/wps/services/preview/PreviewFilterImpl.doFilter(PreviewFilterImpl.java:351(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/wps/mappingurl/impl/URLAnalyzer.doFilter(URLAnalyzer.java:443(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/wps/engine/VirtualPortalFilter.doFilter(VirtualPortalFilter.java:89(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/wps/resolver/servlet/ContentHandlerGzip.internalDoFilter(ContentHandlerGzip.java:704(Compiled Code))
at com/ibm/wps/resolver/servlet/ContentHandlerGzip.doFilter(ContentHandlerGzip.java:451(Compiled Code))
at com/ibm/wps/resolver/servlet/AbstractFilter.doFilter(AbstractFilter.java:96(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/wps/state/filter/StateCleanup.doFilter(StateCleanup.java:103(Compiled Code))
at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:91(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:960(Compiled Code))
at com/ibm/ws/webcontainer/filter/WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1064(Compiled Code))
at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:87(Compiled Code))
at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:914(Compiled Code))
at com/ibm/ws/webcontainer/WSWebContainer.handleRequest(WSWebContainer.java:1662(Compiled Code))
at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:200(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewRequest(HttpInboundLink.java:515(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.processRequest(HttpInboundLink.java:306(Compiled Code))
at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:277(Compiled Code))
at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214(Compiled Code))
at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113(Compiled Code))
at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175(Compiled Code))
at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
at com/ibm/io/async/AsyncChannelFuture$1.run(AsyncChannelFuture.java:205(Compiled Code))
at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1702(Compiled Code))

13

UPDATE: Recently we’ve moved to vaadin 7.7.12 but same problem occurred. Any Ideas ?

14

This is known issue in WebSphere, there is another thread where there is more discussion. Current understanding is that this cannot be eliminated at Vaadin side:


https://vaadin.com/forum/#!/thread/16531334/16531520

15

Fixed by inVaadin 7.7.27 and newer https://github.com/vaadin/framework/pull/12353