Push creating too many Atmosphere threads on Glassfish 4.1 on Solaris

General
1

Hi,

As consequence of problems I have encountered on deploying my Vaadin Application with Push enabled on Glassfish 3.1.2.2 running on Solaris SPARC with JDK 1.6.0_45 (64b) , problem I have stated on
https://vaadin.com/forum/-/message_boards?_19_topLink=recent-posts&_19_groupThreadsUserId=1402172#!/thread/7866320

I tried to deploy my app on Glassfish 4.1 running on Solaris SPARC with JDK 1.7.0_60 having Vaadin v.7.2.7.

Now the situatation is this.

Push is working but after a short time of application excecution I get java.lang.OutOfMemoryError: unable to create new native thread on my Glassfish log and then Glassfish stops working.

After investigating the problem using Visual VM I noticed that when Push is used, Glassfish 4.1 on Solaris creates too many threads of type Atmosphere-Scheduler-xxx and Atmosphere-Shared-AsyncOp-xxx. These threads are created and then fall on a WAITING status kind of DEAD lock and never die. And when the server reach the max number of threads (500 threads) then Glassfish stops working. The only thing I can do is to kill the java process of Glassfish.

I undestand that this seems an Atmosphere problem but as I do not use directly any of the Atmosphere Framework packaged but only through Vaadin Push, I thought that this is the right place to write for.

It is interesting to mention that testing the same application on Glassfish 4.1 on a Windows 8, JDK 1.7.0_67, everything seems to work fine.

Going through Visual VM I can see that I have only 4 threads of type Atmosphere-Scheduler-xxx and 200 threads of type Atmosphere-Shared-AsyncOp-xxx no matter how long I play with the application and Push.

I’ve also posted the problem on https://java.net/jira/browse/GRIZZLY-1694

Thanks

2

Hi, thanks for the report.

200 threads should indeed be the default maximum. Could you check whether setting the following servlet init parameters to some reasonable integer value will keep the thread count below the limit:


org.atmosphere.cpr.broadcaster.maxProcessingThreads
org.atmosphere.cpr.broadcaster.maxAsyncWriteThreads

3

I’m actually using Servlet 3.0 and my Servlet class inside my UI class is as follow:

[code]
@WebServlet(value = “/*”, asyncSupported = true, initParams={@WebInitParam(name=“org.atmosphere.cpr.broadcaster.maxProcessingThreads”, value=“10”),
@WebInitParam(name=“org.atmosphere.cpr.broadcaster.maxAsyncWriteThreads”, value=“10”)})
@VaadinServletConfiguration(productionMode = true, ui = CommandCenterUI.class, widgetset=“us.icitap.cc.widgetset.CommandcenterWidgetset”)
public static class Servlet extends VaadinServlet implements SessionInitListener, SessionDestroyListener{

       ...........

       // code
       .............

}

[/code]But it doesn’t make any difference.

Actually I just took a look to the Glassfish configuration of Thread Pool and I notice that on Glassfish 4.1 official realease, the class used for both http-thread-pool and thread-pool-1 is org.glassfish.grizzly.threadpool.GrizzlyExecutorService.
Searching the GF modules folder I really can not find where this class is.

While on GF 3.2.1.2 the class used for both http-thread-pool and thread-pool-1 is: com.sun.grizzly.http.StatsThreadPool.
I can easly track this class to the jar file grizzly-http.jar located on the modules folder of GF 3.2.1.2

I’m still trying to undestand if the problem is here and how this thing is related to my issue

EDIT: GF 3.2.1.2 is actually GF 3.1.2.2, sorry the mistake

4

I just found out that class org.glassfish.grizzly.threadpool.GrizzlyExecutorService used to implement the thread pools http-thread-pool and thread-pool-1 is located under nucleus-grizzly-all.jar

5

After setting the atmosphere params as Johannes suggested I can see through Visual VM that the Atmosphere-Shared-AsyncOp-X threads and Atmosphere-Shared-DispatchOp-x threads are limited to 10 per each thread type, same as the limit I put to the parameters:
org.atmosphere.cpr.broadcaster.maxProcessingThreads
org.atmosphere.cpr.broadcaster.maxAsyncWriteThreads

But still the threads of type Atmosphere-Scheduler-xx continue to increase in number no matter if I interact with my app or not after I have opened it (the app). Of cource while using my app, these type of threads are created faster, while if I leave the app opened and not use it they are created slower.

I’ve also added some other atmosphere params to see if anything changes but with no success.

@WebInitParam(name=“org.atmosphere.cpr.broadcaster.shareableThreadPool”, value=“true”),
@WebInitParam(name=“org.atmosphere.cpr.broadcasterLifeCyclePolicy”, value=“IDLE”),
@WebInitParam(name=“org.atmosphere.cpr.recoverFromDestroyedBroadcaster”, value=“true”),
@WebInitParam(name=“org.atmosphere.cpr.Broadcaster.writeTimeout”, value=“30000”)

I’ve also update java to JDK 1.7_67 and tried Vaadin 7.3.2.

Nothing helped.

6

So, having my application (Vaadin 7.3.2, Push Enabled) configured with the atmosphere params as below:

@WebServlet(value = “/*”, asyncSupported = true, initParams={
@WebInitParam(name=“org.atmosphere.cpr.broadcaster.maxProcessingThreads”, value=“10”),
@WebInitParam(name=“org.atmosphere.cpr.broadcaster.maxAsyncWriteThreads”, value=“10”),
@WebInitParam(name=“org.atmosphere.cpr.broadcaster.shareableThreadPool”, value=“true”),
@WebInitParam(name=“org.atmosphere.cpr.broadcasterLifeCyclePolicy”, value=“IDLE”),
@WebInitParam(name=“org.atmosphere.cpr.recoverFromDestroyedBroadcaster”, value=“true”),
@WebInitParam(name=“org.atmosphere.cpr.Broadcaster.writeTimeout”, value=“30000”),
@WebInitParam(name=“org.atmosphere.cpr.Broadcaster.supportOutOfOrderBroadcast”, value=“true”)})

and deployed on a Glassfish 4.1, JDK .1.7.0_67 on both Windows 8.1 and Solaris SunOS 5.11 both 64bit, I notice:

On Solaris I have:

247 threads of type Atmosphere-Scheduler-xxx
10 threads of type Atmosphere-Shared-AsyncOp-xxx
10 threads of type Atmosphere-Shared-DispatcherOp-xxx

On WIndows I have:

4 threads of type Atmosphere-Scheduler-xxx
10 threads of type Atmosphere-Shared-AsyncOp-xxx
10 threads of type Atmosphere-Shared-DispatcherOp-xxx

Strange thing is that it was possible to limit the number of threads of type Atmosphere-Shared-AsyncOp-xxx and Atmosphere-Shared-DispatcherOp-xxx with the params:
@WebInitParam(name=“org.atmosphere.cpr.broadcaster.maxProcessingThreads”, value=“10”),
@WebInitParam(name=“org.atmosphere.cpr.broadcaster.maxAsyncWriteThreads”, value=“10”)

on both Windows and Solaris

BUT I can not figure out what does limit the number of threads of type Atmosphere-Scheduler-xxx to 247 on Solaris and to 4 on Windows