ConcurrentModificationException with load test

Hello,

we're running load tests on our application and are seeing the exception below. It happens intermittently: for a set of 4000 tests we have 200 errors, approximately. It seems to be a problem in Vaadin.

Server is a Tomcat, Java is 8.

Any ideas anyone?

Thanks in advance

SCHWERWIEGEND: Servlet.service() for servlet [de.abc.zyx.AppUI$Servlet] in context with path [/app-context] threw exception [com.vaadin.server.ServiceException: java.util.ConcurrentModificationException] with root cause
java.util.ConcurrentModificationException
       at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:711)
       at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:734)
       at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
       at com.vaadin.event.EventRouter.fireEvent(EventRouter.java:188)
       at com.vaadin.server.VaadinService.onVaadinSessionStarted(VaadinService.java:814)
       at com.vaadin.server.VaadinService.createAndRegisterSession(VaadinService.java:771)
       at com.vaadin.server.VaadinService.doFindOrCreateVaadinSession(VaadinService.java:724)
       at com.vaadin.server.VaadinService.findOrCreateVaadinSession(VaadinService.java:665)
       at com.vaadin.server.VaadinService.findVaadinSession(VaadinService.java:524)
       at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1402)
       at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:350)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
       at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
       at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
       at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
       at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
       at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
       at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       at java.lang.Thread.run(Thread.java:745)

SCHWERWIEGEND: Servlet.service() for servlet [de.abc.zyx.AppUI$Servlet] in context with path [/app-context] threw exception [com.vaadin.server.ServiceException: java.lang.NullPointerException] with root cause
java.lang.NullPointerException
       at de.abc.zyx.AppUI.prepareSubmenu(AppUI.java:87)
       at de.abc.zyx.AppUI.init(AppUI.java:69)
       at com.vaadin.ui.UI.doInit(UI.java:675)
       at com.vaadin.server.communication.UIInitHandler.getBrowserDetailsUI(UIInitHandler.java:214)
       at com.vaadin.server.communication.UIInitHandler.synchronizedHandleRequest(UIInitHandler.java:74)
       at com.vaadin.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:41)
       at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1408)
       at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:350)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
       at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
       at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
       at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
       at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
       at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
       at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
       at java.lang.Thread.run(Thread.java:745)

The CME looks like something to worry about. I'm not deeply familiar with the workings of VaadinService, but I suppose there could be a problem with synchronization in the EventRouter when firing those SessionInitEvents. If they are fired simultaneously from multiple threads during load testing, I suppose they could be a problem. Requests are otherwise synchronized on the sessions, and they are also locked on a new session in findOrCreateVaadinSession() that calls the methods eventually firing the events... The session handling looks a bit complex, so I hope there's no problem with how that works... I'd find it suprising if there's such a basic synchronization malfunctioning.

The latter exception is from your code, so maybe it's unrelated?
 

There's some locking asserts around that. You could try to enable asserts with -ea parameter to your server and see if they catch anything.

This Bug was already found. Have a look at EventRouter, Method fireEvent

public void fireEvent(EventObject event, ErrorHandler errorHandler) {
// It is not necessary to send any events if there are no listeners
if (listenerList != null) {

// Make a copy of the listener list to allow listeners to be added
// inside listener methods. Fixes #3605.

Ticket 3605: Patch to avoid concurrent modification exceptions in EventRouter

Hmm. If it was fixed already in 6.2, how are you still experiencing the problem... The ticket seems to deal with a typical cause of CMEs, when you add or delete to a collection while iterating over it. That's maybe different from your case. It's quite possible that load testing would find a locking problem that causes CMEs because of modifications from simultaneously running threads.

In any case, I recommend using the -ea parameter and see if you get more detailed information.

Same problem here. Is there any update on this issue? Since we're currently running on Vaadin 7.5.2 #3605 doesn't seem to fix this issue completely.

Current Setup: Tomcat 8, Java 8, Vaadin 7.5.2, SpringBoot 1.2.5

I tried running the load test and the problem occurs to me as well. The -ea parameter didn't seem to have any effect (if I used it correctly). It's hard to say anything without close debugging, which can be difficult to do for this kind of a problem. I hope the problem is not caused by JMeter testing, but I guess not.

I really suspect that there's some problem with the session locking, which seems to use a bit unusual pattern, and if so, the requests served in different threads could go through unlocked to the EventRouter, where they cause the CMEs.

Please create a ticket, and refer to this thread.

I'm currently working on a patch and making good progress, since I've fixed the issue in my case. The problem is, that the fix provided in #3605 only makes the ConcurrentModificationException more unlikely to occur but doesn't prevent ist completely. When the listenerList is copied by .toArray() a iterator object is still used. Adding a new Listener at the wrong time might cause a CME

I will create a ticket later that day

Thread is now mentioned in https://dev.vaadin.com/ticket/19153

I provided a possible fix as a patch based on Vaadin 7.5.7

Basically I changed the datatype of the listenerList from LinkedHashSet to ConcurrentSkipListSet

OK, great, thanks!

So your fix uses a thread-safe set class... I'd think that the real problem is that the thread lock is leaking, so using a thread-safe set is just fixing the symptom. Well, whoever looks into the issue probably considers it, so it's good to have your patch there as one solution or workaround anyhow.

Note that your patch contains a some unnecessary changes; I hope that you are using the correct code autoformatting rules for the Vaadin project. Oh, looks like the development instructions do not even mention that anywhere anymore... :-(

Yes you might be right. I unfortunately didn't had the time to dig into the thread-locking.

I searched for the autoformatting rules, but didn't found anything. If you have the values I will remake the patch :)

OK, looks like we haven't mentioned the coding conventions info in the GitHub repository info nor in the Contributing Code page. You can find the old and partially outdated version at https://dev.vaadin.com/wiki/CodingConventions. The needed VaadinCleanup.xml and VaadinJavaConventions.xml files can be found under the eclipse folder in the Vaadin Framework repository.

Marko Grönroos: OK, looks like we haven't mentioned the coding conventions info in the GitHub repository info nor in the Contributing Code page. You can find the old and partially outdated version at https://dev.vaadin.com/wiki/CodingConventions. The needed VaadinCleanup.xml and VaadinJavaConventions.xml files can be found under the eclipse folder in the Vaadin Framework repository.

Unfortunately, after doing the step-by-step instructions eclipse keeps formatting lines I didn't touch. Since the patch won't find the way into any official release, I guess it's applicable to extract relevant lines as long as there is no offical fix available. However I'm curious about an offical fix for this issue :)
 

Sure, no need to fix it, it's good the have the patch there for reference, even though it probably won't be used as it is now.