Session replication

General
1

Hi,
I’m trying to setup HTTP session replication on Tomcat.
The replication works only if I place the following code in Application.transactionEnd method:

HttpServletRequest request = (HttpServletRequest) transactionData;
Object attribute = request.getSession().getAttribute("com.vaadin.terminal.gwt.server.WebApplicationContext");
request.getSession().setAttribute("com.vaadin.terminal.gwt.server.WebApplicationContext", attribute);

Unfortunately under load (about 20 sessions simulated by JMeter) I get deadlocks in Tomcat DeltaManager serialization code:

Name: TP-Processor1
State: WAITING on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@287fc766 owned by: TP-Processor8
Total blocked: 1  Total waited: 2

Stack trace: 
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
org.apache.catalina.ha.session.DeltaSession.lock(DeltaSession.java:199)
org.apache.catalina.ha.session.DeltaSession.setAttribute(DeltaSession.java:596)
org.apache.catalina.ha.session.DeltaSession.setAttribute(DeltaSession.java:581)
org.apache.catalina.session.StandardSessionFacade.setAttribute(StandardSessionFacade.java:130)
com.haulmont.cuba.web.App.transactionEnd(App.java:510)
Name: TP-Processor8
State: BLOCKED on java.util.Collections$SynchronizedList@2a8cd366 owned by: TP-Processor1
Total blocked: 1  Total waited: 0

Stack trace: 
java.util.Collections$SynchronizedCollection.writeObject(Collections.java:1602)
sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
org.apache.catalina.ha.session.DeltaRequest$AttributeInfo.writeExternal(DeltaRequest.java:374)
org.apache.catalina.ha.session.DeltaRequest.writeExternal(DeltaRequest.java:272)
org.apache.catalina.ha.session.DeltaRequest.serialize(DeltaRequest.java:287)
org.apache.catalina.ha.session.DeltaManager.serializeDeltaRequest(DeltaManager.java:640)
org.apache.catalina.ha.session.DeltaManager.requestCompleted(DeltaManager.java:1144)
   - locked org.apache.catalina.ha.session.DeltaRequest@642e1d80
org.apache.catalina.ha.session.DeltaManager.requestCompleted(DeltaManager.java:1114)
org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:550)
org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:537)
org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:519)
org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:430)
org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:363)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
java.lang.Thread.run(Thread.java:619)

So I suspect that my attempt to kick the replication by setting session attribute is wrong.

Any help would be greatly appreciated.

Regards,
Konstantin

2

The problem solved.
I have removed explicit setAttribute code from Application.transactionEnd and followed recommendations from
this link
.
Now session serialization on request end takes place in one thread and there are no deadlocks even under heavy load.

Actually this was of course Tomcat’s problem, not Vaadin.

Regards,
Konstantin

3

Hi,

I guess that also synchronize block does the trick.


HttpServletRequest request = (HttpServletRequest) transactionData;
Object attribute = request.getSession().getAttribute("com.vaadin.terminal.gwt.server.WebApplicationContext");
synchronized(attribute) {
    request.getSession().setAttribute("com.vaadin.terminal.gwt.server.WebApplicationContext", attribute);
}

But how about performance. Have you done any measurements? I mean w/o replication and with replication. I have also tried Terracotta with Vaadin but ain’t done any performance test with it.

-pete-

4

Hi Pete,

I guess you are right, but I prefer running in one thread rather than additional synchronization.

As for performance. I have the following environment:

  • real-world application, real-world test case opening and closing several complex Vaadin windows
  • 2 virtual machines 1Ghz CPU, 768M RAM
  • Apache HTTP load-balancer on a separate machine
  • JMeter simulating 50 simultaneous users, total 5450 requests

Results:

Replication is ON
avg = 84 ms, median = 14 ms, 90% line = 188 ms
CPU consumption 40…100%, memory 90…150M

Replication is OFF
avg = 20 ms, median = 3 ms, 90% line = 50 ms
CPU consumption 40…60%, memory 60…70M

So I have about 4 times slower response time and a lot more resource consumption. As expected :slight_smile:

Regards,
Konstantin

5

BTW, GlassFish is faster than Tomcat and clustering is just there. :V


Warning: I am biased towards OpenSolaris and GlassFish.