Reopen: What may cause AbstractCommunicationManager.readRequest() hanged?

Hello,

I have a problem (as below) on AbstractCommunicationManager.readRequest() and have no idea how to resolve it. It is possible for Vaadin core team to provide some clues for me?

My deployment envr. is Redhat 64bit + Tomcat 6.0.18 + Vaadin 6.4.6. Frequently I would meet such a problem that the progress icon doesn’t disappear after I click buttons (Occasionally in my dev. envr. Win7 + Tomcat 7.0 + Vaadin 6.4.6). After a long time I can receive a timeout exception:

java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:466)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:435)
at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:496)
at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:116)
at org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:322)
at org.apache.coyote.Request.doRead(Request.java:427)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:286)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:407)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:309)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:198)
at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.readRequest(AbstractCommunicationManager.java:1199)
at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.handleVariables(AbstractCommunicationManager.java:1008)
at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.doHandleUidlRequest(AbstractCommunicationManager.java:590)
at com.vaadin.terminal.gwt.server.CommunicationManager.handleUidlRequest(CommunicationManager.java:266)
at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.service(AbstractApplicationServlet.java:476)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:243)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:201)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:163)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:108)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:556)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:402)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:249)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:267)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:245)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:260)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Because there is no my code in the exception stack, I modified the core code of VAADIN - com.vaadin.terminal.gwt.server.AbstractCommunicationManager.java to track what may be received at AbstractCommunicationManager.readRequest():


    private static String readRequest(Request request) throws IOException {

        int requestLength = request.getContentLength();

        byte[] buffer = new byte[requestLength]
;
        InputStream inputStream = request.getInputStream();

        int bytesRemaining = requestLength;
        while (bytesRemaining > 0) {
            int bytesToRead = Math.min(bytesRemaining, MAX_BUFFER_SIZE);
            [b]
System.out.println("Len:" + requestLength+" Remain:" + bytesRemaining + " Read:" + bytesToRead);
[/b]
            int bytesRead = inputStream.read(buffer, requestLength
                    - bytesRemaining, bytesToRead);
            if (bytesRead == -1) {
                break;
            }
            [b]
System.out.println("Buffer:" + new String(buffer, "utf-8"));
[/b]
            bytesRemaining -= bytesRead;
        }

        String result = new String(buffer, "utf-8");

        return result;
    }

The log for correct execution is :
Len:5 Remain:5 Read:5
Buffer:init
Len:90 Remain:90 Read:90
Buffer:67f303a0-1a28-41f3-929d-101c2159b74d614PID0heighti1362PID0widthitruePID8stateb
Len:112 Remain:112 Read:112
Buffer:67f303a0-1a28-41f3-929d-101c2159b74d481PID47positionxi196PID47positionyi-1PID51citruePID47closeb
Len:55 Remain:55 Read:55
Buffer:67f303a0-1a28-41f3-929d-101c2159b74dtruePID64stateb

The log for the exception above is:
Len:5 Remain:5 Read:5
Buffer:init
Len:90 Remain:90 Read:90
Buffer:e1a29d7f-dcd1-410d-af2c-69a7f52a3560614PID0heighti1362PID0widthitruePID8stateb
Len:99 Remain:99 Read:99
Buffer:e1a29d7f-dcd1-410d-af2c-69a7f52a3560481PID47positionxi196PID47positionyitruePID47closeb
Len:55 Remain:55 Read:55

I guess that the issue is caused by no input in request inputStream, so tomcat server has to wait till timeout.

Does someone know the root reason?

Thanks,
Watt

Are you by any chance testing with Internet Explorer and are using keep-alive on your server?

I tested IE6/8 and Firefox 3, and surely my server was alive as I can open another browser to access it.

When I have a test the vaadin online demo, occasionally the demo may report “Communication Problem” and “Internal Error” (see two screenshots attached). My test steps are to click features alternately in the menu tree.
11551.doc (299 KB)

Sure sounds like the IE-only problem described at
http://msgroups.net/microsoft.public.internetexplorer.general/XMLHttpRequest-POST-sometimes-fails-when-server-is-usi
.

Solution: Turn off connection keep-alive in the server.

Thank you a lot for your solution.

After I turn off keep-alive in the server, IE works well except one “Internal Error” reported and one reproduced problem. Anyway it’s impoved greatly.

Hello Arthur,

Did you try the solution -Turn off connection keep-alive in the server?
As I wrote days ago, I still have the same problems occasionally after turning of server keep-alive.

To be safe, I plan to use a watchdog monitoring
inputStream.read()
. After an empty body is detected, my application would just send back a JSON message to browser. Accordingly browser can stop displaying the progress icon, and so user knows he can continue his work.

Is there any JSON message can do that? If yes, may I know its format?

On another side, if browser received an incompleted JSON message from server, is it possible to discard the message and hide the progress icon?

Thanks,
Watt

I also received the error. This tends to happen only in the most glorious browser of all time.
Did you come up with a right solution?

SEVERE: Terminal error:
java.io.IOException
at org.apache.jk.common.JkInputStream.receive(JkInputStream.java:205)
at org.apache.jk.common.JkInputStream.refillReadBuffer(JkInputStream.java:265)
at org.apache.jk.common.JkInputStream.doRead(JkInputStream.java:183)
at org.apache.coyote.Request.doRead(Request.java:428)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:304)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:405)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:327)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.getRequestPayload(AbstractCommunicationManager.java:1
374)
at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.handleVariables(AbstractCommunicationManager.java:118
2)
at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.doHandleUidlRequest(AbstractCommunicationManager.java
:730)
at com.vaadin.terminal.gwt.server.CommunicationManager.handleUidlRequest(CommunicationManager.java:296)
at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.service(AbstractApplicationServlet.java:483)

It buged me for a long time, and there is no solution.

Thanks a lot for this thread. It helped us to resolve the random socket read errors we were recieving from past few weeks.

Our server setup includes BigIP load balancer → 2 apache(2.2) web servers → 2 Tomcat (7.0) servers

Issue: When using Vaadin application, we randomly recieve “Communication error” followed by IOException Socket read timeout.

Fix: As mentioned in this thread, we turned keep-alive OFF on apache servers and so far its going good. We do NOT see any more communication or socket read erros.

The IE issue described here is IE 6,7,8 but we were facing this issue with IE9 as well.

I was facing the same problem for a long time. Is there any other solution for this because turning off keep alive is more like workaround than a real solution. It slowes communication and also working with application is much more slower.