Queued Push messages failing occasionally, ideas?

Using any of 7.1.2, 7.1.3 and 7.1.5 , we are seeing a problem under certain circumstance with Push enabled where a queued JavaScriptFunction callback push message appears to be sent according to the js console, but the server never receives the request. I have confirmed that the server isn’t blocked on any particular thread

We are using JBoss and we see that we fallback to streaming, and push normally works fine. But under certain circumstances, reproducible by rapidly bringing up multiple tabs with the same page, the queued message problem kicks in and our page is broken. This failure situation comes up for us frequently when we are using TestBench with GhostDriver. We see it in Chrome as well - haven’t tried other browsers.

In our environment, we run in embedded mode where we wait for the embedded page to be loaded before we complete our Vaadin “view load”, with the following code in our UI class:


    @Override
    protected void init(VaadinRequest vaadinRequest) {
        //setup the communication channel between our shell app and this vaadin app
        JavaScript.getCurrent().addFunction("fooo.vaadin.sendWorkContext", new SendWorkContextJavaScriptFunction());

        //tell the shell that vaadin is currently ready to accept communication from it
        JavaScript.getCurrent().execute("vaadinBridge.onVaadinReady();");

        //Create a loading message while we wait for the shell to start communication with us
        VerticalLayout content = new VerticalLayout();
        content.setSizeFull();
        addStyleName("embedded-app");
    }

Here we init(), then wait for the parent page to load some “work context” details which our Vaadin app requires before it can load its actual content.
Normally this works fine event with push enabled (the js callback is queued waiting for push to init and then gets called), but in the error circumstance, the queued push request never comes back that runs the SendWorkContextJavaScriptFunction.call(), and our page never loads.

I’m curious if this is something anyone has thoughts on. I’m happy to create a Trac issue as well.

Trying to troubleshoot the issue I looked at the js console output (shown below) in the working and failing cases and also watched the servlet.service() calls to see what is different in the two cases. Unfortunately as seen below the only thing that appears different is that in the failing case the push message request is never received by the servlet.

Normal working case:


Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Starting application vaadin-app 0.js:6610
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Using theme: tetris 0.js:6610
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Server to client RPC call: 304:com.vaadin.shared.extension.javascriptmanager.ExecuteJavaScriptRpc.executeJavaScript([vaadinBridge.onVaadinReady();]
) 0.js:6610
Vaadin is ready to start shell communications push:262
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Making UIDL Request with params: 75080f3a-aed1-4822-9a60-a55a30fdddcf
[["304","com.vaadin.ui.JavaScript$JavaScriptCallbackRpc","call",["fooo.vaadin.sendWorkContext",[{--json-snipped--}]
]]] 0.js:6610
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Queuing push message: 75080f3a-aed1-4822-9a60-a55a30fdddcf[["304","com.vaadin.ui.JavaScript$JavaScriptCallbackRpc","call",
["fooo.vaadin.sendWorkContext",[{--json-snipped--}]
]]] 0.js:6610
Vaadin push loaded vaadinPush.js:23
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: vaadinPush.js loaded 0.js:6610
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Establishing push connection 0.js:6610
WebSocket connection to 'ws://localhost:8080/vaadin/PUSH/?v-uiId=12&v-csrfToken=75080f3a-aed1-4822-9a60-a55a30fdddcf&
X-Atmosphere-tracking-id=0&X-Atmosphere-Framework=1.0.14.vaadin4&X-Atmosphere-Transport=websocket&X-Atmosphere-TrackMessageSize=true&
X-Cache-Date=0&Content-Type=application/json;%20charset=UTF-8' failed: Unexpected response code: 501 vaadinPush.js:23
Websocket closed, reason: Connection was closed abnormally (that is, with no close frame being sent). vaadinPush.js:23
Websocket closed, wasClean: false vaadinPush.js:23
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Push connection closed, awaiting reconnection 0.js:6610
Websocket failed. Downgrading to Comet and resending vaadinPush.js:23
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Push connection using primary method (websocket) failed. Trying with streaming 0.js:6610
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Push connection established using streaming 0.js:6610
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Sending push message: 75080f3a-aed1-4822-9a60-a55a30fdddcf[["304","com.vaadin.ui.JavaScript$JavaScriptCallbackRpc","call",
["fooo.vaadin.sendWorkContext",[{--json-snipped--}]
]]] 0.js:6610
Thu Sep 19 10:42:14 GMT-500 2013 com.vaadin.client.VConsole
INFO: Received push message: for(;;);[{"changes" : [["change",{"pid":"303"},["54",{"id":"303","location":"http:\/\/localhost:8080\/solutions\/push#\/p5"}]
],
["change",{"pid":"305"},["55",{"id":"305","v":{"selected":"1"}},["tabs",{},["tab",{"caption":"Push","key":"1","selected":true},["56",{"id":"307","cached":true}]
],
["tab",{"caption":"Broadcast","key":"2"}]
]]]], "state":{"305":{"immediate":true,"height":"100.0%","width":"100.0%","styles":["minimal"]
},"312":{--json-snipped--}, 
"scriptDependencies": ["published:///SimpleToolTipExtension.js"]
, "timings":[64872, 3]
}] 

Failing case:


Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: Starting application vaadin-app 0.js:6610
Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: Using theme: tetris 0.js:6610
Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: Server to client RPC call: 446:com.vaadin.shared.extension.javascriptmanager.ExecuteJavaScriptRpc.executeJavaScript([vaadinBridge.onVaadinReady();]
) 0.js:6610
Vaadin is ready to start shell communications push:262
Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: Making UIDL Request with params: 75080f3a-aed1-4822-9a60-a55a30fdddcf[["446","com.vaadin.ui.JavaScript$JavaScriptCallbackRpc","call",
["fooo.vaadin.sendWorkContext",[{--json-snipped--}]
]]] 0.js:6610
Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: Queuing push message: 75080f3a-aed1-4822-9a60-a55a30fdddcf[["446","com.vaadin.ui.JavaScript$JavaScriptCallbackRpc","call",
["fooo.vaadin.sendWorkContext",[{--json-snipped--}]
]]] 0.js:6610
Vaadin push loaded vaadinPush.js:23
Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: vaadinPush.js loaded 0.js:6610
Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: Establishing push connection 0.js:6610
WebSocket connection to 'ws://localhost:8080/vaadin/PUSH/?v-uiId=26&v-csrfToken=75080f3a-aed1-4822-9a60-a55a30fdddcf&
X-Atmosphere-tracking-id=0&X-Atmosphere-Framework=1.0.14.vaadin4&X-Atmosphere-Transport=websocket&X-Atmosphere-TrackMessageSize=true&
X-Cache-Date=0&Content-Type=application/json;%20charset=UTF-8' failed: Unexpected response code: 501 vaadinPush.js:23
Websocket closed, reason: Connection was closed abnormally (that is, with no close frame being sent). vaadinPush.js:23
Websocket closed, wasClean: false vaadinPush.js:23
Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: Push connection closed, awaiting reconnection 0.js:6610
Websocket failed. Downgrading to Comet and resending vaadinPush.js:23
Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: Push connection using primary method (websocket) failed. Trying with streaming 0.js:6610
Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: Push connection established using streaming 0.js:6610
Thu Sep 19 10:52:36 GMT-500 2013 com.vaadin.client.VConsole
INFO: Sending push message: 75080f3a-aed1-4822-9a60-a55a30fdddcf[["446","com.vaadin.ui.JavaScript$JavaScriptCallbackRpc","call",
["fooo.vaadin.sendWorkContext",[{--json-snipped--}]
]]] 0.js:6610
Thu Sep 19 10:52:37 GMT-500 2013 com.vaadin.client.VConsole
INFO: New window width: 1743 0.js:6610
Thu Sep 19 10:52:37 GMT-500 2013 com.vaadin.client.VConsole
INFO: New window height: 1187 0.js:6610
Thu Sep 19 10:52:37 GMT-500 2013 com.vaadin.client.VConsole
INFO: New parent width: 1729 
..etc with no push message recieved and servlet.service() push request never called
 (eventually heartbeat request does go through successfully, but push request never completes so our UI never loads)

I have the same problem. I’m using vaadin v7.2.5 on jetty v9.1.5. The initial Websocket connection cannot be established thus, sending a login password and username makes the UI hang for around 60 seconds until atmosphere.js establishes a new connection (seems like an internal timeout).

This is, what i am seeing in the webbrowser console.

[code]
“Fri Aug 01 10:46:08 GMT+200 2014 com.vaadin.client.VConsole
INFO: Establishing push connection” 10.199.84.40
GET http://server/PUSH/ [0ms]

Firefox can’t establish a connection to the server at ws://server/PUSH/?v-uiId=4&v-csrfToken=f69e932b-0ac4-480e-a51b-a88e22a6fc9a&X-Atmosphere-tracking-id=0&X-Atmosphere-Framework=2.1.5.vaadin4-jquery&X-Atmosphere-Transport=websocket&X-Atmosphere-TrackMessageSize=true&X-Cache-Date=0&Content-Type=application/json;%20charset=UTF-8&X-atmo-protocol=true. vaadinPush.debug.js:11276
GET http://server/PUSH/ [HTTP/1.1 200 OK 63ms]

“Websocket closed, reason: Connection was closed abnormally (that is, with no close frame being sent).” vaadinPush.debug.js:13176
“Websocket closed, wasClean: false” vaadinPush.debug.js:13176
“Websocket failed. Downgrading to Comet and resending” vaadinPush.debug.js:13176
“Fri Aug 01 10:46:09 GMT+200 2014 com.vaadin.client.VConsole
INFO: Push connection closed” 10.199.84.40
“Fri Aug 01 10:46:09 GMT+200 2014 com.vaadin.client.VConsole
INFO: Push connection using primary method (websocket) failed. Trying with long-polling” 10.199.84.40
syntax error
[/code]However, the new connection then works for several minutes, after which i’m seeing on the console the following:

"Fri Aug 01 10:52:39 GMT+200 2014 com.vaadin.client.VConsole INFO: Reopening push connection" However, the problem goes away when i am using a windows 7 (IE, firefox 31, chrome latest) or debian 7 (iceweasel v10) in a vm which runs on my host system (Mac OS).

I can’t wrap my head around this problem. Unfortunately, in the webserver logs i do not see these abnormally closed websocket connections.

Other than ‘Tetris Dev’ my web application is a simple UI which displays a login window.
After entering the password the UI hangs which is caused by the not working push connection.

I have the same problem as @Tetris Dev.
Happy day scenario:

"Tue Sep 02 12:13:09 GMT+200 2014 com.vaadin.client.VConsole" INFO: Making UIDL Request with params: "Tue Sep 02 12:13:09 GMT+200 2014 com.vaadin.client.VConsole" INFO: Sending push message: "Tue Sep 02 12:13:09 GMT+200 2014 com.vaadin.client.VConsole" INFO: Received push message: "Tue Sep 02 12:13:09 GMT+200 2014 com.vaadin.client.VConsole" INFO: JSON parsing took 1ms app "Tue Sep 02 12:13:09 GMT+200 2014 com.vaadin.client.VConsole" INFO: Handling message from server" Not-so-happy day scenario:

[code]
Tue Sep 02 12:11:00 GMT+200 2014 com.vaadin.client.VConsole INFO: Making UIDL Request with params:
Tue Sep 02 12:11:00 GMT+200 2014 com.vaadin.client.VConsole INFO: Sending push message:
[one minute passes … UI unresponsive now]

Tue Sep 02 12:12:20 GMT+200 2014 com.vaadin.client.VConsole INFO: Sending heartbeat request…
Tue Sep 02 12:12:20 GMT+200 2014 com.vaadin.client.VConsole INFO: Received push message:
Tue Sep 02 12:12:20 GMT+200 2014 com.vaadin.client.VConsole INFO: JSON parsing took 1ms
Tue Sep 02 12:12:20 GMT+200 2014 com.vaadin.client.VConsole INFO: Handling message from server
[UI back to normal]

[/code]Can someone give any hint? The problem doesn’t exist when I access application from localhost.

So the heartbeat request somehow makes the push message to go through? I’ve seen this before… not just sure what the circumstances were. To me this sounds like there might be a proxy somewhere that’s not exactly push-compatible, especially as you mentioned that locally things work fine.

My environment:
Windows 7,
Vaadin 7.1.7
Push (default options)
Osgi/jetty/equinox

We are facing the exact same issue with vaadin 7.3.4. The UI just hangs for minute(s) after a connection was re-established. The issue occurs only to some of our customers.

The vaadin debug output says “INFO: Queuing push message: …”. I think the issue could be related to the apache reverse proxy we’re using, however it is very strange that not all of the users are affected by this problem.

Environment:
Vaadin 7.3.4
Karaf (OSGi), Jetty
Apache Reverse Proxy
Push enabled (Transport.STREAMING)

Are there any hints on this?

Thanks in advance

Florian

I had to disable Push Mode in whole application to get things to work. I suspect that Jetty doesn’t fully support Push Mode and/or Vaadin (or maybe at least some versions of Jetty).

This could be a possible solution if we wouldn’t need the push mode but we need it for our application to work.

Well, we had to rewrite or delete parts of code that used Push Mode. I will be happy to know if you find any other work-around solution.

I have investigated this problem a little bit further. First of all, the following configurations work:

  • Server always runs on a windows 7 in an enterprise network. The server is not registered with the enterprise windows-domain.
  • Mac OS in client network just works
  • Linux in client network works
  • Windows XP in client network works
  • Mac OS per Cisco VPN connected to the client network - have to set the proxy settings in firefox!! to “none” - then works, otherwise not (seems, that the initial http connection is correctly going through the vpn to the server, the push-connection however is tried to be established through the not-available proxy)
  • Windows XP, 7 in VM, per VPN works
  • Linux in VM per VPN works

The only thing that does not work is:

  • Windows 7 in client network within the enterprise-domain

it seems to me, that this is a problem of the domain-settings. However, the client IT is not willingly or not capable of resolving this problem.

Hope this list helps somebody to find a solution for their specific problem.