Vaadin 7.1 Push : Load Testing

Vaadin 6 / 7 / 8
1

Hi,

I’ve had a problem on my mind for some time, and I guess it’s time to air it : will it be possible to perform loadtesting on Vaadin 7.1 when using push technology?

We’ve not had a chance to play with the 7.1 branch yet, or to dabble with push (other than some experiments with 6.x + DontPushOzoneLayer a year ago) - but we have managed to perform some rudimentary load-testing on Vaadin 6 using JMeter (not using push), following the
guide on the wiki
; fairly simple, in that you can record and replay via the HTTP Proxy inside JMeter. We’d do do some customization/parameterization of the HTTP requests, but that’s fairly easy (We’d probably generate the traffic using our QA Test tools, record that, and then mung the recorded JMeter test plan to introduce variables etc).

We want to use push : there many use cases where we’d need to be able to push notifications and updates to the clients. We’ve pushed them to back-burner for now, but I am thinking about them.

If, in 7.1, a push transport is based on WebSockets - how will it be possible to loadtest it? JMeter has a sampler for WebSockets (or somewone somewhere has a WebSocket sample for JMeter) - but how would you be able to record the traffic in the first place? I’m sure that that we’d able to use HTTP request/response with 7.1 - but there’s not point in load testing a transport that we’d not be using in a deployed environment. Has this been considered by the dev team? What would Vaadin’s approach to loadtesting a push based application - or am I worrying about nothing?

Cheers,

Charles.

(Tori just swallowed my original post, by the way: I’ve retyped this in a text editor to avoid losing it again. That’s twice that’s happened to me, and it’s really quite irritating! Time of swallowed post approximately 07.30 AM Weds May 29 (UTC), in case there’s anything in the logs)

2

Hi,

You are absolutely right - loadtesting push is not very easy right now - and yes, the dev team is aware of this problem. As you noticed, testing with JMeter is not very straightforward, even with the WebSocketSampler.

Though it’s in principle possible to make a JMeter test by basically taking a non-push test, and adapting it by looking at the push-specific traffic with e.g firebug (you should find that most of the traffic is the same as during non-push), this is not practical due to some dynamic parameters used by the push implementation.
I think we’ll probably end up making it possible to test with JMeter (e.g by starting your application in a special mode, similar to turning off the security key while loadtesting non-push). This is not done yet, though.

Meanwhile you might want to look into running phantomjs (note however that 1.9 does not yet support websockets, so it’ll fallback to streaming, 2.0 will support websockets).


tl;dr
 Loadtesting you push-based application is currently quite hard, but we’re aware of this problem!

I’m interested in any ideas/thoughts/feedback on this topic, so feel free to chime in, everyone!

Best Regards,
Marc

3

Thanks for the response Marc.

For me, I think making it possible to use JMeter is probably the best approach. I’m not particularly tied to JMeter, but - in the scheme of such things - it is fairly easy to understand and use, widely known, and of course open-source, with loads of third party tools.

Setting/Unsetting a flag in the application-under-test, recording using JMeter Proxy - then possibly transforming the test-case - is probably the best we can hope for.

Our loadtests are actually business oriented in nature. We will usually need to demonstrate that a particular configuration of hardware/software will support a given number of activities at the same time; our approach over the years (with our Swing client/server app - and using our own handrolled load-test tool) has been to take the projected number of concurrent users for a given site, their projected busines ‘volume’ over a year, work out how much per day (e.g. payments, invoices entered, customers created, accounts created), and then per hour, and try and reproduce a 20 minute slot of the pro-rata volume (with weighting to simulate peaks/high-load).

For what it’s worth, we need to be record and replay loadtests - it won’t be feasible for us to manually code/script them. The basic reason is that we’ll need to be creating business logic scripts - high level interaction use-cases - and then parameterize them, and play them with varying levels of load an interaction. These scripts will be need to be maintained over many releases of our product - and so they can’t be dependent on the low-level traffic.

We currently use QFTest to test our application - our basic plan is to create QFTest scripts (which are comparatively high level, and aren’t quite so closely tied to physical hierarchy of the DOM - which, in turn, means it’s less tricky to maintain them over multiple releases of our code), and then to replay those scripts via a proxy to record the traffic; hopefully we can then create a load-test script from the recorded traffic.

This worked quite well for us V6 using JMeter and HTTP request response; it’d be good to take the same basic approach when using push - although we’re open to alternatives.

Essentially, I was/am just checking to see that load-testing with push is actively being considered in Turku - we’re not likely to be doing anything with loadtesting/push for around another 6 months; I am glad to see they are!

Cheers,

Charles.

4

Hi Charles. As far as I know it’s impossible to record websocket traffic using JMeter proxy server. However websockets testing can be done via
JMeter-WebSocketSampler
 extension. See
WebSocket Testing With Apache JMeter
 guide for more details.