Building an application for load testing
You should run load tests against an application that is as close as possible to your production build. Vaadin Flow applications can be built in two different ways—development or production. For load testing, you want to build the application in production mode.
To build and run the application in production mode, open a terminal window, change to the directory in which you extracted the zip file, and run the following command:
cd skeleton-starter-flow-14
mvn package jetty:run -Pproduction -DargLine="-Xmx8m"
The argLine option allows you to configure the JVM. In the example above, we are setting a max heap size of 8MB just to demonstrate more easily how to perform stress testing later in this article. In real-life projects, you should configure the JVM to mimic the production environment.
Check that you can use the application in your browser at http://localhost:8080:
Creating a new test plan in JMeter
Download, install, and run JMeter. You can find the binaries and instructions at https://jmeter.apache.org.
In the JMeter menu, select File > Templates…. Next, select Recording in the combo box and click Create:
Change the following configuration details and click Create:
- hostToRecord: localhost
- schemeToRecord: http
You should see a test plan with preconfigured elements for cookies management, a thread group, and a script recorder, among others. A test plan is equivalent to a project in an IDE. It contains all the configuration required to run the load tests. A thread group contains the sequence of requests you want to perform against your application. Each sequence is run in a separate thread.
Recording a test plan in a web browser
JMeter includes a proxy that allows you to record the interactions with a web application using a web browser. The proxy listens to port 8888. Configure a manual proxy using the network settings of your browser. For example, in Firefox the configuration dialog looks like this:
Before starting the script recording, check your local IP address. For example, in Linux systems run:
ifconfig | grep inet
You'll find a line with your local IP similar to this:
inet 192.168.1.35 netmask 0xffffff00 broadcast 192.168.1.255
Use the reported IP address in your browser (Firefox in this example) to request the application, for example: http://192.168.1.35:8080. You'll get a proxy error and won't see the application at this point.
In JMeter, select HTTP(S) Test Script Recorder in the left view and click the start button:
You'll see a message about an SSL certificate. Click OK. Enter a Transaction name, for example, "home" since we are testing the home page of the web application:
Go to the browser (Firefox) and request the application again. This time you should see the app. Enter a name and click Say hello in the webapp.
In the Recorder: Transactions Control popup window, click Stop to stop recording. In JMeter, you should see the results of the interaction when you select View Results Tree under HTTP(S) Test Script Recorder:
You might have a different number of requests (and generated names), depending on exactly how you interacted with the application. For example, if you hit ENTER or clicked Say hello before a change event was fired on the text field or closed the browser before stopping recording, you might end up with different requests.
You should also be able to see the recorded requests under Test Plan > Thread Group > Recording Controller:
These are the requests that are part of the script to run later.
Extracting data from HTTP responses
To be able to run the test, you have to modify the recorded requests to use the CSRF token that Vaadin Flow uses as a security mechanism. This token is returned to the client in the body of the response to the first request to the application (home-12 in the example) in a JSON object that you can inspect in the results tree. Vaadin Flow returns a different token every time the app is requested for the first time. This token is included on later AJAX calls to the server, for example, when you click a button. You can see the value used during the recording in the results tree by selecting the first request and searching for Vaadin-Security-Key:
To extract the value of the token, right-click the first request in the thread group and select Add > Post Processors > Regular Expression Extractor:
Fill in the following information:
- Name of the created variable: csrfToken
- Regular Expression: "Vaadin-Security-Key":"(.{36})"
- Template: $1$
This creates a variable with the name csrfToken that you can later use with the expression ${csrfToken}. The variable will contain the value of the first capturing group on the regular expression, that is, the 36-character string that contains the value of the CSRF token.
If you click the second request (home-17 in the example), you'll see that the request includes the value of the CSRF token in a JSON object:
You have to replace this "hard-coded" value in all subsequent requests with the expression that "injects" the value of the csrfToken variable extracted by the regular expression from the first request. You can edit the Body Data section directly to replace the content of the string with ${csrfToken} or you can use JMeter’s search and replace feature to do it all at once. For example:
Alternatively, with search and replace:
Running load tests
With the configuration in place, you can now run the test. Select View Results Tree in the thread group and click the start button in the toolbar:
You should see successful requests in the view:
Try simulating many concurrent users by adjusting the number of threads in the Thread Group as follows:
- Number of threads (users): 1000
- Ramp-up period (seconds): 5
- Deselect the Same user on each iteration checkbox
Before running the experiment, right-click Test Plan and select Add > Listener > Summary Report:
Run the test again by clicking the start button on the toolbar. On my machine, the results were a success (0% errors):
This could be different in your case, depending on the resources of your machine. A conclusion of this experiment is that the application can successfully serve 1000 users arriving concurrently (within a time window of 5 seconds) with as little Java heap memory as 8MB.
Running stress tests
Stress testing pushes the limits of your application to see what happens when it is put under pressure. It's a way to test the robustness of your application. You can also use it to find the spot at which you will need to increase computing resources, such as memory, processing units, database pool connections, etc.
Try modifying the number of concurrent users (threads) to a much higher level, for example to 3000, and run the tests again. Remember to clear the results and summaries before running the tests. You can do so by selecting Run > Clear All in the main menu (or use the keyboard shortcut shown in the menu).
On my machine, 3000 users were a little too many and I started to get errors in the responses at some point:
This is most likely a combination of not enough computing power and memory. Let's change the memory parameter (since it's the easiest thing to do in this example), stop the application and run it again with more heap memory:
mvn jetty:run -Pproduction -DargLine="-Xmx256m"
Clear the results and rerun the test. On my machine, that change did the trick:
Again, keep in mind that you might get very different results, depending on the hardware you use to run and test the application and the other programs you have running at the same time. In real-life scenarios, you should have separate environments for running the app and running the tests. For example, you can have a test server where you deploy and run your test application and use your development machine to run JMeter and execute the tests.
JMeter includes many more features than those we explored here. I encourage you to take a look at their documentation to learn more about it.