At Vaadin, we are often asked if it is possible to run performance tests on an application to prove that it fulfills its non-functional requirements.
In this blog series, we answer that question, introduce you to performance testing and help you get started with the process.
The first blog post provides a brief introduction to performance testing—more specifically, load testing. In subsequent posts, we’ll cover the technical details needed to create a test for a Vaadin application and which Vaadin internals are relevant from a load-testing standpoint. We conclude the series with a discussion on various load testing tools that help you to create and execute test scripts, with instructions for using JMeter and Gatling, two widely-used open-source tools.
By the end of the series, you should be able to perform performance testing of your application on your own.
What is performance testing?
Let’s start with a formal definition of performance testing: it’s a non-functional testing type that aims to identify—and where possible eliminate—stability, availability and scalability issues in software.
Performance testing is an umbrella term that includes scalability, load, endurance and stress testing. Together, these have the common goal of verifying all the non-functional requirements of the software. For example, capacity testing is used to carry out a capacity estimation, which includes an evaluation of the random-access memory (RAM) needed to function smoothly and the number of servers on which a new application is deployed. Load testing is often employed, among other methods, to estimate minimum requirements.
Scalability versus load testing
The terms “scalability testing” and “load testing” are often used interchangeably, because the methods and tools used are typically the same, but the purpose of each type is quite different. Scalability tests are conducted to ensure that the system response time remains the same, or increases only slightly, as the number of users increases. The objective is to determine the system’s ability to grow (scale). Load testing, on the other hand, is employed to verify whether the system can support the anticipated (low, average and peak) load. The results of both tests are beneficial and provide insight into the system’s capacity.
What is load testing?
Load testing simulates hundreds or thousands of simultaneous users to test a system’s performance. It investigates metrics, such as latency, central processing unit (CPU) utilization, throughput and memory utilization. For meaningful results, it’s essential to establish performance parameters before conducting the tests. For example, if you’re looking into response time, you should first define the maximum acceptable time against which to measure the system’s actual performance.
To ensure that your results are both valid and feasible, you need to test the most critical and the most common flows in the system. Checking the load and response time of a Contact Us page is less important than verifying that money-transferring operations work consistently under different loads in an online banking app. Examining non-critical paths does not add much value.
Ideally, your tests should be based on a real user interaction or on the insights of a person who knows the system well.
How load is simulated
There are different ways to generate load. The most straightforward is to assign a group of testers to simultaneously test the application manually in the UI. In this scenario, you rely on replaying captured user interaction with the system at the communications-protocol level.
An alternative is to use one of the many load testing tools available on the market. Some of the best known are LoadRunner, JMeter and Gatling. Free versions are available for the latter two, and both include a recorder tool that creates a test script by recording a user interacting with an application in a browser. This script is then executed to simulate and test load on the system under different conditions.
While load testing tools often provide statistical reports on response time (such as minimum, maximum and average), to accurately measure resource utilization on a server during a test run, you need a profiler. JProfiler is a popular choice that makes it easy to measure resource utilization.
Load testing process
Typically, the load testing process involves these steps:
- Decide the purpose and the scope of the test.
- Record a test script.
- Configure the recorded test script.
- Set up a testing environment.
- Run the test script against the environment and gather the results.
- Analyze the results
In this blog post series, the main focus is on steps 2, 3, and 5.
When performance-related problems are found in the tested application, you should do the necessary fixes to the application (based on the test results and possibly also profiling) and then iterate the whole process again.