Scalable web applications

This might sound simplistic, but many times it might be enough to just add as much memory as possible to the server. Modern servers and server operating systems have support for hundreds of gigabytes of physical memory. For instance, again in our example, if the size of a session would be 0.5MB and we had 5000 concurrent users, the memory need would be about 28GB.

You also have to take care that your application server is configured to reserve enough memory. For example, the default heap size for Java is typically 2GB and for example Apache Tomcat will not reserve more memory if you do not ask it to do it with -Xmx JVM argument. You might need a special JVM for extremely large heap sizes. We used the following Java virtual machine parameters in our tests:

The parameters -Xms and -Xmx are for setting the minimum and the maximum heap size for the server (5 GB in the example), the -Xss is used to reduce the stack size of threads to save memory (typically the default is 1MB for 64bit Java) and the -server option tells JVM that the Java process is a server.

The biggest culprit for the big session size in the inventory application is the container (BeanItemContainer) which is filled with all items of the database. Containers, and especially the built in fully featured BeanItemContainer, are typically the most memory hungry parts of Vaadin applications. One can either reduce the number of items loaded in the container at one time or use some lightweight alternatives available from Vaadin Directory (vaadin.com/directory) such as Viritin, MCont, or GlazedLists Vaadin Container. Another approach is to release containers and views to the garbage collection e.g. every time the user switches into another view, though that will slightly increase the CPU load since the views and containers have to be rebuilt again, if the user returns to the view. The feasibility of this option is up to your application design and user flow –usually it’s a good choice.

Since every session in the memory reserves it for as long as it stays there, the shorter the session timeout is, the quicker the memory is freed. Assuming that the average usage time is much shorter than the session timeout, we can state that halving the session timeout approximately halves the memory need, too. Another way to reduce the session’s time in the memory could be instructing users to logout after they are done.

The session of a Vaadin application is kept alive by requests (such as user interactions) made from the client to the server. Besides user interaction, the client side of Vaadin application sends a heartbeat request into the server side, which should keep the session alive as long as the browser window is open. To override this behaviour and to allow closing idle sessions, we recommend that the closeIdleSessions parameter is used in your servlet configuration. For more details, see chapter Application Lifecycle in the Book of Vaadin.

If there is not enough memory, for example if there is no way to reduce the size of a session and the application needs a very long session timeout, then there is only one option left: clustering. We will discuss clustering later in the Out of CPU chapter since clustering is more often needed for increasing CPU power.

If you are not absolutely sure about the origin of the high CPU usage, it is always good to verify it with a performance profiling tool. There are several options for profiling, such as JProfiler, XRebel, and Java VisualVM. We will use VisualVM in this case since it comes freely with every (Oracle’s) JDK since the version 1.5.

Our typical procedure goes like this: 1. Deploy your webapp and start your server, 2. Start VisualVM and double click your server’s process (“e.g. Tomcat (pid 1234)”) on the Applications tab (see Figure 8), 3. Start your load test script with, for instance, 100 concurrent users, 4. Open the Sampler tab to see where the CPU time is spent, 5. Use the filter on the bottom to show the CPU usage of your application (e.g. “biz.mydomain.projectx”) and possible ORM (Object-relational mapping) framework (e.g. “org.hibernate”) separately.

Typically, only a small part (e.g. 0.1 - 2 %) of CPU time is spent on the classes of your webapp, if your application does not contain heavy business logic. Also, CPU time spent on the classes of Vaadin should be very small (e.g. 1%). You can be relaxed about performance bottlenecks of your code if the most time (>90%) is spent on application server’s classes (e.g. “org.apache.tomcat”).

Unfortunately, quite often database functions and ORM frameworks take a pretty big part of CPU time. We will discuss how to tackle heavy database operations in the Database chapter below.

Figure 8: Profiling CPU usage of our inventory application with Java VisualVM

Some application servers (at least Tomcat and Wildfly) allow you to use native (operating system specific) implementation of certain libraries. For example, The Apache Tomcat Native Library gives Tomcat access to certain native resources for performance and compatibility. Here we didn’t test the effect of using native libraries instead of standard ones. With little online research, it seems that the performance benefit of native libraries for Tomcat is visible only if using secured https connections.

We recommended above not to strain a server more than 50% of its total CPU capacity. The reason was that above that level, a garbage collection pause tends to freeze the server for too long a time. That is because it typically starts not before almost all of the available heap is already spent and then it does the full collection. Fortunately, it is possible to tune the Java garbage collector so that it will do its job in short periods. With little online study, we found the following set of JVM parameters for web server optimized garbage collection

The first parameter prevents Java from using its default garbage collection strategy and makes it use CMS (concurrent-mark-sweep) instead. The second parameter tells at which level of “occupancy” the garbage collection should be started. The value 70% for the second parameter is typically a good choice but for optimal performance it should be chosen carefully for each environment e.g. by trial and error.

The CMS collector should be good for heap sizes up to about 4GB. For bigger heaps there is the G1 (Garbage first) collector that was introduced in JDK 7 update 4. G1 collector divides the heap into regions and uses multiple background threads to first scan regions that contain the most of garbage objects. Garbage first collector is enabled with the following JVM parameter.

If you are using Java 8 Update 20 or later, and G1, you can optimize the heap usage of duplicated Strings (i.e. their internal char[] arrays) with the following parameter.

We have now arrived at the point where a single server cannot fulfill our scalability needs whatever tricks we have tried. If a single server is not enough for serving all users, obviously we have to distribute them to two or more servers. This is called clustering.

Clustering has more benefits than simply balancing the load between two or more servers. An obvious additional benefit is that we do not have to trust a single server. If one server dies, the user can continue on the other server. In worst case, the user loses her session and has to log in again, but at least she is not left without the service. You probably have heard the term “session replication” before. It means that the user’s session is copied into other servers (at least into one other) of the cluster. Then, if the server currently used by the user goes down, the load balancer sends subsequent requests to another server and the user should not notice anything.

We will not cover session replication in this article since we are mostly interested in increasing the ability to serve more and more concurrent users with our system. We will show two ways to do clustering below, first with Apache WebServer and Tomcats and then with the Wildfly Undertow server.

Traditionally Java web application clustering is implemented with one Apache Web Server as a load balancer and 2 or more Apache Tomcat servers as nodes. There are a lot of tutorials online, thus we will just give a short summary below.

There are several other options (free and commercial ones) for the load balancer, too. For example, our customers have used at least F5 in several projects.

Using Wildfly Undertow as a load balancer has several advantages over Apache Web Server. First, as Undertow comes with your WildFly server, there is no need to install yet another software for a load balancer. Then, you can configure Undertow with Java (see Figure 8) which minimizes the error prone conf file or xml configurations. Finally, using the same vendor for application servers and for a load balancer reduces the risk of intercompatibility issues. The clustering setup for Wildfly Undertow is presented below. We are using sticky session management to maximize performance.

Figure 9: Simple load balancer with two nodes and sticky sessions.

We already discussed shortly, how to use Java VisualVM for finding CPU bottlenecks. These same instructions also apply for finding out at what level the database consumes the performance. Typically you have several Repository-classes (e.g. CustomerRepository) in your web application, used for CRUD (create, read, update, delete) operations (e.g. createCustomer). Commonly your repository implementations either extend Spring’s JPARepository or use javax.persistence.EntityManager or Spring’s Datasource for the database access. Thus, when profiling, you will probably see one or more of those database access methods in the list of methods that are using most of your CPU’s capacity.

According to our experience, one of the bottlenecks might be that small database queries (e.g. findTaskForTheDay) are executed repeatedly instead of doing more in one query (e.g. findTasksForTheWeek). In some other cases, it might be vice versa: too much information is fetched and only part of it is used (e.g. findAllTheTasks). A real life example of the latter happened recently in a customer project, where we were able to a gain significant performance boost just by using JPA Projections to leave out unnecessary attributes of an entity (e.g. finding only Task’s name and id) in a query.

After performance profiling, you have typically identified a few queries that are taking a big part of the total CPU time. A part of those queries might be the ones that are relatively fast as a single query but they are just done hundreds or thousands of times. Another part of problematic queries are those that are heavy as is. Moreover, there is also the N+1 query problem, when, for example, a query for fetching a Task entity results N more queries for fetching one-to-many members (e.g. assignees, subtasks, etc.) of the Task.

The queries of the first type might benefit from combining to bigger queries as discussed in the previous subchapter (use findTasksForTheWeek instead of findTaskForTheDay). I call this approach custom caching. This approach typically requires changes in your business logic too: you will need to store (cache) yet unneeded entities, for example in a HashMap or List and then handle all these entities sequentially.

The queries of the second type are typically harder to optimize. Typically slow queries can be optimized by adding a certain index or changing the query logic into a little bit different form. The difficult part is to figure out what exactly makes the query slow. I recommend using a logging setting that shows the actual sql query made in your log file or console (e.g. in Hibernate use show_sql=true). Then you can take the query and run it against your database and try to vary it and see how it behaves. You can even use the EXPLAIN keyword to ask MySQL or PostgreSql (EXPLAIN PLAN FOR in Oracle and SHOWPLAN_XML in SQL Server) to explain how the query is executed, what indexes are used etc.

The N+1 queries can be detected by analysing the executed sqls in the log file. The first solution for the issue is redesigning the problematic query to use appropriate join(s) to make it fetch all the members in a single sql query. Sometimes, it might be enough to use FetchType.EAGER instead of LAZY for the problematic cases. Yet another possibility could be your own custom caching as discussed above.

According to Oracle’s Java EE Tutorial: a second-level cache is a local store of entities managed by the persistence provider. It is used to improve the application performance. A second-level cache helps to avoid expensive database queries by keeping frequently used entities in the cache. It is especially useful when you update your database only by your persistence provider (Hibernate or Eclipselink), you read the cached entities much more often than you update them, and you have not clustered your database.

There are different second-level cache vendors such as EHCache, OSCache, and SwarmCache for Hibernate. You can find several tutorials for these online. One thing to keep in mind is that the configuration of, for example, EHCache varies whether you use Spring or not. Our experience of the benefits of second-level caches this far is that in real world applications the benefits might be surprisingly low. The benefit gain depends highly on how much your application uses the kind of data from the database that is mostly read-only and rarely updated.

There are two common options for clustering or replication of the database: master-master replication and master-slave replication. In the master-master scheme any node in the cluster can update the database, whereas in the master-slave scheme only the master is updated and the change is distributed to the slave nodes right after that. Most relational database management systems support at least the master-slave replication. For instance, in MySql and PostgreSQL, you can enable it by few configuration changes and by granting the appropriate master rights for replication. You can find several step-by-step tutorials online by searching with e.g. the keywords “postgresql master slave replication”.

When looking back to the first figure (Figure 1) of the article, you might wonder what kind of database solutions the world’s biggest web application’s use? Most of them use some relation database, partly, and have a NoSQL database (such as Cassandra, MongoDB, and Memcached) for some of the functionality. The big benefit of many NoSQL solutions is that they are typically easier to cluster, and thus help one to achieve extremely scalable web applications. The whole topic of using NoSQL is so big that we do not have the possibility to discuss it in this article.