How to reduce memory consumption of BeanItemContainer or Table?

General
1

I’m trying to do a stress test of vaadin web project, I use BeanItemContainer to wrap a
static
 list of data beans and display it in Table.

It uses over
2G memory with 100 concurrent threads
, and
over 0.75G with 10
, it seems BeanItemContainer clone the list instead of just use it directly.

Are there any configs, settings or other patterns that can display a table with less memory consumption?

My test sample:


web.xml


<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">
    <display-name>Vaadin Web Application</display-name>
    <context-param>
        <description>Vaadin production mode</description>
        <param-name>productionMode</param-name>
        <param-value>true</param-value>
    </context-param>
    <servlet>
        <servlet-name>Vaadin</servlet-name>
        <servlet-class>com.vaadin.server.VaadinServlet</servlet-class>
        <init-param>
            <description>Vaadin UI to display</description>
            <param-name>UI</param-name>
            <param-value>test.vaadin.table.TableUI</param-value>
        </init-param>
        <init-param>
        	<param-name>disable-xsrf-protection</param-name>
        	<param-value>true</param-value>
        </init-param>
    </servlet>
    <servlet-mapping>
        <servlet-name>Vaadin</servlet-name>
        <url-pattern>/*</url-pattern>
    </servlet-mapping>
</web-app>


TableUI.java


package test.vaadin.table;

import java.util.List;

import com.vaadin.annotations.Title;
import com.vaadin.data.util.BeanItemContainer;
import com.vaadin.server.VaadinRequest;
import com.vaadin.ui.Table;
import com.vaadin.ui.UI;

@Title("Table")
public class TableUI extends UI {

	private static final long serialVersionUID = -6777271979317811922L;

	private static List<Data> _list;

	BeanItemContainer<Data> dataBeans = createDummyDatasource();
	private Table dataTable = new Table("test", dataBeans);

	protected void init(VaadinRequest request) {
		initLayout();
	}

	@SuppressWarnings("deprecation")
	private void initLayout() {
		dataTable.setPageLength(50);
		dataTable.setHeight("350px");
		dataTable.setWidth("350px");
		dataTable.setVisibleColumns(new String[] { "fieldOne", "fieldTwo", "fieldThree",
				"fieldFour", "fieldFive", "fieldSix",
				"fieldSeven", "fieldEight", "fieldNine",
				"fieldTen"});
		dataTable.setDebugId("test");
		setContent(dataTable);
	}

	private BeanItemContainer<Data> createDummyDatasource() {

		if (_list == null) {
			try {
				_list = Data.getTestData();
			} catch (Exception e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			}
		}
		BeanItemContainer<Data> bc = new BeanItemContainer<Data>(Data.class, _list);

		return bc;
	}

}


Data.java


package test.vaadin.table;

import java.util.ArrayList;
import java.util.List;

public class Data {
	private static List<Data> _testData;
	private String _fieldOne;
	private String _fieldTwo;
	private String _fieldThree;
	private String _fieldFour;
	private String _fieldFive;
	private String _fieldSix;
	private String _fieldSeven;
	private String _fieldEight;
	private String _fieldNine;
	private String _fieldTen;
	public Data (String fieldOne, String fieldTwo, 
			String fieldThree, String fieldFour, String fieldFive, 
			String fieldSix, String fieldSeven, String fieldEight, 
			String fieldNine, String fieldTen) {
		_fieldOne = fieldOne;
		_fieldTwo = fieldTwo;
		_fieldThree = fieldThree;
		_fieldFour = fieldFour;
		_fieldFive = fieldFive;
		_fieldSix = fieldSix;
		_fieldSeven = fieldSeven;
		_fieldEight = fieldEight;
		_fieldNine = fieldNine;
		_fieldTen = fieldTen;
	}
	public void setFieldOne (String fieldOne) {
		_fieldOne = fieldOne;
	}
	public String getFieldOne () {
		return _fieldOne;
	}
	public void setFieldTwo (String fieldTwo) {
		_fieldTwo = fieldTwo;
	}
	public String getFieldTwo () {
		return _fieldTwo;
	}
	public void setFieldThree (String fieldThree) {
		_fieldThree = fieldThree;
	}
	public String getFieldThree () {
		return _fieldThree;
	}
	public void setFieldFour (String fieldFour) {
		_fieldFour = fieldFour;
	}
	public String getFieldFour () {
		return _fieldFour;
	}
	public void setFieldFive (String fieldFive) {
		_fieldFive = fieldFive;
	}
	public String getFieldFive () {
		return _fieldFive;
	}
	public void setFieldSix (String fieldSix) {
		_fieldSix = fieldSix;
	}
	public String getFieldSix () {
		return _fieldSix;
	}
	public void setFieldSeven (String fieldSeven) {
		_fieldSeven = fieldSeven;
	}
	public String getFieldSeven () {
		return _fieldSeven;
	}
	public void setFieldEight (String fieldEight) {
		_fieldEight = fieldEight;
	}
	public String getFieldEight () {
		return _fieldEight;
	}
	public void setFieldNine (String fieldNine) {
		_fieldNine = fieldNine;
	}
	public String getFieldNine () {
		return _fieldNine;
	}
	public void setFieldTen (String fieldTen) {
		_fieldTen = fieldTen;
	}
	public String getFieldTen () {
		return _fieldTen;
	}
	public static List<Data> getTestData () {
		if (_testData == null) {
			_testData = new ArrayList<Data>();
			for (int i = 0; i < 2000; i++) {
				_testData.add(new Data("fieldOne_" + "i", "fieldTwo_" + "i", "fieldThree_" + "i",
						"fieldFour_" + "i", "fieldFive_" + "i", "fieldSix_" + "i",
						"fieldSeven_" + "i", "fieldEight_" + "i", "fieldNine_" + "i",
						"fieldTen_" + "i"));
			}
		}
		return _testData;
	}
}

Regards,
kabi

2

BeanItemContainer does have some memory overhead, not because of duplication of the data shown but through a fairly large amount of metadata, pre-creating all the BeanItem instances, some lists in the items that are often empty and could be optimized to be null when empty etc. Some of these could be eliminated at the cost of slightly more complex code.

I have been experimenting with some such optimizations (as well as some new features like supporting a mix of multiple bean types in a container) in a hobby project of mine, but haven’t yet published it in the Directory. You could take a look if
MCont
 helps you.

Otherwise, as you have static data, it should be very easy to implement your own container inheriting AbstractInMemoryContainer. That way, you could eliminate most if not all the unnecessary overhead.

3

This sounds to me to be the correct approach here - containers really aren’t that complicated to implement - albeit perhaps a little bit fiddly - and this seems a fairly specific use case.

4

Hi,

Could you please provide a sample or a related article with respect to how to implement AbstractInMemoryContainer?

Thanks,

kabi

5

See the javadoc for AbstractInMemoryContainer.

For a static container, at the minimum one needs to implement getContainerPropertyIds(), getUnfilteredItem(itemId) and add the items using internalAdd*(…) methods in the constructor or in some initialization method.

I would also recommend implementing your custom Item and Property classes if you want to minimize memory usage. The Item could even only have two fields (itemId or some index and a reference to the original data table) and fetch data from there like IndexedContainer does. The item class only really needs to implement getItemPropertyIds() and getItemProperty(…), the other methods can throw UnsupportedOperationException. The property class can likewise contain the aforementioned two fields plus the propertyId or index, and throw UOE from set*().

If you let your IDE generate classes based on the relevant based classes / interfaces, you’ll have the methods to fill in there.

6

Thanks for your suggestion, I understand how to implement an abstract class but I’m not familiar with structure and life-cycle of Vaadin, just wonder know whether there are any official documents or related articles (example, guideline, tips, check list, …, etc).

Thanks for your explanation however, I’ll try it out and let you know the result.

kabi

7

For creating containers, there isn’t really much documentation out there apart from the javadoc and a number of container implementations in the
Directory
, of which many might be too complex to use as examples. Quite a few people have managed to write their own so far, though.

Maybe someday I manage to write a little blog post about it.

8

Again, me,

I’ve implemented AbstractInMemoryContainer as below and going to do stress test again with it, are there anything can be further improved? Especially on performance or memory consumption.


package test.vaadin.table;

import com.vaadin.data.util.AbstractInMemoryContainer;
import com.vaadin.data.util.BeanItem;
import com.vaadin.data.Property;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Hashtable;

public class TestContainer<T> extends AbstractInMemoryContainer<T, Object, BeanItem<T>> {
	private final Class<? extends T> _type;
	private Map<Object, Property> _propMap = new Hashtable<Object, Property>();;
	private java.util.Collection<?> _containerPropertyIds;
	private List<T> _items = new ArrayList<T>();

	public TestContainer (Class<? extends T> clazz, List<? extends T> items) {
		_type = clazz;
		_items.addAll(items);
		for (int i = 0; i < items.size(); i++) {
			T item = items.get(i);
			BeanItem<T> bi = new BeanItem<T>(item);
			
			if (_containerPropertyIds == null) {
				_containerPropertyIds = bi.getItemPropertyIds();
				for (Object o : _containerPropertyIds) {
					_propMap.put(o, bi.getItemProperty(o));
				}
			}
		}
	}
	@Override
	public T addItem () {
		throw new UnsupportedOperationException();
	}
	@Override
	public BeanItem<T> addItem (Object itemId) throws UnsupportedOperationException {
		T item = null;
		if (!_type.isInstance(item)) {
			throw new UnsupportedOperationException();
		}
		item = (T)itemId;
		BeanItem<T> bi = new BeanItem<T>(item);
		_items.add(item);
		return bi;
	}
	@Override
	public T addItemAfter (Object previousItemId) throws UnsupportedOperationException {
		throw new UnsupportedOperationException();
	}
	@Override
	public BeanItem<T> addItemAfter (Object previousItemId, Object newItemId) throws UnsupportedOperationException {
		return internalAddItemAfter ((T)previousItemId, (T)newItemId, new BeanItem(newItemId), false);
	}
	@Override
	public Object addItemAt (int index) throws UnsupportedOperationException {
		throw new UnsupportedOperationException();
	}
	@Override
	public BeanItem<T> addItemAt (int index, Object newItemId) {
		if (index < 0 || index > size()) {
			index = size();
		}
		return internalAddItemAt(index+1, (T)newItemId, new BeanItem<T>((T)newItemId), false);
	}

	@Override
	protected BeanItem<T> internalAddItemAfter (T previousItemId,
			T newItemId, BeanItem<T> item, boolean filter) {
		int index = _items.indexOf(previousItemId); // 0+, or -1 if not found
		if (previousItemId != null) {
			index = _items.indexOf(previousItemId);
		}
		return internalAddItemAt(index+1, (T)newItemId, item, filter);
	}
	@Override
	protected BeanItem<T> internalAddItemAt(int index,
			T newItemId, BeanItem<T> item, boolean filter) {
		if (item != null) {
			_items.add(index, newItemId);
		}
		return item;
	}
	@Override
	protected void internalRemoveAllItems () {
		_items.clear();
		_propMap.clear();
		_containerPropertyIds.clear();
	}
	@Override
	protected boolean internalRemoveItem (Object itemId) {
		boolean exists = _items.remove(itemId);
		return exists;
	}
	@Override
	protected BeanItem<T> internalAddItemAtEnd (T newItemId, BeanItem<T> item, boolean filter) {
		return internalAddItemAt(_items.size(), newItemId, item, filter);
	}
	@Override
	protected List<T> getVisibleItemIds() {
		return _items;
	}
	@Override
	public int size () {
		return _items.size();
	}
	@Override
	public Class<?> getType(Object propertyId) {
		return _propMap.get(propertyId).getClass();
	}
	@Override
	public Property getContainerProperty(java.lang.Object itemId,
            java.lang.Object propertyId) {
		return new BeanItem<T>((T)itemId).getItemProperty(propertyId);
	}
	@Override
	protected BeanItem<T> getUnfilteredItem(java.lang.Object itemId) {
		if (_items.contains(itemId))
			return new BeanItem<T>((T)itemId);
		return null;
	}
	@Override
	public java.util.Collection<?> getContainerPropertyIds() {
		return _containerPropertyIds;
	}
}

Thanks,

kabi

References:


BeanItemContainer


Java doc of AbstractInMemoryContainer

9

A few things I spotted there:

  • The related entries in _propMap are not cleaned when an item is removed, resulting in a potential memory leak.
  • Pre-creating all the items and then keeping all their Property instances in memory can use quite a lot of memory. Property instances could be created on-demand only.
  • BeanItem metadata should probably be cached, which would reduce the overhead of creating new BeanItems significantly. Unfortunately, the API for accessing pre-created metadata is has package visibility so your class would need to be in com.vaadin.data (or some of its subpackages - I cannot recall) for that to work. You should encapsulate all calls to “new BeanItem(…)” to make it possible to use sensible caching etc.
  • getContainerProperty(…) should not create a new Property instance (and BeanItem) but use the old one so that e.g. ValueChangeListeners are registered for the correct Property instance and to avoid creating extra property instances.

One could optimize this further by using custom Items and Property instances that just access the data from the original beans, but that requires some more code - my MCont project does some such things.

10

We use BeanItemContainers for quite a big data and the memomory consuption is terrible - from 1GB allocated for Tomcat 800MB is used
by java.lang.reflection.Method
 (about 6mil. instances for a table that contains about 3000 of records with about 40 properties for 18 sessions). Does it exist a way to reduce number of created instances?

11

My little
MCont
 tries to initialize Properties in a lazy manner, reducing the initial memory consumption. However, once initialized, each property has the same lifecycle it would have in a normal BeanContainer, so if you scan through the whole ExtensibleBeanContainer, you get to a memory consumption approaching that of a BeanContainer.

I planned to optimize this further (significantly) by sharing metadata, but that is still a
TODO item in ExtensibleBeanContainer
. The memory usage optimization might have a small extra CPU cost when accessing properties, but that should be more than offset by the reduction of the memory footprint and all the related cache, GC, creation etc. overhead.

12

I finally got around to
publishing MCont in Vaadin Directory
 - I hope that will make it easier to test it out.

13

After first try looks awesome. It incomparable with BeanItemContainer. :slight_smile:

14

Hi,

You could also check out ListContainer in Maddon add-on. It use apache commons-beanutil tools for bean introspection. And it does it very efficiently.

cheeers,
matti