Table soooo slow when doing mass updates..

[RockyD]

Hello people, this is my first post to your forums.

First of all, I have to tell that
I really like toolkit way of coding
, doing every bit in Java is wonderful! After playing for few days with your samples I decided to write down an apllication of my own. Now I am completely baffled with the following issue because my code is very simple and I can’t see what the heck I am doing wrong…

I got a simple table that contains four columns. All columns are String.class such as:

myTable.addContainerProperty("columnX", String.class, null);

I am adding one thousand lines (4000 cells) there which I want client to load all in one http call so Therefore I disable buffering by:

myTable.setPageLength(0);

And here I add 1000 items to my table in a loop

 for (i=0; i < 999; i++) {
                        Item item = myTable.getItem(myTable.addItem());
                        item.getItemProperty("columnX").setValue("Ciao!!");
...
                    }
// I takes multiple minutes before we get past above for loop :-(

But if I make setPageLength(0) then adding those 1000 items takes
multiple minutes
on the server!! So, I grab my profiler and see that all the time is spent toolkit event classes such as com.itmill.toolkit.ui.AbstractField.fireValueChange. In fact it slows down exponentially based on how many items I already got in my table. This can’t work this way, how can I disable events when I am doing mass update to my table??

[Marko1 (Marko Grönroos)]

@RockyD

I am adding one thousand lines (4000 cells) there which I want client to load all in one http call so Therefore I disable buffering by:

myTable.setPageLength(0);

Hi,

I’m not sure what your use case is, but I suppose you could have such a need when you want to generate reports. An Ajax framework may not be very scalable for that purpose. There’s inevitably a huge amount overhead from creating UI widgets for each table cell. Even if the server-side doesn’t get choked, as was in your case, the browser will at some point. If you want to have more scalable reports, I suggest generating HTML.

for (i=0; i < 999; i++) {
Item item = myTable.getItem(myTable.addItem());
item.getItemProperty(“columnX”).setValue(“Ciao!!”);
…
}
// I takes multiple minutes before we get past above for loop

[/code]

Yeah, setting the value of a property causes a notification to the UI component displaying the value and it seems that the notification travels around a bit. Normally, only the visible items have a dynamically created UI component, but having all the items visible can create more jungle for the notifications to travel around.

I suggest that you do not use the setValue(), but use the conveniency addItem() method that accepts all the properties of an item as an object vector. This way there won’t be any change notifications. For example:

        /* Create the table with a caption. */
        Table table = new Table("This is my Table");
        
        /* Define the names and data types of columns.
         * The "default value" parameter is meaningless here. */
        table.addContainerProperty("First Name", String.class,  null);
        table.addContainerProperty("Last Name",  String.class,  null);
        table.addContainerProperty("Year",       Integer.class, null);

        /* Add a few items in the table. */
        table.addItem(new Object[] {"Nicolaus","Copernicus",new Integer(1473)}, new Integer(1));
        table.addItem(new Object[] {"Tycho",   "Brahe",     new Integer(1546)}, new Integer(2));
        table.addItem(new Object[] {"Giordano","Bruno",     new Integer(1548)}, new Integer(3));
        table.addItem(new Object[] {"Galileo", "Galilei",   new Integer(1564)}, new Integer(4));
        table.addItem(new Object[] {"Johannes","Kepler",    new Integer(1571)}, new Integer(5));
        table.addItem(new Object[] {"Isaac",   "Newton",    new Integer(1643)}, new Integer(6));

This should help when you create the items, but it doesn’t help if you need to make large updates to the property values with the setValue(). I suppose the notifications could be optimized better, but you’d still have scalability problems in the client-side.

[QuentinAstegiano]

I have some cases myself where I have to handle really big lists (something like, from a few thousands entries to 20/30K entries, each of those with multiple attributes and reference to quite a lot of java objects).

The best solution I’ve found (well, a member of my team found, to be honest ^^ ) is to use a custom Container & custom Items.

To add a lot of datas, and to manipulate it afterward (we have a need for filters, search, sorting, … and even on-the-fly CSV export), it is oh so much quicker and easier to do, than what we do previously : using addContainerProperty & addItem .

But, of course, it is much more complicated to setup

Quentin.

[Matti_Tahvonen]

Hi folks!

I have spent some hours today investigating the bad performance RockyD brought up. There seems to be some very serious issues in issues in one of the oldest classes of the framework.

IndexedContainer and its subclasses (Properties, Items) seems to do some very expensive operations in hashCode and equals methods. This makes storing them in standard java collections very slow. And table does this quite a lot if one sets pagelenght = 0 (render all) as it needs to keep list of rendered properties.

Quick workarounds I can see:

• use your own better optimized container (this is often required anyways to optimize hard load or to handle large amount of data)
• Copy-paste IndexedContainer and blow all overridden hashcode + equals methods to get quickly started
• don’t disable row buffering
• configure table first and attach table to component tree as late as possible

Luckily this issue pops up only in some situations (like with pageLength == 0), but I can see why Quentin has chosen to implement own containers.

I made a table specific workaround to trunk and ticket #1980 in case you want to follow actions on this issue.

[Marko1 (Marko Grönroos)]

Sure, if you use a regular scrollable table (fixed page length), a table can easily hold 500 000 (half a million) items in Firefox. With a fast custom container, it’s not even slow, because Table accesses items only when it displays them, so a smart container can even generate the items on the fly when needed.

Table breaks suddenly somewhere after the 500 000 rows, because the contents of a table are really inside a huge

24 pixels per row by default. At 500 000 rows that makes 12 million pixels. Firefox seems to have some limit somewhere above that. The limit could be different with different browsers. I suppose we could try to work around the limit by doing the scrollbar somehow differently at some point.

Anyhow, if you set setPageLength(0), the browser seems to choke up after just 1000 items. Safari seems to execute the client-side code a lot faster than Firefox and IE, so it can hold bigger tables.

The original problem was apparently caused by Property.ValueChangeEvents created by the setValue() calls. If I understood it correctly, each event causes the Table to refresh all visible items, and at step
i
there are
i
items to refresh in the table, so I think you get O(N^2) complexity just from that. But as said, that’s only with setPageLength(0).

[RockyD]

Wow… that was lengthy explanation!

Well, I tried creating my own indexedcontainer first and populating it with my items before setting this indexedcontainer into my table and now my code works a lot faster! Still I notice that following code takes 3 seconds on my server to complete:

myTable.setContainerDataSource(myIndexedContainer);

Over 44 million invocations to java.lang.Object.equals to process my lousy 1000 lines :-), after waiting for couple minutes in my profiler I decided to stop waiting… I guess this is because of poor hashcode coding which Matti mentioned…

@Matti Tahvonen

Hi folks!

• configure table first and attach table to component tree as late as possible

What do you mean with “component tree”?

@Matti Tahvonen

I made a table specific workaround to trunk and ticket #1980 in case you want to follow actions on this issue.

How can I get this #1980 change to the project of mine?

Ciao!

[Jani (Jani Laakso)]

@RockyD

Still I notice that following code takes 3 seconds on my server to complete:

Our developers are inspecting this issue too, I’ve noticed multiple changesets flowing to our repository even today…

@RockyD

What do you mean with “component tree”?

This is internal R&D talk With “Component tree” we mean Toolkit aplication’s user interface state which contains every widget that is currently attached to your main window. Attaching your table as late as possible to “component tree” means that first populate your indexedcontainer / table completely and then call Layout.addComponent(myTable) as late as possible.

@RockyD

How can I get this #1980 change to the project of mine?

Nightly and release builds are located in:

http://dev.itmill.com/wiki/Builds

Check out dev.itmill.com how to access source code repository if that suits your needs

[Matti_Tahvonen]

Hi !

I made some changes for IndexedContainer and Table which should resolve server side performance issues discussed in this thread. Marko is just now preparing a release (5.2.6) that includes these fixes.

Still remember the best practices with data binding (and Table component):

• prepare your data source unattached and then set it to table (changing visible content will fire some expensive events)
• this is ajax framework, use the caching mechanism whenever possible
• “Quentin’s rule”: building your own custom tailored data container will finally be the best solution, although it needs some coding in the beginning

And Jani, thanks for help answering here!

cheers,
matti