DB Access - performance problems

It’s Monday, and I’m having a bad day :frowning: Maybe someone can help…

I am building a web app which lets a user log in, and do some auditing on one of my database systems. The problem at the moment is that after the user logs in, I need to read in a database table which currently contains over 500k records. My web app times out (Communications failure) before the table gets read in. I tried spinning off a thread to do the database read, and then have the user page wait until it was finished before allowing the user to do any processing, but that didn’t seem to work either.

There is nothing fancy about the table - I’m reading 500k records into a List with the sql query “select m from mytable as m”; I’m using EclipseLink…

Can someone give me a hint as to how to read this data in successfully - maybe in the background somehow? I was planning to put it in a static List in the main program so it only has to be read the first time the program is accessed. After that it can be left alone (unless the user specifically requests a refresh…)

I have successfully read the data (in a different program) into a table using the LazyQueryContainer, but in this case, I need all the data before the user starts processing it…

Thanks in advance,


500,000 records? In memory all at once?
Maybe allow your user to filter for records they work on ?
Then read the filtered records.
They do their work and filter for more?

If there
is no other solution than having all this records in memory (I hope you have good reasons for that), then maybe an interactive web application is the wrong place to do it. If your Vaadin web app runs on Apache Tomcat, you could consider to move the 500k records processing code into a separate servlet. You could access this servlet from the Vaadin web app using simple HTTP GET/POST requests, triggering the reading in of the data from the database and triggering other processing of the data. So in the end you would have a small, lightweight web service that does the actual processing on the data, while your Vaadin web app only does the presentation stuff.

I’m looking at ways to reduce the # items in memory - I’ll look into the idea of a servlet. Also, I think I did something wrong in my first attempt at putting in a separate data thread yesterday - I’ll have to revist that.

Thanks for the tips…