Upload component goes OOM

I’m trying to use an upload component to upload huge files (2.5gb). The problem is that I often get Out of memory errors, before the upload has even started (so before the ‘receiveUpload’ method is called). I get following stacktrace in my logfiles:

If I look at the heap dump there is a buffer inside a ByteArrayOutputStream (probably the one from the stacktrace) that holds 512Mb.

I don’t keep the files in memory or something, so I’m wondering where this comes from. It also happens on the first upload after I restart the server, so it’s not as if some data from a previous upload is hanging around. After I press ‘upload’ my disk starts spinning so it seems the browser is sending data from the uploaded file. It happens both in firefox and opera.

I’m using Vaadin 6.2.2 and jetty 6.1.26, and firefox If anybody has experienced the same issue, or knows where this might come from, all help is welcome.

I think that you get to a lot of problems with file uploads with browsers in general when you go over 2GB marker. I think it has to do with 32bit counting. upper bound of 2^32-1 is around the 2GB marked. Also, you really should consider using applets of flash when you get to files of that size, as standard http upload has no resume capabilities etc.

Anyhow, to me it looks like your system is trying to allocate memory for the whole 2.5 GB, and it is running out of space at that point. Maybe you could get a breakpoint somewhere and be able to check the size of that array / input of readLine?

Thanks for your reply! I’ll try to run it through a debugger.

In the mean time, what I’m wondering about: why is the

reading the complete file in memory instead of the upload component ? I would assume that, since I use the upload component, any file content will be handled by the upload component and not by the ‘AbstractCommunicationManager’.

Also, in some cases it works and I immediately see the ‘receiveUpload’ callback being called. What the determining factor ?

To be honest, I’m not a great hero when it comes to gwt etc., so maybe I’m missing the point here somewhere.

I have same problem, in same line with Vaadin 6.7.1, but all time with Glassfish 3.1.2, even with 50KB files; with Glassfish 3.0.1 I’m working very well.

I think that problem is specific to Glassfish 3.1.2, also EasyUpload AddOn have problems with this version.

PS
I’m returning a FileOutputStream in Upload.Receiver.receiveUpload(), also files are 100KB in average with 2 concurrent users and 1GB of heap, so memory shouldn’t be the problem.