Outputtng (streaming) the large amount of data as HTTP response

General
1

Dear all,

In my application I would like to have a “Export to XML” functionality to allow the user to download the big amount of information as XML. As
StreamResource.StreamSource()
 interface dictates, I need to return
InputStream
 for the data to be downloaded by user. Unfortunately:

  • It is not possible to create
    ByteArrayInputStream
     and collect all data, as the data is too big to fit in memory. More over, I find it inefficient to copy the data from
    StreamResource
    's
    InputStream
     to
    HttpServletResponse
     via intermediate buffer, which happens in

    AbstractApplicationServlet#handleDownload()

    … How can I have an access to
    HttpServletResponse
    OutputStream
     directly?
  • I am happy to dump information to the temporary file (as suggested
    here
    ), but the creation time might take up to one hour, so proxy server may abort this download.
  • Another solution (suggested
    here
    ) is to use
    PipedInputStream
     /
    PipedOutputStream
     and additional feeding thread. Does not work for me on some reason: the file downloading aborts in few minutes always at same position, however feeding thread never exits…
    [code]
    myStreamResource.setStreamSource(new StreamResource.StreamSource() {

    /**
  • @see com.vaadin.terminal.StreamResource.StreamSource#getStream()
  • /
    public InputStream getStream() {
    // PipedInputStream and PipedOutputStream should be connected before any read()/write() operation is invoked:
    final PipedInputStream pis = new PipedInputStream();
    final PipedOutputStream pos;

    try {
    pos = new PipedOutputStream(pis);
    }
    catch (IOException e) {
    logger.error(“Failed to create a pipe”, e);
    return null;
    }

    new Thread(new Runnable() {
    /**
  • @see java.lang.Runnable#run()
  • /
    public void run() {
    OutputStream gzip = null;

    try {
    gzip = new GZIPOutputStream(pos);

    myService.exportAllIndexedDocumentsAsXML(gzip);
    }
    catch (IOException e) {
    IOUtils.closeQuietly(gzip);
    IOUtils.closeQuietly(pis);
    logger.error(“Failed to send GZIP XML data to the stream”, e);
    }
    }
    }).start();

    return pis;
    }
    });

    [/code]
    Is this approach recommended? Does it work for somebody?
2

I suggested the PipedInputStream, and it does work for me, for reasonably-sized files (produce PDFs and Excels that dowload in a few seconds.) There is no reason for the code to hang unless there is a proxy or browser timeout that prevents the information from flowing out. There could also be hard limits on the size of transfers, or other configuration settings on the web server proper – there could be copies to disk done without you being aware of them, or some size quota at play. The fact that you always fail at the same place hints at a size limit.

I would suggest that you
a) test locally that the PipedInputStream code works from a local stream to a local stream, for the full sized transfer.
b) test the download from a browser running on the same machine (i.e. localhost) - no network, on a development machine
c) directly on a a lan, without an intervening proxy.

It would also help if you could count how many bytes exactly were transfered before the failure – you might want to use a program like HttpWatch to see what is going on.

3

The problem with launching the feeding thread is that you can’t correctly capture the exceptions, which occur when you write to
HttpServletResponse
 output stream. Do you find it in general handy approach to write directly to output stream without need to create a separate thread? Would you vote for this possibility in API?

Finally I found out that the following exception occurred (which was wrapped in
AbstractApplicationPortlet#handleServiceException()
 into
PortletException
, into container and swallowed):

org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:148)
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:92)
	at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.handleDownload(AbstractApplicationServlet.java:926)
	at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.handleURI(AbstractApplicationServlet.java:1068)
	at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.service(AbstractApplicationServlet.java:517)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:533)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:461)

I also fired
this bug
 which is relevant, but not cause of the problem. It looks like the problem is container-specific, as it occurs for Jetty and seems to work for Tomcat.