Upload file Streaming

Flow
1

Hi,

My goal is to do a stream upload of files in a Vaadin Flow application, i.e. to upload a file to an external storage service without keeping all of the file bytes in the application (memory, temp file, etc). For reference, I’m using Vaadin 25 with Springboot 4.

In this blog post Rethinking uploads and downloads in Vaadin 24.8 - A migration guide, it is mentioned that:

Streaming is currently not supported in Spring Boot applications due to special multipart request handling

I naively implemented an UploadHandler from which I get the UploadEvent. Then the InputStream is consumed chunk by chunk, some validation is done, and the bytes and are send through multipart requests to the external service. Here is the FileStorageUploadHandler:

@RequiredArgsConstructor
@Slf4j
public class FileStorageUploadHandler
    extends TransferProgressAwareHandler<UploadEvent, FileStorageUploadHandler>
    implements UploadHandler {

  private final SerializableConsumer<FileStorageMetadata> successCallback;

  @Override
  public void handleUploadRequest(UploadEvent uploadEvent) {
    try {
      FileStorageMetadata fileStorageMetadata =
          fileStorageService.store(
              new FileContent(uploadEvent.getInputStream(), uploadEvent.getFileName()));
      log.info("fileStorageMetadata: {}", fileStorageMetadata);
      successCallback.accept(fileStorageMetadata);
    } catch (FileStorageException e) {
      notifyError(uploadEvent, e);
    }
  }

  private void notifyError(UploadEvent uploadEvent, FileStorageException e) {
    TransferContext transferContext = getTransferContext(uploadEvent);
    getListeners()
        .forEach(
            listener -> listener.onError(transferContext, new IOException(e.getDefaultMessage())));
  }

  @Override
  protected TransferContext getTransferContext(UploadEvent transferEvent) {
    return new TransferContext(
        transferEvent.getRequest(),
        transferEvent.getResponse(),
        transferEvent.getSession(),
        transferEvent.getFileName(),
        transferEvent.getOwningElement(),
        transferEvent.getFileSize());
  }
}

I’m trying to understand the limitation of the Upload component because I was able to upload a valid big file (>100MB). However, when there’s an exception being thrown, e.g. validation error, a retry occurs, perhaps the browser XHR request retry.

As can be seen, if fileStorageService.store(…) fails, then the listener for errors are notified.

Could you provide some insight, specifically how to prevent the fileStorageService call everytime?

2

In Vaadin 25 we changed the Upload component to not use multi-part requests by default. I assume that is why you did not run into the limitation mentioned in the blog post.

I’m not sure what could cause the retry behavior, though I haven’t much experience with the upload handler API.

1 Like
3

I gave this a quick try with the view below, but could not reproduce any retries. One thing that is probably missing from your implementation is re-throwing the exception, otherwise the upload will not be marked as failed in the Upload component. This is also shown in the JavaDoc for TransferProgressAwareHandler.notifyError. But I could not observe any other side-effects when I left that out.

@Route("")
public class View extends Div {
    public View() {
        Upload upload = new Upload(new CustomUploadHandler());
        add(upload);
    }
    
    static class CustomUploadHandler extends TransferProgressAwareHandler<UploadEvent, CustomUploadHandler> implements UploadHandler {
        @Override
        public void handleUploadRequest(UploadEvent uploadEvent) throws IOException {
            try {
                System.out.println("Starting upload for file: " + uploadEvent.getFileName());
                FakeExternalStorage.upload(uploadEvent.getInputStream());
            } catch (IOException exc) {
                notifyError(uploadEvent, exc);
                throw exc;
            }
        }

        @Override
        protected TransferContext getTransferContext(UploadEvent transferEvent) {
            return new TransferContext(
                    transferEvent.getRequest(),
                    transferEvent.getResponse(),
                    transferEvent.getSession(),
                    transferEvent.getFileName(),
                    transferEvent.getOwningElement(),
                    transferEvent.getFileSize());
        }
    }

    static class FakeExternalStorage {
        private static final int MAX_SIZE = 1024 * 1024; // 1MB

        public static void upload(InputStream inputStream) throws IOException {
            ByteArrayOutputStream buffer = new ByteArrayOutputStream();
            byte[] chunk = new byte[8192];
            int bytesRead;
            int totalBytes = 0;

            while ((bytesRead = inputStream.read(chunk)) != -1) {
                totalBytes += bytesRead;
                if (totalBytes > MAX_SIZE) {
                    throw new IOException("Upload exceeds maximum size of 1MB");
                }
                buffer.write(chunk, 0, bytesRead);
            }
        }
    }
}
4

Thank you for this, @sissbruecker.

I have forked the skeleton-starter-flow-spring to include your View class (renamed to UploadView):

I have tried uploading a 100MB file and I get this error 7 times:

java.io.IOException: Upload exceeds maximum size of 1MB
	at org.vaadin.example.UploadView$FakeExternalStorage.upload(UploadView.java:58) 
...

Can you try with my fork, please?

Thanks in advance

5

test with this

spring:
  #jpa:
  #  hibernate:
  #    ddl-auto: update
  servlet:
    multipart.max-file-size: 50MB
    multipart.max-request-size: 50MB
7

Hi Rubén,

Thanks for you suggestion

Unfortunately, that doesn’t solve my issue and I think the reason is what @sissbruecker mentioned in his first answer: in Vaadin 25, multipart is disabled for the Upload component (by default)

8

It has an if statement that prevents you from uploading more than 1MB.

I remember with Vaadin version 8, it processed up to gigabytes of .iso files, but it wrote to disk first.

This post is interesting

9

Thanks for the reproduction. I can confirm the issue when using larger files. In my testing I used one that was barely above the 1MB limit, which didn’t reproduce.

I’ve filed and issue here: UploadHandler.handleUploadRequest called multiple times after throwing exception · Issue #23186 · vaadin/flow · GitHub

1 Like
10

Which browser are you using BTW? I can only reproduce this in Firefox, but not in Chrome. Safari seems to have a different issue that the upload component shows the file as “stalled”, so apparently it doesn’t receive an error on the client.

11

Thank you for following this up @sissbruecker

I’m testing in Firefox, mostly. I can confirm that Chrome has the expected behavior.

About showing the file as “stalled”, I think it’s important to report what I observe in the UI.

For a file of 100MB, I see the progress bar and the text inform the user that the upload has started. Then it increases to about 10%, then it shows as “stalled”, then it restarts as many times as I see exceptions in the logs. Hence why I initially mentioned XHR retries.