Upload: OutOfMemory

Hi,

I have a problem with the Upload component. I get in every case a OutOf Memory.

My environment:

- JRE/JDK6u31
- glassfish 3.1 (with all updates)
- vaadin-6.7.6.jar

I have extends the upload example from the Vaadin book. My Upload example class implements also the StartedListener so I hope I will be informed if the upload process is started. And i have packed the Upload example inside of a application class so that I can start these class. I have also added some interesting debug outputs (as System.out.println):

@SuppressWarnings("serial")
public class UploadExampleApplication extends Application 
implements Receiver, SucceededListener, FailedListener, StartedListener {

  Panel root;         // Root element for contained components.
  Panel imagePanel;   // Panel that contains the uploaded image.
  File  file;         // File to write to.

  /** {@inheritDoc}
   */
  @Override
  public void init() {
    System.out.println("________________ INIT APP");
    
    final Window root = new Window("My Upload Component");
    setMainWindow(root);

    // Create the Upload component.
    final Upload upload = new Upload("Upload the file here", this);

    // Use a custom button caption instead of plain "Upload".
    upload.setButtonCaption("Upload Now");

    // Listen for events regarding the success of upload.
    upload.addListener((Upload.SucceededListener) this);
    upload.addListener((Upload.FailedListener) this);
    upload.addListener((Upload.StartedListener) this);

    root.addComponent(upload);
    root.addComponent(new Label("Click 'Browse' to "+
            "select a file and then click 'Upload'."));

    // Create a panel for displaying the uploaded image.
    imagePanel = new Panel("Uploaded image");
    imagePanel.addComponent(
                     new Label("No image uploaded yet"));
    root.addComponent(imagePanel);
  }

  // Callback method to begin receiving the upload.
  public OutputStream receiveUpload(String filename,
                                    String MIMEType) {
    System.out.println("________________ RECEIVED UPLOAD");
      FileOutputStream fos = null; // Output stream to write to
      file = new File("/tmp/uploads/" + filename);
      try {
          // Open the file for writing.
          fos = new FileOutputStream(file);
      } catch (final java.io.FileNotFoundException e) {
          // Error while opening the file. Not reported here.
          e.printStackTrace();
          return null;
      }

      return fos; // Return the output stream to write to
  }

  // This is called if the upload is finished.
  public void uploadSucceeded(Upload.SucceededEvent event) {
    System.out.println("________________ UPLOAD SUCCEEDED");
      // Log the upload on screen.
      root.addComponent(new Label("File " + event.getFilename()
              + " of type '" + event.getMIMEType()
              + "' uploaded."));
      
      // Display the uploaded file in the image panel.
      final FileResource imageResource = new FileResource(file, this);
      imagePanel.removeAllComponents();
      imagePanel.addComponent(new Embedded("", imageResource));
  }

  // This is called if the upload fails.
  public void uploadFailed(Upload.FailedEvent event) {
    System.out.println("________________ UPLOAD FAILED");
      // Log the failure on screen.
      root.addComponent(new Label("Uploading "
              + event.getFilename() + " of type '"
              + event.getMIMEType() + "' failed."));
  }

  /** {@inheritDoc}
   */
  @Override
  public void uploadStarted(StartedEvent event) {
    System.out.println("________________ UPLOAD STARTET");
  }
}

But if I try to uploading a 17KB big (image) file I get:

INFO: ________________ INIT APP

WARNUNG: StandardWrapperValve[c-onDocFlowVaadin]: PWC1406: Servlet.service() for servlet c-onDocFlowVaadin threw exception
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2786)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:71)
	at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.readLine(AbstractCommunicationManager.java:386)
	at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.doHandleSimpleMultipartFileUpload(AbstractCommunicationManager.java:427)
	at com.vaadin.terminal.gwt.server.CommunicationManager.handleFileUpload(CommunicationManager.java:257)
	at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.service(AbstractApplicationServlet.java:495)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
	at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
	at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
	at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
	at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
	at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
	at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
	at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
	at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
	at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
	at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
	at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
	at java.lang.Thread.run(Thread.java:662)

Anybody knows whats wrong here?

Thanks and regards,
Steffen

PS: I have still increased memory settings by specifying "-Xms2000m" and "-Xmx2000m" in the JVM settings of glassfish (and of course restart glassfish after doing that ...)

Have you tried increasing the max permsize as well? For instance -XX:MaxPermSize=512m

Hi,

I have increase perm size to 512m. But without success. The OutOfMem ist still there (on GlassFish).

After I had seen that it's running on tomcat6 very well (with the default memory settings) I don't think this is a memory problem. I think there is a bug in the Upload component may be related with a special jar library which is contained together with GlassFish and which is in tomcat isn't there ...

Thanks and regards,
Steffen

Hi Steffen,

Take a look to Java Tuning White Paper and Oracle GlassFish Server 3.1 Performance Tuning Guide.

HTH,

Javi.

I have figured out that this also works in JBoss 7.1.1 and in a older glassfish version (I have tested it successful in V3.1.1 B12). Seems is a glassfish bug. I have added a bug:
http://java.net/jira/browse/GLASSFISH-18505

Regards,
Steffen

This probably isn't Glassfish's fault, but Vaadin AbstractCommunicationManager class is just assuming too much things.

There is a bit awkward implementation of "readLine" in the class:

370 private static String readLine(InputStream stream) throws IOException {
371 ByteArrayOutputStream bout = new ByteArrayOutputStream();
372 int readByte = stream.read();
373 while (readByte != LF) {
374 bout.write(readByte);
375 readByte = stream.read();
376 }
377 byte[] bytes = bout.toByteArray();
378 return new String(bytes, 0, bytes.length - 1, UTF8);
379 }

What if there is no LF character in the stream? This is where out of memory comes from...

A quick look around shows that this is most likely called from here:

414 while (!atStart) {
415 String readLine = readLine(inputStream);
416 contentLength -= (readLine.length() + 2);
417 if (readLine.startsWith("Content-Disposition:")
418 && readLine.indexOf("filename=") > 0) {
419 rawfilename = readLine.replaceAll(".*filename=", "");
420 String parenthesis = rawfilename.substring(0, 1);
421 rawfilename = rawfilename.substring(1);
422 rawfilename = rawfilename.substring(0,
423 rawfilename.indexOf(parenthesis));
424 firstFileFieldFound = true;
425 } else if (firstFileFieldFound && readLine.equals("")) {
426 atStart = true;
427 } else if (readLine.startsWith("Content-Type")) {
428 rawMimeType = readLine.split(": ")[1];
429 }
430 }

Here Vaadin is assuming that the browsers and servers are all playing nice and set all the headers just in the way Vaadin is expecting them. If "Content-Disposition" header will not be there, then this will just call its readLine to infinity which will eventually run out of real data, after this happens it will spin away happily looking for its "LF" and filling the ByteArrayOutputStream with the EOF return value :)

I will have a look at what data GlassFish actually provides to see where exacly and why this hangs.
Also this kind of parsing code just begs denial of service attacks.

I experienced the same issue here (readline sometimes allocates the complete 2.5Gb when I upload a file of that size), and your explanation seems to be consistent with the fact that it sometimes works, and sometimes doesn't (it's a binary file, and probably doesn't contain any LF when you're unlucky).

Kim

Had a look what is coming from GlassFish...

And the strangest thing is that the InputStream passed to Vaadin's readline always returns -1 on read() calls immediately.
So with glassfish 3.1.2 the out of memory exception happens because the Vaadin's readLine tries to build an infinitely large ByteArrayOutputStream which is getting filled by -1.

I would look into Glassfish part, but there is no easy way how to get sources for the binary distribution. And I am currrently too lazy to build it myself.

Maybe someday...

This problem with Upload and GF 3.1.2 seems to be a GF "feature".

Details and fix available here:
http://java.net/jira/browse/GLASSFISH-18444

i have the same exception, if the Vaadin Upload is used and i try to analyse with the commons fileupload the http request in protected void service(HttpServletRequest request, HttpServletResponse response) method.
The commons fileupload reads the input stream of the http request. So if uploading wiill be performed, the input stream end is already reached. Thus the -1 will be returned. But the current version (vaadin 6.8.4) of AbstractCommunicationManager ignores it!

java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2271)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:122)
	at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.readLine(AbstractCommunicationManager.java:383)
	at com.vaadin.terminal.gwt.server.AbstractCommunicationManager.doHandleSimpleMultipartFileUpload(AbstractCommunicationManager.java:424)
	at com.vaadin.terminal.gwt.server.CommunicationManager.handleFileUpload(CommunicationManager.java:257)
	at com.vaadin.terminal.gwt.server.AbstractApplicationServlet.service(AbstractApplicationServlet.java:495)

P.S i use the Apache Tomcat as web server.

Please create a ticket for the issue at http://dev.vaadin.com.

P.S i'm created a simple example without using of commonupload to to reproduce that problem (teh vaadin source snippets were used).

private static final int EOF = -1;
	
		private static final int LF = "\n".getBytes()[0];
	

	public static final boolean isMultipartContent(HttpServletRequest request) {
			if (!("post".equals(request.getMethod().toLowerCase()))) {
				return false;
			}
			String contentType = request.getContentType();
			if (contentType == null) {
				return false;
			}
	
			return (contentType.toLowerCase().startsWith("multipart/"));
		}
	
		public static String getFileName(HttpServletRequest request) {
			String result = "";
			try {

				InputStream inputStream = request.getInputStream();
	
				boolean firstFileFieldFound = false;
	
				/*
				 * Read the stream until the actual file starts (empty line). Read
				 * filename and content type from multipart headers.
				 */
				while (!firstFileFieldFound) {
					String readLine = readLine(request);
	
					if (readLine.startsWith("Content-Disposition:") && readLine.indexOf("filename=") > 0) {
						String rawfilename = readLine.replaceAll(".*filename=", "");
						String parenthesis = rawfilename.substring(0, 1);
						rawfilename = rawfilename.substring(1);
						rawfilename = rawfilename.substring(0, rawfilename.indexOf(parenthesis));
						result = rawfilename;
						firstFileFieldFound = true;
					} 
				}

			} catch (IOException e1) {
				// TODO Auto-generated catch block
				e1.printStackTrace();
			}
	
			return result;
		}
	
		private static String readLine(InputStream stream) throws IOException {
			ByteArrayOutputStream bout = new ByteArrayOutputStream();
			int readByte = stream.read();
			while (readByte != LF && readByte != EOF) {
				bout.write(readByte);
				readByte = stream.read();
			}
			byte[] bytes = bout.toByteArray();
			return new String(bytes, 0, bytes.length - 1, Charsets.UTF_8);
		}

This example must be called within of service method of an own servlet extending servlet ApplicationServlet before super.service (.) was called

a-ka

public class OwnServlet extends ApplicationServlet {....

	protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
		synchronized (getSyncObject(request.getSession(true))) {// make thread safe

			//tst

			boolean isMultipart = HttpServletUtility.isMultipartContent(request);
			if (isMultipart) {
				String fileName=HttpServletUtility.getFileName(request);
		}
		//tst end
.....
	private static synchronized Object getSyncObject(HttpSession session) {
		Object syncObj = session.getAttribute(SYNC_OBJECT_KEY);
		if (syncObj == null) {
			syncObj = new Object();
			session.setAttribute(SYNC_OBJECT_KEY, syncObj);
		}
		return syncObj;
	}

...}


}

Just want to let you know that I added a comment to http://dev.vaadin.com/ticket/10096:

If the default charset of the JVM uses a multi-byte encoding (e.g. UTF-16), AbstractCommunicationManager:369 sets the line feed integer LF to something different than 0x0a (first byte of multi-byte encoding). Since the readLine() method reads bytes from the request until LF is encountered, it reads infinitely. Even after the request has no more data, readLine() fills the buffer with -1 since InputStream.read() returns -1 if the stream has no more data. This goes as long as memory is available...

The GlassFish 3.1 based on Servlet 3.0.1 (as the Tomcat 7, but the Tomcat 6 isn't).

I replaced the original com.vaadin.terminal.gwt.server.CommunicationManager with an own class:

[size=2]

(...)
	/**
	 * {@inheritDoc}
	 */
	@Override
	protected void doHandleSimpleMultipartFileUpload(Request request, Response response, StreamVariable streamVariable, String variableName, VariableOwner owner, String boundary)
			throws IOException {

		// Servlet 3.x extension !!
		Part part = getFirstPart(request);

		InputStream inputStream = part.getInputStream(); // --> because we won't have data if we call the request.getInputStream() !!
		int contentLength = new Long(part.getSize()).intValue();
		String rawfilename = getFileName(part);
		String rawMimeType = part.getContentType();

		/*
		 * Should report only the filename even if the browser sends the path
		 */
		final String filename = removePath(rawfilename);
		final String mimeType = rawMimeType;

		try {
			/*
			 * safe cast as in GWT terminal all variable owners are expected to
			 * be components.
			 */
			Component component = (Component) owner;
			if (component.isReadOnly()) {
				throw new UploadException("Warning: file upload ignored because the componente was read-only");
			}
			// boolean forgetVariable = streamToReceiver(simpleMultiPartReader, streamVariable, filename, mimeType, contentLength);
			boolean forgetVariable = streamToReceiver(inputStream, streamVariable, filename, mimeType, contentLength);
			if (forgetVariable) {
				cleanStreamVariable(owner, variableName);
			}
		} catch (Exception e) {
			synchronized (getApplication()) {
				handleChangeVariablesError(getApplication(), (Component) owner, e, new HashMap<String, Object>());
			}
		}
		sendUploadResponse(request, response);
	}

	/**
	 * Return the first part of the multipart/form-data. This is a Servlet 3.x extension.
	 * 
	 * @param request
	 *            the Vaadin request
	 * @return the fist part from the request, it may be null
	 * @throws IOException
	 */
	private Part getFirstPart(Request request) throws IOException {
		Part part = null;
		HttpServletRequest originalRequest = (HttpServletRequest) request.getWrappedRequest();
		try {
			Collection<Part> parts = originalRequest.getParts();
			// select the first part only
			if (parts != null && !parts.isEmpty()) {
				for (Part p : parts) {
					part = p;
					break;
				}
			}
		} catch (ServletException e) {
			throw new IOException(e.getMessage(), e);
		}

		return part;
	}

	/**
	 * Returns the name of the uploaded file. This is a Servlet 3.x extension.
	 * 
	 * @param part
	 *            the uploaded part.
	 * @return the name of the uploaded file.
	 */
	private String getFileName(Part part) {
		String result = null;
		String partHeader = part.getHeader("content-disposition");
		for (String cd : partHeader.split(";")) {
			if (cd.trim().startsWith("filename")) {
				result = cd.substring(cd.indexOf('=') + 1).trim().replace("\"", "");
			}
		}

		return result;
	}
(...)
[/size]

This workaround works for me (Tomcat 7 / Servlet 3.0.1 / 64bit / Win7).

Regards,
cx.chico

We had the same problem with weblogic 12.

Chico's workaround seems to be doing the job.

The problem still occurs on version of vaadin 7.1.11 (I checked also sources for newest version and it looks like that is still not fixed). We got it very rarely on Tomcat. 
So my hack was creating new handler that extends FileUploadHandler with below changes:
  1. readLine method 

private static String readLine(InputStream stream) throws IOException {
        ByteArrayOutputStream bout = new ByteArrayOutputStream();
        int readByte = stream.read();
        // We changed below line, we need to check if bytes are still available
        // otherwise it could be infinitive loop and later Java Heap Space
        while (readByte != LF && readByte > -1) {
            bout.write(readByte);
            readByte = stream.read();
        }
        byte[] bytes = bout.toByteArray();
        // bytes length could be 0
        if (bytes.length == 0) {
            return "";
        } else {
            return new String(bytes, 0, bytes.length - 1, UTF8);
        }
    }

2. inside doHandleSimpleMultipartFileUpload method

// It could be infinitive loop and later Java Heap Space if there will be no "Content-Disposition:" and
        //  "filename=" in the line so we provide counter
        int count = 0;
        while (!atStart && count < CONTENT_DISPOSITION_COUNTER) {
            String readLine = readLine(inputStream);
            contentLength -= (readLine.getBytes(UTF8).length + CRLF.length());
            if (readLine.startsWith("Content-Disposition:")
                    && readLine.indexOf("filename=") > 0) {
                rawfilename = readLine.replaceAll(".*filename=", "");
                char quote = rawfilename.charAt(0);
                rawfilename = rawfilename.substring(1);
                rawfilename = rawfilename.substring(0,
                        rawfilename.indexOf(quote));
                firstFileFieldFound = true;
            } else if (firstFileFieldFound && readLine.equals("")) {
                atStart = true;
            } else if (readLine.startsWith("Content-Type")) {
                rawMimeType = readLine.split(": ")[1];
            }
            count++;
        }

Value for CONTENT_DISPOSITION_COUNTER we set = 200 (I think this is enough).

Idea is that we are checking first "n" line and if no "Content-Disposition" and "filename=" than we are using default values for rawfilename and rawMimeType (no infinitive loop).

Of course we need to add custom FileUploadHandler. Changes in VaadinServlet:

@Override
    protected VaadinServletService createServletService(DeploymentConfiguration deploymentConfiguration) throws ServiceException {
        VaadinServletService service = new VaadinServletService(this,deploymentConfiguration) {
            @Override
            protected List<RequestHandler> createRequestHandlers() throws ServiceException {
                List<RequestHandler> requestHandlers = super.createRequestHandlers();
                requestHandlers.add(new FixedFileUploadHandler());
                return requestHandlers;
            }
        };
        service.init();
        return service;
    }