The file upload component has setMaxFileSizeUpload component | Vaadin components and it would be great to apply the same restriction to files uploaded via RTA
The brute force solution would be to restrict upload size on the servlet container or in e.g. proxy configuration; that will fail in an ugly way, though (the user will get a message like “Server Connection lost”). You could open a feature ticket for a component API enhancement. Note that you will probably want the servlet container and/or proxy limits set in place regardless.