Grid & Grid Pro IndexOutOfBoundsException

Flow
1

Dear Team,

I am trying to implement a lazy loading grid based on the masterdetail example from start.vaadin.com (Version 14.0.4).

Therefore we added a postgres Database as the datasource and added a dataprovider taken from the documentation.
We added 100000 datasets. When we scroll through the data slowly, the grid works fine but when we scroll fast we get an IndexOutOfBoundsException.

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at java.util.ArrayList.rangeCheck(ArrayList.java:657) ~[na:1.8.0_191]

	at java.util.ArrayList.get(ArrayList.java:433) ~[na:1.8.0_191]

	at com.vaadin.flow.data.provider.DataCommunicator.lambda$getJsonItems$3(DataCommunicator.java:611) ~[flow-data-2.0.11.jar:2.0.11]

	at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250) ~[na:1.8.0_191]

	at java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110) ~[na:1.8.0_191]

	at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693) ~[na:1.8.0_191]

	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_191]

	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_191]

	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_191]

	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_191]

	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_191]

	at com.vaadin.flow.data.provider.DataCommunicator.getJsonItems(DataCommunicator.java:613) ~[flow-data-2.0.11.jar:2.0.11]

	at com.vaadin.flow.data.provider.DataCommunicator.collectChangesToSend(DataCommunicator.java:556) ~[flow-data-2.0.11.jar:2.0.11]

	at com.vaadin.flow.data.provider.DataCommunicator.flush(DataCommunicator.java:473) ~[flow-data-2.0.11.jar:2.0.11]

	at com.vaadin.flow.data.provider.DataCommunicator.lambda$requestFlush$2f364bb9$1(DataCommunicator.java:421) ~[flow-data-2.0.11.jar:2.0.11]

	at com.vaadin.flow.internal.StateTree.lambda$runExecutionsBeforeClientResponse$1(StateTree.java:364) ~[flow-server-2.0.11.jar:2.0.11]

	at java.util.ArrayList.forEach(ArrayList.java:1257) ~[na:1.8.0_191]

	at com.vaadin.flow.internal.StateTree.runExecutionsBeforeClientResponse(StateTree.java:361) ~[flow-server-2.0.11.jar:2.0.11]

	at com.vaadin.flow.server.communication.UidlWriter.encodeChanges(UidlWriter.java:392) ~[flow-server-2.0.11.jar:2.0.11]

	at com.vaadin.flow.server.communication.UidlWriter.createUidl(UidlWriter.java:182) ~[flow-server-2.0.11.jar:2.0.11]

	at com.vaadin.flow.server.communication.UidlRequestHandler.writeUidl(UidlRequestHandler.java:116) ~[flow-server-2.0.11.jar:2.0.11]

	at com.vaadin.flow.server.communication.UidlRequestHandler.synchronizedHandleRequest(UidlRequestHandler.java:89) ~[flow-server-2.0.11.jar:2.0.11]

	at com.vaadin.flow.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40) ~[flow-server-2.0.11.jar:2.0.11]

	at com.vaadin.flow.server.VaadinService.handleRequest(VaadinService.java:1540) ~[flow-server-2.0.11.jar:2.0.11]

	at com.vaadin.flow.server.VaadinServlet.service(VaadinServlet.java:246) [flow-server-2.0.11.jar:2.0.11]

	at com.vaadin.flow.spring.SpringServlet.service(SpringServlet.java:95) [vaadin-spring-12.0.5.jar:na]

	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) [javax.servlet-api-4.0.1.jar:4.0.1]

	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:712) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:459) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:352) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:312) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.springframework.web.servlet.mvc.ServletForwardingController.handleRequestInternal(ServletForwardingController.java:141) [spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177) [spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:52) [spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039) [spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) [spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) [spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908) [spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665) [javax.servlet-api-4.0.1.jar:4.0.1]

	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) [spring-webmvc-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) [javax.servlet-api-4.0.1.jar:4.0.1]

	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) [tomcat-embed-websocket-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) [spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) [spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) [spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) [spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) [spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:118) [spring-web-5.1.9.RELEASE.jar:5.1.9.RELEASE]

	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.StandardContextValve.__invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:41002) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_191]

	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]

	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.22.jar:9.0.22]

	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_191]

This is the java code:

package com.example.application.spring.views.dashboard;

import com.example.application.spring.BroadcastMessage;
import com.example.application.spring.Broadcaster;
import com.example.application.spring.MainView;
import com.example.application.spring.backend.BackendService;
import com.example.application.spring.backend.Employee;
import com.example.application.spring.views.dashboard.DashboardView.DashboardViewModel;
import com.vaadin.flow.component.AttachEvent;
import com.vaadin.flow.component.DetachEvent;
import com.vaadin.flow.component.Tag;
import com.vaadin.flow.component.UI;
import com.vaadin.flow.component.button.Button;
import com.vaadin.flow.component.dependency.JsModule;
import com.vaadin.flow.component.grid.Grid;
import com.vaadin.flow.component.gridpro.GridPro;
import com.vaadin.flow.component.polymertemplate.Id;
import com.vaadin.flow.component.polymertemplate.PolymerTemplate;
import com.vaadin.flow.component.textfield.PasswordField;
import com.vaadin.flow.component.textfield.TextField;
import com.vaadin.flow.data.binder.Binder;
import com.vaadin.flow.data.provider.CallbackDataProvider;
import com.vaadin.flow.data.provider.DataProvider;
import com.vaadin.flow.router.AfterNavigationEvent;
import com.vaadin.flow.router.AfterNavigationObserver;
import com.vaadin.flow.router.PageTitle;
import com.vaadin.flow.router.Route;
import com.vaadin.flow.router.RouteAlias;
import com.vaadin.flow.shared.Registration;
import com.vaadin.flow.templatemodel.TemplateModel;

import java.io.Serializable;
import java.util.List;

@Route(value = "dashboard", layout = MainView.class)
@RouteAlias(value = "", layout = MainView.class)
@PageTitle("Dashboard")
@JsModule("./src/views/dashboard/dashboard-view.js")
@Tag("dashboard-view")
public class DashboardView extends PolymerTemplate<DashboardViewModel> implements AfterNavigationObserver, Serializable {

    public static interface DashboardViewModel extends TemplateModel {
    }

    private final BackendService service;

    @Id
    private GridPro<Employee> employees;

    @Id
    private TextField firstname;

    @Id
    private TextField lastname;

    @Id
    private TextField email;

    @Id
    private PasswordField password;

    @Id
    private Button cancel;

    @Id
    private Button save;

    private Binder<Employee> binder;

    private Employee selected;

    Registration broadcasterRegistration;

    public DashboardView(BackendService service) {
        this.service = service;

        configureGrid(service);

        // Configure Form
        binder = new Binder<>(Employee.class);

        // Bind fields. This where you'd define e.g. validation rules
        binder.bindInstanceFields(this);
        // note that password field isn't bound since that property doesn't exist in
        // Employee

        // the grid valueChangeEvent will clear the form too
        cancel.addClickListener(e -> {
            employees.asSingleSelect().clear();
            selected = null;
        });

        save.addClickListener(e -> {
            binder.writeBeanIfValid(selected);
            service.save(selected);

            CallbackDataProvider<Employee, Void> employeeList = (CallbackDataProvider<Employee, Void>) employees.getDataProvider();
            employeeList.refreshItem(selected);
            Broadcaster.broadcast(new BroadcastMessage(selected, BroadcastMessage.BroadcastMessageType.REFRESH_DASHBOARD));
        });
    }

    private void configureGrid(BackendService service) {
        // Configure Grid
        employees.setDataProvider(DataProvider.fromCallbacks(
                // First callback fetches items based on a query
                query -> {
                    // The index of the first item to load
                    int offset = query.getOffset();

                    // The number of items to load
                    int limit = query.getLimit();

                    List<Employee> persons = service.getEmployees(offset, limit);

                    return persons.stream();
                },
                // Second callback fetches the number of items
                // for a query
                query -> service.getEmployeeCount()));
        employees.addColumn(Employee::getFirstname).setHeader("First name");
        employees.addColumn(Employee::getLastname).setHeader("Last name");
        employees.addColumn(Employee::getEmail).setHeader("Email");
        employees.setSelectionMode(Grid.SelectionMode.SINGLE);
        //when a row is selected or deselected, populate form
        employees.asSingleSelect().addValueChangeListener(event -> populateForm(event.getValue()));
    }

    @Override
    public void afterNavigation(AfterNavigationEvent event) {
    }

    private void populateForm(Employee value) {
        this.selected = value;
        // Value can be null as well, that clears the form
        binder.readBean(value);
        // The password field isn't bound through the binder, so handle that
        password.setValue("");
    }

    @Override
    protected void onAttach(AttachEvent attachEvent) {
        UI ui = attachEvent.getUI();
        broadcasterRegistration = Broadcaster.register(broadcastMessage -> {
            if (BroadcastMessage.BroadcastMessageType.REFRESH_DASHBOARD.equals(broadcastMessage.getMessageType())) {
                ui.access(() -> employees.getDataProvider().refreshItem(broadcastMessage.employee));
            }
        });
    }

    @Override
    protected void onDetach(DetachEvent detachEvent) {
        broadcasterRegistration.remove();
    }
}

The dataservice is a Spring Service Class with two Methods:

    public List<Employee> getEmployees(int offset, int limit) {
        Pageable pageable = PageRequest.of(offset, limit);
        return employeeRepository.findAllBy(pageable);
    }

    public int getEmployeeCount() {
        int employees = Math.toIntExact(employeeRepository.count());
        return employees;
    }

Can you please advise how to solve the issue?

Kind regards
Florian

2

Is your getEmployeeCount() guaranteed to return size of the list you get from getEmployees(…)?

3

I am experiencing something similar myself. What seems to be the case is that the Spring-data PageRequest.of does not work on the same basis as Vaadin DataProvider.fromCallbacks. The definition of PageRequest.of (see https://docs.spring.io/spring-data/commons/docs/current/api/org/springframework/data/domain/PageRequest.html#of-int-int-) requires a page number and a page size. The description is a bit brief but I can only assume it means that the page size should be fixed. For Vaadin Flow Grid this seems to be 50, though I couldn’t find methods to get or set this value. The first parameter is the page number in terms of blocks of 50 records, starting at 0.

So, I think the PageRequest.of should be:

final int PAGE_SIZE = 50;
Pageable pageable = PageRequest.of(offset/PAGE_SIZE, PAGE_SIZE);

So the limit is not actually used. However, if you don’t call query.getLimit(); Vaadin will throw an error saying you should call it. Even though the limit is not used anywhere I think it will still work because the actually number of records returned will match the limit (because the count from the DB should be correct).

I’m happy to be corrected if wrong!

I am actually having a different problem which means I can’t prove the above. I am using DB2/400 and the first page of data returned works fine. Hibernate just gets the first 50 records. For the second page however, hibernate creates a more complex query which gets the first 100 records and then selects records 51 and up. This select however returns no records and so I get the error above. I think this is a Hibernate issue which I am trying to resolve.

4

For completeness sake. I have resolved my problem mentioned at the end of my last post above. The issue was with using an IBMi logical to perform a UNION of 10 very similar tables. See my Stack Overflow question for the details:

[Hibernate paging query returns no records with DB2/400 dialect]
(https://stackoverflow.com/questions/58624972/hibernate-paging-query-returns-no-records-with-db2-400-dialect)

5

I think I am now experiencing Florian’s problem. From my post above I suggested fixing the (limit) page size to 50 as this is what Vaadin seems to use. I then checked out a dataset with 591 records. If I scroll down slowly with my mouse wheel then my logging tells me that the (limit) page size is 50 and everything works fine. If however I click the vertical scroll bar about two thirds of the way down then the offset is logged as 350 and the limit as 150. If it click right on the end of the vertical scroll bar the offset is 500 and the offset 91.

I can’t figure out how to map those numbers to a Spring-data-jpa PageRequest.of(pageNumber, pageSize) call. I guess I could stick with a page size of 50 and make multiple requests to collect the rows I need before returning to Vaadin.

The only fix from Vaadin that I can think of would be to be able to get and set the page (limit) size to a specific value. I’m not sure it makes a whole lot of sense for it to jump to 150 as in my example above, or indeed 91. Just getting that value would be enough, as long as it didn’t change for a Grid instance.

6

I have just come across:

[Grid.setPageSize(int pageSize)]
(https://vaadin.com/api/platform/14.0.12/com/vaadin/flow/component/grid/Grid.html#setPageSize-int-)

I thought it might solve the problem above so I set it to 50 for all the grids I was using. Testing reveals though that in the DataProvider.fromCallbacks the limit is still set to 150 on occasion (and offset is not always a multiple of 50). Perhaps I am doing something wrong or misunderstand.

7

Good morning!

I’ve been facing the same problem. Somewhere in the code there is a number breaking the lazy loading. If you use a pageSize of 50 it works, but using 20 fails in ArrayList rangeCheck.

I’m now using Vaadin 14.1.3. Please fix it.

Thank you!

8

Can you check the existing tickets, like

https://github.com/vaadin/flow/issues/6631#issuecomment-539661604 or https://github.com/vaadin/flow/issues/6996

Also

https://github.com/vaadin/flow/issues/6072

https://github.com/vaadin/flow/issues/6631

I’m now using Vaadin 14.1.3. Please fix it.

If your case is different, then you should open a new ticket in GitHub and include example that reproduces the problems, so that it can be debugged.

9

From my work on it a couple of months back I seem to recall that the basic problem was that Vaadin’s strategy was more flexible than Spring-data-jpa’s. If it was the other way around things would be fine of course. I did solve it for myself in that if Vaadin asks for 150 records, then I would loop my jpa code to get 3 x 50 records. So far Vaadin has always asked for multiples of 50 records, apart from at the end of the table, which works fine anyway. I think the very simple solution would be for Vaadin to fix the records read to 50, or whatever was set by the coder using setPageSize(n). I believe it is the flexibility that is the problem. Vaadin says “start at offset x and read y records”, JPA says “tell me your page size and which page you want”.

10

Mark Wheadon:
From my work on it a couple of months back I seem to recall that the basic problem was that Vaadin’s strategy was more flexible than Spring-data-jpa’s. If it was the other way around things would be fine of course. I did solve it for myself in that if Vaadin asks for 150 records, then I would loop my jpa code to get 3 x 50 records. So far Vaadin has always asked for multiples of 50 records, apart from at the end of the table, which works fine anyway. I think the very simple solution would be for Vaadin to fix the records read to 50, or whatever was set by the coder using setPageSize(n). I believe it is the flexibility that is the problem. Vaadin says “start at offset x and read y records”, JPA says “tell me your page size and which page you want”.

PageRequest.of(offset/PAGE_SIZE, PAGE_SIZE)

  • works perfect