asked on May 11, 2021
We plan to use Flexmonster to display flat data, but the data source should be very large.
Is there built-in pagination if I use say custom API data source?
I would expect events would be triggered on scroll and also upon changing sorting/filtering options.
If you do not have built-in support, how can I achieve that?
Will I need to override cell to add my own backend sorting mechanism and also add a pagination toolbar?
Thanks!
⋅ May 13, 2021
Hello, Rodrigo,
Thank you for writing to us.
As of right now, Flexmonster does not provide a built-in pagination feature due to particular technical specificities of the component.
At the same time, your own pagination implementation can be built with the custom data source API protocol. The custom API approach allows you to load the data partially using separate requests, however, all data will be loaded to the browser at once.
Thus in the flat view, the whole dataset will be loaded to Flexmonster. Therefore, if you have a large dataset, we suggest prefiltering the data on the backend to reduce the client-side load.
We would like to also mention that our component supports virtual scrolling which allows working with large datasets without paginating them. The core of this approach is that only visible rows are rendered. Please find further details here: https://www.flexmonster.com/blog/flexmonster-pivot-performance-million-rows-in-seconds/.
Hope our answer helped!
Best regards,
Milena
⋅ May 13, 2021
Hi Milena. Hope you are doing!
Initially the datasets we were expecting would be about 5k-8k. But even for smaller datasets like 5k records loading time is slow (see attached screenshot measured on an average machine). We are also planning to use it to drill-down aggregate data, and the number of rows are often 15k, 20k, 25k, or even double/triple of that.
I guess the only option is server-side pagination since we plan to run it against very large data sets
So, since there are no callback events for virtual scrolling, or server-side pagination as a whole, one route we can take to add server-side pagination would be adding a classic pagination toolbar, and for sorting we will need to override headers by using the flexmonster customizeCell callback.
Does that sound correct? Do you have other suggestions/recommendations for how to implement server-side pagination?
Thanks!
⋅ May 14, 2021
Hi, Rodrigo,
Thank you for providing further details about the situation on your side.
Our team would like to confirm that it is possible to implement your own server-side pagination with a custom pagination toolbar and sorting.
We have some suggestions on the implementation, however, they will depend on the data source you plan to use.
As we understood from the video you sent, currently Flexmonster is connected to the JSON/CSV data. On the other hand, in this thread, you mentioned a custom data source API. Therefore, we would like to ask you some questions regarding your data source:
Have you referred to custom API only because of pagination? Or you plan a migration to custom API in any case?
This info will help us to provide you with an appropriate suggestion.
Looking forward to your response.
Kind regards,
Milena
⋅ May 14, 2021
Hi Milena,
To give you more background:
Yes, in the video we sent you we load data from one of our APIs and pass down to flexmonster as JSON data (dataSource.type=json / grid type flat). We also use flexmonster for table pivoting. Here though we use custom API data source.
But answering your question
"Have you referred to custom API only because of pagination? Or you plan a migration to custom API in any case?"
I mean, the reference implementation has limit/page parameters so I thought it could have server-side pagination as well.
To be honest I'm trying to figure out what is the best way to achieve server-side pagination if we would display flat data with flexmonster component.
We want to have sort, multi-sorting column, and possibly column filtering as well (optional)
----
-EDITED-
I tried showing flat data with custom api pagination. It behaved very well for 20k records. The problem is that we learned we will need it to solve for 100k records in some scenarios, so I guess we will need server-side pagination anyways or resort to caching.
⋅ May 18, 2021
Hello!
Thank you for your answer.
In this case, we would like to recommend implementing server-side pagination for the CSV/JSON data source.
Please find some general steps below.
filename property, add a query parameter with a page number. Please have a look at the following code snippet:
flexmonster.updateData({
type: 'csv',
filename: 'http://example.com/get-data?page=2'
});
updateData on click, passing desirable filter or sorting to filename as a parameter:
flexmonster.updateData(
type: 'csv',
filename: 'http://example.com/get-data?page=2&sort=price_asc'
});
Let us know if our suggestions were useful for you.
Best regards,
Milena
⋅ June 4, 2021
Hello, Rodrigo,
We are reaching out to ask if our suggestion was helpful.
Looking forward to hearing from you.
Kind regards,
Illia
⋅ June 8, 2021
Hi Illia.
Yes, that helped. Thanks for reaching out.
Taking the opportunity let me ask you.
How can we get back the filtering selection like the JSON below that we see being passed down for custom api.
"filter": [
{
"field": {
"uniqueName": "ffiec_band"
},
"include": [
{
"member": "C&D"
}
]
},
{
"field": {
"uniqueName": "days_past_due"
},
"query": {
"greater": 3
}
}
]
I ask because when the final user clicks on a Pivot table cell, we display our own drilldown and we need to take into account filtering information which we are not doing today.
When I try to retrieve it programmatically, I get this
"caption": "Band",
"filter": {
"members": [
"ffiec_band.[c&d]"
]
}
Thanks
⋅ June 9, 2021
Hello,
Thank you for your question.
If we understand correctly, you want to get the filter in the custom data source API format rather than the internal representation retrieved using the getFilter API call.
In this case, please note that this representation is only generated when composing the request to the custom data source API endpoint. It cannot be accessed from the outside and serves only for communication.
Still, there are two workarounds that you may find helpful.
The first one implies manual translating the internal Filter Object into the Custom Data Source API Filter Object. Please use these links to our API reference to learn more about the structure of both objects.
Another approach is to let Flexmosnter generate a complete drill-through request for you and prevent the display of the default drill-through view. The generated request with all the filters will be available for further operations. Here is a JSFiddle for your reference: https://jsfiddle.net/flexmonster/pbqjvd0u/. You will find a detailed explanation of this approach in the following forum thread (the link points to the specific question rather than the entire thread): https://www.flexmonster.com/question/custom-datasource-api-lazy-loading-of-flat-table-view/#!answer-37435.
Please let us know if it works for you.
Looking forward to hearing your feedback.
Kind regards,
Illia
⋅ June 18, 2021
Hi Illia,
I wen't a slightly different route due to how my Drilldown React component works and so used customizeAPIRequest and everything worked perfectly.
I tried also your suggestion, which was a good option too... just not for my case.
Thanks for the guidance and quick replies
⋅ June 22, 2021
Hello, Rodrigo,
Thank you for your feedback.
We are happy to hear you found a suitable solution for your case.
Feel free to reach out if any additional questions arise.
Best regards,
Illia