asked on January 20, 2021
Hi flexmonster,
My question might seem a bit strange, so I'll give some contextual elements to help the understanding. Sorry it's a bit long :).
In the context of the integration of the flexmonster component into our Emasphere software, we want to give the possibility to our users to pre-configure their pivot table. We already use flexmonster for some time to display the details of an aggregation, but its starting configuration is pretty much standardized (always the same rows and columns). The user can then configure the pivot the way he likes, but the result is not saved for later usage.
Now we would like the user to pre-configure it, by specifying the fields (hierarchies) he'd like to use as the aggregation, row and columns... and be able to set filters (this is where we have an issue). To allow this, we have a "configurator" screen in our app which also leverage on flexmonster. The current principle is to give flexmonster a portion of the client's data (as a JSON data source), let the user configure everything, then save the flexmonster report configuration in our database. After that configuration step, the user is able to use that pre-configured report in his visuals.
The key here is that we give only a portion of the data to flexmonster when in the configurator. This is to avoid retrieving the whole client's database, with all the possible fields, and transport it to the front-end. Our clients inject their data daily or monthly into our application, so we ensure there is always a time window in every request we do. Indeed, the total amount of data per client is potentially infinite. In the case of our configurator, we display the complete data of the current month by default.
This all goes fine when configuring aggregations, rows and columns. However, when configuring filters, we come to the limitation that flexmonster only propose the values (members) he can find in the data portion (i.e. the current month). We have a mechanic to change the time window to something larger (e.g. the current year), but it doesn't really fix the issue. The user will find more values (maybe not all of them) and will configure his filter properly, but then that filter will be broken again when he comes back later on in this configurator to change the setup. Even if he doesn't touch the filter, the values not being in the current month data will be wiped out.
So, we are looking for a way to specify all the possible values of a field, either when building up the report object or at runtime via an API call. This would allow setting up filters using all the possible values of that field, and not only what is visible in the data.
Reading your nice documentation, we came across the custom data source API. It does everything we need except that it expects the back-end to return the already aggregated pivot table. We actually selected flexmonster because it is capable of aggregating raw data on the client side, so we would like to keep it that way.
Would there be any way we can use to specify all fields and their values (like with the custom data source) but still be able to provide raw data to flexmonster and aggregation to happen on the client side ?
Thanks for reading this long question 🙂
Olivier
⋅ January 21, 2021
Hello, Olivier,
Thank you for describing your case in such detail.
Indeed, one of the main advantages of using the custom data source API approach is that not all of the data is loaded into the browser at once. Only the part needed for the current representation is loaded. This makes it possible to work with larger data volumes.
For your case, it seems the Flexmonster Data Server is the right tool for the task at hand. Created by our team, the Flexmonster Data Server is a ready-to-use implementation of the custom data source API. The Flexmonster Data Server supports connecting to different data sources (CSV, JSON, SQL Databases). All the aggregation logic and custom data source API response structure generation has been taken care of. This approach would require minimal configuration from your end.
Using the Flexmonster Data Server, it is possible to retrieve all the possible members of a hierarchy without loading the entire data set into the browser.
Please see the following JSFiddle we have prepared for you: https://jsfiddle.net/flexmonster/gdk8Lctb/.
In the example, the getMembers() API call is used to get all the possible members of the "Category" hierarchy.
Please let us know if such an approach would work for you.
If any questions arise, please feel free to reach out - we will be happy to help.
Looking forward to your reply.
Kind regards,
Vera
⋅ January 28, 2021
Hello, Olivier,
We would like to kindly take an interest if you found our suggestion helpful.
Do you think the approach using Flexmonster Data Server would work for your case?
Please share your thoughts with us.
Looking forward to your reply.
Kind regards,
Vera
⋅ February 1, 2021
Hello Vera,
Sorry for the late answer, I've been quite busy last week.
The approach looks quite promising indeed. I had a closer look at the component, and it raises a few questions.
As of now, the json we give to the flexmonster component comes from our java services. It results from a database query (including the time window as a "where" clause) and is manipulated before being returned to the front-end. The manipulations mainly consist in removing some fields not visible to the user, adding field translations, pre-configure the aggregation field, etc. All that to point to the fact that it's not a simple database query. It needs input (the time window) and some manipulations.
So, the main question is how to integrate Flexmonster Data Server with our 100% java back-end. Reading the documentation, it looks feasible to use FDS as a library, which would be the preferred way, so we can keep our logic and use FDS as a "facade" for the communication with the front-end. The thing is... its a .NET library. I don't have any experience with integrating a .NET dll into a java service. Do you have some experience on your side ?
I might be wrong, but I don't see a good way to have FDS as a standalone component part of our back-end services. I see several issues:
That's why I would prefer the library option. I just don't see how to do that exactly with java.
What do you think ?
Kind regards,
Olivier
⋅ February 3, 2021
Hello, Olivier,
Thank you for your reply and for providing further details.
Considering that you are using dynamic queries, we would like to kindly explain that the best solution would be to implement the custom data source API.
Another solution would be to create custom filter controls outside of Flexmonster and load the required members for the custom filters by selecting the necessary data on your server.
Please consider that Flexmonster Data Server relies on the set of predefined indexes. Our team kindly recommends implementing your own custom data source API approach when it comes to dynamic queries. We would like to mention that using your own implementation of the custom data source API would provide more flexibility over how the data is processed.
We have a special guide with steps on how to create your own custom data source API implementation.
Our team has also prepared two examples of the custom data source API implementation: a Node.js sample server and a .NET Core sample server.
In fact, the Flexmonster Data Server is a further development of our sample .Net Core server with several improvements and optimizations added to it. You are welcome to check out the .Net Core sample server's source code on GitHub: https://github.com/flexmonster/api-data-source.
Although the sample servers are not written in Java, you are welcome to check out the custom data Source API server implementation tutorial and inspect the existing implementations' structure to build your own Java custom data source API implementation.
Please feel free to reach out if any questions arise - we will be happy to help.
Kind regards,
Vera
⋅ February 16, 2021
Hello, Olivier,
Our team is wondering if you found our previous response helpful.
Do you think one of the suggested approaches would work for you?
Please let us know if there are any questions that we could help you with.
Looking forward to your response.
Kind regards,
Vera
⋅ February 16, 2021
Hello Vera,
Yes the answer has been helpful for us, at least we know we should not investigate that data server component anymore.
I'll definitely have a look at the material you provided in your answer, thanks a lot for this.
So far we see 3 solutions to our problem:
The third bullet point leads me to another question for you: is it possible to completely deactivate the filtering of a flexmonster report ? It's not critical, but of course we don't want our user to be confused about where to set up filters (in the flexmonster report or in the emasphere surroundings)
What do you think ?
⋅ February 17, 2021
Hello, Olivier,
Thank you for your reply.
Our team is currently working on your case.
We will make sure to reach out to you shortly.
Kind regards,
Vera
⋅ February 18, 2021
Hello, Olivier,
Thank you for your response.
Out of the suggested solutions, we would recommend the custom data source API as the most performance-efficient approach when it comes to large data volumes.
We understand that implementing the custom data source API could take some time. Still, it would provide more flexibility in data processing and a more reliable way to work with large data sets in the long run.
In addition, we have detailed guides and two example implementations (Node.js and .NET Core) that could be used as a base for your project.
As for hiding Flexmonster's UI filter controls, we would like to confirm that it is possible.
This can be achieved by specifying the showFilter: false report option.
Here is a JSFiddle example for illustration: https://jsfiddle.net/flexmonster/ten6vu95/
Please let us know if this helps. If any further questions arise, please feel free to reach out.
Looking forward to your response.
Kind regards,
Vera
⋅ February 19, 2021
Ok, thanks a lot for your help !
