asked on September 13, 2024
Hi Flexmonster team,
We have run into a scale issue that has brought us to a halt. Here is our scenario. We have a customer's dataset we are aggregating and displaying in a pivot table that is made up of 30M rows. We are currently using the Flexmonster Custom Data Source API.
The most granular dimensions in our pivot table are contract IDs - there are about 3M unique values for these. Anytime we add this as a row or a report filter, loading the members takes minutes because the client library requests all the members contained in the displayed result set. The thing is, we already have them in the select response when we have it as a row.
We saw this improvement back in 2022 to not need to request members when the data element is set to be not filterable: https://www.flexmonster.com/question/custom-datasource-api-avoid-members-call-on-some-fields/
There is also a filter option to prevent live search and instead require someone to type in a search for a filter and press enter to see the results. Here is Flexmonster's fiddle example: https://jsfiddle.net/flexmonster/w0e1aykt/
Our customers need to be able to search and filter by contract IDs, but they don't need to see all the contract IDs expanded for a non-filtered dataset. What we want to do is make it so ContractID isn't needed by members, but does allow for server side searching, only returning back the members that match the non-live search box to then allow the user to select or deselect. Is there a way to achieve this or make this an improvement to the library?
Thanks,
Bill
⋅ September 16, 2024
Hello, Bill!
Thank you for reaching out to us.
We have tested the described use case on our side and recommend sending an empty members list for the "Id" field to the client.
The idea is that for the field located in the Report filters no members are displayed on the grid. So, you can send an empty array with the response for /members request for the "Id" field.
This way, the label filter for the "Id" field remains working as it uses the client input for filtering, and no members are sent to the client.
The example response should look as follows:
{
members: []
}
Please try the suggested approach on your side and let us know if it works well for you.
Looking forward to hearing from you.
Kind regards,
Solomiia
⋅ September 16, 2024
Solomiia,
I work with Bill and took a look at your recommendation. There are a couple issues with this solution that would prevent us from using it.
First, this will only work when the field is a Report Filter. Once it is moved into a row(or column), the client will not expand any rows for the given field. Previously, in this thread it looks like there was a solution for this by setting `"filters": false` in the mapping config for the field. This fixes the problem when the field is a row, Unfortunately when we do this, the field cannot be used as a report filter. This is the biggest blocker for us since we need to support adding this field as a row at times.
Second, It is somewhat confusing when I open the report filter that it shows me the selection filter with "0 of 0 selected". The user has to have the knowledge to click into the labels to be able to apply a filter. They are still able to use the search bar but it doesn't do anything.
Third, when setting members to [], there appears to be a bug that once I set a filter via the "label" filter I am unable to ever remove the filter. If I clear the label filter, I am unable to click "apply" to close the filter modal with no filters applied.
Is there a solution that would allow us to use the field as either a row or a report filter without needing to make a members request that returns every single possible value for the field? Given its possible to not select all members when using `"filters": false`. It seems like it should be possible to add the ability to add a "labels" only filter for a field without making a members call for all possible values. An example of what this might look like is here(I edit this video to what it might look like, it doesn't work like this today).
Thanks,
Brian
⋅ September 17, 2024
Hello, Brian!
Thank you for your feedback and additional details about the use case.
To maintain the best user experience for this use case we suggest replacing the default filter pop-up with your own custom filter control. This way you can provide the filter inputs you need (for example, have only a labels filter) and use Flexmonster API calls to apply the filter underneath. And it would be easy to add your own search with server-side filtering logic.
We have prepared a JSFiddle sample with predefined members to illustrate the idea: https://jsfiddle.net/flexmonster/9tg68b4w/.
Also, our team is going to research possible ways to avoid loading all the members to the pivot table, to cover the use cases where the whole list of members is not needed immediately. We will be back to you with the research results with ETA October 15th.
Please let us know if you need our further assistance in configuring a custom filter control in Flexmonster.
Looking forward to hearing from you.
Kind regards,
Solomiia
⋅ September 17, 2024
Hi Solomiia,
Thank you for the response. While the custom filter example is interesting, it really doesn't buy us anything because we are currently in a catch-22. If we allow the column with massive amounts of unique data to be filterable, the members API call the library makes under the covers kills our performance. If we make the column as not filterable, then the gear icon disappears and we can even leverage the event override.
Is there a way to tell the library to show the gear icon on a non-filterable column, even if it doesn't trigger the default event? That would allow us to then implement our own custom event handler to make a server side call outside of flexmonster without us having to do crazy hacking on the javascript to try to determine what cells should have a filter icon but dont.
Ultimately preventing the call to members for members we already have in the selected result object would be preferrable, but we could at least work around this for now if we can renable the gear icon for filtering without it making the call to members.
Thanks,
Bill
⋅ September 18, 2024
Hello, Bill and Brian!
Thank you for your swift response.
Please note that with the filters:false set for the field, this field won't be displayed in the Report Filters area.
So, we assume the approach where "Id" field is located only in Rows/Columns and there are no /members request works for you.
Then, it is possible to query the necessary HTML elements to restore the gear icon on the cell header and add an event listener to it: https://jsfiddle.net/flexmonster/cx8gknub/.
Regarding our improvements research, our team will get back to you if there are any updates on the matter within the given ETA.
Feel free to reach out in case of any other questions.
Kind regards,
Solomiia
⋅ October 16, 2024
Hello, Bill and Brian!
Hope you are doing well.
Our team is happy to inform you about our research updates. As a result, a new filters.members property for the custom data source API was added to the mapping.
When filters.members is set to false, the /members request won't be sent for that field. Instead, the members will be read from the /select response to display the data on the grid.
In addition, the filter icons will remain visible and the field can be used in the report filters area. This way you can use the filter icons and replace the default filter pop-up with your custom one.
You can see how the new property works in this JSFiddle: https://jsfiddle.net/flexmonster/k2L90rpb/.
Also, kindly note that in the next minor release, the selection filter won't be available in the default filter pop-up window for fields that have filters.members set to false. Only the labels and values filters will be available.
The feature is included in the 2.9.88 version of Flexmonster: https://www.flexmonster.com/release-notes/version-2-9-88/.
You are welcome to update the component: https://www.flexmonster.com/doc/updating-to-the-latest-version/.
Please let us know if the new property is helpful for your use case.
Kind regards,
Vera
⋅ October 16, 2024
We are going to be busy for the next week on some important projects, but once we get a chance to breath we will implement this change and let you know how it works!
⋅ October 17, 2024
Hello, Bill,
Thank you for the update.
We will be looking forward to hearing from you.
Kind regards,
Vera
⋅ October 29, 2024
Vera,
I took a look at this and it works as I would expect when I set `filters.members` to false in report configuration mapping json that is load, but it is not working as expected when returning the `filters.members` = false via the API as part of the fields response here. When returned as part of the fields response it is still sending the members request, although it is correctly not displaying the list of values in the filter but that doesn't help with the performance issue. Is there a way to allow us to return this value as part of the fields API response and have it behave the same as if it were configured as part of the report json? We were trying to make this dynamic per field and having this required as part of the hardcoded configuration mapping makes that a lot harder to handle.
A sample response we are sending back as part of the fields request is:
```
{
"fields": [
{
"uniqueName": "field_with_lots_of_values",
"type": "string",
"caption": null,
"folder": null,
"aggregations": null,
"filters": {
"members": false,
"query": null,
"valueQuery": null
}
}
}
```
⋅ October 29, 2024
Hello, Brian!
Thank you for getting back to us.
We are glad to hear the new property works as expected.
Our Dev team will fix the members property in the FilterConfigObject for custom data source API in the next minor version with an ETA November 11th.
Upd: After the fix, /members request won't be sent when filters.members: false is set in the FilterConfigObject. We will notify you about the update.
Feel free to reach out to us in case of any other questions.
Kind regards,
Solomiia
⋅ October 29, 2024
thank you! To be clear it appears the "members" property is in the FilterConfigObject for the custom data source API but it doesn't seem to prevent the members call like it does when adding members = false in the FieldMappingObject does.
⋅ October 30, 2024
Hello, Brian!
Thank you for the clarification.
We understand the case and will double-check that the /members request is not sent when filters.members are set to false in both FieldMappingObject and in FilterConfigObject.
Our team will notify you about any updates on the matter.
Feel free to ask if there are any further questions.
Kind regards,
Solomiia
⋅ November 12, 2024
Hello, Bill and Brian!
Hope everything is well.
Our team is glad to let you know that if the selection filter is disabled for a particular field, the /members request will now not be sent to the server when opening the filter view for that field.
The fix is included into 2.9.90 version of Flexmonster: https://www.flexmonster.com/release-notes/version-2-9-90/.
You are welcome to update the component: https://www.flexmonster.com/doc/updating-to-the-latest-version/.
Please let us know if the filters.members: false works as expected after the fix.
Looking forward to hearing your feedback.
Kind regards,
Solomiia
⋅ November 18, 2024
Solomiia,
I took a look at this and it isn't working as I would expect. When I set filters.members: false in the API response and then I expand the field in the pivot, it acts as if there are no values for that field and displays nothing. For example here I am returning false and when I expand "Accounting Date" nothing shows. Here I am not returning false and you can see when I expand "Accounting Date" there is a value displayed as I would expect.
When I set filters.members: false in the report mapping json it works as expected and is only not displaying the values when the fields response returns filters.members: false for the field. I attached an example of what our fields response looks like, please let me know if this is an issue with my response or something internal to the flexmonster code.
Thanks,
Brian
⋅ November 19, 2024
Solomiia, we have upgraded our support plan, so if it is easier for us to video conference with an engineering directly to show them the problem real-time in our production environment, we would be happy to do that.
⋅ November 19, 2024
Hello, Bill!
We are happy to hear about your upgrade.
Please note that our Tech team is currently working on your case and trying to reproduce the described behavior on our side.
We will be back to you soon.
Kind regards,
Solomiia
⋅ November 19, 2024
Hello, Bill and Brian!
Thank you for giving us some time.
Our team has reproduced the described behavior of filters.members: false when set in the /fields response. We will fix it with the ETA November 25th.
We will notify you as soon as the fix is ready.
Best regards,
Solomiia
⋅ November 26, 2024
Hello, Bill and Brian!
Hope you are doing well.
Our team is glad to let you know that the issue with expands not working when the selection filter is disabled on the server side was fixed.
The fix is included in the 2.9.91 version of Flexmonster: https://www.flexmonster.com/release-notes/version-2-9-91/.
You are welcome to update the component: https://www.flexmonster.com/doc/updating-to-the-latest-version/.
Please let us know if everything works as expected after the fix.
Best regards,
Solomiia
⋅ December 5, 2024
Hello, Bill and Brian!
Hope you are having a great week.
We are wondering if you had a chance to test the latest fix with the selection filter disabled on the server side. Could you please let us know if everything works well now?
Looking forward to hearing your feedback.
Kind regards,
Solomiia
⋅ December 5, 2024
Solomiia,
Thank you! I was about to confirm that this is working as expected. Thanks again for adding support for this!
Best,
Brian
⋅ December 6, 2024
Hello, Brian!
Thank you for your feedback.
We are happy to hear everything works well now.
Feel free to reach out to us in case of any other questions.
Kind regards,
Solomiia
