asked on April 13, 2022
Hi,
I am testing out the data server. One thing I noticed is when I load one query and create index. It takes lot of memory. Is there any way to optimize this memory usage ?
Regards,
Vaibhav
⋅ April 13, 2022
Hello, Vaibhav!
Thank you for your question.
Kindly note that Flexmonster Data Server loads the data to RAM. In this context, we would like to know if you are using a "Database" type data source? If so, you can filter data in your query so that FDS will load only the necessary data.
Also, we recommend checking out these FDS configurations, which may be helpful for memory optimization:
"Settings" tab in Flexmonster Admin Panel.0, which means no cache will be stored."Keep data on refresh" disabled, so old data won't be saved after refreshing the index.Please let us know if this advice was helpful.
Best Regards,
Maksym
⋅ April 20, 2022
Hello, Vaibhav!
We would like to know if you tried any of the suggested approaches to optimize Flexmonster Data Server memory consumption. Were they helpful for your case?
Best Regards,
Maksym
⋅ May 18, 2024
Hello Maksym, on addition i have the same issue. I use sql server as my data source and my server has memory of 64GB. Below is my conf file
{
"DataSources": [
{
"Indexes": {},
"Type": "csv"
},
{
"DatabaseType": "mssql",
"ConnectionString": "MY_CONNECTION",
"Indexes": {
"TotalSum": {
"Query": "MY_QUERY",
"Status": "enabled",
"RefreshTime": 320
}
},
"Type": "database"
}
],
"Security": {
"Authorization": {
"Enabled": false
},
"CORS": {
"AllowOrigin": "*"
}
},
"HTTPS": {
"Enabled": true,
"Certificate": {
"Path": "C:\\xampp\\htdocs\\flexmonsterkeys\\certificate.pfx",
"AllowInvalid": false
},
"Protocols": "Http1AndHttp2"
},
"DataStorageOptions": {
"DataRefreshTime": 0,
"RetryAttempts": 30,
"CacheSizeLimit": 0,
"KeepDataOnRefresh": false
},
"AdminSafeList": "127.0.0.1;::1",
"Port": 3942,
"LoggerMinLevel": "Info",
"ReportSharingOptions": {
"EndpointEnabled": false,
"ReportExpirationTime": 0,
"MaxSizeOfFolder": 0,
"AutoDeletingEnabled": false
}
}
When refreshing the memory used by flexmonster server is very high >54GB which causes my server to stack at some point, can you include my issue in this same ticket?
⋅ May 18, 2024
See attached image of my task manager memory usage
⋅ May 20, 2024
Hello, Daniel!
Thank you for your question.
As previously mentioned, the Flexmonster Data Server loads the entire dataset into RAM for greater performance. Additionally, it creates in-memory structures for optimizing performance with pivot tables. From our experience, the Flexmonster Data Server tends to use 200%- 300% of memory compared with the original file size, depending on the dataset specifics. Hence, if the raw data volume of a dataset loaded with the query is approximately 20+GB, the resulting total volume of data is expected.
Given that you have already disabled caching and the KeepDataOnRefresh option, the only way to substantially reduce memory usage is to filter the data. If possible, do not select columns that do not need to be visualized or remove certain records with a WHERE statement in a connection query. Alternatively, the Data Sever can be run on a more powerful machine. If the options above are not viable, we suggest using a different data source that does not directly rely on your server's RAM. Please check our article for guidance on selecting the most suitable data source: https://www.flexmonster.com/blog/how-to-choose-the-best-data-source-to-use-with-flexmonster/
We are looking forward to hearing your feedback.
Best Regards,
Maksym
⋅ May 30, 2024
Hello, Daniel,
Hope you are doing well.
Just checking in to ask if you had a chance to review our previous suggestions regarding the Flexmonster Data Server's memory usage.
Please let us know if any further questions arise.
Best Regards,
Maksym