@Alejandro Decchi it depends on the number of otel-collectors running. Each instance opens 3 connections to clickhouse (for metrics, traces and logs) and 1 connection from query-service. The more instances of otel-collector you have, the more will be the number of connections.

The metric you shared is for the server which is not really big given ClickHouse accepts a much higher number of connections. I assume the first image is from the dashboards which are provided by the query service. There is no way to configure the max conn through files. Is this happening frequently?

Hi @Ankit Nayan, we are running 3 instances of otel-collector currently. Concerning your question @Srikanth Chekuri, we were aware of this issue last Thursday and since that, it has been happening continuously. However, it only happens in one of the dashboards we have configured inside the company. The pattern that I guess is producing this behavior is related to the fact that this dashboard is composed of 32 panels trying to retrieve their data at the same time. Indeed some of the panels retrieve their data successfully.

Thanks for info. Is it the same panels that timeout err with this or is it random?

@Mariano Mirabelli 3 instances should not cause an issue. Can you plot a chart to monitor the number of connections. Would like to see how the connections change with this query being fired. Apart from that all 32 panels should not load at once and there should be lazy loading based on user's viewport. We have picked up this issue https://github.com/SigNoz/signoz/issues/2005

@Srikanth Chekuri No, this is random. Indeed, I realized that depending on the time selected to retrieve the data (30 minutes, 1 hour, and so on) the panels shown as unavailable are different. In addition to that, the pattern tends to be that in the first load of the dashboard, we use to observe more panels as unavailable in comparison with clicking the refresh button in the SigNoz UI once the dashboard has been rendered.

any feedback ?