How much memory should the query service need? I am running it with 512MB and it seems to be dying due to OOM errors. Does it need to scale according to how much clickhouse data there is or maybe there is a memory leak? On version 10.2

Right now, it should scale based on the number of timeseries. This will be improved soon in upcoming versions.

Does that correlate to the number of services?

can you run

select count() from signoz_metrics.time_series_v2;

You should try upgrading the memory allocation of the query-service. @Srikanth Chekuri any requirements estimate assuming twice the above mentioned number to be on the safer side?

I went from 512 to 1024 and seems to be stable for now

cool

This is the rough estimation based on the past observations and generic requirements. For 100k time series with max of 30 labels it will be in the range of 190-240MB. The query service also does other things + process runtime needs some memory. So in your case the time series alone will eat up the all the memory you allocated and cause OOM.

I see, is there some kind of caching going on in the service? Look forward to this being optimized as our entire company will depend on this 1 service being up to send alerts to our Pagerduty. If it goes down we are blind.

@Ankit Nayan we should add a metrics summary which provides cardinality details of each metric. That would be a) useful to devs to understand their cardinality numbers / sample tag values. b) track cardinality explosions.

nice idea...

Sure. Gimme a min.

We should also have some OSS design contributors to pick up a few design tasks or atleast suggest a few mockups

Here you go: https://github.com/SigNoz/signoz/issues/1541

I added a few more services and had to increase memory again to 2048. Will see how much more I need