bcz of this our query-service pods in crash back loop

That shouldn’t be the case. Can you share the output of describe pods for query-service?

Ok

can you also check if clickhouse is in a healthy state ? or it’s restarting as well ?

No click house is in healthy state no restart

That will help to know the correct reason. The background upload fail to s3 wouldn’t crash the DB and (should be) unrelated to query service getting into crashbackloopoff.

describe pods output

I can see

OOMKilled

as the reason for termination, we will have to increase the limits. cc @Ankit Nayan @Srikanth Chekuri

@sudhanshu dev Can you share the output for this from clickhouse

select count() from signoz_metrics.time_series_v2;

so we can give some rough estimate of how much RAM is needed for query service to not crash?

Got it

@sudhanshu dev there is some inefficiency in loading timeseries right now. We should be fixing this within 3-4 weeks. Right now we are trying a temp fix and estimate.

ok

Is it possible to not limit the query service in resources and run it for a few mins (15-30 mins should be good), we want to collect pprof data

ok

also, I see you running in

0.11.0

, we have raised a fix to reduce memory usage of query-service in

v0.11.1

. Can you give it a try?

Sure

Let us know if the query-service does not run within 4GB of memory

will also do that

thanks

Yes it would help us

@Ankit Nayan @sudhanshu dev Port-forward pprof port

6060

of query-service container:

kubectl -n platform port-forward pod/my-release-signoz-query-service-0 6060:6060

In another terminal, run the following to obtain pprof data: • CPU Profile

curl "<http://localhost:6060/debug/pprof/profile?seconds=30>" -o query-service.pprof -v

• Heap Profile

curl "<http://localhost:6060/debug/pprof/heap>" -o query-service-heap.pprof -v

Got it