Hello <@U02SS3ZAMKQ> I have configured Signoz on ...
# support
s
Hello @Prashant Shahi I have configured Signoz on GKE cluster in GCP for tracing purpose only. I have monitor the Signoz resources and found that the below pod is using high memory and CPU resources. signoz-otel-collector Memory: 45-51 GB approx (peak hours) CPU: 6-7 core (peak hours) chi-signoz-clickhouse-cluster Memory: 9-10 GB approx (peak hours) CPU: 5-6 core (peak hours) Also, signoz clickhouse cluster is consuming high disk usages around 1 TB in last 24 hours. I have set only 1 replica of both above component. Should i need to increase replica to reduce memory and CPU resources? or else can you please provide settings or signoz parameter to reduce the resources? Thanks
s
Something is wrong with this. Please share the heap profile result.
Copy code
kubectl -n <your-namespace> port-forward pod/<collector-pod-with-high-memory-usage> 1777:1777
Copy code
go tool pprof <http://localhost:1777/debug/pprof/heap>
And share the result of
top
from heap profile
s
Hello @Srikanth Chekuri Thank you for your response. As per your request, I have attached heap profile. Moreover, can we use bucket instead of disk with chi-signoz-clickhouse-cluster component? If yes, can you please share the information how we can configure it?
s
How many collectors are you running? How many spans do you ingest perminute?
s
Thank you for your response. I have decrease spans limit which has fixed the issue.