This message was deleted.

Ok, I've made some progress on this. The query-service is a statefulset and not a deployment, so running:

kubectl -n platform edit statefulset my-release-signoz-query-service

I edited the memory limit from 1000Mi to 2000Mi. Is this advisable? What are the recommendations for a production deployment?

hey @Chris Ahern what are the CPU/RAM you have allocated to the cluster where SigNoz is running - and whats the current load in terms of request per second

I'll get the stats but the requests per second on the whole is very low (avg less than 1 per second), although there are bursts now and then. I am see big variations in the /api/v1/version calls, which I would expect to be consistently very fast:

2022-08-12T00:41:46.023Z        INFO    app/server.go:157       /api/v1/version timeTaken: 5.741997ms
2022-08-12T00:41:54.232Z        INFO    app/server.go:157       /api/v1/version timeTaken: 307.99µs
2022-08-12T00:41:56.024Z        INFO    app/server.go:157       /api/v1/version timeTaken: 307.645µs
time="2022-08-12T00:41:59Z" level=warning msg="Ignoring hint {StepMs:0 Func:rate StartMs:1660264799657 EndMs:1660264919657} for query [1660264799657,1660264919657,{span_kind=\"SPAN_KIND_SERVER\",__name__=\"signoz_latency_count\"}]." component=clickhouse
2022-08-12T00:42:04.233Z        INFO    app/server.go:157       /api/v1/version timeTaken: 926.051µs
2022-08-12T00:42:06.017Z        INFO    app/server.go:157       /api/v1/version timeTaken: 291.9µs
2022-08-12T00:42:14.233Z        INFO    app/server.go:157       /api/v1/version timeTaken: 885.899µs

ok, that load should be easily handled. @Prashant Shahi do yo have any inputs on this

Is this advisable? What are the recommendations for a production deployment?

Yeah, it should work fine provided K8s cluster has enough resources. We are running testbed with 100+ RPS and shikhandi with default resource limits