Hi, so I have a kubernetes deployment I was experimenting with, the dashboards are too slow to load, I assume it's because of the query service, but when I try to increase the query service replica count, I'm unable to log back in and it asks me to sign up, am I scaling it wrong or am I missing something?

At this point, horizontal scaling of query-service is not supported. You mentioned that dashboards are slow? Can you elaborate on the sort of setup you have? What are the resources allocated to clickhouse? What dashboards are you trying to open?

yes, Thanks for responding. so, my setup has 6 shards of clickhouse, each with 8cpu and 32 gigs of ram. It's ingesting data fine. the collectors are on autoscaling with 15 pods to start with resources of 1cpu and 2 gigs ram. The dashboard is the services dashboard. Although I do see the warning "Too Many Top Level Operations" (i've taken steps to address this now) when I click on an application on the services page. it loads the top level operations, but loading the other metrics like p99s, apdex, etc gets cancelled.

Right, how is the resource utilization of the overall setup looking? Is there any 90-100% cpu or mem utilization?

not particularly, whatever resource related issues I saw, I did squash them and ensure there is enough resources. Particularly the issue i see is, with these dashboard queries in the query-service give an error like this

{"level":"INFO","timestamp":"2024-07-02T19:57:10.098Z","caller":"v2/helper.go:255","msg":"cache retrieve status","status":"key miss"}                                                                                                                                                                                  {"level":"ERROR","timestamp":"2024-07-02T19:57:11.207Z","caller":"app/http_handler.go:3359","msg":"Error while adding temporality for metrics","error":"context canceled","stacktrace":"<http://go.signoz.io/signoz/pkg/query-service/app.(*APIHandler).QueryRangeV4|go.signoz.io/signoz/pkg/query-service/app.(*APIHandler).QueryRangeV4>\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/http_handler.go:3359\ngo.signoz.io/signoz/pkg/query-service/app.(*APIHandler).RegisterQueryRangeV4Routes.(*AuthMiddleware).ViewAccess.func1\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/auth.go:50\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/net/http/server.go:2136\ngo.signoz.io/signoz/ee/query-service/app.loggingMiddleware.func1\n\t/home/runner/work/signoz/signoz/ee/query-service/app/server.go:379\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/net/http/server.go:2136\ngo.signoz.io/signoz/ee/query-service/app.(*Server).analyticsMiddleware-fm.(*Server).analyticsMiddleware.func1\n\t/home/runner/work/signoz/signoz/ee/query-service/app/server.go:534\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/net/http/server.go:2136\ngo.signoz.io/signoz/ee/query-service/app.setTimeoutMiddleware.func1\n\t/home/runner/work/signoz/signoz/ee/query-service/app/server.go:565\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/net/http/server.go:2136\ngo.signoz.io/signoz/pkg/query-service/app.LogCommentEnricher.func1\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/server.go:360\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/net/http/server.go:2136\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212\ngo.signoz.io/signoz/ee/query-service/app.(*Server).createPublicServer.(*Cors).Handler.func2\n\t/home/runner/go/pkg/mod/github.com/rs/cors@v1.11.0/cors.go:289\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/net/http/server.go:2136\ngo.signoz.io/signoz/ee/query-service/app.(*Server).createPublicServer.CompressHandler.CompressHandlerLevel.func3\n\t/home/runner/go/pkg/mod/github.com/gorilla/handlers@v1.5.1/compress.go:141\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/net/http/server.go:2136\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/net/http/server.go:2009"}

Let me get back to you on this

Thanks! looking forward

Although I do see the warning "Too Many Top Level Operations" (i've taken steps to address this now) when I click on an application on the services page. it loads the top level operations, but loading the other metrics like p99s, apdex, etc gets cancelled.

It will be addressed here soon https://github.com/SigNoz/signoz/pull/5404