1. Logging into knowledgebase.signoz.io doesn't work, keeps redirecting back to the login screen so I came here 😄
2. I am querying data in the signoz dashboard over a longer timeframe (6h) and we have quite a bit of spans added. The UI always shows "Something went wrong" and I get a 504 gateway error for the "getTagFilters" and "getSpanFilters" xhr calls, cutoff is always at 3 seconds. Is there some way to set a higher timeout from the frontend to the query service? Or could some other config be at play here?
Answers much appreciated
12/01/2023, 1:06 PM
hey @Andy David can you share how much data (number of spans) you have for 6h and what is the memory/CPU allocated to SigNoz instance?
Might be case of underprovisioning
12/04/2023, 7:10 AM
hi Pranay, thank you for your response. It might have been the case for underprovisioning, we increased the hardware under clickhouse and eventually it did stabilize (not initially though because we had the signoz-otel-collectors load spans into clickhouse from a ±100GB kafka topic) The UI now only throws this error for getting the tags for the 1 week period. Are there some recommendations with regard to sizing clickhouse based on ingested number of entities?
hello @Pranay I checked the usage explorer, looks like we have "Total count is 1155708235". Whenever I try to load the dashboard for one week, I get the same issue, timeout from both APIs called by the frontendm seemingly cut at 3.04s. Out clickhouse cluster has since relaxed quite a bit, but for long running queries I still get this. Any idea?
nevermind, found a way for my api gateway to wait for backend calls longer than default 3000 ms