https://signoz.io logo
Join Slack
Powered by
The rate and apdex graph is now empty starting 2 w...
# support
j
The rate and apdex graph is now empty starting 2 weeks ago. I have tried removing all data, reinstalling and it is still now showing up for new data. Any idea on where I should be looking at?
s
Please share the logs of query-service when you load this page and also share the version of SigNoz query-service and otel-collector
j
Untitled
Untitled
Running these using Docker compose inside a VM signoz/query-service:0.52.0 signoz/signoz-otel-collector:0.102.4 Running this inside Kubernetes using k8s-infra chart docker.io/otel/opentelemetry-collector-contrib:0.88.0
s
Nothing unusal in the logs. Do you still face this issue?
j
Ya. Still facing this issue.
Any thought of any other places to look at?
s
Let's look at the logs of signoz-otel-collector and see if we can find something there.
j
Untitled
Untitled
s
Please share the otel collector config?
j
I mainly added timeout and memory_limiter after cloning it from the signoz repository
Untitled
s
On this page can you open the n/w tab of browser and share the response for all 
query_range
api calls?
j
Untitled
Untitled
Just sharing, this seems like a load issue. I purposely reducing the number of logs that get sent to 
otel-collector
and restarted it. Its memory usage then get back to normal range. Rate and Apdex are showing up regularly again as well.
Update again. Not sure if it is load as nothing increased in the last few days.
But the memory usage pattern changed again. Previously it maintains around 5-6GB. Now it keeps increasing after starting up around 8GB. Then around 9GB onward, metrics starts to get delayed and not posted again.
otelcol_exporter_queue_size
- seems like issue is with the 
clickhousemetricwrite
exporter being too slow and the queue is building up here
3 Views
Open in Slack
PreviousNext
Powered by