Hi there! Pleasure to meet you all! I use self-hosted singoz 0.68.1, on 4 nodes 8CPU 32GB each.
My app pushes around 30mln log records per day. Less traces, for debug purposes traces are off.
And we observed a significant loss of logs and traces, we have another logs collector service we use as a baseline. Same query and around 20% logs lost in signoz.
Although I see this log around 13 times an hour
Copy code
2025-02-12T20:39:35.953Z error internal/queue_sender.go:92 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "logs", "name": "clickhouselogsexporter", "error": "no more retries left: StatementSend:write: write tcp 10.52.30.56:51856->10.0.1.103:9000: write: connection reset by peer", "dropped_items": 835}
Im super confused and will appreciate any help, thanks a ton in advance!
k
KingPin
02/13/2025, 1:53 AM
hmmm I realize im seeing this error also.... maybe I need to check my system for loss....
s
Srikanth Chekuri
02/15/2025, 3:18 AM
What was the CPU usage by ClickHouse when you see these errors?
SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.