I am missing lot of logs from signoz, is there a way to find out why this is happening? I associate logs with traces, and even though traces are present, many times logs are not there. I am having k8s deployment, and sending logs from node.js application using pino-otel-transporter https://github.com/pinojs/pino-opentelemetry-transport

Please share more details, like is it the first time you are instrumenting the application or it was working previously and suddenly stopped working due to a change/upgrade that you made?

we are already using the instrumentation and signoz for more than 2 months in production, and we have always observed some missing logs. But now we are seeing that it is happening little bit more frequently. With traces I don’t observe this drop, but with logs it is much more evident

Did you check logs of otel-collector which emits that it is dropping data?

any particular keywords I should look for?

look for “drop/dropping/failed”

nope, not able to find any such thing in logs

There is a metric named

otelcol_exporter_send_failed_log_records

, which you can scrape from otel collector to see if it has dropped any log.

no failed records here as well

then there is some issue with instrumentation layer, you will have to locally reproduce it to debug it.

is writing to disk, and then using otel agent a better way?

for instrumentation, can you suggest what is the general best practice?

using the SDK is always better as it natively transforms the data, the last resort should be using files.

understood

Try checking what kind of logs are missing to get a better idea if it’s certain logs or random.

mostly error logs, as we check for error logs from traces