https://signoz.io logo
Powered by
a

Andrew Grishin

09/13/2022, 2:03 PM
Hello, We're using SigNoz on k8s (helm) to monitor some nodejs apps. signoz-otel-collector spams this in logs:
Copy code
2022-09-13T13:34:21.183Z	error	exporterhelper/queued_retry.go:183	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "name": "clickhousemetricswrite", "error": "Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality 
temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}], "dropped_items": 1024}
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send>
	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/queued_retry.go:183
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send>
	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/metrics.go:134
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1|go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1>
	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/queued_retry_inmemory.go:105
<http://go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume|go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume>
	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/internal/bounded_memory_queue.go:99
<http://go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2|go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2>
	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/internal/bounded_memory_queue.go:78
A quick glance at source code suggests there's a possibility to debug https://github.com/SigNoz/opentelemetry-collector-contrib/blob/develop/exporter/clickhousemetricsexporter/exporter.go#L164 but I'm not sure how to configure this zap.S() logger, it seems to be noop by default. Any way to enable this logging to get more detailed error messages?
p

Prashant Shahi

09/13/2022, 2:06 PM
cc @Ankit Nayan @Srikanth Chekuri
s

Srikanth Chekuri

09/14/2022, 6:18 AM
@Andrew Grishin we have see this happen with empty data in the past. did you have docker stats or something similar in your receivers? I am not totally sure why sugaredlogger is not showing the log. Will look into it.
a

Andrew Grishin

09/14/2022, 2:54 PM
@Srikanth Chekuri we have most of our apps with standard nodejs instrumentations and one app with dotnet core. We didnt configure docker or anything else to send to Signoz. Thanks for looking into!
s

Srikanth Chekuri

09/14/2022, 3:25 PM
What’s SDK versions you are using?
a

Andrew Grishin

09/14/2022, 3:30 PM
Copy code
"@opentelemetry/auto-instrumentations-node": "^0.31.0",
    "@opentelemetry/exporter-metrics-otlp-grpc": "^0.29.2",
    "@opentelemetry/exporter-otlp-grpc": "^0.26.0",
    "@opentelemetry/exporter-trace-otlp-grpc": "^0.29.2",
    "@opentelemetry/sdk-metrics-base": "^0.29.2",
    "@opentelemetry/sdk-node": "^0.29.2",
    "nestjs-otel": "^3.0.4",
s

Srikanth Chekuri

09/14/2022, 3:52 PM
It looks like otlp grpc metrics exporter is only compatible until <= 
0.53.0
version of collector according to js docs.
a

Ankit Nayan

09/15/2022, 8:14 AM
@Srikanth Chekuri so any nodejs user won't be able to send metric via grpc to current otel-collector version of signoz?
s

Srikanth Chekuri

09/15/2022, 8:41 AM
Not totally sure about that. It could be that they haven’t updated the docs since last time when they made it 0.53. Otherwise it also happens when the data is empty. Let me check with example.
3 Views
Powered by