https://signoz.io logo
Join Slack
Powered by
Hello, We're using SigNoz on k8s (helm) to monito...
# support
a
Hello, We're using SigNoz on k8s (helm) to monitor some nodejs apps. signoz-otel-collector spams this in logs:
Copy code
2022-09-13T13:34:21.183Z	error	exporterhelper/queued_retry.go:183	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "name": "clickhousemetricswrite", "error": "Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality 
temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}], "dropped_items": 1024}
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send>
	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/queued_retry.go:183
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send>
	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/metrics.go:134
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1|go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1>
	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/queued_retry_inmemory.go:105
<http://go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume|go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume>
	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/internal/bounded_memory_queue.go:99
<http://go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2|go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2>
	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/internal/bounded_memory_queue.go:78
A quick glance at source code suggests there's a possibility to debug https://github.com/SigNoz/opentelemetry-collector-contrib/blob/develop/exporter/clickhousemetricsexporter/exporter.go#L164 but I'm not sure how to configure this zap.S() logger, it seems to be noop by default. Any way to enable this logging to get more detailed error messages?
p
cc @Ankit Nayan @Srikanth Chekuri
s
@Andrew Grishin we have see this happen with empty data in the past. did you have docker stats or something similar in your receivers? I am not totally sure why sugaredlogger is not showing the log. Will look into it.
a
@Srikanth Chekuri we have most of our apps with standard nodejs instrumentations and one app with dotnet core. We didnt configure docker or anything else to send to Signoz. Thanks for looking into!
s
What’s SDK versions you are using?
a
Copy code
"@opentelemetry/auto-instrumentations-node": "^0.31.0",
    "@opentelemetry/exporter-metrics-otlp-grpc": "^0.29.2",
    "@opentelemetry/exporter-otlp-grpc": "^0.26.0",
    "@opentelemetry/exporter-trace-otlp-grpc": "^0.29.2",
    "@opentelemetry/sdk-metrics-base": "^0.29.2",
    "@opentelemetry/sdk-node": "^0.29.2",
    "nestjs-otel": "^3.0.4",
s
It looks like otlp grpc metrics exporter is only compatible until <= 
0.53.0
version of collector according to js docs.
a
@Srikanth Chekuri so any nodejs user won't be able to send metric via grpc to current otel-collector version of signoz?
s
Not totally sure about that. It could be that they haven’t updated the docs since last time when they made it 0.53. Otherwise it also happens when the data is empty. Let me check with example.
4 Views
Open in Slack
PreviousNext
Powered by