a

    Andrew Grishin

    1 week ago
    Hello, We're using SigNoz on k8s (helm) to monitor some nodejs apps. signoz-otel-collector spams this in logs:
    2022-09-13T13:34:21.183Z	error	exporterhelper/queued_retry.go:183	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "name": "clickhousemetricswrite", "error": "Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality 
    temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}], "dropped_items": 1024}
    <http://go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send>
    	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/queued_retry.go:183
    <http://go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send>
    	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/metrics.go:134
    <http://go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1|go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1>
    	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/queued_retry_inmemory.go:105
    <http://go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume|go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume>
    	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/internal/bounded_memory_queue.go:99
    <http://go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2|go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2>
    	/go/pkg/mod/go.opentelemetry.io/collector@v0.43.0/exporter/exporterhelper/internal/bounded_memory_queue.go:78
    A quick glance at source code suggests there's a possibility to debug https://github.com/SigNoz/opentelemetry-collector-contrib/blob/develop/exporter/clickhousemetricsexporter/exporter.go#L164 but I'm not sure how to configure this zap.S() logger, it seems to be noop by default. Any way to enable this logging to get more detailed error messages?
    Prashant Shahi

    Prashant Shahi

    1 week ago
    cc @Ankit Nayan @Srikanth Chekuri
    Srikanth Chekuri

    Srikanth Chekuri

    1 week ago
    @Andrew Grishin we have see this happen with empty data in the past. did you have docker stats or something similar in your receivers? I am not totally sure why sugaredlogger is not showing the log. Will look into it.
    a

    Andrew Grishin

    1 week ago
    @Srikanth Chekuri we have most of our apps with standard nodejs instrumentations and one app with dotnet core. We didnt configure docker or anything else to send to Signoz. Thanks for looking into!
    Srikanth Chekuri

    Srikanth Chekuri

    1 week ago
    What’s SDK versions you are using?
    a

    Andrew Grishin

    1 week ago
    "@opentelemetry/auto-instrumentations-node": "^0.31.0",
        "@opentelemetry/exporter-metrics-otlp-grpc": "^0.29.2",
        "@opentelemetry/exporter-otlp-grpc": "^0.26.0",
        "@opentelemetry/exporter-trace-otlp-grpc": "^0.29.2",
        "@opentelemetry/sdk-metrics-base": "^0.29.2",
        "@opentelemetry/sdk-node": "^0.29.2",
        "nestjs-otel": "^3.0.4",
    Srikanth Chekuri

    Srikanth Chekuri

    1 week ago
    It looks like otlp grpc metrics exporter is only compatible until <=
    0.53.0
    version of collector according to js docs.
    Ankit Nayan

    Ankit Nayan

    1 week ago
    @Srikanth Chekuri so any nodejs user won't be able to send metric via grpc to current otel-collector version of signoz?
    Srikanth Chekuri

    Srikanth Chekuri

    1 week ago
    Not totally sure about that. It could be that they haven’t updated the docs since last time when they made it 0.53. Otherwise it also happens when the data is empty. Let me check with example.