This message was deleted.

Detailed error looks like this

2024-01-03T04:51:24.701Z	error	exporterhelper/retry_sender.go:145	Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "data_type": "logs", "name": "otlp", "error": "Permanent error: rpc error: code = Unknown desc = data refused due to high memory usage", "dropped_items": 823}
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send>
	<http://go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/retry_sender.go:145|go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/retry_sender.go:145>
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send|go.opentelemetry.io/collector/exporter/exporterhelper.(*logsExporterWithObservability).send>
	<http://go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/logs.go:176|go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/logs.go:176>
<http://go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start.func1|go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start.func1>
	<http://go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/queue_sender.go:126|go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/queue_sender.go:126>
<http://go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1|go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1>
	<http://go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/internal/bounded_memory_queue.go:52|go.opentelemetry.io/collector/exporter@v0.87.0/exporterhelper/internal/bounded_memory_queue.go:52>

Can it because of the

max_recv_msg_size_mib

set for signoz otlp receiver

No What is your signoz collector config?

Mostly base config.

config:
      receivers:
        otlp: 
          protocols: 
            http:
              # Since this collector needs to receive data from the web, enable cors for all origins
              # `allowed_origins` can be refined for your deployment domain
              cors:
                allowed_origins:
                  - "http://*"
                  - "https://*"
            grpc:

Share the full config bro

Collector config 👇 Let me know if you need more

otelCollector: 
    autoscaling:
      enabled: true
      keda: 
        enabled: true
        maxReplicaCount: "10"
    config:
      receivers:
        otlp: 
          protocols: 
            http:
              # Since this collector needs to receive data from the web, enable cors for all origins
              # `allowed_origins` can be refined for your deployment domain
              cors:
                allowed_origins:
                  - "http://*"
                  - "https://*"
            grpc:
    resources:
      requests:
        cpu: 512m
        memory: 1024Mi

You are probably sharing the override values part. This is not it. Use kubectl describe configmap for otel collector pod and share the full config.

This is my otel collector config. The one that is exporting to signoz

It's erroring because of the memory limiter

But that shouldnt happen at the exporter level right. Also it should be temporary. Isnt it ?

Yes, it should be temporary. The main reason is memory limiter put in place and I am not sure about the stacktrace part. Might have to look into detail.

Any suggestions on how do we handke this. Does not having memory limiter helps ? I see that's the default

You should better estimate the potential memory usage based on the load you expect to generate and set some better upper limits for memory. Or flush the batches more often than 2s etc...