Title
s
sudhanshu dev
09/14/2022, 6:59 AMand images is otel/opentelemetry-collector-contrib:0.43.0
p
Prashant Shahi
09/14/2022, 8:31 AMIt should be 0.55.0
s
sudhanshu dev
09/14/2022, 8:42 AMIs there any dependency with eks version?
n
nitya-signoz
09/14/2022, 8:58 AMThere are a lot of breaking changes between v0.43.0 and 0.55.0 . Some protocol buffers are also updated. For exact details, you can visit the releases https://github.com/open-telemetry/opentelemetry-collector/releases
s
sudhanshu dev
09/14/2022, 9:10 AMLet me try with updated one
@nitya-signoz Will signoz repo supports
0.55.0
As I upgraded tag
this deployment otelcontribcol start giving error
p
Prashant Shahi
09/14/2022, 9:55 AMOh, right!
That version should be updated.
@sudhanshu dev can you try re-installing otel-collector-k8s components?
n
nitya-signoz
09/14/2022, 9:56 AMYeah, we will have to make change in the otel-collecot-k8s
s
sudhanshu dev
09/14/2022, 9:57 AMDO I need to take update from ur git and then reploye
with new tag
'?
bcz in signoz repo image tag is 0.45.0
@Prashant Shahi could u plz let me know what r u trying to say
redeploy means
taking updated configuration and then
p
Prashant Shahi
09/14/2022, 9:59 AM# cleanup
kubectl -n signoz-infra-metrics delete -Rf agent
kubectl -n signoz-infra-metrics delete -Rf deployment
# re-install
kubectl -n signoz-infra-metrics apply -Rf agent
kubectl -n signoz-infra-metrics apply -Rf deployment@sudhanshu dev You have updated the version in both agent and deployment to
0.55.0s
sudhanshu dev
09/14/2022, 10:00 AMyes
rror exporterhelper/queued_retry.go:183 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "Permanent error: rpc error: code = ResourceExhausted desc = grpc: received message after decompression larger than max (4526957 vs. 4194304)", "dropped_items": 19305}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector@v0.55.0/exporter/exporterhelper/queued_retry.go:183
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector@v0.55.0/exporter/exporterhelper/metrics.go:132
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector@v0.55.0/exporter/exporterhelper/queued_retry_inmemory.go:119
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
go.opentelemetry.io/collector@v0.55.0/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
go.opentelemetry.io/collector@v0.55.0/exporter/exporterhelper/internal/bounded_memory_queue.go:69
same error
p
Prashant Shahi
09/14/2022, 10:12 AM@Srikanth Chekuri any insights here?
cc @Ankit Nayan
s
sudhanshu dev
09/14/2022, 10:24 AMOk let me try
Added
data:
otel-collector-config: |
receivers:
otlp:
protocols:
grpc:
max_recv_msg_size_mib: 300
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
now deploying
@nitya-signoz same error
I did changes in infra-metrics.yaml
file
I hope u referred to the same
n
nitya-signoz
09/14/2022, 10:36 AMJust confirming here
• You are running multiple otel collectors and the otel collectors which are sending the data to singoz otel collector is throwing the error.
• You applied the changes on the signoz otel collector which is accepting the data from other otel collectors right ?
Let me know if the above to assumptions are correct.
s
sudhanshu dev
09/14/2022, 10:41 AMyes one demaon set and one deployment is running
I am getting error in deployment
In the last pod
I am getting error
n
nitya-signoz
09/14/2022, 10:42 AMOhh thats interesting. I expected this error to occur on the daemon set collectors.
s
sudhanshu dev
09/14/2022, 10:43 AMyes
That why
I sent u file
bcz we did changes in demon configmap
and getting error in deployment
no error in demonset
n
nitya-signoz
09/14/2022, 10:53 AMAhh the number is high.
s
sudhanshu dev
09/14/2022, 10:54 AMWhere should we define this
n
nitya-signoz
09/14/2022, 10:55 AMTry increasing the value to a bigger number like 10000 for max_recv_msg_size_mib and see if you get the error again ?
s
sudhanshu dev
09/14/2022, 10:56 AMIn the same
File
Infra-metrics.yaml
n
nitya-signoz
09/14/2022, 11:03 AMNot sure about the file but the otel collector which is receiving the data and from other otel collectors. So it should be for the deployment. @Prashant Shahi can you correct me if that points to the infra-metrics.yaml.
s
Srikanth Chekuri
09/14/2022, 11:05 AMI didn’t follow the whole conversation but the keep increasing the receiver msg size is not the correct thing. You should be able to batch the export size that fits within some limit without hampering the receiver side stuff. Can you try configuring
send_batch_max_sizes
sudhanshu dev
09/14/2022, 11:43 AMSame error
I start getting this error when upgraded to 0.55.0
from 0.43.0
@nitya-signoz
s
Srikanth Chekuri
09/14/2022, 3:25 PMWhere and how much did you configure? If the batch size is not reasonable the total payload size will sill be big and receiver throws error. This was the case for user who I know faced similar problem and they got it fixed by using
send_batch_max_sizea
Ankit Nayan
09/15/2022, 8:18 AM@sudhanshu dev is this error fixed now?
s
sudhanshu dev
09/16/2022, 11:30 AM@Prashant Shahi After adding properties
seems issue fixed
kubectl logs -f otelcontribcol-65f7cbcd58-wcr9c -n signoz-infra-metrics
2022/09/16 11:01:09 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/09/16 11:01:09 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-09-16T11:01:09.879Z info service/telemetry.go:103 Setting up own telemetry...
2022-09-16T11:01:09.879Z info service/telemetry.go:138 Serving Prometheus metrics {"address": ":8888", "level": "basic"}
2022-09-16T11:01:09.913Z info extensions/extensions.go:42 Starting extensions...
2022-09-16T11:01:09.913Z info pipelines/pipelines.go:74 Starting exporters...
2022-09-16T11:01:09.913Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "metrics", "name": "otlp"}
2022-09-16T11:01:09.913Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "metrics", "name": "otlp"}
2022-09-16T11:01:09.913Z info pipelines/pipelines.go:86 Starting processors...
2022-09-16T11:01:09.913Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2022-09-16T11:01:09.913Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "batch", "pipeline": "metrics"}
2022-09-16T11:01:09.913Z info pipelines/pipelines.go:98 Starting receivers...
2022-09-16T11:01:09.913Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "k8s_cluster", "pipeline": "metrics"}
2022-09-16T11:01:09.914Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "k8s_cluster", "pipeline": "metrics"}
2022-09-16T11:01:09.914Z info service/collector.go:215 Starting otelcol-contrib... {"Version": "0.55.0", "NumCPU": 32}
2022-09-16T11:01:09.914Z info service/collector.go:128 Everything is ready. Begin running and processing data.
2022-09-16T11:01:09.914Z info k8sclusterreceiver@v0.55.0/receiver.go:59 Starting shared informers and wait for initial cache sync. {"kind": "receiver", "name": "k8s_cluster", "pipeline": "metrics"}
2022-09-16T11:01:14.916Z info k8sclusterreceiver@v0.55.0/receiver.go:80 Completed syncing shared informer caches. {"kind": "receiver", "name": "k8s_cluster", "pipeline": "metrics"}
No more message drop error