https://signoz.io
Join Slack
Hello everyone recently we started getting this error from `signoz-k8s-infra-otel-agent` to `signoz...
# support
r
Hello everyone recently we started getting this error from 
signoz-k8s-infra-otel-agent
to 
signoz-otel-collector
:
Copy code
{"level":"info","ts":1746611437.7966254,"msg":"MetricsExporter","kind":"exporter","data_type":"metrics","name":"logging","resource metrics":61,"metrics":628,"data points":817}
2025/05/07 09:50:37 traces export: Post "<http://signoz-otel-collector.monitoring.svc.cluster.local:4317/v1/traces>": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00"
2025/05/07 09:50:42 traces export: Post "<http://signoz-otel-collector.monitoring.svc.cluster.local:4317/v1/traces>": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00"
2025/05/07 09:50:47 traces export: Post "<http://signoz-otel-collector.monitoring.svc.cluster.local:4317/v1/traces>": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00"
2025/05/07 09:50:52 failed to upload metrics: Post "<http://signoz-otel-collector.monitoring.svc.cluster.local:4317/v1/metrics>": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x06\x04\x00\x00\x00\x00\x00\x00\x05\x00\x00@\x00"
{"level":"info","ts":1746611452.796582,"msg":"LogsExporter","kind":"exporter","data_type":"logs","name":"logging","resource logs":4,"log records":5}
the port 4317 is open and collector is successfully running
n
Hey @Roman Levytskyi was it working earlier?
r
yes, it was working perfectly
@Nagesh Bansal
n
what's the endpoint you're using?
the 4317 is for grpc endpoint
r
yes, we use grpc. a bit of info here: we use 
k8s-infra
chart and forward data to the 
signoz-otelcol
main collector to be stored in clickhouse. the part of the config looks like:
Copy code
otelDeployment:
    exporters:
      otlp:
        endpoint: signoz-otel-collector.monitoring.svc.cluster.local:4317
and
Copy code
otelAgent:
  config:
    exporters:
      otlp:
        endpoint: signoz-otel-collector.monitoring.svc.cluster.local:4317
n
Could you please share the values you're using for helm chart
r
for k8s infra or main signoz chart?
n
both if possible
r
@Nagesh Bansal take a look please also, does it make any difference if I use otel exporter per Agent and Deployment like in the files or the otelCollectorEndpoint global property?
n
yeah, Could you please try with the setttingup the otelCollectorEndpoint in the global. although the issue seems more to be that otel-collector is not able to reach the endpoint, so either it can be related to your sevices.
r
i just tried and it’s the same exact error
i also checked if the service is running at that address and it is
n
let me check it once. adding @Vibhu Pandey for any suggestions
h
Me too facing same issue from last 5days. I have deployed signoz infra 10days back 5 days it was given me every insight of the cluster but from last Saturday, I am not even able to see anything in signoz-ui regarding to k8s-infra.
r
@Hari Prasad exactly the same thing! ours got broken 2-3 days ago
h
@Nagesh Bansal did any changes related to infra went live in signoz
n
what's the version of signoz you're running @Hari Prasad @Roman Levytskyi
r
latest for me @Nagesh Bansal
both signoz and infra
h
I think this is the latest version available in official docs
n
@Hari Prasad the latest version is: v0.81.0 Although I'm checking whether the issue with v0.80.0
h
@Nagesh Bansal from when this version is available for the people
n
we release new version every wednesday of the week
h
Okay, Can you also confirm any issue related to v0.80.0
n
Hey @Roman Levytskyi just a heads up the endpoint here should be an HTTP endpoint
selfTelemetry:
endpoint: signoz-otel-collector.monitoring.svc.cluster.local:4317
r
you mean point to 
4318
instead? what if omitted? will not be reported or 
otelCollectorEndpoint
will be used?
n
the otelColletorEndpoint will be used
@Roman Levytskyi @Hari Prasad Confirmed the issue: if you're enabling the selfTelemetry then please provide the HTTP endpoint in the config:
Copy code
presets:
  selfTelemetry:
    endpoint: signoz-otel-collector.<NAMESPACE>.svc.cluster.local:4318
    traces:
      enabled: true
    metrics:
      enabled: true
    logs:
      enabled: true
by default selfTelemetry takes the OTEL_COLLECTOR_ENDPOINT ( which will be a grpc endpoint, if you're setting up the otelCollectorEndpoint to grpc )
r
so ideally it should accept a GRPC endpoint not only HTTP, but for now it’s an issue, correct?
n
r
ok, this is clear, but if NOT provided will it still work and report to GRPC or still fail and expect either to be disabled or provided with correct HTTP-only endpoint?
n
it shouldnt be any issue if you're not enabling the selfTelemetry and not providingv the HTTP endpoint Also if possible please try with the solution and let me know if you're still facing the issue
r
i’m trying now
let me summarize though that we’re on the same page:
Copy code
if no endpoint is provided to selfTelemetry but it is enabled, it will successfully export to GRPC configured globally or it will not report selfTelemetry at all?
sorry for too many questioins, I want to make sure in understand 100% how it works and what to choose
h
@Nagesh Bansal i am getting following error if i use 4318
Copy code
{"level":"info","ts":1746633012.3878634,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"otlp","error":"rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"","interval":"23.231427349s"}
{"level":"info","ts":1746633013.4247258,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"otlp","error":"rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"","interval":"39.002784418s"}
{"level":"info","ts":1746633015.7429657,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"otlp","error":"rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"","interval":"25.885274693s"}
{"level":"info","ts":1746633016.9920123,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"otlp","error":"rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"","interval":"9.351378477s"}
{"level":"info","ts":1746633026.3439534,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"otlp","error":"rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"","interval":"10.083124388s"}
{"level":"info","ts":1746633027.342186,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"otlp","error":"rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"","interval":"36.205781874s"}
{"level":"info","ts":1746633027.700545,"caller":"exporterhelper/retry_sender.go:118","msg":"Exporting failed. Will retry the request after interval.","kind":"exporter","data_type":"metrics","name":"otlp","error":"rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: http2: frame too large\"","interval":"23.927454572s"}
n
@Roman Levytskyi if self-telemetry is enabled and no endpoint/grpc endpoint is provided it will not report anything, and throw the same error as you share, if you're enabling selfTelemetry you need to provide the http endpoint.
if no endpoint is provided to selfTelemetry but it is enabled, it will successfully export to GRPC configured globally or it will not report selfTelemetry at all?
@Hari Prasad I request you to please share the issue in a separate thread, I'll check it out there.
r
@Nagesh Bansal got you, thanks
i can confirm that this error is gone after switching to HTTP
thanks a LOT
17 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by