Hello, team! We are setting up SigNoz and have identified the following issue
gRPC vs HTTPS Protocol Mismatch When Exporting OTLP Traces via GCP Private Service Connect
1. Environment
Exporter (Project A) - OpenTelemetry exporter running as a GKE service
Receiver (Project B) - SigNoz OTLP Collector on GKE
Network - Google Cloud Private Service Connect between the two projects
External HTTP(S) Load Balancer in front of the SigNoz Collector
Backend service protocol: HTTP/2
Health checks: passing
2. Problem
When the exporter streams traces over gRPC (port 4317) through the HTTPS load balancer, the collector logs:
rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers.
reset reason: remote connection failure, transport failure reason: TLS_error:
268435703:SSL routines
OPENSSL internalWRONG_VERSION_NUMBER
Root cause: the LB is terminating TLS (HTTPS) and forwarding HTTP/2, but the SigNoz Collector expects raw gRPC over HTTP/2 on port 4317—so the connection is reset before it reaches the pod.
3. What Already Works
Using the HTTP endpoint on port 4318 (HTTP/1.1) with TLS termination succeeds:
otlp:
endpoint: "<*
s*ignoz-endpoint>"
tls:
insecure: true
min_version: "1.2"
4. Goal
Find a deployment or configuration pattern that allows OTLP gRPC traffic (port 4317) → Private Service Connect → Google Cloud HTTPS LB → SigNoz Collector, without protocol mismatches or connection resets.
5. Questions
Load Balancer / PSC settings
Are there additional flags or annotations on the GCP HTTPS load balancer or PSC service attachment needed to proxy gRPC (HTTP/2) correctly?
SigNoz Collector configuration
Should the Collector itself terminate TLS on the gRPC port (i.e. accept HTTPS directly on 4317)?
Any specific Helm chart values or Kubernetes Service annotations to enable HTTP/2 gRPC behind an HTTPS LB?
6. Best practices
Recommended patterns for exposing gRPC OTLP endpoints via Google Cloud external HTTPS load balancers.
Example YAML snippets or community-tested configs that work end-to-end.
Any pointers, sample configs, or experiences would be extremely helpful—thanks in advance!