https://signoz.io
Join Slack
Anyone can share me the config of signoz and k8s-infra helm value. I try config otelCollectorEndpoin...
# support
t
Anyone can share me the config of signoz and k8s-infra helm value. I try config otelCollectorEndpoint using http port 4318. But the otel can not push the metrics to this endpoint
n
Could you please share the URL for the endpoint here
t
The custom values. for collector install in cluster A using signoz chart
Copy code
global:
  storageClass: storage1
clickhouse:
  enabled: true
  zookeeper:
    enabled: true
    image:
    replicaCount: 3
  persistence:
    size: 200Gi
signoz:
  name: "signoz"
  replicaCount: 1
  image:
    registry: docker.io
    repository: signoz/signoz
    tag: v0.82.1
    pullPolicy: IfNotPresent
  persistence:
    enabled: true
    size: 50Gi
otelCollector:
  ingress:
    enabled: true
    kubernetes.io/ingress.class: nginx
    hosts:
      - host: otelcollector.dev.wiki
        paths:
          - path: /
            pathType: ImplementationSpecific
            port: 4318
    tls:
     - secretName: otel-tls
       hosts:
         - otelcollector.dev.wiki
and collector with chart k8s-infra install in other k8s cluster (cluster B)
Copy code
global:
  storageClass: storage1
  clusterName: "k8s-prod"
  deploymentEnvironment: "production"
  cloud: other
otelCollectorEndpoint: otelcollector.dev.wiki:443
insecureSkipVerify: true
n
why are you using port 443?
t
I want using HTTP
not using GRPC, because I need open GRPC on cluster A
so want to using 443 to replace that
n
For using the HTTP protocol, you should use port 4318, not 443
t
have you see I config ingress ?
that point to port 4318 already
n
grpc port: 4317 http port: 4318
t
So what is ingress mention?
in signoz chart
Copy code
# Configuration for ports
  ports:
    otlp:
      # -- Whether to enable service port for OTLP gRPC
      enabled: true
      # -- Container port for OTLP gRPC
      containerPort: 4317
      # -- Service port for OTLP gRPC
      servicePort: 4317
      # -- Node port for OTLP gRPC
      nodePort: ""
      # -- Protocol to use for OTLP gRPC
      protocol: TCP
    otlp-http:
      # -- Whether to enable service port for OTLP HTTP
      enabled: true
      # -- Container port for OTLP HTTP
      containerPort: 4318
      # -- Service port for OTLP HTTP
      servicePort: 4318
      # -- Node port for OTLP HTTP
      nodePort: ""
      # -- Protocol to use for OTLP HTTP
      protocol: TCP
default config for otlp-http is 4318 and already enabled
then I create ingress to mapping that pod to domain
so ingress only listen 80/443, so here I use 443
Then in other cluster, I using chart k8s-infra and config otelCollectorEndpoint to this domain
n
Got it!!
What's the exact error you're seeing while trying to push the metrics
t
that error log on otel agent
Copy code
{
  "level": "warn",
  "ts": 1748590001.1368558,
  "caller": "grpc@v1.66.0/clientconn.go:1379",
  "msg": "[core] [Channel #1 SubChannel #8]grpc: addrConn.createTransport failed to connect to {Addr: \"10.10.10.12:443\", ServerName: \"otelcollector.dev.wiki:443\", }. Err: connection error: desc = \"error reading server preface: http2: frame too large\"",
  "grpc_log": true
}
n
try with increasing the 
max_recv_msg_size_mib
setting on the receiver: https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md
t
looks like otel-agent is still trying to push the request to grpc endpoint (but it is https)
your config use for GRPC, you can see in default values of signoz chart
Copy code
config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
            max_recv_msg_size_mib: 16
          http:
            endpoint: 0.0.0.0:4318
n
Hey @Thomas E. Dewey, let me check once and i'll get back to you
I'll try to reproduce this and see if i'm facing this
t
ok thank you
@Nagesh Bansal any update for this issue ?
a
i have experienced this issue, for your k8s infra otel agent and deployment, you need to expose the port 4317, the grpc port is the only work that will work
Copy code
global:
  storageClass: storage1
  clusterName: "k8s-prod"
  deploymentEnvironment: "production"
  cloud: other
otelCollectorEndpoint: otelcollector-gprc.dev.wiki:443
insecureSkipVerify: true
see my own config, so regardless, you need to expose the gprc port for the k8-infra otel agent and deployment, the only time it differs is when you enable self-telemetry in your k8-infra, then you can specify the exposed otel-http
Copy code
resource "kubectl_manifest" "ingress_otel_grpc_signoz" {
  yaml_body = <<-EOF
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  name: signoz-otel-collector-grpc-ingress
  namespace: ${kubernetes_namespace.platform.id}
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: nginx
    <http://nginx.ingress.kubernetes.io/grpc-backend|nginx.ingress.kubernetes.io/grpc-backend>: "true"
    <http://nginx.ingress.kubernetes.io/enable-access-log|nginx.ingress.kubernetes.io/enable-access-log>: "false"
    <http://nginx.ingress.kubernetes.io/backend-protocol|nginx.ingress.kubernetes.io/backend-protocol>: "GRPC"
    <http://nginx.ingress.kubernetes.io/proxy-buffer-size|nginx.ingress.kubernetes.io/proxy-buffer-size>: "128k"
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "false"
    <http://nginx.ingress.kubernetes.io/proxy-body-size|nginx.ingress.kubernetes.io/proxy-body-size>: "0"
    <http://nginx.ingress.kubernetes.io/proxy-connect-timeout|nginx.ingress.kubernetes.io/proxy-connect-timeout>: "300"
    <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "300"
    <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "300"
    <http://nginx.ingress.kubernetes.io/upstream-keepalive-timeout|nginx.ingress.kubernetes.io/upstream-keepalive-timeout>: "600"
    <http://nginx.ingress.kubernetes.io/upstream-keepalive-requests|nginx.ingress.kubernetes.io/upstream-keepalive-requests>: "100"
spec:
  rules:
  - host: <http://otelcollector-gprc.mydomain.com|otelcollector-gprc.mydomain.com>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: signoz-otel-collector
            port:
              number: 4317
      - path: /opentelemetry.proto.collector.logs.v1.LogsService/Export
        pathType: ImplementationSpecific
        backend:
          service:
            name: signoz-otel-collector
            port:
              number: 4317
      - path: /opentelemetry.proto.collector.metrics.v1.MetricsService/Export
        pathType: ImplementationSpecific
        backend:
          service:
            name: signoz-otel-collector
            port:
              number: 4317
      - path: /opentelemetry.proto.collector.trace.v1.TraceService/Export
        pathType: ImplementationSpecific
        backend:
          service:
            name: signoz-otel-collector
            port:
              number: 4317
   EOF
}
t
@Abdulmalik Salawu So that mean I need expose grpc port on otel-agent, not expose on signoz(otel-colletor) ? For my understand, the work flow is that
can you share your workflow
a
no its on the signoz(otel-colletor) that you will expose the grpc port, so that your cluster b(otel-agent) will use it
8 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by