https://signoz.io logo
Join Slack
Powered by
dear SigNoz community, I wish to take advantage of...
# support
a
dear SigNoz community, I wish to take advantage of the signoz 
k8s-infa
helm chart to add a receiver that will scrape prometheus metrics. I use the k8s-infra helm chart as a subchart to my own chart and here are the values that I used:
Copy code
k8s-infra:
  global:
    cloud: others
    clusterName: aws-il
    deploymentEnvironment: dev
  otelCollectorEndpoint: signoz-otel-collector.obs.svc.cluster.local:4317
  otelInsecure: true
  presets:
    otlpExporter:
      enabled: true
    loggingExporter:
      enabled: false
  otelAgent:
    receivers:
      otlp: # copied from k8s-infra values
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
            max_recv_msg_size_mib: 4
          http:
            endpoint: 0.0.0.0:4318
      prometheus: # Added the Prometheus receiver
        config:
          global:
            scrape_interval: 15s
          scrape_configs:
            - job_name: 'sessions-server'
              static_configs:
                - targets: [ 'hera-sessions-server:8000' ]
    service:
      pipelines:
        metrics:
          receivers: [otlp, prometheus]  # added prometheus
          processors: [batch]
          exporters: [otlp]
the prometheus metrics are available on service:port 
hera-sessions-server:8000
as written in the targets field. I see no errors about prometheus in the 
k8s-infra-otel-agent
pod. I would expect my metrics to be available in the SigNoz UI when creating a new dashboard, but they are not. Any information on how to troubleshoot this and get my own first metric into SigNoz would be greatly appreciated.
Actually I had a few error messages in my otel agent pod logs. I had to add a few roles. Below is my updated values. Now I have no errors in my logs. Unfortunately I also have no mention about prometheus nor urls being scraped and I don't see my extra metric name in the signoz UI. Updated values:
Copy code
k8s-infra:
  global:
    cloud: others
    clusterName: aws-il
    deploymentEnvironment: dev
  otelCollectorEndpoint: signoz-otel-collector.obs.svc.cluster.local:4317
  otelInsecure: true
  presets:
    otlpExporter:
      enabled: true
    loggingExporter:
      enabled: false
  otelAgent:
    receivers:
      otlp: # copied from k8s-infra values
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
            max_recv_msg_size_mib: 4
          http:
            endpoint: 0.0.0.0:4318
      prometheus: # Added the Prometheus receiver
        config:
          global:
            scrape_interval: 15s
          scrape_configs:
            - job_name: 'sessions-server'
              static_configs:
                - targets: [ 'hera-sessions-server:8000' ]
    service:
      pipelines:
        metrics:
          receivers: [otlp, prometheus]  # added prometheus
          processors: [batch]
          exporters: [otlp]
    clusterRole:
      rules: # copied from k8s-infra, added PVs, PVCs ...
        - apiGroups: [""]
          resources:
            - events
            - namespaces
            - namespaces/status
            - nodes
            - nodes/spec
            - pods
            - pods/status
            - replicationcontrollers
            - replicationcontrollers/status
            - resourcequotas
            - services
            - persistentvolumes
            - persistentvolumeclaims
            - nodes/stats
            - nodes/proxy
          verbs: [ "get", "list", "watch" ]
        - apiGroups: [ "apps" ]
          resources: [ "daemonsets", "deployments", "replicasets", "statefulsets" ]
          verbs: [ "get", "list", "watch" ]
        - apiGroups: [ "extensions" ]
          resources: [ "daemonsets", "deployments", "replicasets" ]
          verbs: [ "get", "list", "watch" ]
        - apiGroups: [ "batch" ]
          resources: [ "jobs", "cronjobs" ]
          verbs: [ "get", "list", "watch" ]
        - apiGroups: [ "autoscaling" ]
          resources: [ "horizontalpodautoscalers" ]
          verbs: [ "get", "list", "watch" ]
digging into this I discovered that the final config is created by a function in a template and that I cannot override the final result by changing the values unfortunately.
s
Srikanth to look into this
a
I ended up using the open-telemetry/opentelemetry-collector helm chart instead of tweaking the infra-k8s chart. Thanks!
2 Views
Open in Slack
PreviousNext
Powered by