https://signoz.io logo
Join Slack
Powered by
Hello, I'm running into an issue with exported lab...
# support
e
Hello, I'm running into an issue with exported labels. I have a kind cluster with a few services exposing metrics on a prometheus /metrics endpoint. I want SigNoz to run in this cluster, discover these services, and scrape from the metrics endpoints. With the following value overrides, I can see the correct metrics and services in SigNoz: 
otelCollectorMetrics:
ports:
prometheus:
enabled: true
containerPort: 9464
servicePort: 9464
protocol: TCP
config:
receivers:
prometheus:
config:
scrape_configs:
- job_name: kubernetes-services
kubernetes_sd_configs:
- role: service
exporters:
prometheus:
endpoint: 0.0.0.0:9464
service:
pipelines:
metrics:
exporters: [clickhousemetricswrite]
I also want the OpenTelemetry Collector to expose a Prometheus endpoint. But as soon as I add "prometheus" to the list of exporters, I see recursively added labels with an "exported" prefix. For example, "exported_exported_job", "exported_exported_exported_job", "exported_exported_exported_exported_job", etc. Am I configuring the prometheus exporter incorrectly?
e
I solved this with a different pattern. I created a otel-collector who's entire job is to scrape prometheus endpoints of various products and ship them to the signoz otel collector. Works great.
Copy code
apiVersion: <http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>
kind: OpenTelemetryCollector
metadata:
  name: prometheus-scraper
spec:
  mode: deployment # This configuration is omittable.
  config: |
    receivers:
      otlp:
        protocols:
          grpc: {}
          http: {}
      prometheus:
        config:
          global:
            scrape_interval: 20s # Adjust this interval as needed
          scrape_configs:
            - job_name: 'kong'
              static_configs:
                - targets: ['kong-metrics:9119']
            - job_name: 'mongodb'
              static_configs:
                - targets: ['mongodb-metrics:9216']
            - job_name: 'nats'
              static_configs:
                - targets: ['nats-metrics:7777']

    processors:
      batch:
        send_batch_size: 1000
        timeout: 10s
      attributes/upsert:
        attributes:
          - key: deployment.environment
            value: dev
            action: insert
        actions:
          - key: k8s_cluster_name
            value: dev
            action: upsert
          - key: k8s_namespace_name
            value: dev
            action: upsert

    exporters:
      otlp:
        endpoint: "ingest.signoz.host.name:4317"
        tls:
          insecure: true
        timeout: 20s # Adjust the timeout value as needed
      logging:
        verbosity: detailed

    service:
      telemetry:
        metrics:
          address: localhost:8888
      pipelines:
        metrics:
          receivers: [otlp, prometheus]
          processors: [attributes/upsert, batch]
          #processors: [batch]
          exporters: [otlp]
👍 1
As an example ^
👍 1
13 Views
Open in Slack
PreviousNext
Powered by