https://signoz.io logo
Join Slack
Powered by
Hi All, I need to scrap data from prometheus, do...
# general
s
Hi All, I need to scrap data from prometheus, done the changes in 
my-release-signoz-otel-collector.yaml
config file by adding below snippet
Copy code
prometheus:
        config:
          scrape_configs:
            - job_name: "otel-collector"
              scrape_interval: 30s
              static_configs:
                - targets: ["otel-collector:8889", "prometheus-operated:9090"]
my prometheus is running in 
prometheus-operated:9090
 i have port-forward locally and confirmed on the service it is fine and restarted otel-deployment and otel-collector deployment pods and getting below error in collector pods
Copy code
{"level":"error","timestamp":"2024-04-04T11:36:23.737Z","caller":"opamp/server_client.go:268","msg":"Collector failed for restart during rollback","component":"opamp-server-client","error":"failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:\n\n* error decoding 'exporters': error reading configuration for \"prometheus\": 1 error(s) decoding:\n\n* '' has invalid keys: config","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload|github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:268\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).applyRemoteConfig\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:173\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).Apply\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:159\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:209\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
{"level":"error","timestamp":"2024-04-04T11:36:23.737Z","caller":"opamp/server_client.go:216","msg":"failed to apply config","component":"opamp-server-client","error":"failed to reload config: /var/tmp/collector-config.yaml: collector failed to restart: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:\n\n* error decoding 'exporters': error reading configuration for \"prometheus\": 1 error(s) decoding:\n\n* '' has invalid keys: config","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler|github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:216\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
cc @Prashant Shahi @Srikanth Chekuri what am i missing ?
p
I think you have added the prometheus exporter, where it should have been the receiver.
s
Copy code
apiVersion: v1
data:
  otel-collector-config.yaml: "exporters:\n  clickhouselogsexporter:\n    dsn: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}\n
    \   timeout: 10s\n  clickhousemetricswrite:\n    endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}\n
    \   resource_to_telemetry_conversion:\n      enabled: true\n    timeout: 15s\n
    \ clickhousetraces:\n    datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}\n
    \   low_cardinal_exception_grouping: ${LOW_CARDINAL_EXCEPTION_GROUPING}\n  prometheus:\n
    \   config:\n      scrape_configs:\n        - job_name: \"otel-collector\"\n          scrape_interval:
    30s\n          static_configs:\n            - targets: [\"otel-collector:8889\",
    \"prometheus-operated:9090\"]\n\nextensions:\n  health_check:\n    endpoint: 0.0.0.0:13133\n
    \ pprof:\n    endpoint: localhost:1777\n  zpages:\n    endpoint: localhost:55679\nprocessors:\n
    \ batch:\n    send_batch_size: 50000\n    timeout: 1s\n  k8sattributes:\n    extract:\n
    \     metadata:\n      - k8s.namespace.name\n      - k8s.pod.name\n      - k8s.pod.uid\n
    \     - k8s.pod.start_time\n      - k8s.deployment.name\n      - k8s.node.name\n
    \   filter:\n      node_from_env_var: K8S_NODE_NAME\n    passthrough: false\n
    \   pod_association:\n    - sources:\n      - from: resource_attribute\n        name:
    k8s.pod.ip\n    - sources:\n      - from: resource_attribute\n        name: k8s.pod.uid\n
    \   - sources:\n      - from: connection\n  memory_limiter: null\n  resourcedetection:\n
    \   detectors:\n    - env\n    - system\n    system:\n      hostname_sources:\n
    \     - dns\n      - os\n    timeout: 2s\n  signozspanmetrics/cumulative:\n    dimensions:\n
    \   - default: default\n      name: service.namespace\n    - default: default\n
    \     name: deployment.environment\n    - name: signoz.collector.id\n    dimensions_cache_size:
    100000\n    latency_histogram_buckets:\n    - 100us\n    - 1ms\n    - 2ms\n    -
    6ms\n    - 10ms\n    - 50ms\n    - 100ms\n    - 250ms\n    - 500ms\n    - 1000ms\n
    \   - 1400ms\n    - 2000ms\n    - 5s\n    - 10s\n    - 20s\n    - 40s\n    - 60s\n
    \   metrics_exporter: clickhousemetricswrite\n  signozspanmetrics/delta:\n    aggregation_temporality:
    AGGREGATION_TEMPORALITY_DELTA\n    dimensions:\n    - default: default\n      name:
    service.namespace\n    - default: default\n      name: deployment.environment\n
    \   - name: signoz.collector.id\n    dimensions_cache_size: 100000\n    latency_histogram_buckets:\n
    \   - 100us\n    - 1ms\n    - 2ms\n    - 6ms\n    - 10ms\n    - 50ms\n    - 100ms\n
    \   - 250ms\n    - 500ms\n    - 1000ms\n    - 1400ms\n    - 2000ms\n    - 5s\n
    \   - 10s\n    - 20s\n    - 40s\n    - 60s\n    metrics_exporter: clickhousemetricswrite\nreceivers:\n
    \ hostmetrics:\n    collection_interval: 30s\n    scrapers:\n      cpu: {}\n      disk:
    {}\n      filesystem: {}\n      load: {}\n      memory: {}\n      network: {}\n
    \ httplogreceiver/heroku:\n    endpoint: 0.0.0.0:8081\n    source: heroku\n  httplogreceiver/json:\n
    \   endpoint: 0.0.0.0:8082\n    source: json\n  jaeger:\n    protocols:\n      grpc:\n
    \       endpoint: 0.0.0.0:14250\n      thrift_http:\n        endpoint: 0.0.0.0:14268\n
    \ otlp:\n    protocols:\n      grpc:\n        endpoint: 0.0.0.0:4317\n        max_recv_msg_size_mib:
    16\n      http:\n        endpoint: 0.0.0.0:4318\n  otlp/spanmetrics:\n    protocols:\n
    \     grpc:\n        endpoint: localhost:12345\nservice:\n  extensions:\n  - health_check\n
    \ - zpages\n  - pprof\n  pipelines:\n    logs:\n      exporters:\n      - clickhouselogsexporter\n
    \     processors:\n      - batch\n      receivers:\n      - otlp\n      - httplogreceiver/heroku\n
    \     - httplogreceiver/json\n    metrics:\n      exporters:\n      - clickhousemetricswrite\n
    \     processors:\n      - batch\n      receivers:\n      - otlp\n      - prometheus\n
    \   metrics/internal:\n      exporters:\n      - clickhousemetricswrite\n      processors:\n
    \     - resourcedetection\n      - k8sattributes\n      - batch\n      receivers:
    \n      - hostmetrics\n    traces:\n      exporters:\n      - clickhousetraces\n
    \     processors:\n      - signozspanmetrics/cumulative\n      - signozspanmetrics/delta\n
    \     - batch\n      receivers:\n      - otlp\n      - jaeger\n  telemetry:\n
    \   metrics:\n      address: 0.0.0.0:8888"
  otel-collector-opamp-config.yaml: 'server_endpoint: "<ws://my-release-signoz-query-service:4320/v1/opamp>"'
kind: ConfigMap
metadata:
  annotations:
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: my-release
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: platform
  labels:
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: otel-collector
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: my-release
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: signoz
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: 0.41.1
    <http://helm.sh/chart|helm.sh/chart>: signoz-0.37.1
    k8slens-edit-resource-version: v1
  name: my-release-signoz-otel-collector
  namespace: platform
only receiver , anythin g wrong in config
p
prometheus: ...
should be under 
receivers
and not 
exporters
s
oh
will try
Copy code
warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1712237452481, "target_labels": "{__name__=\"up\", instance=\"prometheus-operated:9090\", job=\"otel-collector\"}"}
prometheus is in different namespace can it scrape the data ?
??
i need to edit -> my-release-signoz-otel-collector.yaml config only right
p
it is not recommended to edit the configmap directly, but via Helm override value file
s
okay, but i deployed prometehus via helm only community version i need to pass the service name /metrics endpoint or /graph endpoint ?
okay will edit via helm and see
https://github.com/SigNoz/charts/blob/main/charts/signoz/values.yaml i am not sure where i need to add the prometheus config, little confusing...can you let me know the line number and snippet pls
@Prashant Shahi sorry to bother u much, i can see many ppl had this doubt before and still i am not able to figure out i have prometheus running in monitoring namespace , how i can scrape the data and what changes i need to do helm k8s signoz try to help me bro
p
Try including the following in your override config:
Copy code
otelCollectorMetrics:
  config:
    receivers:
      prometheus/custom:
        config:
          scrape_configs:
            - job_name: "custom-collector"
              scrape_interval: 30s
              static_configs:
                - targets:
                  - prometheus-operated.monitoring.svc.cluster.local:9090
    service:
      pipelines:
        metrics:
          receivers: [prometheus, prometheus/custom]
Followed by helm upgrade:
Copy code
helm upgrade my-release -n platform signoz/signoz -f override-config.yaml
s
great thanks
p
Use 
otelCollectorMetrics
instead of 
otelCollector
as multiple instances of otelCollector will cause redundant data.
s
will try
23 Views
Open in Slack
PreviousNext
Powered by