Hello! I'm trying to set up a simple test of Prometheus/Node Exporter to demo internally to our company (so we can decide to start using SigNoz more generally). I'm running Node Exporter on a host and can confirm that I see Prometheus successfully getting those metrics. However, I can't seem to get SigNoz to receive those same metrics. I've followed this (https://signoz.io/docs/userguide/send-metrics/#enable-a-prometheus-receiver) and added the endpoint where Node Exporter is exposing these metrics to my

deploy/docker/clickhouse-setup/otel-collector-metrics-config.yaml

. However, I can't seem to find any of these metrics showing up in SigNoz. Could I get some help troubleshooting? more info in 🧵

Was it the case of the imported dashboard not being able to show the data, or was it the no data scraped? Can you use the query builder tab and check for the node exporter metric names and see if they show up?

i do not see them show up in the query builder. i think the ones i do see are metrics that SigNoz collects already.

Can you share the logs of the collector metrics receiver? And what was the scrape config?

here's my scrape_config:

receivers:
  otlp:
    protocols:
      grpc:
      http:
  prometheus:
    config:
      scrape_configs:
        # otel-collector-metrics internal metrics
        - job_name: otel-collector-metrics
          scrape_interval: 60s
          static_configs:
            - targets: ["localhost:8888", "pacific:9100"]
              labels:
                job_name: otel-collector-metrics
        # SigNoz span metrics
        - job_name: signozspanmetrics-collector
          scrape_interval: 60s
          static_configs:
            - targets:
              - otel-collector:8889

pacific:9100

is the machine i want to hit. it's using tailscale's MagicDNS. OH. well now i know i should have checked the

clickhouse-setup_otel-collector-metrics_1

logs.

$ docker logs clickhouse-setup_otel-collector-metrics_1
...
...
2023-01-13T01:03:18.173Z	info	prometheusreceiver@v0.66.0/metrics_receiver.go:288	Starting scrape manager	{"kind": "receiver", "name": "prometheus", "pipeline": "metrics"}
2023-01-13T01:04:19.900Z	warn	internal/transaction.go:120	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "pipeline": "metrics", "scrape_timestamp": 1673571859900, "target_labels": "{__name__=\"up\", instance=\"pacific:9100\", job=\"otel-collector-metrics\", job_name=\"otel-collector-metrics\"}"}

It says failed to scrape; that’s the reason they aren’t showing up. How does the pod/container know about the tailescale’s magic?

hmm.. it's running on a host that has tailscale configured. i assumed it would just work.

No, that would have been correct if you running the collector binary directly. Since you are running a docker container it doesn’t work.

alright. i'll play with it. thanks for your help!

You can enable debug logging here https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/otel-collector-metrics-config.yaml#L58

logging:
  loglevel: debug

sorry... where does the

logging

key go?

'service.telemetry' has invalid keys: logging
2023/01/17 16:56:13 application run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* 'service.telemetry' has invalid keys: logging

i think i've tried nesting it under all of them -- service, telemetry, or metrics, with no luck.

service:
  telemetry:
    logging:
      loglevel: debug
    metrics:
      address: 0.0.0.0:8888

Ah sorry it should be logs instead

service:
  telemetry:
    logs:
      level: debug

i finally got around to working on this again. works.

$ sudo ufw allow 9100

🤦‍♂️