I deployed the signoz and k8s-infra helm chart and am seeing issues with metrics, the signoz-otel-collector-metrics pod is throwing this error in particular:

{
  "level": "warn",
  "ts": 1741997691.754477,
  "caller": "internal/transaction.go:128",
  "msg": "Failed to scrape Prometheus endpoint",
  "kind": "receiver",
  "name": "prometheus",
  "data_type": "metrics",
  "scrape_timestamp": 1741997691753,
  "target_labels": "{__name__=\"up\", instance=\"10.0.146.7:8889\", job=\"generic-collector\", job_name=\"generic-collector\", k8s_container_name=\"signoz-otel-collector-k8s-infra-otel-agent\", k8s_namespace_name=\"signoz\", k8s_node_name=\"ip-10-0-173-254.us-west-2.compute.internal\", k8s_pod_name=\"signoz-otel-collector-k8s-infra-otel-agent-csrnr\", k8s_pod_phase=\"Running\", k8s_pod_ready=\"true\", k8s_pod_uid=\"b943d3b4-4786-492a-839d-41e98c6d8643\", signoz_k8s_component=\"otel-agent\", signoz_k8s_instance=\"signoz-otel-collector\", signoz_k8s_name=\"k8s-infra\"}"
}

Any ideas?

@Sam Currie i am also getting this warning, i think this service collects metrics of signoz collector only, other metrics are coming fine. what metrics you are having issue with?

{
  "level": "warn",
  "ts": 1742014234.0471196,
  "caller": "internal/transaction.go:128",
  "msg": "Failed to scrape Prometheus endpoint",
  "kind": "receiver",
  "name": "prometheus",
  "data_type": "metrics",
  "scrape_timestamp": 1742014234044,
  "target_labels": "{__name__=\"up\", instance=\"10.10.4.230:8888\", job=\"generic-collector\", job_name=\"generic-collector\", k8s_container_name=\"collector\", k8s_namespace_name=\"monitoring\", k8s_node_name=\"ip-10-10-6-159.ap-south-1.compute.internal\", k8s_pod_name=\"signoz-infra-otel-collector-8574b654ff-rgdtb\", k8s_pod_phase=\"Running\", k8s_pod_ready=\"true\", k8s_pod_uid=\"d05d2d35-5461-4755-936b-f8bbae119faa\", signoz_k8s_component=\"otel-collector\", signoz_k8s_instance=\"signoz-infra\", signoz_k8s_name=\"signoz\"}"
}

@Abhay I’m not seeing pod & container metrics in signoz, I am seeing host metrics however

Nice, did it start to populate?

@Sam Currie can you share screenshot of where you are looking ?

@Vibhu Pandey I upgraded the app from 0.74 -> 0.76 and metrics started flowing; I am still getting the scraping error but I believe @Abhay is correct and that’s just the collector itself