Can signoz pick up metrics from a pod that has /metrics enabled (and the prometheus-annotations in place) Or how do I best get signoz to pickup on exported metrics from my pods? (signoz runs in k8s)

You can use prometheus receiver: https://signoz.io/docs/userguide/send-metrics-cloud/#enable-a-prometheus-receiver

@Vishal Sharma Awesome, but that one does not really cover how to do it when running in k8s.

Open the deploy/docker/clickhouse-setup/otel-collector-config.yaml file in a plain-text editor.

I deploy using your helm-charts.

Have you installed k8s otel agent?

yes

Edit k8s infra chart values

# -- Configurations for OtelAgent

We recommend using otel-deployment instead to scrap Prometheus metrics. Your override values will include something like:

otelDeployment:
  config:
    receivers:
      prometheus/scraper:
        config:
          scrape_configs:
            - job_name: "otel-deployment"
              scrape_interval: 60s
              kubernetes_sd_configs:
                - role: pod
              relabel_configs:
                - source_labels: [__meta_kubernetes_pod_annotation_signoz_io_scrape]
                  action: keep
                  regex: true
                - source_labels: [__meta_kubernetes_pod_annotation_signoz_io_path]
                  action: replace
                  target_label: __metrics_path__
                  regex: (.+)
                - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_annotation_signoz_io_port]
                  action: replace
                  separator: ":"
                  target_label: __address__
                - target_label: job_name
                  replacement: otel-deployment
                - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
                  action: replace
                  target_label: signoz_k8s_name
                - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
                  action: replace
                  target_label: signoz_k8s_instance
                - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
                  action: replace
                  target_label: signoz_k8s_component
                - source_labels: [__meta_kubernetes_namespace]
                  action: replace
                  target_label: k8s_namespace_name
                - source_labels: [__meta_kubernetes_pod_name]
                  action: replace
                  target_label: k8s_pod_name
                - source_labels: [__meta_kubernetes_pod_uid]
                  action: replace
                  target_label: k8s_pod_uid
                - source_labels: [__meta_kubernetes_pod_node_name]
                  action: replace
                  target_label: k8s_node_name
                - source_labels: [__meta_kubernetes_pod_ready]
                  action: replace
                  target_label: k8s_pod_ready
                - source_labels: [__meta_kubernetes_pod_phase]
                  action: replace
                  target_label: k8s_pod_phase
    service:
      pipelines:
        metrics/internal:
          receivers: [prometheus/scraper]

To use Prometheus scraper annotations, replace

signoz_io

with

prometheus_io

from

relabel_configs

section.

Yea, I'm not fuzzed about using prometheus, I just want to scrape metrics and get them into signoz. Thank you @Prashant Shahi I will try this later today.