https://signoz.io logo
Join Slack
Powered by
as we are facing an issue when enabling the Promet...
# support
v
as we are facing an issue when enabling the Prometheus port in Otel collector and adding istio scrape configs in Prometheus configs in default _values.yaml file 2024-02-08T093134.259Z warn internal/transaction.go:123 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1707384694258, "target_labels": "{name=\"up\", app=\"istiod\", install_operator_istio_io_owning_resource=\"unknown\", instance=\":\", istio=\"pilot\", istio_io_rev=\"default\", job=\"kubernetes-pods\", kubernetes_namespace=\"istio-system\", kubernetes_pod_name=\"istiod-854d97f5-256pr\", operator_istio_io_component=\"Pilot\", pod_template_hash=\"854d97f5\", sidecar_istio_io_inject=\"false\"}"} 2024-02-08T093140.161Z warn internal/transaction.go:123 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1707384700158, "target_labels": "{name=\"up\", beta_kubernetes_io_arch=\"amd64\", beta_kubernetes_io_instance_type=\"c5.xlarge\", beta_kubernetes_io_os=\"linux\", eks_amazonaws_com_capacityType=\"ON_DEMAND\", eks_amazonaws_com_nodegroup=\"c5-xlarge-20240208091000231000000013\", eks_amazonaws_com_nodegroup_image=\"ami-0f7abd12e02335c8d\", eks_amazonaws_com_sourceLaunchTemplateId=\"lt-013efec8b1cd57673\", eks_amazonaws_com_sourceLaunchTemplateVersion=\"1\", failure_domain_beta_kubernetes_io_region=\"us-east-1\", failure_domain_beta_kubernetes_io_zone=\"us-east-1a\", instance=\"ip-10-15-1-6.ec2.internal\", job=\"kubernetes-nodes\", k8s_io_cloud_provider_aws=\"8cc336c7f8f631cc2813e68cba2b02da\", kubernetes_io_arch=\"amd64\", kubernetes_io_hostname=\"ip-10-15-1-6.ec2.internal\", kubernetes_io_os=\"linux\", node_kubernetes_io_instance_type=\"c5.xlarge\", topology_ebs_csi_aws_com_zone=\"us-east-1a\", topology_kubernetes_io_region=\"us-east-1\", topology_kubernetes_io_zone=\"us-east-1a\"}"}
r
Hi 🙂 This error seems to indicate a failure in being able to reach the target of scraping Please share your otel collector config
v
Copy code
prometheus:
        config:
          scrape_configs:
            # generic prometheus metrics scraper (scrapped when <http://signoz.io|signoz.io> pod annotations are set)
            - job_name: prometheus
              static_configs:
                - targets:
                    - localhost:9090
            - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              job_name: kubernetes-apiservers
              kubernetes_sd_configs:
                - role: endpoints
                  namespaces:
                    names:
                      - istio-system
              relabel_configs:
                - action: keep
                  regex: default;kubernetes;https
                  source_labels:
                    - __meta_kubernetes_namespace
                    - __meta_kubernetes_service_name
                    - __meta_kubernetes_endpoint_port_name
              scheme: https
              tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
            - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              job_name: kubernetes-nodes
              kubernetes_sd_configs:
                - role: node
              relabel_configs:
                - action: labelmap
                  regex: __meta_kubernetes_node_label_(.+)
                - replacement: kubernetes.default.svc:443
                  target_label: __address__
                - regex: (.+)
                  replacement: /api/v1/nodes/$1/proxy/metrics
                  source_labels:
                    - __meta_kubernetes_node_name
                  target_label: __metrics_path__
              scheme: https
              tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
            - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              job_name: kubernetes-nodes-cadvisor
              kubernetes_sd_configs:
                - role: node
                  namespaces:
                    names:
                      - istio-system
              relabel_configs:
                - action: labelmap
                  regex: __meta_kubernetes_node_label_(.+)
                - replacement: kubernetes.default.svc:443
                  target_label: __address__
                - regex: (.+)
                  replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
                  source_labels:
                    - __meta_kubernetes_node_name
                  target_label: __metrics_path__
              scheme: https
              tls_config:
                ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                insecure_skip_verify: true
            - job_name: kubernetes-service-endpoints
              kubernetes_sd_configs:
                - role: endpoints
                  namespaces:
                    names:
                      - istio-system
              relabel_configs:
                - action: keep
                  regex: true
                  source_labels:
                    - __meta_kubernetes_service_annotation_prometheus_io_scrape
                - action: replace
                  regex: (https?)
                  source_labels:
                    - __meta_kubernetes_service_annotation_prometheus_io_scheme
                  target_label: __scheme__
                - action: replace
                  regex: (.+)
                  source_labels:
                    - __meta_kubernetes_service_annotation_prometheus_io_path
                  target_label: __metrics_path__
                - action: replace
                  regex: ([^:]+)(?::\d+)?;(\d+)
                  replacement: $1:$2
                  source_labels:
                    - __address__
                    - __meta_kubernetes_service_annotation_prometheus_io_port
                  target_label: __address__
                - action: labelmap
                  regex: __meta_kubernetes_service_label_(.+)
                - action: replace
                  source_labels:
                    - __meta_kubernetes_namespace
                  target_label: kubernetes_namespace
                - action: replace
                  source_labels:
                    - __meta_kubernetes_service_name
                  target_label: kubernetes_name
                - action: replace
                  source_labels:
                    - __meta_kubernetes_pod_node_name
                  target_label: kubernetes_node
            - job_name: kubernetes-service-endpoints-slow
              kubernetes_sd_configs:
                - role: endpoints
                  namespaces:
                    names:
                      - istio-system
              relabel_configs:
                - action: keep
                  regex: true
                  source_labels:
                    - __meta_kubernetes_service_annotation_prometheus_io_scrape_slow
                - action: replace
                  regex: (https?)
                  source_labels:
                    - __meta_kubernetes_service_annotation_prometheus_io_scheme
                  target_label: __scheme__
                - action: replace
                  regex: (.+)
                  source_labels:
                    - __meta_kubernetes_service_annotation_prometheus_io_path
                  target_label: __metrics_path__
                - action: replace
                  regex: ([^:]+)(?::\d+)?;(\d+)
                  replacement: $1:$2
                  source_labels:
                    - __address__
                    - __meta_kubernetes_service_annotation_prometheus_io_port
                  target_label: __address__
                - action: labelmap
                  regex: __meta_kubernetes_service_label_(.+)
                - action: replace
                  source_labels:
                    - __meta_kubernetes_namespace
                  target_label: kubernetes_namespace
                - action: replace
                  source_labels:
                    - __meta_kubernetes_service_name
                  target_label: kubernetes_name
                - action: replace
                  source_labels:
                    - __meta_kubernetes_pod_node_name
                  target_label: kubernetes_node
              scrape_interval: 5m
              scrape_timeout: 30s
            - honor_labels: true
              job_name: prometheus-pushgateway
              kubernetes_sd_configs:
                - role: service
                  namespaces:
                    names:
                      - istio-system
              relabel_configs:
                - action: keep
                  regex: pushgateway
                  source_labels:
                    - __meta_kubernetes_service_annotation_prometheus_io_probe
            - job_name: kubernetes-services
              kubernetes_sd_configs:
                - role: service
                  namespaces:
                    names:
                      - istio-system
              metrics_path: /probe
              params:
                module:
                  - http_2xx
              relabel_configs:
                - action: keep
                  regex: true
                  source_labels:
                    - __meta_kubernetes_service_annotation_prometheus_io_probe
                - source_labels:
                    - __address__
                  target_label: __param_target
                - replacement: blackbox
                  target_label: __address__
                - source_labels:
                    - __param_target
                  target_label: instance
                - action: labelmap
                  regex: __meta_kubernetes_service_label_(.+)
                - source_labels:
                    - __meta_kubernetes_namespace
                  target_label: kubernetes_namespace
                - source_labels:
                    - __meta_kubernetes_service_name
                  target_label: kubernetes_name
            - job_name: kubernetes-pods
              kubernetes_sd_configs:
                - role: pod
                  namespaces:
                    names:
                      - istio-system
              relabel_configs:
                - action: keep
                  regex: true
                  source_labels:
                    - __meta_kubernetes_pod_annotation_prometheus_io_scrape
                - action: replace
                  regex: (.+)
                  source_labels:
                    - __meta_kubernetes_pod_annotation_prometheus_io_path
                  target_label: __metrics_path__
                - action: replace
                  regex: ([^:]+)(?::\d+)?;(\d+)
                  replacement: $1:$2
                  source_labels:
                    - __address__
                    - __meta_kubernetes_pod_annotation_prometheus_io_port
                  target_label: __address__
                - action: labelmap
                  regex: __meta_kubernetes_pod_label_(.+)
                - action: replace
                  source_labels:
                    - __meta_kubernetes_namespace
                  target_label: kubernetes_namespace
                - action: replace
                  source_labels:
                    - __meta_kubernetes_pod_name
                  target_label: kubernetes_pod_name
                - action: drop
                  regex: Pending|Succeeded|Failed
                  source_labels:
                    - __meta_kubernetes_pod_phase

            - job_name: kubernetes-pods-slow
              kubernetes_sd_configs:
                - role: pod
                  namespaces:
                    names:
                      - istio-system
              relabel_configs:
                - action: keep
                  regex: true
                  source_labels:
                    - __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow
                - action: replace
                  regex: (.+)
                  source_labels:
                    - __meta_kubernetes_pod_annotation_prometheus_io_path
                  target_label: __metrics_path__
                - action: replace
                  regex: ([^:]+)(?::\d+)?;(\d+)
                  replacement: $1:$2
                  source_labels:
                    - __address__
                    - __meta_kubernetes_pod_annotation_prometheus_io_port
                  target_label: __address__
                - action: labelmap
                  regex: __meta_kubernetes_pod_label_(.+)
                - action: replace
                  source_labels:
                    - __meta_kubernetes_namespace
                  target_label: kubernetes_namespace
                - action: replace
                  source_labels:
                    - __meta_kubernetes_pod_name
                  target_label: kubernetes_pod_name
                - action: drop
                  regex: Pending|Succeeded|Failed
                  source_labels:
                    - __meta_kubernetes_pod_phase
              scrape_interval: 5m
              scrape_timeout: 30s

            - job_name: "generic-collector"
              scrape_interval: 60s
              kubernetes_sd_configs:
                - role: pod
              relabel_configs:
                - source_labels:
@Raj Kamal can you go through this issue we are stucked here with istio metrics scrape configs
@Srikanth Chekuri @Raj Kamal @Prashant Shahi @Pranay are adding scrape configs of istio in Prometheus scrape configs as we are unable to send istio scrape metrics to otel mentioned error above... We tried adding Prometheus server in istio namespace and sent metrics to otel service as istio namespace Prometheus pushing metrics to otel but otel didn't respond. How to integrate istio and signoz? If there is any solution to get default dashboards while applying the helm.
Copy code
added extenison in istio                                spec:
  profile: default
  meshConfig:
    accessLogEncoding: JSON
    accessLogFile: /dev/stdout
    extensionProviders:
      - name: otel
        envoyOtelAls:
          service: signoz-otel-collector-metrics.monitoring.svc.cluster.local
          port: 4317
    defaultProviders:
      accessLogging:
        - envoy
        - otel
    outboundTrafficPolicy:
Copy code
istio logs 
	ads	RDS: PUSH request for node:productpage-v1-d4f8dfd97-76ssq.bookinfo resources:33 size:23.3kB cached:17/33
2024-02-12T13:09:20.698105Z	info	ads	ADS: "10.15.2.91:38554" istio-internal-ingressgateway-796d4fc74-hn2zz.istio-system-20 terminated
2024-02-12T13:09:20.778712Z	info	ads	ADS: new connection for node:istio-internal-ingressgateway-796d4fc74-hn2zz.istio-system-24
2024-02-12T13:09:20.779490Z	info	ads	CDS: PUSH request for node:istio-internal-ingressgateway-796d4fc74-hn2zz.istio-system resources:55 size:54.6kB cached:45/54
2024-02-12T13:09:20.779758Z	info	ads	EDS: PUSH request for node:istio-internal-ingressgateway-796d4fc74-hn2zz.istio-system resources:54 size:11.1kB empty:0 cached:54/54
2024-02-12T13:09:20.780092Z	info	ads	LDS: PUSH request for node:istio-internal-ingressgateway-796d4fc74-hn2zz.istio-system resources:0 size:0B
Copy code
signoz-otel-collector-metrics logs               {"level":"info","timestamp":"2024-02-12T12:28:31.201Z","caller":"service/service.go:69","msg":"Starting service"}
{"level":"info","timestamp":"2024-02-12T12:28:31.202Z","caller":"opamp/simple_client.go:26","msg":"Starting simple client","component":"simple-client"}
2024-02-12T12:28:31.217Z	info	service@v0.88.0/telemetry.go:84	Setting up own telemetry...
2024-02-12T12:28:31.218Z	info	service@v0.88.0/telemetry.go:201	Serving Prometheus metrics	{"address": "0.0.0.0:8888", "level": "Basic"}
2024-02-12T12:28:31.218Z	info	exporter@v0.88.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite/hostmetrics"}
time="2024-02-12T12:28:31Z" level=info msg="Shard count changed from 0 to 1. Resetting time series map." component=clickhouse
2024-02-12T12:28:31.226Z	info	kube/client.go:107	k8s filtering	{"kind": "processor", "name": "k8sattributes/hostmetrics", "pipeline": "metrics/hostmetrics", "labelSelector": "", "fieldSelector": "spec.nodeName=ip-10-15-2-161.ec2.internal"}
2024-02-12T12:28:31.230Z	info	exporter@v0.88.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite"}
2024-02-12T12:28:31.230Z	warn	filesystemscraper/factory.go:60	No `root_path` config set when running in docker environment, will report container filesystem stats. See <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver#collecting-host-metrics-from-inside-a-container-linux-only>	{"kind": "receiver", "name": "hostmetrics", "data_type": "metrics"}
time="2024-02-12T12:28:31Z" level=info msg="Shard count changed from 0 to 1. Resetting time series map." component=clickhouse
2024-02-12T12:28:31.235Z	info	service@v0.88.0/service.go:143	Starting /signoz-collector...	{"Version": "dev", "NumCPU": 2}
2024-02-12T12:28:31.235Z	info	extensions/extensions.go:33	Starting extensions...
2024-02-12T12:28:31.235Z	info	extensions/extensions.go:36	Extension is starting...	{"kind": "extension", "name": "health_check"}
2024-02-12T12:28:31.235Z	info	healthcheckextension@v0.88.0/healthcheckextension.go:35	Starting health_check extension	{"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2024-02-12T12:28:31.237Z	warn	internal@v0.88.0/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "extension", "name": "health_check", "documentation": "<https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks>"}
2024-02-12T12:28:31.239Z	info	extensions/extensions.go:43	Extension started.	{"kind": "extension", "name": "health_check"}
2024-02-12T12:28:31.239Z	info	extensions/extensions.go:36	Extension is starting...	{"kind": "extension", "name": "zpages"}
2024-02-12T12:28:31.239Z	info	zpagesextension@v0.88.0/zpagesextension.go:53	Registered zPages span processor on tracer provider	{"kind": "extension", "name": "zpages"}
2024-02-12T12:28:31.239Z	info	zpagesextension@v0.88.0/zpagesextension.go:63	Registered Host's zPages	{"kind": "extension", "name": "zpages"}
2024-02-12T12:28:31.239Z	info	zpagesextension@v0.88.0/zpagesextension.go:75	Starting zPages extension	{"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":"localhost:55679"}}}
2024-02-12T12:28:31.239Z	info	extensions/extensions.go:43	Extension started.	{"kind": "extension", "name": "zpages"}
2024-02-12T12:28:31.239Z	info	extensions/extensions.go:36	Extension is starting...	{"kind": "extension", "name": "pprof"}
2024-02-12T12:28:31.239Z	info	pprofextension@v0.88.0/pprofextension.go:60	Starting net/http/pprof server	{"kind": "extension", "name": "pprof", "config": {"TCPAddr":{"Endpoint":"localhost:1777"},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
2024-02-12T12:28:31.240Z	info	extensions/extensions.go:43	Extension started.	{"kind": "extension", "name": "pprof"}
2024-02-12T12:28:31.240Z	info	prometheusreceiver@v0.88.0/metrics_receiver.go:239	Starting discovery manager	{"kind": "receiver", "name": "prometheus", "data_type": "metrics"}
2024-02-12T12:28:31.240Z	info	prometheusreceiver@v0.88.0/metrics_receiver.go:230	Scrape job added	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "jobName": "generic-collector"}
2024-02-12T12:28:31.243Z	info	kubernetes/kubernetes.go:329	Using pod service account via in-cluster config	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "discovery": "kubernetes", "config": "generic-collector"}
@nitya-signoz can you go through this
52 Views
Open in Slack
PreviousNext
Powered by