https://signoz.io logo
Join Slack
Powered by
This message was deleted.
# support
s
This message was deleted.
a
we are creating APM metrics from traces and exposing them at otel-collector level using prometheus exporter at otel-collector config
we use the otel-collector-metrics to scrape apm from other otel collectors and write to clickhouse
b
So if I want to collect prometheus metrics would I configure the otel collector or otel metrics collecotr?
a
otel metrics collector
would be a better place
it would be like a prometheus scraping instance
and since prometheus receiver works on a scrape config it cannot be horizontally scaled. It will created duplicated metrics as the prom config will get passed on every otel-collector-metrics instance
b
see I would have thought that as well however look at this
on the helm chart
Copy code
otelCollectorMetrics:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      # Data sources: metrics
      prometheus:
        config:
          scrape_configs:
            - job_name: otel-collector
              scrape_interval: 30s
              static_configs:
                - targets:
                  - prometheus.k8s.ops.us-west-2.aws.wna.cloud:80
                  # the following string is replaced with OTel service name using the helper template
                  - $OTEL_COLLECTOR_PROMETHEUS
a
can you check if 
kuberhealthy_check
metric is available at suggestions of query builder?
this would confirm data is ingested at clickhouse
if not, we should check logs of otel collector metrics and look for errors in scraping the endpoint specified
so it's not ingested yet I think
b
ahh well that does make sense
Copy code
2022-07-11T13:21:26.387Z	warn	internal/otlp_metricsbuilder.go:159	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "scrape_timestamp": 1657545676379, "target_labels": "map[__name__:up instance:prometheus.k8s.ops.us-west-2.aws.wna.cloud:80 job:otel-collector]"}
a
yeah
b
hmm that should be the correct url however
I also tried the 443 port still nada
a
you should be able to see the metrics when you ssh into the otel-collector-metrics container and run
Copy code
curl prometheus.k8s.ops.us-west-2.aws.wna.cloud:80/metrics
try running 
curl otel-collector:8889/metrics
to confirm
b
Copy code
kubectl exec -it signoz-otel-collector-metrics-6b8897cb8b-qz6w9 -- bash 
Defaulted container "signoz-otel-collector-metrics" out of: signoz-otel-collector-metrics, signoz-otel-collector-metrics-init (init)
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "3fecef848eab715e20bb6523233f610bf8a3dec3b0f0874ad17b85052586d3d6": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "bash": executable file not found in $PATH: unknown
fyi this error has been very annoying haha
also curl doesn’t seem to be in the container either
Copy code
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "15120171d65129eaa116c987781c453f52ea5171531aa4ab35f1b2e45051bdf0": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "curl": executable file not found in $PATH: unknown
a
yeah..I remember doing that for debugging 😛
@Prashant Shahi any help here?
p
fyi this error has been very annoying haha
yeah, 
scratch
that... 😅
Yeah, like @Ankit Nayan said.. are you sure the following command returns prometheus metrics?
Copy code
curl prometheus.k8s.ops.us-west-2.aws.wna.cloud:80/metrics
b
Copy code
curl prometheus.k8s.ops.us-west-2.aws.wna.cloud:80/metrics
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
👀 1
p
can you try this?
Copy code
kubectl -n platform run -i --tty busybox --image=busybox --restart=Never -- sh
Copy code
wget -q -O- prometheus.k8s.ops.us-west-2.aws.wna.cloud:80/metrics
b
yup I get stuff back
p
can you verify if those metrics are accessible from signoz dashboard?
you have the scrapper config set properly, right?
b
the interesting thing is I am using the otel collector helm chart in another cluster with this config
Copy code
prometheus:
      config:
        scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 5s
            static_configs:
              - targets: ["prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud:80"]
and it works fine
p
I see.. it could be caused by 
https
redirect. can you create another instance of prometheus receiver, and also set 
scheme: https
?
b
Copy code
curl prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud:80   
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
it isn’t tho
p
can you enable verbose flag?
and share output with both the endpoints..
b
Copy code
curl prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud:80 -v
*   Trying 10.200.217.61:80...
* Connected to prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud (10.200.217.61) port 80 (#0)
> GET / HTTP/1.1
> Host: prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Date: Mon, 11 Jul 2022 15:09:06 GMT
< Content-Type: text/html
< Content-Length: 164
< Connection: keep-alive
< Location: <https://prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud/>
< 
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud left intact
Copy code
curl prometheus.k8s.ops.us-west-2.aws.wna.cloud:80 -v                      
*   Trying 172.16.4.164:80...
* Connected to prometheus.k8s.ops.us-west-2.aws.wna.cloud (172.16.4.164) port 80 (#0)
> GET / HTTP/1.1
> Host: prometheus.k8s.ops.us-west-2.aws.wna.cloud
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Date: Mon, 11 Jul 2022 15:09:28 GMT
< Content-Type: text/html
< Content-Length: 164
< Connection: keep-alive
< Location: <https://prometheus.k8s.ops.us-west-2.aws.wna.cloud/>
< 
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host prometheus.k8s.ops.us-west-2.aws.wna.cloud left intact
p
both of them seems to be identical..
b
yup
p
only difference in config I see is 
targets
using 
[ ]
and 
-
with line breaks.. Or perhaps it could be double quotes.
so, the latter one you have it deployed using Helm chart, while first one using Operator, right?
b
I can add the quotes and see
I am not seeing errors in logs. But not seeing those kuberhealthy data points in the suggestion.
p
was 
kuberhealthy
present when you tried curl?
b
oop I think I am getting data now
p
nice..
b
also if y’all have any ideas about the kube-prom thing that’d be awesome 🙂
p
Also, you can run the following to get unique metrics from that particular endpoint..
Copy code
wget -q -O- prometheus.k8s.ops.us-west-2.aws.wna.cloud:80/metrics | grep "^\w*" -o | uniq | sort
also if y’all have any ideas about the kube-prom thing that’d be awesome
not sure about that..
b
yeah I have been digging in the docs on that and not super effective so far
@Prashant Shahi Regarding OpenTelemetry Collector and Prometheus Operator. it looks like Prom Operator scrapes things via service monitors and then the prometheus instance allows you to get that data and query from it. I am trying to figure out how with Otel to use those targets it grabs and send them to Otel.
p
@Blake Romano any particular reason to go with prometheus operator?
Because there are other ways to collect k8s metrics like: • kubeletstatsreceiver and other k8s receivers • use 
kubernetes_sd_configs
in otelCollector prometheus receiver: refer to these links: ◦ https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md ◦ https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml
b
That's just what we have so that's what I'm stuck with
17 Views
Open in Slack
PreviousNext
Powered by