https://signoz.io logo
#support
Title
# support
b

Blake Romano

07/11/2022, 1:21 PM
Also on a different but similar question. What is the difference between the Otel collector, and Otel Metrics Collector with the Helm Chart? Why not have one Otel Collector for everything? I am a bit confused on distinguishment.
a

Ankit Nayan

07/11/2022, 1:28 PM
we are creating APM metrics from traces and exposing them at otel-collector level using prometheus exporter at otel-collector config
we use the otel-collector-metrics to scrape apm from other otel collectors and write to clickhouse
b

Blake Romano

07/11/2022, 1:29 PM
So if I want to collect prometheus metrics would I configure the otel collector or otel metrics collecotr?
a

Ankit Nayan

07/11/2022, 1:30 PM
otel metrics collector
would be a better place
it would be like a prometheus scraping instance
and since prometheus receiver works on a scrape config it cannot be horizontally scaled. It will created duplicated metrics as the prom config will get passed on every otel-collector-metrics instance
b

Blake Romano

07/11/2022, 1:32 PM
see I would have thought that as well however look at this
on the helm chart
Copy code
otelCollectorMetrics:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      # Data sources: metrics
      prometheus:
        config:
          scrape_configs:
            - job_name: otel-collector
              scrape_interval: 30s
              static_configs:
                - targets:
                  - prometheus.k8s.ops.us-west-2.aws.wna.cloud:80
                  # the following string is replaced with OTel service name using the helper template
                  - $OTEL_COLLECTOR_PROMETHEUS
a

Ankit Nayan

07/11/2022, 1:34 PM
can you check if
kuberhealthy_check
metric is available at suggestions of query builder?
this would confirm data is ingested at clickhouse
if not, we should check logs of otel collector metrics and look for errors in scraping the endpoint specified
b

Blake Romano

07/11/2022, 1:36 PM
a

Ankit Nayan

07/11/2022, 1:36 PM
so it's not ingested yet I think
b

Blake Romano

07/11/2022, 1:36 PM
ahh well that does make sense
Copy code
2022-07-11T13:21:26.387Z	warn	internal/otlp_metricsbuilder.go:159	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "scrape_timestamp": 1657545676379, "target_labels": "map[__name__:up instance:prometheus.k8s.ops.us-west-2.aws.wna.cloud:80 job:otel-collector]"}
a

Ankit Nayan

07/11/2022, 1:36 PM
yeah
b

Blake Romano

07/11/2022, 1:38 PM
hmm that should be the correct url however
I also tried the 443 port still nada
a

Ankit Nayan

07/11/2022, 1:43 PM
you should be able to see the metrics when you ssh into the otel-collector-metrics container and run
Copy code
curl prometheus.k8s.ops.us-west-2.aws.wna.cloud:80/metrics
try running
curl otel-collector:8889/metrics
to confirm
b

Blake Romano

07/11/2022, 1:45 PM
Copy code
kubectl exec -it signoz-otel-collector-metrics-6b8897cb8b-qz6w9 -- bash 
Defaulted container "signoz-otel-collector-metrics" out of: signoz-otel-collector-metrics, signoz-otel-collector-metrics-init (init)
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "3fecef848eab715e20bb6523233f610bf8a3dec3b0f0874ad17b85052586d3d6": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "bash": executable file not found in $PATH: unknown
fyi this error has been very annoying haha
also curl doesn’t seem to be in the container either
Copy code
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "15120171d65129eaa116c987781c453f52ea5171531aa4ab35f1b2e45051bdf0": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "curl": executable file not found in $PATH: unknown
a

Ankit Nayan

07/11/2022, 1:46 PM
yeah..I remember doing that for debugging 😛
@Prashant Shahi any help here?
p

Prashant Shahi

07/11/2022, 2:49 PM
fyi this error has been very annoying haha
yeah,
scratch
that... 😅
Yeah, like @Ankit Nayan said.. are you sure the following command returns prometheus metrics?
Copy code
curl prometheus.k8s.ops.us-west-2.aws.wna.cloud:80/metrics
b

Blake Romano

07/11/2022, 2:55 PM
Copy code
curl prometheus.k8s.ops.us-west-2.aws.wna.cloud:80/metrics
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
👀 1
p

Prashant Shahi

07/11/2022, 2:55 PM
can you try this?
Copy code
kubectl -n platform run -i --tty busybox --image=busybox --restart=Never -- sh
Copy code
wget -q -O- prometheus.k8s.ops.us-west-2.aws.wna.cloud:80/metrics
b

Blake Romano

07/11/2022, 2:57 PM
yup I get stuff back
p

Prashant Shahi

07/11/2022, 2:58 PM
can you verify if those metrics are accessible from signoz dashboard?
you have the scrapper config set properly, right?
b

Blake Romano

07/11/2022, 3:05 PM
the interesting thing is I am using the otel collector helm chart in another cluster with this config
Copy code
prometheus:
      config:
        scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 5s
            static_configs:
              - targets: ["prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud:80"]
and it works fine
p

Prashant Shahi

07/11/2022, 3:07 PM
I see.. it could be caused by
https
redirect. can you create another instance of prometheus receiver, and also set
scheme: https
?
b

Blake Romano

07/11/2022, 3:08 PM
Copy code
curl prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud:80   
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
it isn’t tho
p

Prashant Shahi

07/11/2022, 3:08 PM
can you enable verbose flag?
and share output with both the endpoints..
b

Blake Romano

07/11/2022, 3:09 PM
Copy code
curl prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud:80 -v
*   Trying 10.200.217.61:80...
* Connected to prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud (10.200.217.61) port 80 (#0)
> GET / HTTP/1.1
> Host: prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Date: Mon, 11 Jul 2022 15:09:06 GMT
< Content-Type: text/html
< Content-Length: 164
< Connection: keep-alive
< Location: <https://prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud/>
< 
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host prom-doolittle-dev.k8s.doolittle-nonprod.us-west-2.aws.wna.cloud left intact
Copy code
curl prometheus.k8s.ops.us-west-2.aws.wna.cloud:80 -v                      
*   Trying 172.16.4.164:80...
* Connected to prometheus.k8s.ops.us-west-2.aws.wna.cloud (172.16.4.164) port 80 (#0)
> GET / HTTP/1.1
> Host: prometheus.k8s.ops.us-west-2.aws.wna.cloud
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Date: Mon, 11 Jul 2022 15:09:28 GMT
< Content-Type: text/html
< Content-Length: 164
< Connection: keep-alive
< Location: <https://prometheus.k8s.ops.us-west-2.aws.wna.cloud/>
< 
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host prometheus.k8s.ops.us-west-2.aws.wna.cloud left intact
p

Prashant Shahi

07/11/2022, 3:10 PM
both of them seems to be identical..
b

Blake Romano

07/11/2022, 3:12 PM
yup
p

Prashant Shahi

07/11/2022, 3:14 PM
only difference in config I see is
targets
using
[ ]
and
-
with line breaks.. Or perhaps it could be double quotes.
so, the latter one you have it deployed using Helm chart, while first one using Operator, right?
b

Blake Romano

07/11/2022, 3:14 PM
I can add the quotes and see
I am not seeing errors in logs. But not seeing those kuberhealthy data points in the suggestion.
p

Prashant Shahi

07/11/2022, 3:19 PM
was
kuberhealthy
present when you tried curl?
b

Blake Romano

07/11/2022, 3:21 PM
oop I think I am getting data now
p

Prashant Shahi

07/11/2022, 3:22 PM
nice..
b

Blake Romano

07/11/2022, 3:22 PM
also if y’all have any ideas about the kube-prom thing that’d be awesome 🙂
p

Prashant Shahi

07/11/2022, 3:23 PM
Also, you can run the following to get unique metrics from that particular endpoint..
Copy code
wget -q -O- prometheus.k8s.ops.us-west-2.aws.wna.cloud:80/metrics | grep "^\w*" -o | uniq | sort
also if y’all have any ideas about the kube-prom thing that’d be awesome
not sure about that..
b

Blake Romano

07/11/2022, 3:36 PM
yeah I have been digging in the docs on that and not super effective so far
@Prashant Shahi Regarding OpenTelemetry Collector and Prometheus Operator. it looks like Prom Operator scrapes things via service monitors and then the prometheus instance allows you to get that data and query from it. I am trying to figure out how with Otel to use those targets it grabs and send them to Otel.
p

Prashant Shahi

07/12/2022, 6:14 AM
@Blake Romano any particular reason to go with prometheus operator?
Because there are other ways to collect k8s metrics like: • kubeletstatsreceiver and other k8s receivers • use
kubernetes_sd_configs
in otelCollector prometheus receiver: refer to these links: ◦ https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.mdhttps://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml
b

Blake Romano

07/12/2022, 11:08 AM
That's just what we have so that's what I'm stuck with
3 Views