https://signoz.io logo
#support
Title
# support
a

Alexei Zenin

07/26/2022, 8:25 PM
Do we need to run 2 Otel collectors or can we just use the one collector instead of running the other one solely to scrape the prometheus endpoint from the other? Basically can we switch this to write to clickhouse metrics instead?
Copy code
signozspanmetrics/prometheus:
    metrics_exporter: prometheus <== change this to clickhousemetricswrite
p

Pranay

07/27/2022, 4:54 AM
@Ankit Nayan any insights on this?
a

Ankit Nayan

07/27/2022, 5:04 AM
@Alexei Zenin that was what I did when building it. But then the writing of metrics will be directly proportional to the rate of incoming spans as spans create those metrics. This will be very write heavy for metrics and unnecessary. This could have also been solved by keeping metrics in memory for say 60s and then write back to clickhouse using clickhousemetricswrite but I chose the other trusted way of doing it prometheus style.
If you do not need APM metrics, you can remove the otel-collector-metrics though I don't think otel-collector-metrics will be consuming high resource
Another use of otel-collector-metrics is to add all prometheus style scraping config there and think of it as the scraper part of prometheus.
a

Alexei Zenin

07/27/2022, 1:49 PM
I see, so if I do not run it I just won’t get custom metrics? Or will parts of the SigNoz UI be missing certain information for spans?
ah i see
a

Ankit Nayan

07/27/2022, 1:56 PM
do you use these metrics?
a

Alexei Zenin

07/27/2022, 1:59 PM
I think we would want to keep these, to have the full functionality of the UI. It would be nice if it was all in one collector
a

Ankit Nayan

07/27/2022, 2:02 PM
yeah..I understand but we have architected it considering scalability too.. the other otel-collectors can be horizontally scaled up to handle load but otel-collector metrics will contain prometheus scrape config which cannot be horizontally scaled. And keeping all cluster level metrics in just 1 otel-collector-metrics rather than all otel-collectors receiving the same config and hence same data and duplicate entries to clickhouse
a

Alexei Zenin

07/27/2022, 4:12 PM
So in my setup I plan on running 2 otel collectors per ECS host. 1 for accepting traces on 4317 and the other for doing the metrics scraping as you say (so only scrape the local collector). I need to reserve a bunch of host ports for this, namely 4317, 13133 and 8889. I think i can get away with the otel-collector-metrics not needing to expose any ports and do an internal HEALTHCHECK via docker instead of an ALB external ping.
Unfortunately your image extends scratch so I will need to add curl into the image along with possibly a shell to get that to work
a

Ankit Nayan

07/27/2022, 4:39 PM
aren't container ports abstracted from docker networks?
a

Alexei Zenin

07/27/2022, 5:33 PM
in my case i think i need to bind to a well-known host port otherwise I wouldn’t know which port to configure when using the
172.17.0.1
bridge network IP. This way my services on the host can configure their opentelemetry agents with that IP and the well known port of 4317 for example (without worrying about how to find out the dynamic port that the collector would be running on if it wasn’t hardcoded). This is only for the collector not collector-metrics (collector metrics would set the scrape config to
172.17.0.1:8889
)
ah actually maybe i can not publish the ports and use the name of the containers instead
ill post back here what I end up doing, thanks for all the help!
18 Views