This message was deleted.

@Ankit Nayan any insights on this?

@Alexei Zenin that was what I did when building it. But then the writing of metrics will be directly proportional to the rate of incoming spans as spans create those metrics. This will be very write heavy for metrics and unnecessary. This could have also been solved by keeping metrics in memory for say 60s and then write back to clickhouse using clickhousemetricswrite but I chose the other trusted way of doing it prometheus style.

I see, so if I do not run it I just won’t get custom metrics? Or will parts of the SigNoz UI be missing certain information for spans?

nothing missing from spans. APM metrics will not come. Overview will show only top-left and bottom most, no db metrics, no external calls metrics

ah i see

do you use these metrics?

I think we would want to keep these, to have the full functionality of the UI. It would be nice if it was all in one collector

yeah..I understand but we have architected it considering scalability too.. the other otel-collectors can be horizontally scaled up to handle load but otel-collector metrics will contain prometheus scrape config which cannot be horizontally scaled. And keeping all cluster level metrics in just 1 otel-collector-metrics rather than all otel-collectors receiving the same config and hence same data and duplicate entries to clickhouse

So in my setup I plan on running 2 otel collectors per ECS host. 1 for accepting traces on 4317 and the other for doing the metrics scraping as you say (so only scrape the local collector). I need to reserve a bunch of host ports for this, namely 4317, 13133 and 8889. I think i can get away with the otel-collector-metrics not needing to expose any ports and do an internal HEALTHCHECK via docker instead of an ALB external ping.

aren't container ports abstracted from docker networks?

in my case i think i need to bind to a well-known host port otherwise I wouldn’t know which port to configure when using the

172.17.0.1

bridge network IP. This way my services on the host can configure their opentelemetry agents with that IP and the well known port of 4317 for example (without worrying about how to find out the dynamic port that the collector would be running on if it wasn’t hardcoded). This is only for the collector not collector-metrics (collector metrics would set the scrape config to

172.17.0.1:8889

)