Given this we will probably be ingesting hundreds of millions of spans per hour

cc @Ankit Nayan if you have any thoughts

its looking like 1 span ~1KB in clickhouse

Noticed i can change the OTEL sampler to ratio it to help reduce costs. Didnt know by default it records all traces

last we tested needed around 80 CPUs for 350K spans/s ingestion rate. And probably 8CPUs were good to handle 10K spans/s

and 130 CPUs for handling 500K spans/s

it will be like an agent otel-collector creating APM metrics and dropping data before sending to SigNoz. And all those APM metrics will also be forwarded to SigNoz.

So to confirm if I don’t send all traces to the signoz collector which collects metrics, the metrics such as Error rate + Operations per second will be inaccurate and off by a lot in the SigNoz UI? Seems I might want these to be 100% accurate so your suggestion makes more sense. In general i found the Datadog agent has different sampling rate thresholds which I haven’t found an equivalent in open telemetry. They even have different sample rates for errors which allows them to still capture a good amount of them via traces (do you know if its at all possible to replicate their way of being able to capture error traces separate from all traces in OTEL?). It seems they have a similar setup to your approach to capture all span metrics accurately.

So I ended up running the signoz span metrics collector on the agents along with the probabilistic sampler set to 20%. Everything seems to be working well still. Is there any code that would propagate all errors still through to SigNoz out of curiosity? It seems every exception has a trace, im assuming this is by design though and that page is generated based off of available traces and has nothing to do with the

signozspanmetrics/prometheus

processor which looks at all traces. Im assuming that processor only calculates the various PXX metrics with error rates, req rates etc

For anyone following my pipeline setup on the OTEL agent:

pipelines:
                      traces:
                        receivers: [otlp]
                        processors: [memory_limiter, signozspanmetrics/prometheus, probabilistic_sampler, batch]
                        exporters: [otlphttp]
                      # exports metrics which are scraped in-process instead of running separate container
                      metrics/spanmetrics:
                        receivers: [otlp/spanmetrics]
                        exporters: [prometheus]
                      metrics:
                        receivers: [prometheus]
                        processors: [memory_limiter, batch]
                        exporters: [otlphttp]

cc @Andrew Uken