https://signoz.io
Join Slack
This message was deleted.
# support
s
This message was deleted.
a
Given this we will probably be ingesting hundreds of millions of spans per hour
cc @Ankit Nayan if you have any thoughts
its looking like 1 span ~1KB in clickhouse
Noticed i can change the OTEL sampler to ratio it to help reduce costs. Didnt know by default it records all traces
a
@Alexei Zenin you should use probabilisticsamplerprocessor after 
signozspanmetrics/prometheus
at https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/otel-collector-config.yaml#L135 for sampling to not affect APM metrics as we create APM metrics from traces
last we tested needed around 80 CPUs for 350K spans/s ingestion rate. And probably 8CPUs were good to handle 10K spans/s
and 130 CPUs for handling 500K spans/s
a
I see thanks for the pointers. I think we might want to set our sampling to a low rate early on before it hits the network to avoid paying hefty regional data transfer charges. From my understanding your approach would send all traces from the apps over the network to then be dropped after in the signoz collector gateway.
a
I think we might want to set our sampling to a low rate early on before it hits the network
you will be using the otel-collector for this right? Even before hitting the network, you can use 
signoz/signoz-otel-collector:0.55.3
which has 
signozspanmetrics/prometheus
processor
it will be like an agent otel-collector creating APM metrics and dropping data before sending to SigNoz. And all those APM metrics will also be forwarded to SigNoz.
a
i see, i was using the standard distro of OpenTelemetry in an agent mode where it forwarded to the SigNoz collectors. Wondering on why SigNoz can’t derive the various span metrics with just a sampling of the traces? I know sampling by definition makes it less accurate, is that the case here or are you saying the signoz span metrics would break without 100% sampling?
a
No No....it won't break..it will work based on the traces & spans it receives
a
Ah ok, phew 😅. So it will just be less accurate, thanks for the help and pointers!
So to confirm if I don’t send all traces to the signoz collector which collects metrics, the metrics such as Error rate + Operations per second will be inaccurate and off by a lot in the SigNoz UI? Seems I might want these to be 100% accurate so your suggestion makes more sense. In general i found the Datadog agent has different sampling rate thresholds which I haven’t found an equivalent in open telemetry. They even have different sample rates for errors which allows them to still capture a good amount of them via traces (do you know if its at all possible to replicate their way of being able to capture error traces separate from all traces in OTEL?). It seems they have a similar setup to your approach to capture all span metrics accurately.
a
Interesting... let me give it a read. Can you please create an issue for this? We are planning to work on sampling soon
🙌 1
a
Found a really good blog post showing someones real setup in PROD for sampling: https://opensearch.org/blog/technical-post/2021/12/distributed-tracing-pipeline-with-opentelemetry/
🙌 1
➕ 1
So I ended up running the signoz span metrics collector on the agents along with the probabilistic sampler set to 20%. Everything seems to be working well still. Is there any code that would propagate all errors still through to SigNoz out of curiosity? It seems every exception has a trace, im assuming this is by design though and that page is generated based off of available traces and has nothing to do with the 
signozspanmetrics/prometheus
processor which looks at all traces. Im assuming that processor only calculates the various PXX metrics with error rates, req rates etc
For anyone following my pipeline setup on the OTEL agent:
Copy code
pipelines:
                      traces:
                        receivers: [otlp]
                        processors: [memory_limiter, signozspanmetrics/prometheus, probabilistic_sampler, batch]
                        exporters: [otlphttp]
                      # exports metrics which are scraped in-process instead of running separate container
                      metrics/spanmetrics:
                        receivers: [otlp/spanmetrics]
                        exporters: [prometheus]
                      metrics:
                        receivers: [prometheus]
                        processors: [memory_limiter, batch]
                        exporters: [otlphttp]
cc @Andrew Uken
13 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by