Starting in 45! Our next OTel Webinar on sending OTLP via Apache Kafka https://www.youtube.com/live/ncOu44NtAec?si=Az1xsTx0vXCeQpuQ

@Nočnica Mellifera Unfortunately I wasn't able to attend the webinar live. This is an amazing one! Love it, Thanks ton guys!!! If I were there I would have loved to ask about your thoughts on: 1- The consumers from Kafka to the backend platform for example SigNoz! What are the different options for consumers and the scalability of them? 2- As you said would be good to have queueing systems, for such a case what could be a good arch consider k8s env • OTEL agent as Dasmonset > Kafka > some consumer running as something > SigNoz • OTEL gateway as Deployment > Kafka > some consumer running as something > SigNoz • OTEL agent as Dasmonset > OTEL gateway as Deployment/Statefulset > Kafka > some consumer running as something > SigNoz • 3- fault tolerance in case of downtime in SigNoz

^^ These are great questions! I'm working on a post now that will help answer, at least some of these

Can’t wait for hour post! @Nočnica Mellifera Please tag me as soon as it is available!

soon! BTW the first article on collector architecture and Kafka is up here: https://signoz.io/blog/maximizing-scalability-apache-kafka-and-opentelemetry/

@Nočnica Mellifera Thanks for the article. My point was that in the webinar ( at the 18th min ) you were talking about partitioning by trace ID and sending to different kafka topic, then you could have multiple consumers to read from these topics. From your article I see you mentioned the consumers to be OTEL collector. Then how would the scalability of OTEL collector from one Kafka topic would be ? For example, would it have to only be one OTEL collector consuming from one Kafka topics? What if we want to scale that OTEL collector? I think this is a similar question you had on session 1 of the collector around pull based and push based systems? By default OTEL collectos is push based where the apps are pushing to it, but now I assume the challenge would be with the collector pulling from Kafka topic

so in the config for the Kafka receiver you can set it to read from different Kafka topics, so it should be possible to have multiple collectors reading that way. If your concern is how well this works in practice, or how well it works if you have multiple collectors subscribed to the same topic. I'd take that to the collector channel in the CNCF Slack, I haven't tried any load testing with this config.

yes, the concern is that the horizontal scalability of OTEL collectors reading from the same Kafka topic.