This message was deleted.

Quarkus does it OOTB and it’s helpful. https://quarkus.io/guides/opentelemetry#smallrye-reactive-messaging-kafka Previously, we had also considered adding trace data to sqs messages. So the need is there. Currently the lifecycle of a message is captured in logs which does the job but isn’t as smooth. Some systems might require alerting on events as the SLO would require consuming events within a certain time period.

A challenge that I see is retaining all the relevant spans (i.e. tail-based sampling) and if all the relevant spans for an async transaction (producer producing at time t1, all related sync operations at or before t1, consumer consuming at time t1 + delta) can be retained in the collector itself. Usually if the delta is more than a few minutes / seconds - probably we won't even see the complete trace with all the spans.

@Romil Punetha otel tracing works for kafka too. We are able to pass context across kafka and identify producer and consumer services from the traces. @Srikanth Chekuri can you see anything otel does not enable which Quarkus does out-of-box?

Haven't faced it yet due to low scale. We work with MPL and they face this issue which is only caught through customer tickets or internal tickets incase of analytical event lag.

got it..and what is MPL? Mobile Premier League?

Yes

@Aditya KP yeah.. tail-sampling would be difficult in such scenarios. I was thinking more from a post-processing angle. We would try constructing a tree from a traceID using a job and if the tree is complete we shall note the time taken for each complete trace. Each tree now will have a flowID (a flow will be a bucketing strategy of traces where we assign a flow id to 1 structure of a tree). So, whenever we open a traceID, we shall check the avg flow completion duration and when alerting, we shall wait for that flow completion duration also.

@Ankit Nayan do you mind giving a concrete example for traceflow you mentioned?

@Srikanth Chekuri something like in below diagram https://danluu.com/tracing-analytics/

Is this generally applicable technique or specifically for async event driven systems?

Each tree now will have a flowID (a flow will be a bucketing strategy of traces where we assign a flow id to 1 structure of a tree). So, whenever we open a traceID, we shall check the avg flow completion duration and when alerting, we shall wait for that flow completion duration also.

Doing all this will require preserving all the spans for a specific flow (smaller for transactions / sync processes, longer for async processes) - which is effectively tail based sampling. So in the context of a specific flow - we can for example have baselines and say that specific parts of the flow are under-performant / have errors. This is extremely helpful for large systems where you have multiple oncalls - now they can simply look at this "deviation" and say that okay this is the cause of degradation.

So in the context of a specific flow - we can for example have baselines and say that specific parts of the flow are under-performant / have errors.

Yeah, was thinking in those lines.

I was curious to hear more about async workflows and pain points