Is there anywhere that documents the cause of the "This trace has missing spans" error? We are sending trace data from all of our APIs at this point and so we are lost on where to start with this issue since Googling for this error only netted an open github issue with another person asking for similar information (https://github.com/SigNoz/signoz/issues/1526). We also checked all the signoz containers and none are throwing any errors either.
09/26/2022, 6:02 PM
hey @Eric Rodriguez
"This trace has missing spans" is caused if all the spans in a trace has not reached SigNoz otel collector yet. Sometimes, this shows for new traces which have long spans
are you looking at traces which are generated in the last 1-2 minutes?
Other reason could be that some of your spans are getting dropped - may be because of high number of spans sent which your otel collector is not able to handle ( because of low resources)
Can you share a screenshot of what you see in the trace detail page - might help understand whats happening
09/26/2022, 6:14 PM
We checked the server resource utilization for the VM running the signoz services and we are at 15% CPU and 25% memory utilization
09/26/2022, 6:25 PM
I see. What is the request per second for your applications? Services page of SigNoz should help you find that number
also adding @Vishal Sharma here - he may have more insights on this
09/26/2022, 6:34 PM
.06 to .10 RPS depending on the service
09/26/2022, 6:43 PM
I see, so high scale is not a problem. I will let @Vishal Sharma to get back and see if he has any more insights
09/27/2022, 6:13 AM
@Eric Rodriguez Since traffic is low and even resource utilization is low it’s possible that instrumentation has some issues.
Which language instrumentation are you using and how have you instrumented? I think it will be faster to debug if we can get on a call sometime.