Is there anywhere that documents the cause of the "This trace has missing spans" error? We are sending trace data from all of our APIs at this point and so we are lost on where to start with this issue since Googling for this error only netted an open github issue with another person asking for similar information (https://github.com/SigNoz/signoz/issues/1526). We also checked all the signoz containers and none are throwing any errors either.

hey @Eric Rodriguez "This trace has missing spans" is caused if all the spans in a trace has not reached SigNoz otel collector yet. Sometimes, this shows for new traces which have long spans are you looking at traces which are generated in the last 1-2 minutes? Other reason could be that some of your spans are getting dropped - may be because of high number of spans sent which your otel collector is not able to handle ( because of low resources) Can you share a screenshot of what you see in the trace detail page - might help understand whats happening

We checked the server resource utilization for the VM running the signoz services and we are at 15% CPU and 25% memory utilization

I see. What is the request per second for your applications? Services page of SigNoz should help you find that number

.06 to .10 RPS depending on the service

I see, so high scale is not a problem. I will let @Vishal Sharma to get back and see if he has any more insights

@Eric Rodriguez Since traffic is low and even resource utilization is low it’s possible that instrumentation has some issues. Which language instrumentation are you using and how have you instrumented? I think it will be faster to debug if we can get on a call sometime.

I had the same issue in aspnetcore application with manual instrumentation when all my spans related to incoming HTTP requests appeared in signoz with warning "This trace has missing spans" After digging in, I found that the reason was in Microsoft.AspNetCore.Hosting.HostingApplicationDiagnostics.BeginRequest: https://github.com/dotnet/aspnetcore/blob/481e472ea22a53fe96f11ba001c2dd4fbdc59ed5/src/Hosting/Hosting/src/Internal/HostingApplicationDiagnostics.cs#L88, which initialized object

Activity.Current

with

Microsoft.AspNetCore.Hosting.HttpRequestIn

span assigned as parent to all subsequent spans. So siznoz considered this parent span as missing because in my code only "child" spans were submitted to telemetry endpoint (I didn't suspect that they even had any parent). To get rid of this parent span, I had to disable it on Serilog level in appsettings.json:

{ "MinimumLevel": { "Override": { "Microsoft.AspNetCore.Hosting": "6" }}}

As a result, signoz started showing my spans as expected without annoying warning.