I am unable to capture any traces. The UI never sh...
# support
c
I am unable to capture any traces. The UI never shows any activity. I have signoz installed with kubernetes (by helm chart). I've also setup an ingress (with tls cert) to direct to ports 3301 for the UI and 4317 for the collector. The UI works fine opening on https. I also have test opentelemetry clients for .NET Core and python. My problem is that I so far can not see any traces in the UI. As a test, if I wget the collector URL (https://sigznos.api.myserver.local), I can see an error in the collector log, so can confirm the message is getting to the backend (I obviously don't expect it to generate a span). Both client apps are instrumented and I can see the output in the console. For the .NET Core app, the only exporter when I see any activity in the log is ZipkinExporter. Including/excluding this line makes no difference: AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true); Here's my .NET code: static void Main(string[] args) { //AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true); using var tracer2Provider = Sdk.CreateTracerProviderBuilder() .SetResourceBuilder(ResourceBuilder.CreateDefault()) .AddSource("MyTest.Console") .AddAspNetCoreInstrumentation() .SetResourceBuilder(ResourceBuilder.CreateDefault() .AddTelemetrySdk()) .AddZipkinExporter(otlpOptions => { otlpOptions.Endpoint = new Uri("https://signoz.api.myserver.local"); otlpOptions.ExportProcessorType = ExportProcessorType.Simple; }) .AddConsoleExporter() .Build(); using (Activity activity = source.StartActivity("RunProgram")) { activity?.AddTag("CMA-app-Name", "MyTest.App.Name"); var program = new Program(); program.SigNozTest(); // Makes an http call to an API } Here's the output on the console: Activity.TraceId: 14d3c904ec9ee6583014177882f9f590 Activity.SpanId: 9603d4cf46d8c592 Activity.TraceFlags: Recorded Activity.ActivitySourceName: MyTest.Console Activity.DisplayName: RunProgram Activity.Kind: Internal Activity.StartTime: 2022-06-01T004043.5888893Z Activity.Duration: 000001.4237944 Activity.Tags: CMA-app-Name: MyTest.App.Name Resource associated with Activity: telemetry.sdk.name: opentelemetry telemetry.sdk.language: dotnet telemetry.sdk.version: 1.3.0.498 service.name: MYTEST application: MYTEST The .NET app (code and trace above) generates this error in the collector log: 2022-05-31T230351.352Z warn zapgrpc/zapgrpc.go:191 [core] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST / HTTP/1.1\\r\\nHost: s\"" {"grpc_log": true} The python client app shows no sign of connecting to the backend. It's the sample flask app from https://signoz.io/blog/opentelemetry-flask/ with the following env vars set: OTEL_EXPORTER_OTLP_ENDPOINT="https://signoz.api.myserver.local" OTEL_EXPORTER_OTLP_INSECURE=true OTEL_RESOURCE_ATTRIBUTES=service.name=my-flask-app I've also tried setting OTEL_EXPORTER_OTLP_CERTIFICATE to the value of the cert. The message I get on the flask console is: Transient error StatusCode.UNAVAILABLE encountered while exporting span batch, retrying in 2s. What am I missing?
👆 1
👀 1
s
Why is there no port number in your endpoint setting env/param?
c
It's mapped in the ingress file: rules: - host: signoz.api.myserver.local http: paths: - path: / pathType: ImplementationSpecific backend: service: name: my-release-signoz-otel-collector port: number: 4317
s
Transient error StatusCode.UNAVAILABLE encountered while exporting span batch, retrying in 2s.
This warning makes it clear. The collector service is not reachable to exporter from app. There is certainly an issue with ingress setup. @Prashant Shahi might help you.
c
Thanks for looking into this. As mentioned, I am able to hit the collector using wget and the .NET client - and can see a resulting error message in the collector's log from the attempt. I suspect the problem has to do with SSL but not sure why.
Ok .. here are some developments: 1. I added the local cert to the python venv and after that it gives the following (very similar error as the .NET client) 2022-06-01T225821.251Z warn zapgrpc/zapgrpc.go:191 [core] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST /v1/traces HTTP/1.1\"" {"grpc_log": true} 2. I set up port forwarding: kubectl -n platform port-forward svc/my-release-signoz-otel-collector 4317:4317 and set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 I get the same error as in 1 above. 3. With port forwarding on, I set OTEL_EXPORTER_OTLP_ENDPOINT=https://localhost:4317 (note https), when retrying it gives the following (same error but garbled/encrypted payload): 2022-06-01T233000.301Z warn zapgrpc/zapgrpc.go:191 [core] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x02\\x00\\x01\\x00\\x01\\xfc\\x03\\x03\\xfd\\x02\\xf7\\x9e(LTs\\x15\\xc53<\\x81\"" {"grpc_log": true} I'm running out of options. Hopefully this is enough info to point to where the problem is!