Hi all, I am new to Signoz. Am facing an issue with sending traces to it from python code. Signoz

v0.10.2

was deployed on a GKE Autopilot cluster. I have added an Ingress each for the frontend (443 -> 3313) and for the collector (80 -> 4318) to access the frontend and the collector over the internet. To see if I can send traces, I used this code

<https://gist.github.com/dfrankow/f91aefd683ece8e696c26e183d696c29>

and started it like so and did a POST and GET to it as shared in that gist -

❯ python3 -m venv simple_server
❯ source simple_server/bin/activate
❯ python -V
Python 3.9.13
❯ pip install opentelemetry-distro
❯ pip install opentelemetry-exporter-otlp
❯ wget <https://gist.githubusercontent.com/dfrankow/f91aefd683ece8e696c26e183d696c29/raw/016714b21cb2172b43611aabe03e53249027bd84/simple_server.py>
❯ OTEL_RESOURCE_ATTRIBUTES=service.name=test_apm OTEL_EXPORTER_OTLP_ENDPOINT="http://<sub.domain.here>" opentelemetry-instrument --traces_exporter otlp_proto_http python3 simple_server.py 7000 127.0.0.1

The traces do not get sent and the logs seem to suggest that I necessarily need to open the GRPC port even though I am specifying

otlp_proto_http

above. Please suggest if I am missing something.

What's the

OTEL_EXPORTER_OTLP_ENDPOINT="<http://example.com:4317>"

? Did you meant localhost instead of example.com?

No, it is the domain where the endpoint is hosted. I just redacted it before posting here. I also tried it with the public IP address of the VM.

Is that address reachable for the app?

Yes, TCP ports 4317 and 4318 are open to all IPs.

Ok port are open but is the host reachable for the app? Could you try this troubleshooting guide https://signoz.io/docs/install/troubleshooting/?

The

troubleshoot

tool seems to have access.

2022-08-31T17:45:55.293+0530	INFO	workspace/main.go:28	STARTING!
2022-08-31T17:45:55.293+0530	INFO	checkEndpoint/checkEndpoint.go:41	checking reachability of SigNoz endpoint
2022-08-31T17:45:55.405+0530	INFO	workspace/main.go:46	Successfully sent sample data to signoz ...

I noticed that the flask traces are being sent now, though these exceptions also show sometimes -

❯ OTEL_RESOURCE_ATTRIBUTES=service.name=pythonApp OTEL_EXPORTER_OTLP_ENDPOINT="<http://example.com:4317>" opentelemetry-instrument python3 app.py

127.0.0.1 - - [31/Aug/2022 13:13:02] "POST /action HTTP/1.1" 500 -
127.0.0.1 - - [31/Aug/2022 13:13:04] "GET /completed HTTP/1.1" 200 -
127.0.0.1 - - [31/Aug/2022 13:13:04] "GET /static/assets/style.css HTTP/1.1" 304 -
127.0.0.1 - - [31/Aug/2022 13:13:04] "GET /static/assets/emoji.css HTTP/1.1" 304 -
127.0.0.1 - - [31/Aug/2022 13:13:04] "GET /static/assets/twemoji.min.js HTTP/1.1" 304 -
127.0.0.1 - - [31/Aug/2022 13:13:04] "GET /static/assets/emoji.js HTTP/1.1" 304 -
127.0.0.1 - - [31/Aug/2022 13:13:06] "GET /list HTTP/1.1" 500 -
127.0.0.1 - - [31/Aug/2022 13:13:06] "GET /static/assets/style.css HTTP/1.1" 304 -
127.0.0.1 - - [31/Aug/2022 13:13:06] "GET /static/assets/emoji.css HTTP/1.1" 304 -
127.0.0.1 - - [31/Aug/2022 13:13:06] "GET /static/assets/twemoji.min.js HTTP/1.1" 304 -
127.0.0.1 - - [31/Aug/2022 13:13:06] "GET /static/assets/emoji.js HTTP/1.1" 304 -
Transient error StatusCode.UNAVAILABLE encountered while exporting metrics, retrying in Nones.
Exception while exporting metrics an integer is required (got type NoneType)
Traceback (most recent call last):
  File "/Users/username/Downloads/test_apm/simple_server/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 305, in _export
    self._client.Export(
  File "/Users/username/Downloads/test_apm/simple_server/lib/python3.9/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/username/Downloads/test_apm/simple_server/lib/python3.9/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Connection reset by peer"
        debug_error_string = "{"created":"@1661938573.017058000","description":"Error received from peer ipv4:<ip-address-here>:4317","file":"src/core/lib/surface/call.cc","file_line":967,"grpc_message":"Connection reset by peer","grpc_status":14}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/username/Downloads/test_apm/simple_server/lib/python3.9/site-packages/opentelemetry/sdk/metrics/_internal/export/__init__.py", line 482, in _receive_metrics
    self._exporter.export(metrics_data, timeout_millis=timeout_millis)
  File "/Users/username/Downloads/test_apm/simple_server/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py", line 248, in export
    return self._export(metrics_data)
  File "/Users/username/Downloads/test_apm/simple_server/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/grpc/exporter.py", line 345, in _export
    sleep(delay)
TypeError: an integer is required (got type NoneType)

Copy code

python3 simple_server.py 7000 127.0.0.1

What does this do? Does it have some reload mechanism? OTEL It has issues reloader so make sure to disable if that's enabled.

https://gist.github.com/dfrankow/f91aefd683ece8e696c26e183d696c29 here is the code

There is nothing instrumented so no data is sent. What were you expecting? The OTEL auto instrumentation only supports some list of popular frameworks https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation.

Ah, got it. I should have looked at that link before. Thanks @Srikanth Chekuri!