Hey folks, Weirdly for my standalone docker deployment, i am seeing errors in my application logs during trace export in production recently. This happens occasionally. Tried running the application locally by setting the

OTEL_EXPORTER_OTLP_ENDPOINT

to the prod signoz deployments and got some error. Below is my django logs

Traceback (most recent call last):
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/sdk/_logs/export/__init__.py", line 259, in _export_batch
    self._exporter.export(self._log_records[:idx])  # type: ignore
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py", line 142, in export
    resp = self._export(serialized_data)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py", line 113, in _export
    return <http://self._session.post|self._session.post>(
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 128, in instrumented_request
    return _instrumented_requests_call(
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 245, in _instrumented_requests_call
    raise exception.with_traceback(exception.__traceback__)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 209, in _instrumented_requests_call
    result = call_wrapped()  # *** PROCEED
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 126, in call_wrapped
    return wrapped_request(self, method, url, *args, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 148, in instrumented_send
    return _instrumented_requests_call(
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 159, in _instrumented_requests_call
    return call_wrapped()
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 146, in call_wrapped
    return wrapped_send(self, request, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

And i cant find any way of checking what exactly is happening. I am assuming the otel-collector is somehow not able to ingest the spans in clickhouse but i couldn’t find any way of viewing debug logs for the collector to understand the root cause. Is this probably a scale issue? I checked the troubleshooting guide and running the trouble shooting command gave following error.

> sudo docker run -it --rm signoz/troubleshoot checkEndpoint --endpoint=172.17.0.1:4318
2023-06-13T11:36:21.026Z	INFO	troubleshoot/main.go:28	STARTING!
2023-06-13T11:36:21.026Z	INFO	checkEndpoint/checkEndpoint.go:41	checking reachability of SigNoz endpoint
Error: not able to send data to SigNoz endpoint ...
rpc error: code = Unavailable desc = connection closed before server preface received
Usage:
  signoz checkEndpoint [flags]

Examples:
checkEndpoint -e localhost:4317

Flags:
  -e, --endpoint string   URL to SigNoz with port
  -h, --help              help for checkEndpoint

Been stuck on this and google searches aren’t returning anything helpful.

Hey @Dipen Bhatt 👋 Troubleshooting only supports gRPC endpoint that would be

172.17.0.1:4317

instead.

Generating telemetry data isn;t the problem, the problem is knowing whats happening inside collector. Is there any debug log option in collector docker-compose which can enable some logs telling where exactly otel-collector might be failing?

You can use logging exporter with

loglevel

set to either

warn

or

error

. Ref: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/loggingexporter/README.md