Hey folks, Weirdly for my standalone docker depl...
# general
Hey folks, Weirdly for my standalone docker deployment, i am seeing errors in my application logs during trace export in production recently. This happens occasionally. Tried running the application locally by setting the
to the prod signoz deployments and got some error. Below is my django logs
Copy code
Traceback (most recent call last):
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/sdk/_logs/export/__init__.py", line 259, in _export_batch
    self._exporter.export(self._log_records[:idx])  # type: ignore
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py", line 142, in export
    resp = self._export(serialized_data)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/exporter/otlp/proto/http/_log_exporter/__init__.py", line 113, in _export
    return <http://self._session.post|self._session.post>(
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/sessions.py", line 578, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 128, in instrumented_request
    return _instrumented_requests_call(
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 245, in _instrumented_requests_call
    raise exception.with_traceback(exception.__traceback__)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 209, in _instrumented_requests_call
    result = call_wrapped()  # *** PROCEED
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 126, in call_wrapped
    return wrapped_request(self, method, url, *args, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 148, in instrumented_send
    return _instrumented_requests_call(
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 159, in _instrumented_requests_call
    return call_wrapped()
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 146, in call_wrapped
    return wrapped_send(self, request, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Users/deepsea/Documents/Dukaan/py-order/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
And i cant find any way of checking what exactly is happening. I am assuming the otel-collector is somehow not able to ingest the spans in clickhouse but i couldn’t find any way of viewing debug logs for the collector to understand the root cause. Is this probably a scale issue? I checked the troubleshooting guide and running the trouble shooting command gave following error.
Copy code
> sudo docker run -it --rm signoz/troubleshoot checkEndpoint --endpoint=
2023-06-13T11:36:21.026Z	INFO	troubleshoot/main.go:28	STARTING!
2023-06-13T11:36:21.026Z	INFO	checkEndpoint/checkEndpoint.go:41	checking reachability of SigNoz endpoint
Error: not able to send data to SigNoz endpoint ...
rpc error: code = Unavailable desc = connection closed before server preface received
  signoz checkEndpoint [flags]

checkEndpoint -e localhost:4317

  -e, --endpoint string   URL to SigNoz with port
  -h, --help              help for checkEndpoint
Been stuck on this and google searches aren’t returning anything helpful.
We have a avg 14 Million spans generated per hour
Hey @Dipen Bhatt 👋 Troubleshooting only supports gRPC endpoint that would be
Copy code
Connection aborted.', RemoteDisconnected('Remote end closed connection without response'
it is possible that either the endpoint is not right or the signoz otel-collector is not healthy.
Generating telemetry data isn;t the problem, the problem is knowing whats happening inside collector. Is there any debug log option in collector docker-compose which can enable some logs telling where exactly otel-collector might be failing?
You can use logging exporter with
set to either
. Ref: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/loggingexporter/README.md
Update otel-collector config to include the
exporter. Remember to include it in all desired pipelines as well.