https://signoz.io
Join Slack
Menu
Home
Archive
Hi folks, any advise on how to debug Logs Pipelines authored in the UI? My team has not been able to...
# support
n
Hi folks, any advise on how to debug Logs Pipelines authored in the UI? My team has not been able to implement pipelines that consistently parse our JSON-formatted logs, yet (1) I see some examples of logs that were correctly processed and (2) I don't see any errors when simulating the pipeline in the UI
Are logs expected to show up in their unprocessed form when first collected, and then get overwritten by their processed form at some later point? If so, is there an SLA?
n
@Raj Kamal ^
r
Hi Nicolas 🙂 The logs get processed before being stored in the DB. there is no delayed processing Can you show us the pipeline you have created (the filter you used and the operators) and also send us some logs that were processed and not processed
n
@Raj Kamal here's a sequence of screenshots showing the filter + sequence of processors:
Here's an example log that I would expect to be processed by the pipeline but is in fact just showing up in its unprocessed form:
Copy code
{
  "body": "{\"body\": \"Failed to check if collection '78bd1dd942e906e3317702a226d99e3a' exists: [Errno -2] Name or service not known\", \"severity_number\": \"<SeverityNumber.ERROR: 17>\", \"severity_text\": \"ERROR\", \"attributes\": {\"otelSpanID\": \"fece67bda8c3b42c\", \"otelTraceID\": \"ba8e358e31367dc3e43f76ce6e349db5\", \"otelTraceSampled\": true, \"otelServiceName\": \"retrieval\", \"logger_name\": \"retrieval.change_localization.knn.vectordb\", \"code.filepath\": \"/server/retrieval/change_localization/knn/vectordb.py\", \"code.function\": \"_physical_collection_exists\", \"code.lineno\": 232, \"exception.type\": \"ResponseHandlingException\", \"exception.stacktrace\": \"Traceback (most recent call last):\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpx/_transports/default.py\\\", line 66, in map_httpcore_exceptions\\n    yield\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpx/_transports/default.py\\\", line 228, in handle_request\\n    resp = self._pool.handle_request(req)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py\\\", line 216, in handle_request\\n    raise exc from None\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py\\\", line 196, in handle_request\\n    response = connection.handle_request(\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpcore/_sync/connection.py\\\", line 99, in handle_request\\n    raise exc\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpcore/_sync/connection.py\\\", line 76, in handle_request\\n    stream = self._connect(request)\\n             ^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpcore/_sync/connection.py\\\", line 122, in _connect\\n    stream = self._network_backend.connect_tcp(**kwargs)\\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpcore/_backends/sync.py\\\", line 205, in connect_tcp\\n    with map_exceptions(exc_map):\\n  File \\\"/usr/local/lib/python3.11/contextlib.py\\\", line 158, in __exit__\\n    self.gen.throw(typ, value, traceback)\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpcore/_exceptions.py\\\", line 14, in map_exceptions\\n    raise to_exc(exc) from exc\\nhttpcore.ConnectError: [Errno -2] Name or service not known\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \\\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py\\\", line 103, in send_inner\\n    response = self._client.send(request)\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpx/_client.py\\\", line 901, in send\\n    response = self._send_handling_auth(\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpx/_client.py\\\", line 929, in _send_handling_auth\\n    response = self._send_handling_redirects(\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpx/_client.py\\\", line 966, in _send_handling_redirects\\n    response = self._send_single_request(request)\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpx/_client.py\\\", line 1002, in _send_single_request\\n    response = transport.handle_request(request)\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/opentelemetry/instrumentation/httpx/__init__.py\\\", line 370, in handle_request\\n    response = self._transport.handle_request(*args, **kwargs)\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpx/_transports/default.py\\\", line 227, in handle_request\\n    with map_httpcore_exceptions():\\n  File \\\"/usr/local/lib/python3.11/contextlib.py\\\", line 158, in __exit__\\n    self.gen.throw(typ, value, traceback)\\n  File \\\"/opt/venv/lib/python3.11/site-packages/httpx/_transports/default.py\\\", line 83, in map_httpcore_exceptions\\n    raise mapped_exc(message) from exc\\nhttpx.ConnectError: [Errno -2] Name or service not known\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n  File \\\"/server/retrieval/change_localization/knn/vectordb.py\\\", line 229, in _physical_collection_exists\\n    return self._qdrant_client.collection_exists(collection_name)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/qdrant_client/qdrant_client.py\\\", line 1591, in collection_exists\\n    return self._client.collection_exists(collection_name=collection_name, **kwargs)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/qdrant_client/qdrant_remote.py\\\", line 2063, in collection_exists\\n    result: Optional[models.CollectionExistence] = self.http.collections_api.collection_exists(\\n                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api/collections_api.py\\\", line 1157, in collection_exists\\n    return self._build_for_collection_exists(\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api/collections_api.py\\\", line 87, in _build_for_collection_exists\\n    return self.api_client.request(\\n           ^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py\\\", line 76, in request\\n    return self.send(request, type_)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py\\\", line 93, in send\\n    response = self.middleware(request, self.send_inner)\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py\\\", line 202, in __call__\\n    return call_next(request)\\n           ^^^^^^^^^^^^^^^^^^\\n  File \\\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py\\\", line 105, in send_inner\\n    raise ResponseHandlingException(e)\\nqdrant_client.http.exceptions.ResponseHandlingException: [Errno -2] Name or service not known\\n\"}, \"dropped_attributes\": 0, \"timestamp\": \"2024-05-15T23:26:34.424250Z\", \"observed_timestamp\": \"2024-05-15T23:26:34.424287Z\", \"trace_id\": \"0xba8e358e31367dc3e43f76ce6e349db5\", \"span_id\": \"0xfece67bda8c3b42c\", \"trace_flags\": 1, \"resource\": \"{'telemetry.sdk.language': 'python', 'telemetry.sdk.name': 'opentelemetry', 'telemetry.sdk.version': '1.24.0', 'service.name': 'retrieval'}\"}",
  "id": "2gTN3e4BOAj0rqyoof0ZOLWUMdc",
  "timestamp": "2024-05-15T23:26:36.257053843Z",
  "attributes": {
    "log.file.path": "/var/log/pods/eval_nico-0515-ephemeral-0c5284189a1cc5026afbd3cf7a784e89-deplo8vng7_34c471d6-3dba-4203-a96a-ebfb2e4bf56a/retrieval-pod/0.log",
    "log.iostream": "stdout",
    "logtag": "F",
    "time": "2024-05-15T23:26:36.257053843+00:00"
  },
  "resources": {
    "k8s.cluster.name": "eval",
    "k8s.container.name": "retrieval-pod",
    "k8s.container.restart_count": "0",
    "k8s.namespace.name": "eval",
    "k8s.node.name": "10.0.126.222",
    "k8s.pod.name": "nico-0515-ephemeral-0c5284189a1cc5026afbd3cf7a784e89-deplo8vng7",
    "k8s.pod.start_time": "2024-05-15 22:33:13 +0000 UTC",
    "k8s.pod.uid": "34c471d6-3dba-4203-a96a-ebfb2e4bf56a"
  },
  "severity_text": "",
  "severity_number": 0,
  "span_id": "",
  "trace_flags": 0,
  "trace_id": ""
}
@Raj Kamal any pointers on this? At the moment the Pipelines feature isn't working for us
@Raj Kamal @nitya-signoz any input on this issue? This is becoming a showstopper for our continued usage of SigNoz
Gentle ping here @Raj Kamal @nitya-signoz
n
What you can do is try debugging by enabling just one processor. Disable all the processors and enable one, run the simulator. If things look good enable the second one and try simulating.
n
In the simulator, all samples look good even when I have all processors enabled. Is this the best debug tool available?
r
Hi Nicolas 🙂 Thanks for sending over the logs. I tried to reproduce your issue with the log body below (extracted from the log you provided above), but everything worked fine - both in the preview and the actual processing
Copy code
{"body": "Failed to check if collection '78bd1dd942e906e3317702a226d99e3a' exists: [Errno -2] Name or service not known", "severity_number": "<SeverityNumber.ERROR: 17>", "severity_text": "ERROR", "attributes": {"otelSpanID": "fece67bda8c3b42c", "otelTraceID": "ba8e358e31367dc3e43f76ce6e349db5", "otelTraceSampled": true, "otelServiceName": "retrieval", "logger_name": "retrieval.change_localization.knn.vectordb", "code.filepath": "/server/retrieval/change_localization/knn/vectordb.py", "code.function": "_physical_collection_exists", "code.lineno": 232, "exception.type": "ResponseHandlingException", "exception.stacktrace": "Traceback (most recent call last):\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpx/_transports/default.py\\", line 66, in map_httpcore_exceptions\\n    yield\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpx/_transports/default.py\\", line 228, in handle_request\\n    resp = self._pool.handle_request(req)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py\\", line 216, in handle_request\\n    raise exc from None\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py\\", line 196, in handle_request\\n    response = connection.handle_request(\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpcore/_sync/connection.py\\", line 99, in handle_request\\n    raise exc\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpcore/_sync/connection.py\\", line 76, in handle_request\\n    stream = self._connect(request)\\n             ^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpcore/_sync/connection.py\\", line 122, in _connect\\n    stream = self._network_backend.connect_tcp(**kwargs)\\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpcore/_backends/sync.py\\", line 205, in connect_tcp\\n    with map_exceptions(exc_map):\\n  File \\"/usr/local/lib/python3.11/contextlib.py\\", line 158, in __exit__\\n    self.gen.throw(typ, value, traceback)\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpcore/_exceptions.py\\", line 14, in map_exceptions\\n    raise to_exc(exc) from exc\\nhttpcore.ConnectError: [Errno -2] Name or service not known\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py\\", line 103, in send_inner\\n    response = self._client.send(request)\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpx/_client.py\\", line 901, in send\\n    response = self._send_handling_auth(\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpx/_client.py\\", line 929, in _send_handling_auth\\n    response = self._send_handling_redirects(\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpx/_client.py\\", line 966, in _send_handling_redirects\\n    response = self._send_single_request(request)\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpx/_client.py\\", line 1002, in _send_single_request\\n    response = transport.handle_request(request)\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/opentelemetry/instrumentation/httpx/__init__.py\\", line 370, in handle_request\\n    response = self._transport.handle_request(*args, **kwargs)\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpx/_transports/default.py\\", line 227, in handle_request\\n    with map_httpcore_exceptions():\\n  File \\"/usr/local/lib/python3.11/contextlib.py\\", line 158, in __exit__\\n    self.gen.throw(typ, value, traceback)\\n  File \\"/opt/venv/lib/python3.11/site-packages/httpx/_transports/default.py\\", line 83, in map_httpcore_exceptions\\n    raise mapped_exc(message) from exc\\nhttpx.ConnectError: [Errno -2] Name or service not known\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n  File \\"/server/retrieval/change_localization/knn/vectordb.py\\", line 229, in _physical_collection_exists\\n    return self._qdrant_client.collection_exists(collection_name)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/qdrant_client/qdrant_client.py\\", line 1591, in collection_exists\\n    return self._client.collection_exists(collection_name=collection_name, **kwargs)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/qdrant_client/qdrant_remote.py\\", line 2063, in collection_exists\\n    result: Optional[models.CollectionExistence] = self.http.collections_api.collection_exists(\\n                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api/collections_api.py\\", line 1157, in collection_exists\\n    return self._build_for_collection_exists(\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api/collections_api.py\\", line 87, in _build_for_collection_exists\\n    return self.api_client.request(\\n           ^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py\\", line 76, in request\\n    return self.send(request, type_)\\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py\\", line 93, in send\\n    response = self.middleware(request, self.send_inner)\\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py\\", line 202, in __call__\\n    return call_next(request)\\n           ^^^^^^^^^^^^^^^^^^\\n  File \\"/opt/venv/lib/python3.11/site-packages/qdrant_client/http/api_client.py\\", line 105, in send_inner\\n    raise ResponseHandlingException(e)\\nqdrant_client.http.exceptions.ResponseHandlingException: [Errno -2] Name or service not known\\n"}, "dropped_attributes": 0, "timestamp": "2024-05-15T23:26:34.424250Z", "observed_timestamp": "2024-05-15T23:26:34.424287Z", "trace_id": "0xba8e358e31367dc3e43f76ce6e349db5", "span_id": "0xfece67bda8c3b42c", "trace_flags": 1, "resource": "{'telemetry.sdk.language': 'python', 'telemetry.sdk.name': 'opentelemetry', 'telemetry.sdk.version': '1.24.0', 'service.name': 'retrieval'}"}
I am guessing you have already done other sanity checks like making sure the pipeline is enabled The next step would be to check the collector logs to see if there are errors there Though these are just best guesses and it is hard to figure out the cause without having access to your deployment Without having access to your deployment, another thing we could try is to get the exact pipeline configuration you are using to double check things You should be able to get a dump of your pipeline config from your network inspector in the response to a POST request to 
api/v1/logs/pipelines/latest
when you load the pipelines page. I have tried to illustrate that below This will give us the entire picture of all your pipelines and maybe that can help us find what is leading to your problem
12 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by