https://signoz.io logo
#support
Title
# support
n

Nick Burrett

04/04/2023, 3:35 PM
Where an application runs on Kubernetes but does not have OTEL support for emitting logs (like anything written in Rust), it seems to me that the way to get the log data into SigNoz is via the k8s-infra collectors. If I choose to make Rust apps output in JSON and generally make all apps output in JSON, then I have a consistent logging method to stdout. However the filtering and extraction of attributes from JSON objects, like can be done here https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml#L94-L156 would only be successful if the same rule can be generally applied to all received logs. There is no application-specific interpretation of attributes in JSON methods and the timestamp extraction for parser-crio and parser-containerd demonstrates that because the regexes appear to be tried in order until something succeeds seems to work by luck rather than targeted at a specific application. The rules for renaming attributes would only work if the attribute names are semantically consistent across all applications running on a cluster. Even the routing choice here is super-vague by assuming (incorrectly) that any line beginning with a open-curly-brace is in some parse-able JSON format: https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml#L96-L104 It seems that the most reasonable solution is to avoid using k8s-infra and only use FluentBit, since one can then be explicit about the parser in the Pod configs and then pair accurate parsing with the expected Pod output: https://docs.fluentbit.io/manual/pipeline/filters/kubernetes#suggest-a-parser
n

nitya-signoz

04/04/2023, 4:40 PM
The config is written for general use case to ingest k8s pod logs. The config doesn’t parse the log itself. If you want to parse all your json logs you should apply more operators ex: in this case I wanted to parse specific format of nginx logs https://github.com/SigNoz/nginx-logs-parsing/blob/c14a2178a050293a50549b79991ed3858e295c78/clickhouse-setup/otel-collector-config.yaml#L24 If you want to parse your json logs you can use a router to capture your json logs and then parse it using a pipeline https://github.com/SigNoz/logs-benchmark/blob/0b2451e6108d8fa5fdd5808c4e174bd52b9d55d3/signoz/signoz-client/otel-collector-config.yaml#L12
Even the routing choice here is super-vague by assuming (incorrectly) that any line beginning with a open-curly-brace is in some parse-able JSON format: https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml#L96-L104
This is incorrect. When your logs are stored by k8s into a file it follows a specific format for storing logs. There won’t be a case where your k8s pod logs are captured but it is not stored in docker or some other format.
n

Nick Burrett

04/04/2023, 6:38 PM
Ok, so I can use
json_parser
to parse the
message.body
then I will have to find a way of applying a conditional expression that allows for different mapping rules dependent on the source app, for example where
message.body.xid
for one application would map to
session_id
and
message.body.xid
for another application would map to
transaction_id
.
n

nitya-signoz

04/05/2023, 2:35 PM
Right, you can change and create mappings according to your business logic using those pipelines. We also have a UI-based parser coming up but is delayed due to priorities. https://github.com/SigNoz/signoz/pull/2457 But till then you can create them manually in your collector config. You can read more about them here https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/types/pipeline.md. Try writing one using the example I shared and using the link above and let me know if you face any issues.
2 Views