Where an application runs on Kubernetes but does not have OTEL support for emitting logs (like anything written in Rust), it seems to me that the way to get the log data into SigNoz is via the k8s-infra collectors. If I choose to make Rust apps output in JSON and generally make all apps output in JSON, then I have a consistent logging method to stdout. However the filtering and extraction of attributes from JSON objects, like can be done here https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml#L94-L156 would only be successful if the same rule can be generally applied to all received logs. There is no application-specific interpretation of attributes in JSON methods and the timestamp extraction for parser-crio and parser-containerd demonstrates that because the regexes appear to be tried in order until something succeeds seems to work by luck rather than targeted at a specific application. The rules for renaming attributes would only work if the attribute names are semantically consistent across all applications running on a cluster. Even the routing choice here is super-vague by assuming (incorrectly) that any line beginning with a open-curly-brace is in some parse-able JSON format: https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml#L96-L104 It seems that the most reasonable solution is to avoid using k8s-infra and only use FluentBit, since one can then be explicit about the parser in the Pod configs and then pair accurate parsing with the expected Pod output: https://docs.fluentbit.io/manual/pipeline/filters/kubernetes#suggest-a-parser

The config is written for general use case to ingest k8s pod logs. The config doesn’t parse the log itself. If you want to parse all your json logs you should apply more operators ex: in this case I wanted to parse specific format of nginx logs https://github.com/SigNoz/nginx-logs-parsing/blob/c14a2178a050293a50549b79991ed3858e295c78/clickhouse-setup/otel-collector-config.yaml#L24 If you want to parse your json logs you can use a router to capture your json logs and then parse it using a pipeline https://github.com/SigNoz/logs-benchmark/blob/0b2451e6108d8fa5fdd5808c4e174bd52b9d55d3/signoz/signoz-client/otel-collector-config.yaml#L12

Ok, so I can use

json_parser

to parse the

message.body

then I will have to find a way of applying a conditional expression that allows for different mapping rules dependent on the source app, for example where

message.body.xid

for one application would map to

session_id

and

message.body.xid

for another application would map to

transaction_id

.

Right, you can change and create mappings according to your business logic using those pipelines. We also have a UI-based parser coming up but is delayed due to priorities. https://github.com/SigNoz/signoz/pull/2457 But till then you can create them manually in your collector config. You can read more about them here https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/types/pipeline.md. Try writing one using the example I shared and using the link above and let me know if you face any issues.