Hello i m using fluent bit to send logs to signoz The log is

Question

Hello i m using fluent bit to send logs to signoz The log is in the json format and this json contains an attribute named other info that contains an escaped json string I need to convert this other info attribute to appear structured in the SigNoz log viewer Where else should I configure besides fluentbit log example ``` timestamp 2023 09 21T15 38 31 950Z log level INFO message OK GET mnt index do ecs version 1 2 0 service name the app event dataset the app process thread name http nio 8080 exec 4 log logger the app logger app module app module duration 1442235497 http request method GET http url path mnt index do user id 123456 span id b6f1a671a3b298fe trace flags 01 trace id 38a8eb9837cce5bb0e7f9276018b1985 other info a 123 b 456 c 789 d 987 e 654 f 321 ``` fluent bit config ``` INPUT Name tail Tag log4j2 ecs Path path to logs app json log Parser log4j2 ecs json parser OUTPUT Name forward Match Host localhost Port 24224``` parser config ``` PARSER Name log4j2 ecs json parser Format json Time Key timestamp Time Format Y m dT H M S LZ Decode Field As escaped other info do next Decode Field As json other info```

Prashant Shahi · Answer

cc < nitya signoz>

nitya-signoz · Answer

Please use the json parser operator <https github com open telemetry opentelemetry collector contrib blob main pkg stanza docs operators json parser md>

Ulisses Silva · Answer

The config bellow results in error ```processors logstransform operators type trace parser trace id parse from attributes trace id span id parse from attributes span id type json parser parse from other info type remove field attributes trace id type remove field attributes span id``` Error ```signoz otel collector | Error failed to get config cannot unmarshal the configuration 1 error s decoding signoz otel collector | signoz otel collector | error decoding processors error reading configuration for logstransform 1 error s decoding signoz otel collector | signoz otel collector | error decoding operators 1 unmarshal to json parser 1 error s decoding signoz otel collector | signoz otel collector | error decoding parse from unrecognized prefix```

Ulisses Silva · Answer

I changed the app logger to log the field other info as a json instead of a formatted json string and removed the json parser The field is show in the logs explorer as a json formated string other info Type Tag Data type String a 123 b 456 c 789 d 987 e 654 f 321 Added the json parser againg to the logstransformer ``` type json parser parse from attributes other info``` Results in ```signoz otel collector | 2023 09 22T12 32 36 175Z error helper transformer go 99 Failed to process entry kind processor name logstransform pipeline logs operator id json parser operator type json parser error type map string interface cannot be parsed as JSON action send entry ```

Ulisses Silva · Answer

full error from signoz otel collector container ```signoz otel collector | 2023 09 22T12 43 01 273Z error helper transformer go 99 Failed to process entry kind processor name logstransform pipeline logs operator id json parser operator type json parser error type map string interface cannot be parsed as JSON action send entry observed timestamp 2023 09 22T12 43 01 173937618Z timestamp 2023 09 22T12 43 01 087Z body ERROR POST some url test attributes app module module a duration 10222951 ecs version 1 2 0 error message Some error message error type some java ExceptionClass event dataset the app fluent tag other info a 123 b 456 c 789 d 987 e 654 f 321 http request method POST http url path some url test log level INFO log logger json logger process thread name http nio 8080 exec 3 service name the app status code ERROR trace flags 01 user agent Mozilla 5 0 X11 Linux x86 64 AppleWebKit 537 36 KHTML like Gecko Chrome 116 0 0 0 Safari 537 36 user id kj4l5j23lç4j62jlkj5lk user ip address 0 0 0 0 0 0 0 1 uuid a66e242b 95a3 484b b9c8 8d730289baf6 severity 0 scope name signoz otel collector | signoz otel collector | home runner go pkg mod github com open telemetry opentelemetry collector contrib pkg stanza v0 79 0 operator helper transformer go 99 signoz otel collector | signoz otel collector | home runner go pkg mod github com open telemetry opentelemetry collector contrib pkg stanza v0 79 0 operator helper parser go 140 signoz otel collector | signoz otel collector | home runner go pkg mod github com open telemetry opentelemetry collector contrib pkg stanza v0 79 0 operator helper parser go 112 signoz otel collector | signoz otel collector | home runner go pkg mod github com open telemetry opentelemetry collector contrib pkg stanza v0 79 0 operator helper parser go 98 signoz otel collector | signoz otel collector | home runner go pkg mod github com open telemetry opentelemetry collector contrib pkg stanza v0 79 0 operator parser json json go 62 signoz otel collector | signoz otel collector | home runner go pkg mod github com open telemetry opentelemetry collector contrib processor logstransformprocessor v0 79 0 processor go 143 signoz otel collector | 2023 09 22T12 43 01 273Z error logstransformprocessor v0 79 0 processor go 144 processor encountered an issue with the pipeline kind processor name logstransform pipeline logs error type map string interface cannot be parsed as JSON signoz otel collector | signoz otel collector | home runner go pkg mod github com open telemetry opentelemetry collector contrib processor logstransformprocessor v0 79 0 processor go 144```

nitya-signoz · Answer

You will have to pass the entire body to json parser and the move it here is an example <https github com SigNoz logs benchmark blob 0b2451e6108d8fa5fdd5808c4e174bd52b9d55d3 signoz signoz client otel collector config yaml L12>