Ulisses Silva
09/21/2023, 6:50 PM{
"@timestamp": "2023-09-21T15:38:31.950Z",
"log.level": "INFO",
"message": "OK GET /mnt/index.do",
"ecs.version": "1.2.0",
"service.name": "the_app",
"event.dataset": "the_app",
"process.thread.name": "http-nio-8080-exec-4",
"log.logger": "the_app.logger",
"app_module": "app_module",
"duration": "1442235497",
"http_request_method": "GET",
"http_url_path": "/mnt/index.do",
"user_id": "123456",
"span_id": "b6f1a671a3b298fe",
"trace_flags": "01",
"trace_id": "38a8eb9837cce5bb0e7f9276018b1985",
"other_info": "{\"a\":\"123\",\"b\":\"456\",\"c\":\"789\",\"d\":\"987\",\"e\":\"654\",\"f\":\"321\"}"
}
fluent-bit config:
[INPUT]
Name tail
Tag log4j2.ecs.*
Path /path/to/logs/app-json.log
Parser log4j2.ecs.json.parser
[OUTPUT]
Name forward
Match *
Host localhost
Port 24224
parser config
[PARSER]
Name log4j2.ecs.json.parser
Format json
Time_Key @timestamp
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
Decode_Field_As escaped other_info do_next
Decode_Field_As json other_info
Prashant Shahi
09/21/2023, 7:06 PMnitya-signoz
09/22/2023, 4:42 AMUlisses Silva
09/22/2023, 12:25 PMprocessors:
logstransform:
operators:
- type: trace_parser
trace_id:
parse_from: attributes.trace_id
span_id:
parse_from: attributes.span_id
- type: json_parser
parse_from: other_info
- type: remove
field: attributes.trace_id
- type: remove
field: attributes.span_id
Error:
signoz-otel-collector | Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
signoz-otel-collector |
signoz-otel-collector | * error decoding 'processors': error reading configuration for "logstransform": 1 error(s) decoding:
signoz-otel-collector |
signoz-otel-collector | * error decoding 'operators[1]': unmarshal to json_parser: 1 error(s) decoding:
signoz-otel-collector |
signoz-otel-collector | * error decoding 'parse_from': unrecognized prefix
- type: json_parser
parse_from: attributes.other_info
Results in:
signoz-otel-collector | 2023-09-22T12:32:36.175Z error helper/transformer.go:99 Failed to process entry
{
"kind": "processor",
"name": "logstransform",
"pipeline": "logs",
"operator_id": "json_parser",
"operator_type": "json_parser",
"error": "type map[string]interface {} cannot be parsed as JSON",
"action": "send",
"entry": {
...
signoz-otel-collector | 2023-09-22T12:43:01.273Z error helper/transformer.go:99 Failed to process entry {"kind": "processor", "name": "logstransform", "pipeline": "logs", "operator_id": "json_parser", "operator_type": "json_parser", "error": "type map[string]interface {} cannot be parsed as JSON", "action": "send", "entry": {"observed_timestamp": "2023-09-22T12:43:01.173937618Z", "timestamp": "2023-09-22T12:43:01.087Z", "body": "ERROR POST /some/url/test", "attributes": {"app_module": "module_a", "duration": "10222951", "ecs.version": "1.2.0", "error_message": "Some error message", "error_type": "some.java.ExceptionClass", "event.dataset": "the_app", "fluent.tag": "...", "other_info": {"a": "123", "b": "456", "c": "789", "d": "987", "e": "654", "f": "321"}, "http_request_method": "POST", "http_url_path": "/some/url/test", "log.level": "INFO", "log.logger": "json.logger", "process.thread.name": "http-nio-8080-exec-3", "service.name": "the_app", "status_code": "ERROR", "trace_flags": "01", "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36", "user_id": "kj4l5j23lç4j62jlkj5lk", "user_ip_address": "0:0:0:0:0:0:0:1", "uuid": "a66e242b-95a3-484b-b9c8-8d730289baf6"}, "severity": 0, "scope_name": ""}}
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.79.0/operator/helper/transformer.go:99
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ParseWith|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ParseWith>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.79.0/operator/helper/parser.go:140
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.79.0/operator/helper/parser.go:112
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.79.0/operator/helper/parser.go:98
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/json.(*Parser).Process|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/json.(*Parser).Process>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.79.0/operator/parser/json/json.go:62
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor.(*logsTransformProcessor).converterLoop|github.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor.(*logsTransformProcessor).converterLoop>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor@v0.79.0/processor.go:143
signoz-otel-collector | 2023-09-22T12:43:01.273Z error logstransformprocessor@v0.79.0/processor.go:144 processor encountered an issue with the pipeline {"kind": "processor", "name": "logstransform", "pipeline": "logs", "error": "type map[string]interface {} cannot be parsed as JSON"}
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor.(*logsTransformProcessor).converterLoop|github.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor.(*logsTransformProcessor).converterLoop>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor@v0.79.0/processor.go:144
nitya-signoz
09/23/2023, 4:39 AM