trying to get log pipelines working. self hosted i...
# supports
Sascha Grossenbacher
09/17/2024, 8:25 PMtrying to get log pipelines working. self hosted install using docker. logs are sent as a json string. set up a pipeline that works correctly in simulate and does what I expect. but on new log entries, nothing happens. in docker logs output, I can see an an error like this
Failed to process entry {"kind": "processor", "name": "logstransform/pipeline_test", "pipeline": "logs", "operator_id": "...", "operator_type": "json_parser", "error": "running if expr: interface conversion: interface {} is map[string]interface {}, not string (1:21)\n | body != nil && body matches \"^\\\\s*{.*}\\\\s*$\"\n | ....................^", "action": "send"}Sascha Grossenbacher
09/17/2024, 8:26 PM Copy code
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).HandleEntryError>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.102.0/operator/helper/transformer.go:102
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.102.0/operator/helper/parser.go:105
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.102.0/operator/helper/parser.go:98
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/json.(*Parser).Process|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/json.(*Parser).Process>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.102.0/operator/parser/json/parser.go:24
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/router.(*Transformer).Process|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/router.(*Transformer).Process>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.102.0/operator/transformer/router/transformer.go:57
signoz-otel-collector | <http://github.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor.(*logsTransformProcessor).converterLoop|github.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor.(*logsTransformProcessor).converterLoop>
signoz-otel-collector | /home/runner/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/processor/logstransformprocessor@v0.102.0/processor.go:213s
Srikanth Chekuri
09/18/2024, 3:37 AMIssue here https://github.com/SigNoz/signoz/issues/5993
n
nitya-signoz
09/18/2024, 5:50 AMHow are you sending data to signoz, it seems your body is already a map.
s
Sascha Grossenbacher
09/18/2024, 6:28 AMthanks for the feedback. I'm using the opentelemetry sdk through the drupal opentelemetry module. I didn't know that things can already be objects. the UI shows it as a json string on top for the body, in the attributes it's a clickable tree view thing, but I just assumed that's a json viewer
Sascha Grossenbacher
09/18/2024, 6:29 AMI removed the json processory and directly read things from body.property now, that worked. I did have to completely remove the json processor, it was not enough to just disable it apparenty. the preview is now broken though
n
nitya-signoz
09/18/2024, 7:27 AMThanks for sharing, yeah the preview is broken due to the issue linked above. So once the data is received it is converted to string during storage, and pipelines run before that so since your data wasn’t actually a string it was causing the issue
62 Views