Good afternoon team, today we had a very curious case in Signoz, since we were not able to see any data in the service logs and this was due to a test configuration that we made with Signoz pipelines, where we were obtaining the argocd-server logs in order to clean them using pipelines and obtain clean and necessary data for the log audit within the argocd. This configuration exists in Signoz pipelines, why could it have caused the lack or ingestion of data in Signoz from other services by these Signoz pipelines, please?

Hi José 🙂 Can you share logs from the signoz-otel-collector and query-service containers please? It will help us get to the exact reason why

@Raj Kamal

{"level":"ERROR","timestamp":"2024-12-16T20:15:52.721Z","caller":"model/agent.go:279","msg":"could not generate config recommendation for agent","agentID":"01JF8HSSW206AFVGQT5ZYC0GEK","error":"failed to generate agent config recommendation for log_pipelines: could not marshal yaml for updated conf: could not prepare otel collector processors for log pipelines: failed to prepare operators: couldn't generate nil check for field to be removed by op RemoveProccesor: invalid fieldPath body attributes.grpc.request.claims.at_hash attributes.grpc.request.claims.aud attributes.grpc.request.claims.c_hash attributes.grpc.request.claims.email_verified attributes.grpc.request.claims.exp attributes.grpc.request.claims.iat attributes.grpc.request.claims.iss attributes.grpc.request.claims.sub attributes.grpc.request.content.appNamespace attributes.grpc.request.content.other_irrelevant_fields: unexpected token Identifier(\"attributes\") (1:6)\n | body attributes.grpc.request.claims.at_hash attributes.grpc.request.claims.aud attributes.grpc.request.claims.c_hash attributes.grpc.request.claims.email_verified attributes.grpc.request.claims.exp attributes.grpc.request.claims.iat attributes.grpc.request.claims.iss attributes.grpc.request.claims.sub attributes.grpc.request.content.appNamespace attributes.grpc.request.content.other_irrelevant_fields\n | .....^","errorVerbose":"couldn't generate nil check for field to be removed by op RemoveProccesor: invalid fieldPath body attributes.grpc.request.claims.at_hash attributes.grpc.request.claims.aud attributes.grpc.request.claims.c_hash attributes.grpc.request.claims.email_verified attributes.grpc.request.claims.exp attributes.grpc.request.claims.iat attributes.grpc.request.claims.iss attributes.grpc.request.claims.sub attributes.grpc.request.content.appNamespace attributes.grpc.request.content.other_irrelevant_fields: unexpected token Identifier(\"attributes\") (1:6)\n | body attributes.grpc.request.claims.at_hash attributes.grpc.request.claims.aud attributes.grpc.request.claims.c_hash attributes.grpc.request.claims.email_verified attributes.grpc.request.claims.exp attributes.grpc.request.claims.iat attributes.grpc.request.claims.iss attributes.grpc.request.claims.sub attributes.grpc.request.content.appNamespace attributes.grpc.request.content.other_irrelevant_fields\n | .....^\nfailed to prepare operators\<http://ngo.signoz.io/signoz/pkg/query-service/app/logparsingpipeline.PreparePipelineProcessor|ngo.signoz.io/signoz/pkg/query-service/app/logparsingpipeline.PreparePipelineProcessor>\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/logparsingpipeline/pipelineBuilder.go:40\ngo.signoz.io/signoz/pkg/query-service/app/logparsingpipeline.GenerateCollectorConfigWithPipelines\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/logparsingpipeline/collector_config.go:175\ngo.signoz.io/signoz/pkg/query-service/app/logparsingpipeline.(*LogParsingPipelineController).RecommendAgentConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/logparsingpipeline/controller.go:235\ngo.signoz.io/signoz/pkg/query-service/agentConf.(*Manager).RecommendAgentConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/agentConf/manager.go:114\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).updateRemoteConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:277\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).processStatusUpdate\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:255\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).UpdateStatus\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:213\ngo.signoz.io/signoz/pkg/query-service/app/opamp.(*Server).OnMessage\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/opamp_server.go:104\ngithub.com/open-telemetry/opamp-go/server.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/server/callbacks.go:34\ngithub.com/open-telemetry/opamp-go/server.(*server).handleWSConnection\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/server/serverimpl.go:225\nruntime.goexit\n\t/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.7.linux-amd64/src/runtime/asm_amd64.s:1695\ncould not prepare otel collector processors for log pipelines\<http://ngo.signoz.io/signoz/pkg/query-service/app/logparsingpipeline.GenerateCollectorConfigWithPipelines|ngo.signoz.io/signoz/pkg/query-service/app/logparsingpipeline.GenerateCollectorConfigWithPipelines>\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/logparsingpipeline/collector_config.go:177\ngo.signoz.io/signoz/pkg/query-service/app/logparsingpipeline.(*LogParsingPipelineController).RecommendAgentConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/logparsingpipeline/controller.go:235\ngo.signoz.io/signoz/pkg/query-service/agentConf.(*Manager).RecommendAgentConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/agentConf/manager.go:114\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).updateRemoteConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:277\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).processStatusUpdate\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:255\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).UpdateStatus\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:213\ngo.signoz.io/signoz/pkg/query-service/app/opamp.(*Server).OnMessage\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/opamp_server.go:104\ngithub.com/open-telemetry/opamp-go/server.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/server/callbacks.go:34\ngithub.com/open-telemetry/opamp-go/server.(*server).handleWSConnection\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/server/serverimpl.go:225\nruntime.goexit\n\t/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.7.linux-amd64/src/runtime/asm_amd64.s:1695\ncould not marshal yaml for updated conf\<http://ngo.signoz.io/signoz/pkg/query-service/model.WrapApiError|ngo.signoz.io/signoz/pkg/query-service/model.WrapApiError>\n\t/home/runner/work/signoz/signoz/pkg/query-service/model/response.go:124\ngo.signoz.io/signoz/pkg/query-service/app/logparsingpipeline.(*LogParsingPipelineController).RecommendAgentConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/logparsingpipeline/controller.go:239\ngo.signoz.io/signoz/pkg/query-service/agentConf.(*Manager).RecommendAgentConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/agentConf/manager.go:114\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).updateRemoteConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:277\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).processStatusUpdate\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:255\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).UpdateStatus\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:213\ngo.signoz.io/signoz/pkg/query-service/app/opamp.(*Server).OnMessage\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/opamp_server.go:104\ngithub.com/open-telemetry/opamp-go/server.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/server/callbacks.go:34\ngithub.com/open-telemetry/opamp-go/server.(*server).handleWSConnection\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/server/serverimpl.go:225\nruntime.goexit\n\t/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.7.linux-amd64/src/runtime/asm_amd64.s:1695\nfailed to generate agent config recommendation for log_pipelines\<http://ngo.signoz.io/signoz/pkg/query-service/agentConf.(*Manager).RecommendAgentConfig|ngo.signoz.io/signoz/pkg/query-service/agentConf.(*Manager).RecommendAgentConfig>\n\t/home/runner/work/signoz/signoz/pkg/query-service/agentConf/manager.go:118\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).updateRemoteConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:277\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).processStatusUpdate\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:255\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).UpdateStatus\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:213\ngo.signoz.io/signoz/pkg/query-service/app/opamp.(*Server).OnMessage\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/opamp_server.go:104\ngithub.com/open-telemetry/opamp-go/server.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/server/callbacks.go:34\ngithub.com/open-telemetry/opamp-go/server.(*server).handleWSConnection\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/server/serverimpl.go:225\nruntime.goexit\n\t/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.7.linux-amd64/src/runtime/asm_amd64.s:1695","stacktrace":"go.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).updateRemoteConfig\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:279\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).processStatusUpdate\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:255\ngo.signoz.io/signoz/pkg/query-service/app/opamp/model.(*Agent).UpdateStatus\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/model/agent.go:213\ngo.signoz.io/signoz/pkg/query-service/app/opamp.(*Server).OnMessage\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/opamp/opamp_server.go:104\ngithub.com/open-telemetry/opamp-go/server.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/server/callbacks.go:34\ngithub.com/open-telemetry/opamp-go/server.(*server).handleWSConnection\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/server/serverimpl.go:225"}