Hello everyone, I'm facing issue replate to Otel collector it's stuck in crashloopbackoff, when i checked the logs got the following.

{"level":"info","timestamp":"2025-01-08T05:19:57.460Z","caller":"signozcol/collector.go:168","msg":"Restarting collector service"}
{"level":"info","timestamp":"2025-01-08T05:19:57.460Z","caller":"signozcol/collector.go:144","msg":"Shutting down collector service"}
{"level":"info","timestamp":"2025-01-08T05:19:57.460Z","caller":"signozcol/collector.go:154","msg":"Collector service is not running"}
{"level":"info","timestamp":"2025-01-08T05:19:57.461Z","caller":"signozcol/collector.go:103","msg":"Starting collector service"}
{"level":"info","timestamp":"2025-01-08T05:19:57.470Z","caller":"signozcol/collector.go:168","msg":"Restarting collector service"}
{"level":"info","timestamp":"2025-01-08T05:19:57.470Z","caller":"signozcol/collector.go:144","msg":"Shutting down collector service"}
{"level":"info","timestamp":"2025-01-08T05:19:57.470Z","caller":"signozcol/collector.go:152","msg":"Collector service is shut down"}
{"level":"info","timestamp":"2025-01-08T05:19:57.471Z","caller":"signozcol/collector.go:103","msg":"Starting collector service"}
{"level":"error","timestamp":"2025-01-08T05:19:57.478Z","caller":"opamp/server_client.go:268","msg":"Collector failed for restart during rollback","component":"opamp-server-client","error":"failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:\n\n* error decoding 'exporters': error reading configuration for \"clickhousetraces\": 1 error(s) decoding:\n\n* '' has invalid keys: use_new_schema","stacktrace":"github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:268\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).applyRemoteConfig\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:173\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).Apply\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:159\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:209\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
{"level":"error","timestamp":"2025-01-08T05:19:57.478Z","caller":"opamp/server_client.go:216","msg":"failed to apply config","component":"opamp-server-client","error":"failed to reload config: /var/tmp/collector-config.yaml: collector failed to restart: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:\n\n* error decoding 'exporters': error reading configuration for \"clickhousetraces\": 1 error(s) decoding:\n\n* '' has invalid keys: use_new_schema","stacktrace":"github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:216\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}

when i checked with GPT got the following analysis Invalid Key in Configuration: • The configuration file (

collector-config.yaml

) contains an invalid key

use_new_schema

under the

clickhousetraces

exporter. This key is not supported by the exporter. FYI using SigNoz 0.56, deployed in Kubernetes AKS cluster with helm chart. also tried to changed the value use_new_schema to "true" but still it's not shipping the logs

clickhouselogsexporter:
        dsn: tcp://${CLICKHOUSE_USER}:${CLICKHOUSE_PASSWORD}@${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/${CLICKHOUSE_LOG_DATABASE}
        timeout: 10s
        use_new_schema: true

Please any one help me here to troubleshoot the otelcollector, and take it back to working condition.

Try removing the key from

clickhousetraces

. In the snippet above, you're trying to do it in

clickhouselogsexporter

.

it was "false" earlier for hit and try i made to "true" you mean to delete the key from clickhouse database?

No I meant the

clickhousetraces

exporter configuration

Hello @Vibhu Pandey i have tried but it's still not working

exporters:
      clickhousetraces:
        datasource: tcp://${env:CLICKHOUSE_USER}:${env:CLICKHOUSE_PASSWORD}@${env:CLICKHOUSE_HOST}:${env:CLICKHOUSE_PORT}/${env:CLICKHOUSE_TRACE_DATABASE}
        low_cardinal_exception_grouping: ${env:LOW_CARDINAL_EXCEPTION_GROUPING}
        use_new_schema: false
      clickhousemetricswrite:
        endpoint: tcp://${env:CLICKHOUSE_USER}:${env:CLICKHOUSE_PASSWORD}@${env:CLICKHOUSE_HOST}:${env:CLICKHOUSE_PORT}/${env:CLICKHOUSE_DATABASE}
        timeout: 15s
        resource_to_telemetry_conversion:
          enabled: true
      clickhouselogsexporter:
        dsn: tcp://${env:CLICKHOUSE_USER}:${env:CLICKHOUSE_PASSWORD}@${env:CLICKHOUSE_HOST}:${env:CLICKHOUSE_PORT}/${env:CLICKHOUSE_LOG_DATABASE}
        timeout: 10s
        use_new_schema: true

Please remove the

use_new_schema

key altogether from clickhousetraces exporter configuration. see the error:

Collector failed for restart during rollback","component":"opamp-server-client","error":"failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:\n\n* error decoding 'exporters': error reading configuration for \"clickhousetraces\": 1 error(s) decoding:\n\n* '' has invalid keys: use_new_schema"

Thanks man for help, it's working now 🙂