hi all, After using signoz for almost 2 month in K...
# general
n
hi all, After using signoz for almost 2 month in Kubernetes. otel-collector failed to run. when i tried to troubleshoot that i saw that in clickhouse db "signoz_metrics, signoz_traces, signoz_logs" database where missing. following i'm getting
Copy code
2024-07-08T17:37:49.360Z	info	service@v0.88.0/telemetry.go:84	Setting up own telemetry...
2024-07-08T17:37:49.361Z	info	service@v0.88.0/telemetry.go:201	Serving Prometheus metrics	{"address": "0.0.0.0:8888", "level": "Basic"}
2024-07-08T17:37:49.361Z	info	exporter@v0.88.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite"}
2024-07-08T17:37:49.361Z	info	exporter@v0.88.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"}
time="2024-07-08T17:37:49Z" level=error msg="code: 81, message: Database signoz_metrics does not exist" component=clickhouse
{"level":"error","timestamp":"2024-07-08T17:37:49.363Z","caller":"opamp/server_client.go:268","msg":"Collector failed for restart during rollback","component":"opamp-server-client","error":"failed to build pipelines: failed to create \"clickhousetraces\" exporter for data type \"traces\": error connecting to primary db: code: 81, message: Database signoz_traces does not exist","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload|github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:268\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).applyRemoteConfig\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:173\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).Apply\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:159\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:209\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
{"level":"error","timestamp":"2024-07-08T17:37:49.363Z","caller":"opamp/server_client.go:216","msg":"failed to apply config","component":"opamp-server-client","error":"failed to reload config: /var/tmp/collector-config.yaml: collector failed to restart: failed to build pipelines: failed to create \"clickhousetraces\" exporter for data type \"traces\": error connecting to primary db: code: 81, message: Database signoz_traces does not exist","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler|github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:216\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
{"level":"info","timestamp":"2024-07-08T17:37:49.364Z","logger":"agent-config-manager","caller":"opamp/config_manager.go:172","msg":"Config has changed, reloading","path":"/var/tmp/collector-config.yaml"}
{"level":"info","timestamp":"2024-07-08T17:37:49.364Z","caller":"signozcol/collector.go:168","msg":"Restarting collector service"}
{"level":"info","timestamp":"2024-07-08T17:37:49.364Z","caller":"signozcol/collector.go:144","msg":"Shutting down collector service"}
{"level":"info","timestamp":"2024-07-08T17:37:49.364Z","caller":"signozcol/collector.go:152","msg":"Collector service is shut down"}
{"level":"info","timestamp":"2024-07-08T17:37:49.364Z","caller":"signozcol/collector.go:103","msg":"Starting collector service"}
2024-07-08T17:37:49.373Z	info	service@v0.88.0/telemetry.go:84	Setting up own telemetry...
2024-07-08T17:37:49.374Z	info	service@v0.88.0/telemetry.go:201	Serving Prometheus metrics	{"address": "0.0.0.0:8888", "level": "Basic"}
2024-07-08T17:37:49.374Z	info	exporter@v0.88.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite"}
2024-07-08T17:37:49.374Z	info	kube/client.go:107	k8s filtering	{"kind": "processor", "name": "k8sattributes", "pipeline": "metrics/internal", "labelSelector": "", "fieldSelector": "spec.nodeName=aks-ondemand-26236767-vmss00000h"}
2024-07-08T17:37:49.374Z	info	exporter@v0.88.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"}
time="2024-07-08T17:37:49Z" level=error msg="code: 81, message: Database signoz_metrics does not exist" component=clickhouse
{"level":"info","timestamp":"2024-07-08T17:37:49.377Z","caller":"signozcol/collector.go:168","msg":"Restarting collector service"}
{"level":"info","timestamp":"2024-07-08T17:37:49.377Z","caller":"signozcol/collector.go:144","msg":"Shutting down collector service"}
a
Hi @Nagesh Rathod, It looks like the SigNoz init migration job did not run correctly? Can you double check?