Hello team, I am trying to get signoz running in Nomad and I am getting the following error when ote...
r

Raphaël Grenier

over 1 year ago
Hello team, I am trying to get signoz running in Nomad and I am getting the following error when otel-collector tries to connect to clickhouse:
2024.04.30 19:33:38.290111 [ 750 ] {} <Error> TCPHandler: Code: 115. DB::Exception: Unknown setting database. (UNKNOWN_SETTING), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse
1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007219a43 in /usr/bin/clickhouse
2. DB::BaseSettingsHelpers::throwSettingNotFound(std::basic_string_view<char, std::char_traits<char>>) @ 0x000000001051fbab in /usr/bin/clickhouse
3. DB::BaseSettings<DB::SettingsTraits>::read(DB::ReadBuffer&, DB::SettingsWriteFormat) @ 0x000000001057a6e8 in /usr/bin/clickhouse
4. DB::TCPHandler::receiveQuery() @ 0x0000000012930419 in /usr/bin/clickhouse
5. DB::TCPHandler::receivePacket() @ 0x000000001292789b in /usr/bin/clickhouse
6. DB::TCPHandler::runImpl() @ 0x000000001291b4fb in /usr/bin/clickhouse
7. DB::TCPHandler::run() @ 0x0000000012933eb9 in /usr/bin/clickhouse
8. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in /usr/bin/clickhouse
9. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in /usr/bin/clickhouse
10. Poco::PooledThread::run() @ 0x000000001549f047 in /usr/bin/clickhouse
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse
12. ? @ 0x00007f46509ea609
13. ? @ 0x00007f465090f353
Any idea on what could be wrong?
hi all, After using signoz for almost 2 month in Kubernetes. otel-collector failed to run. when i tr...
n

Nagesh Rathod

over 1 year ago
hi all, After using signoz for almost 2 month in Kubernetes. otel-collector failed to run. when i tried to troubleshoot that i saw that in clickhouse db "signoz_metrics, signoz_traces, signoz_logs" database where missing. following i'm getting
2024-07-08T17:37:49.360Z	info	service@v0.88.0/telemetry.go:84	Setting up own telemetry...
2024-07-08T17:37:49.361Z	info	service@v0.88.0/telemetry.go:201	Serving Prometheus metrics	{"address": "0.0.0.0:8888", "level": "Basic"}
2024-07-08T17:37:49.361Z	info	exporter@v0.88.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite"}
2024-07-08T17:37:49.361Z	info	exporter@v0.88.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"}
time="2024-07-08T17:37:49Z" level=error msg="code: 81, message: Database signoz_metrics does not exist" component=clickhouse
{"level":"error","timestamp":"2024-07-08T17:37:49.363Z","caller":"opamp/server_client.go:268","msg":"Collector failed for restart during rollback","component":"opamp-server-client","error":"failed to build pipelines: failed to create \"clickhousetraces\" exporter for data type \"traces\": error connecting to primary db: code: 81, message: Database signoz_traces does not exist","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload|github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:268\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).applyRemoteConfig\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:173\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).Apply\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:159\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:209\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
{"level":"error","timestamp":"2024-07-08T17:37:49.363Z","caller":"opamp/server_client.go:216","msg":"failed to apply config","component":"opamp-server-client","error":"failed to reload config: /var/tmp/collector-config.yaml: collector failed to restart: failed to build pipelines: failed to create \"clickhousetraces\" exporter for data type \"traces\": error connecting to primary db: code: 81, message: Database signoz_traces does not exist","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler|github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:216\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"}
{"level":"info","timestamp":"2024-07-08T17:37:49.364Z","logger":"agent-config-manager","caller":"opamp/config_manager.go:172","msg":"Config has changed, reloading","path":"/var/tmp/collector-config.yaml"}
{"level":"info","timestamp":"2024-07-08T17:37:49.364Z","caller":"signozcol/collector.go:168","msg":"Restarting collector service"}
{"level":"info","timestamp":"2024-07-08T17:37:49.364Z","caller":"signozcol/collector.go:144","msg":"Shutting down collector service"}
{"level":"info","timestamp":"2024-07-08T17:37:49.364Z","caller":"signozcol/collector.go:152","msg":"Collector service is shut down"}
{"level":"info","timestamp":"2024-07-08T17:37:49.364Z","caller":"signozcol/collector.go:103","msg":"Starting collector service"}
2024-07-08T17:37:49.373Z	info	service@v0.88.0/telemetry.go:84	Setting up own telemetry...
2024-07-08T17:37:49.374Z	info	service@v0.88.0/telemetry.go:201	Serving Prometheus metrics	{"address": "0.0.0.0:8888", "level": "Basic"}
2024-07-08T17:37:49.374Z	info	exporter@v0.88.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite"}
2024-07-08T17:37:49.374Z	info	kube/client.go:107	k8s filtering	{"kind": "processor", "name": "k8sattributes", "pipeline": "metrics/internal", "labelSelector": "", "fieldSelector": "spec.nodeName=aks-ondemand-26236767-vmss00000h"}
2024-07-08T17:37:49.374Z	info	exporter@v0.88.0/exporter.go:275	Stability level of component is undefined	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"}
time="2024-07-08T17:37:49Z" level=error msg="code: 81, message: Database signoz_metrics does not exist" component=clickhouse
{"level":"info","timestamp":"2024-07-08T17:37:49.377Z","caller":"signozcol/collector.go:168","msg":"Restarting collector service"}
{"level":"info","timestamp":"2024-07-08T17:37:49.377Z","caller":"signozcol/collector.go:144","msg":"Shutting down collector service"}