Slackbot
11/03/2022, 12:30 PMShivvinay Kanswal
11/03/2022, 12:31 PM2022-11-03T12:31:13.835Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "267.505856ms"}
2022-11-03T12:31:14.119Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "243.540015ms"}
2022-11-03T12:31:14.386Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "174.956545ms"}
Shivvinay Kanswal
11/03/2022, 12:32 PM2022.11.03 12:31:14.130077 [ 11 ] {e4dbe723-2948-4109-a052-f6c35dd7d214} <Error> TCPHandler: Code: 81. DB::Exception: Database signoz_metrics doesn't exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6fc2fa in /usr/bin/clickhouse
1. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d6705a in /usr/bin/clickhouse
2. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d69478 in /usr/bin/clickhouse
3. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x15d00d9e in /usr/bin/clickhouse
4. ? @ 0x1642acc6 in /usr/bin/clickhouse
5. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x16429535 in /usr/bin/clickhouse
6. DB::TCPHandler::runImpl() @ 0x16fe632a in /usr/bin/clickhouse
7. DB::TCPHandler::run() @ 0x16ff6959 in /usr/bin/clickhouse
8. Poco::Net::TCPServerConnection::start() @ 0x1b3eadef in /usr/bin/clickhouse
9. Poco::Net::TCPServerDispatcher::run() @ 0x1b3ed241 in /usr/bin/clickhouse
10. Poco::PooledThread::run() @ 0x1b5b3c89 in /usr/bin/clickhouse
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x1b5b0fe0 in /usr/bin/clickhouse
12. ? @ 0x7f24132cc609 in ?
13. __clone @ 0x7f24131f1163 in ?
2022.11.03 12:31:14.386011 [ 11 ] {0495bb23-53dd-4608-8765-2931d6fa2ba7} <Error> executeQuery: Code: 81. DB::Exception: Database signoz_metrics doesn't exist. (UNKNOWN_DATABASE) (version 22.4.5.9 (official build)) (from 10.0.105.89:57496) (in query: INSERT INTO signoz_metrics.time_series_v2 (metric_name, timestamp_ms, fingerprint, labels) VALUES), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6fc2fa in /usr/bin/clickhouse
1. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d6705a in /usr/bin/clickhouse
2. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d69478 in /usr/bin/clickhouse
3. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x15d00d9e in /usr/bin/clickhouse
4. ? @ 0x1642acc6 in /usr/bin/clickhouse
5. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x16429535 in /usr/bin/clickhouse
Shivvinay Kanswal
11/03/2022, 12:32 PMShivvinay Kanswal
11/03/2022, 12:37 PM<http://clickhouseinstallations.clickhouse.altinity.com|clickhouseinstallations.clickhouse.altinity.com> 2022-11-01T09:44:50Z
<http://clickhouseinstallationtemplates.clickhouse.altinity.com|clickhouseinstallationtemplates.clickhouse.altinity.com> 2022-11-01T09:44:50Z
<http://clickhouseoperatorconfigurations.clickhouse.altinity.com|clickhouseoperatorconfigurations.clickhouse.altinity.com> 2022-11-01T09:44:50Z
Ankit Nayan
Ankit Nayan
signoz/signoz-otel-collector:0.55.3
with the config at https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/otel-collector-config.yaml it should workhashfyre
11/03/2022, 12:43 PMAnkit Nayan
Ankit Nayan
hashfyre
11/03/2022, 12:44 PMhashfyre
11/03/2022, 12:45 PMCLICKHOUSE_DATABASE
is being passed to the exporter here:
exporters:
clickhouse:
datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
hashfyre
11/03/2022, 12:45 PM.tpl
file _clickhouse.tpl
hashfyre
11/03/2022, 12:47 PMsignoz_metrics
db init by the exporter is failing smh, so we thought may be this value is being passed to the clickhouse
subchart and / or the clickhouse
operator.Ankit Nayan
hashfyre
11/03/2022, 12:50 PMhashfyre
11/03/2022, 12:51 PMAnkit Nayan
hashfyre
11/03/2022, 12:51 PMhashfyre
11/03/2022, 12:51 PMAnkit Nayan
hashfyre
11/03/2022, 12:52 PMhashfyre
11/03/2022, 12:53 PMnc -zv
, the signoz-otel-col
service is exposed behind an internal load balancerhashfyre
11/03/2022, 12:53 PMAnkit Nayan
Ankit Nayan
hashfyre
11/03/2022, 12:55 PMAnkit Nayan
Ankit Nayan
hashfyre
11/03/2022, 12:56 PMAnkit Nayan
Ankit Nayan
hashfyre
11/03/2022, 12:57 PMvalues.yaml
we are missingAnkit Nayan
hashfyre
11/03/2022, 1:35 PMk8s-infra
subchart have anything to do with the clickhouse-exporter
?Ankit Nayan
does theI don't think sosubchart have anything to do with thek8s-infra
?clickhouse-exporter
Ankit Nayan
hashfyre
11/03/2022, 1:38 PMk8s-infra
subchart. I think it was a PVC/PV issueAnkit Nayan
k8s-infra
chart need not be enabled when installing signoz. It's optional. It is made ideally to be installed in other cluster and if user wants to collect data from the cluster where signoz is installedPrashant Shahi
11/03/2022, 2:13 PMPrashant Shahi
11/03/2022, 2:15 PMhashfyre
11/03/2022, 4:20 PMclickhouse
exporter, right?Prashant Shahi
11/03/2022, 4:49 PMclickhouse
exporters from https://github.com/SigNoz/signoz-otel-collector/tree/main/exporter