https://signoz.io logo
Join Slack
Powered by
Hi Team, clickhouse database initialisation is fai...
# support
s
Hi Team, clickhouse database initialisation is failing on signoz clickhouse statefulset. signoz collector pods are getting error that database signoz_metrics doesn't exist. Could you guide us on how clickouse database is being created by the clickhouse chart, on the chart config we see signoz_metrics database getting used but dont see it getting created anywhere Is the database creation being handled internally?
Logs from otel collector:
Copy code
2022-11-03T12:31:13.835Z	info	exporterhelper/queued_retry.go:215	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "267.505856ms"}
2022-11-03T12:31:14.119Z	info	exporterhelper/queued_retry.go:215	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "243.540015ms"}
2022-11-03T12:31:14.386Z	info	exporterhelper/queued_retry.go:215	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "174.956545ms"}
Logs from clickhouse pod:
Copy code
2022.11.03 12:31:14.130077 [ 11 ] {e4dbe723-2948-4109-a052-f6c35dd7d214} <Error> TCPHandler: Code: 81. DB::Exception: Database signoz_metrics doesn't exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6fc2fa in /usr/bin/clickhouse
1. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d6705a in /usr/bin/clickhouse
2. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d69478 in /usr/bin/clickhouse
3. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x15d00d9e in /usr/bin/clickhouse
4. ? @ 0x1642acc6 in /usr/bin/clickhouse
5. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x16429535 in /usr/bin/clickhouse
6. DB::TCPHandler::runImpl() @ 0x16fe632a in /usr/bin/clickhouse
7. DB::TCPHandler::run() @ 0x16ff6959 in /usr/bin/clickhouse
8. Poco::Net::TCPServerConnection::start() @ 0x1b3eadef in /usr/bin/clickhouse
9. Poco::Net::TCPServerDispatcher::run() @ 0x1b3ed241 in /usr/bin/clickhouse
10. Poco::PooledThread::run() @ 0x1b5b3c89 in /usr/bin/clickhouse
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x1b5b0fe0 in /usr/bin/clickhouse
12. ? @ 0x7f24132cc609 in ?
13. __clone @ 0x7f24131f1163 in ?

2022.11.03 12:31:14.386011 [ 11 ] {0495bb23-53dd-4608-8765-2931d6fa2ba7} <Error> executeQuery: Code: 81. DB::Exception: Database signoz_metrics doesn't exist. (UNKNOWN_DATABASE) (version 22.4.5.9 (official build)) (from 10.0.105.89:57496) (in query: INSERT INTO signoz_metrics.time_series_v2 (metric_name, timestamp_ms, fingerprint, labels) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6fc2fa in /usr/bin/clickhouse
1. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d6705a in /usr/bin/clickhouse
2. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d69478 in /usr/bin/clickhouse
3. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x15d00d9e in /usr/bin/clickhouse
4. ? @ 0x1642acc6 in /usr/bin/clickhouse
5. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x16429535 in /usr/bin/clickhouse
is the user supposed to create these databases?
Do these crds create the databases?
Copy code
<http://clickhouseinstallations.clickhouse.altinity.com|clickhouseinstallations.clickhouse.altinity.com>             2022-11-01T09:44:50Z
<http://clickhouseinstallationtemplates.clickhouse.altinity.com|clickhouseinstallationtemplates.clickhouse.altinity.com>     2022-11-01T09:44:50Z
<http://clickhouseoperatorconfigurations.clickhouse.altinity.com|clickhouseoperatorconfigurations.clickhouse.altinity.com>    2022-11-01T09:44:50Z
a
@Shivvinay Kanswal the respective clickhouse exporter at https://github.com/SigNoz/signoz-otel-collector/tree/main/exporter are responsible for creating dbs and tables
so if you use 
signoz/signoz-otel-collector:0.55.3
with the config at https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/otel-collector-config.yaml it should work
h
so the current helm-chart is non-functional?
a
it works fine
not able to make signoz run using the current helm chart?
h
wait, my bad
so 
CLICKHOUSE_DATABASE
is being passed to the exporter here:
Copy code
exporters:
 clickhouse:
   datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
but these are coming from a 
.tpl
file 
_clickhouse.tpl
in our case the 
signoz_metrics
db init by the exporter is failing smh, so we thought may be this value is being passed to the 
clickhouse
subchart and / or the 
clickhouse
operator.
a
h
It is not @Ankit Nayan 😄
let me dump the arch once:
a
😂 @Prashant Shahi can you get on a huddle/zoom and checkout the setup
h
Goal: trying to send EC2 metrics to signoz hosted on EKS
the EC2s and EKS are on the same VPC, and are reachable via security group rules
a
got it...the communication from ec2 to eks is still the next step as I get. SigNoz first needs to run properly using helm in EKS, right?
h
so, we have EC2(otel-agent, fluent-bit) ->-> EKS (signoz-otel-col, clickhouse)
that's verified, we have tested with 
nc -zv
, the 
signoz-otel-col
service is exposed behind an internal load balancer
all otel-col ports on k8s LB are reachable from EC2
a
cool
I remember there was some volume mounting issue with the new k8s version in EKS 1.23. Are you running there? 1.22 works fine
h
we are at 1.23
a
but the error didn't seem like a mounting issue ... otherwise the PVCs would be in pending state
aah...possible to try with 1.22?
h
will do
a
I have been using the chart almost everyday and it works fine
though I mostly work on the local version of chart
h
of course, mostly it has to be some silly config issue on 
values.yaml
we are missing
a
please give it a go with 1.22 and if it doesn't work let us know
h
@Prashant Shahi does the 
k8s-infra
subchart have anything to do with the 
clickhouse-exporter
?
a
does the 
k8s-infra
subchart have anything to do with the 
clickhouse-exporter
?
I don't think so
did 1.22 work ?
h
1.23 itself work after a fresh install of chart, last time we had disabled 
k8s-infra
subchart. I think it was a PVC/PV issue
a
k8s-infra
chart need not be enabled when installing signoz. It's optional. It is made ideally to be installed in other cluster and if user wants to collect data from the cluster where signoz is installed
p
@hashfyre I hope the issue is resolved. Do let me know if you have any doubt.
@Shivvinay Kanswal OtelCollector is reponsible for creating those database in ClickHouse.
h
you mean to say the 
clickhouse
exporter, right?
p
5 Views
Open in Slack
PreviousNext
Powered by