https://signoz.io logo
Powered by
Title
s

Shivvinay Kanswal

11/03/2022, 12:30 PM
Hi Team, clickhouse database initialisation is failing on signoz clickhouse statefulset. signoz collector pods are getting error that database signoz_metrics doesn't exist. Could you guide us on how clickouse database is being created by the clickhouse chart, on the chart config we see signoz_metrics database getting used but dont see it getting created anywhere Is the database creation being handled internally?
Logs from otel collector: 
2022-11-03T12:31:13.835Z	info	exporterhelper/queued_retry.go:215	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "267.505856ms"}
2022-11-03T12:31:14.119Z	info	exporterhelper/queued_retry.go:215	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "243.540015ms"}
2022-11-03T12:31:14.386Z	info	exporterhelper/queued_retry.go:215	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "174.956545ms"}
Logs from clickhouse pod: 
2022.11.03 12:31:14.130077 [ 11 ] {e4dbe723-2948-4109-a052-f6c35dd7d214} <Error> TCPHandler: Code: 81. DB::Exception: Database signoz_metrics doesn't exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6fc2fa in /usr/bin/clickhouse
1. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d6705a in /usr/bin/clickhouse
2. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d69478 in /usr/bin/clickhouse
3. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x15d00d9e in /usr/bin/clickhouse
4. ? @ 0x1642acc6 in /usr/bin/clickhouse
5. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x16429535 in /usr/bin/clickhouse
6. DB::TCPHandler::runImpl() @ 0x16fe632a in /usr/bin/clickhouse
7. DB::TCPHandler::run() @ 0x16ff6959 in /usr/bin/clickhouse
8. Poco::Net::TCPServerConnection::start() @ 0x1b3eadef in /usr/bin/clickhouse
9. Poco::Net::TCPServerDispatcher::run() @ 0x1b3ed241 in /usr/bin/clickhouse
10. Poco::PooledThread::run() @ 0x1b5b3c89 in /usr/bin/clickhouse
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x1b5b0fe0 in /usr/bin/clickhouse
12. ? @ 0x7f24132cc609 in ?
13. __clone @ 0x7f24131f1163 in ?

2022.11.03 12:31:14.386011 [ 11 ] {0495bb23-53dd-4608-8765-2931d6fa2ba7} <Error> executeQuery: Code: 81. DB::Exception: Database signoz_metrics doesn't exist. (UNKNOWN_DATABASE) (version 22.4.5.9 (official build)) (from 10.0.105.89:57496) (in query: INSERT INTO signoz_metrics.time_series_v2 (metric_name, timestamp_ms, fingerprint, labels) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6fc2fa in /usr/bin/clickhouse
1. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d6705a in /usr/bin/clickhouse
2. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d69478 in /usr/bin/clickhouse
3. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x15d00d9e in /usr/bin/clickhouse
4. ? @ 0x1642acc6 in /usr/bin/clickhouse
5. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x16429535 in /usr/bin/clickhouse
is the user supposed to create these databases?
Do these crds create the databases? 
<http://clickhouseinstallations.clickhouse.altinity.com|clickhouseinstallations.clickhouse.altinity.com>             2022-11-01T09:44:50Z
<http://clickhouseinstallationtemplates.clickhouse.altinity.com|clickhouseinstallationtemplates.clickhouse.altinity.com>     2022-11-01T09:44:50Z
<http://clickhouseoperatorconfigurations.clickhouse.altinity.com|clickhouseoperatorconfigurations.clickhouse.altinity.com>    2022-11-01T09:44:50Z
a

Ankit Nayan

11/03/2022, 12:41 PM
@Shivvinay Kanswal the respective clickhouse exporter at https://github.com/SigNoz/signoz-otel-collector/tree/main/exporter are responsible for creating dbs and tables
so if you use 
signoz/signoz-otel-collector:0.55.3
with the config at https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/otel-collector-config.yaml it should work
h

hashfyre

11/03/2022, 12:43 PM
so the current helm-chart is non-functional?
a

Ankit Nayan

11/03/2022, 12:43 PM
it works fine
not able to make signoz run using the current helm chart?
h

hashfyre

11/03/2022, 12:44 PM
wait, my bad
so 
CLICKHOUSE_DATABASE
is being passed to the exporter here: 
exporters:
 clickhouse:
   datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
but these are coming from a 
.tpl
file 
_clickhouse.tpl
in our case the 
signoz_metrics
db init by the exporter is failing smh, so we thought may be this value is being passed to the 
clickhouse
subchart and / or the 
clickhouse
operator.
a

Ankit Nayan

11/03/2022, 12:50 PM
this should work seamless https://signoz.io/docs/install/kubernetes/aws/
h

hashfyre

11/03/2022, 12:50 PM
It is not @Ankit Nayan 😄
let me dump the arch once:
a

Ankit Nayan

11/03/2022, 12:51 PM
😂 @Prashant Shahi can you get on a huddle/zoom and checkout the setup
h

hashfyre

11/03/2022, 12:51 PM
Goal: trying to send EC2 metrics to signoz hosted on EKS
the EC2s and EKS are on the same VPC, and are reachable via security group rules
a

Ankit Nayan

11/03/2022, 12:52 PM
got it...the communication from ec2 to eks is still the next step as I get. SigNoz first needs to run properly using helm in EKS, right?
h

hashfyre

11/03/2022, 12:52 PM
so, we have EC2(otel-agent, fluent-bit) ->-> EKS (signoz-otel-col, clickhouse)
that's verified, we have tested with 
nc -zv
, the 
signoz-otel-col
service is exposed behind an internal load balancer
all otel-col ports on k8s LB are reachable from EC2
a

Ankit Nayan

11/03/2022, 12:53 PM
cool
I remember there was some volume mounting issue with the new k8s version in EKS 1.23. Are you running there? 1.22 works fine
h

hashfyre

11/03/2022, 12:55 PM
we are at 1.23
a

Ankit Nayan

11/03/2022, 12:55 PM
but the error didn't seem like a mounting issue ... otherwise the PVCs would be in pending state
aah...possible to try with 1.22?
h

hashfyre

11/03/2022, 12:56 PM
will do
a

Ankit Nayan

11/03/2022, 12:56 PM
I have been using the chart almost everyday and it works fine
though I mostly work on the local version of chart
h

hashfyre

11/03/2022, 12:57 PM
of course, mostly it has to be some silly config issue on 
values.yaml
we are missing
a

Ankit Nayan

11/03/2022, 12:57 PM
please give it a go with 1.22 and if it doesn't work let us know
h

hashfyre

11/03/2022, 1:35 PM
@Prashant Shahi does the 
k8s-infra
subchart have anything to do with the 
clickhouse-exporter
?
a

Ankit Nayan

11/03/2022, 1:37 PM
does the 
k8s-infra
subchart have anything to do with the 
clickhouse-exporter
?
I don't think so
did 1.22 work ?
h

hashfyre

11/03/2022, 1:38 PM
1.23 itself work after a fresh install of chart, last time we had disabled 
k8s-infra
subchart. I think it was a PVC/PV issue
a

Ankit Nayan

11/03/2022, 1:39 PM
k8s-infra
chart need not be enabled when installing signoz. It's optional. It is made ideally to be installed in other cluster and if user wants to collect data from the cluster where signoz is installed
p

Prashant Shahi

11/03/2022, 2:13 PM
@hashfyre I hope the issue is resolved. Do let me know if you have any doubt.
@Shivvinay Kanswal OtelCollector is reponsible for creating those database in ClickHouse.
h

hashfyre

11/03/2022, 4:20 PM
you mean to say the 
clickhouse
exporter, right?
p

Prashant Shahi

11/03/2022, 4:49 PM
yes, 
clickhouse
exporters from https://github.com/SigNoz/signoz-otel-collector/tree/main/exporter
3 Views
#supportJoin Slack
Powered by