Join Slack

This message was deleted.

# support

Slackbot

11/03/2022, 12:30 PM

This message was deleted.

Shivvinay Kanswal

11/03/2022, 12:31 PM

Logs from otel collector:

Copy code

2022-11-03T12:31:13.835Z	info	exporterhelper/queued_retry.go:215	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "267.505856ms"}
2022-11-03T12:31:14.119Z	info	exporterhelper/queued_retry.go:215	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "243.540015ms"}
2022-11-03T12:31:14.386Z	info	exporterhelper/queued_retry.go:215	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 81, message: Database signoz_metrics doesn't exist", "interval": "174.956545ms"}

Shivvinay Kanswal

11/03/2022, 12:32 PM

Logs from clickhouse pod:

Copy code

2022.11.03 12:31:14.130077 [ 11 ] {e4dbe723-2948-4109-a052-f6c35dd7d214} <Error> TCPHandler: Code: 81. DB::Exception: Database signoz_metrics doesn't exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6fc2fa in /usr/bin/clickhouse
1. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d6705a in /usr/bin/clickhouse
2. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d69478 in /usr/bin/clickhouse
3. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x15d00d9e in /usr/bin/clickhouse
4. ? @ 0x1642acc6 in /usr/bin/clickhouse
5. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x16429535 in /usr/bin/clickhouse
6. DB::TCPHandler::runImpl() @ 0x16fe632a in /usr/bin/clickhouse
7. DB::TCPHandler::run() @ 0x16ff6959 in /usr/bin/clickhouse
8. Poco::Net::TCPServerConnection::start() @ 0x1b3eadef in /usr/bin/clickhouse
9. Poco::Net::TCPServerDispatcher::run() @ 0x1b3ed241 in /usr/bin/clickhouse
10. Poco::PooledThread::run() @ 0x1b5b3c89 in /usr/bin/clickhouse
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x1b5b0fe0 in /usr/bin/clickhouse
12. ? @ 0x7f24132cc609 in ?
13. __clone @ 0x7f24131f1163 in ?

2022.11.03 12:31:14.386011 [ 11 ] {0495bb23-53dd-4608-8765-2931d6fa2ba7} <Error> executeQuery: Code: 81. DB::Exception: Database signoz_metrics doesn't exist. (UNKNOWN_DATABASE) (version 22.4.5.9 (official build)) (from 10.0.105.89:57496) (in query: INSERT INTO signoz_metrics.time_series_v2 (metric_name, timestamp_ms, fingerprint, labels) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6fc2fa in /usr/bin/clickhouse
1. DB::DatabaseCatalog::assertDatabaseExistsUnlocked(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d6705a in /usr/bin/clickhouse
2. DB::DatabaseCatalog::getDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0x15d69478 in /usr/bin/clickhouse
3. DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x15d00d9e in /usr/bin/clickhouse
4. ? @ 0x1642acc6 in /usr/bin/clickhouse
5. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x16429535 in /usr/bin/clickhouse

Shivvinay Kanswal

11/03/2022, 12:32 PM

is the user supposed to create these databases?

Shivvinay Kanswal

11/03/2022, 12:37 PM

Do these crds create the databases?

Copy code

<http://clickhouseinstallations.clickhouse.altinity.com|clickhouseinstallations.clickhouse.altinity.com>             2022-11-01T09:44:50Z
<http://clickhouseinstallationtemplates.clickhouse.altinity.com|clickhouseinstallationtemplates.clickhouse.altinity.com>     2022-11-01T09:44:50Z
<http://clickhouseoperatorconfigurations.clickhouse.altinity.com|clickhouseoperatorconfigurations.clickhouse.altinity.com>    2022-11-01T09:44:50Z

Ankit Nayan

11/03/2022, 12:41 PM

@Shivvinay Kanswal the respective clickhouse exporter at https://github.com/SigNoz/signoz-otel-collector/tree/main/exporter are responsible for creating dbs and tables

Ankit Nayan

11/03/2022, 12:42 PM

so if you use

signoz/signoz-otel-collector:0.55.3

with the config at https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/otel-collector-config.yaml it should work

hashfyre

11/03/2022, 12:43 PM

so the current helm-chart is non-functional?

Ankit Nayan

11/03/2022, 12:43 PM

it works fine

Ankit Nayan

11/03/2022, 12:43 PM

not able to make signoz run using the current helm chart?

hashfyre

11/03/2022, 12:44 PM

wait, my bad

hashfyre

11/03/2022, 12:45 PM

CLICKHOUSE_DATABASE

is being passed to the exporter here:

Copy code

exporters:
 clickhouse:
   datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_TRACE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}

hashfyre

11/03/2022, 12:45 PM

but these are coming from a

.tpl

file

_clickhouse.tpl

hashfyre

11/03/2022, 12:47 PM

in our case the

signoz_metrics

db init by the exporter is failing smh, so we thought may be this value is being passed to the

clickhouse

subchart and / or the

clickhouse

operator.

Ankit Nayan

11/03/2022, 12:50 PM

this should work seamless https://signoz.io/docs/install/kubernetes/aws/

hashfyre

11/03/2022, 12:50 PM

It is not @Ankit Nayan 😄

hashfyre

11/03/2022, 12:51 PM

let me dump the arch once:

Ankit Nayan

11/03/2022, 12:51 PM

😂 @Prashant Shahi can you get on a huddle/zoom and checkout the setup

hashfyre

11/03/2022, 12:51 PM

Goal: trying to send EC2 metrics to signoz hosted on EKS

🆗 1

hashfyre

11/03/2022, 12:51 PM

the EC2s and EKS are on the same VPC, and are reachable via security group rules

Ankit Nayan

11/03/2022, 12:52 PM

got it...the communication from ec2 to eks is still the next step as I get. SigNoz first needs to run properly using helm in EKS, right?

✅ 1

hashfyre

11/03/2022, 12:52 PM

so, we have EC2(otel-agent, fluent-bit) ->-> EKS (signoz-otel-col, clickhouse)

🆗 2

hashfyre

11/03/2022, 12:53 PM

that's verified, we have tested with

nc -zv

, the

signoz-otel-col

service is exposed behind an internal load balancer

hashfyre

11/03/2022, 12:53 PM

all otel-col ports on k8s LB are reachable from EC2

Ankit Nayan

11/03/2022, 12:53 PM

cool

Ankit Nayan

11/03/2022, 12:54 PM

I remember there was some volume mounting issue with the new k8s version in EKS 1.23. Are you running there? 1.22 works fine

hashfyre

11/03/2022, 12:55 PM

we are at 1.23

Ankit Nayan

11/03/2022, 12:55 PM

but the error didn't seem like a mounting issue ... otherwise the PVCs would be in pending state

✅ 1

Ankit Nayan

11/03/2022, 12:56 PM

aah...possible to try with 1.22?

hashfyre

11/03/2022, 12:56 PM

will do

Ankit Nayan

11/03/2022, 12:56 PM

I have been using the chart almost everyday and it works fine

Ankit Nayan

11/03/2022, 12:57 PM

though I mostly work on the local version of chart

hashfyre

11/03/2022, 12:57 PM

of course, mostly it has to be some silly config issue on

values.yaml

we are missing

Ankit Nayan

11/03/2022, 12:57 PM

please give it a go with 1.22 and if it doesn't work let us know

👍 1

hashfyre

11/03/2022, 1:35 PM

@Prashant Shahi does the

k8s-infra

subchart have anything to do with the

clickhouse-exporter

🙅 1

Ankit Nayan

11/03/2022, 1:37 PM

does the
k8s-infra
subchart have anything to do with the
clickhouse-exporter
?

I don't think so

✅ 1

Ankit Nayan

11/03/2022, 1:37 PM

did 1.22 work ?

hashfyre

11/03/2022, 1:38 PM

1.23 itself work after a fresh install of chart, last time we had disabled

k8s-infra

subchart. I think it was a PVC/PV issue

Ankit Nayan

11/03/2022, 1:39 PM

k8s-infra

chart need not be enabled when installing signoz. It's optional. It is made ideally to be installed in other cluster and if user wants to collect data from the cluster where signoz is installed

✅ 2

Prashant Shahi

11/03/2022, 2:13 PM

@hashfyre I hope the issue is resolved. Do let me know if you have any doubt.

Prashant Shahi

11/03/2022, 2:15 PM

@Shivvinay Kanswal OtelCollector is reponsible for creating those database in ClickHouse.

hashfyre

11/03/2022, 4:20 PM

you mean to say the

clickhouse

exporter, right?

Prashant Shahi

11/03/2022, 4:49 PM

yes,

clickhouse

exporters from https://github.com/SigNoz/signoz-otel-collector/tree/main/exporter

✅ 2

38 Views

Open in Slack

Previous Next

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.