#signoz #support #installation I'm doing a setup in the AKS of signoz i've used below helm charts helm repo add signoz https://charts.signoz.io changed the namespace to signoz and used our own standalone clickhouse cluster setup along with zookeeper Post installation i'm obeserving below issues in OTEL_COLLECTOR Error is below PS D:\Signoz> kubectl logs po/signoz-otel-collector-65fbf66f4f-zdntv -n signoz Defaulted container "signoz-otel-collector" out of: signoz-otel-collector, signoz-otel-collector-init (init) 2023-03-17T105829.571Z info service/telemetry.go:111 Setting up own telemetry... 2023-03-17T105829.572Z info service/telemetry.go:141 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "Basic"} 2023-03-17T105829.572Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"} time="2023-03-17T105829Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics ON CLUSTER cluster\n" component=clickhouse time="2023-03-17T105829Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.samples_v2 ON CLUSTER cluster (\n\t\t\tmetric_name LowCardinality(String),\n\t\t\tfingerprint UInt64 Codec(DoubleDelta, LZ4),\n\t\t\ttimestamp_ms Int64 Codec(DoubleDelta, LZ4),\n\t\t\tvalue Float64 Codec(Gorilla, LZ4)\n\t\t)\n\t\tENGINE = MergeTree\n\t\t\tPARTITION BY toDate(timestamp_ms / 1000)\n\t\t\tORDER BY (metric_name, fingerprint, timestamp_ms)\n\t\t\tTTL toDateTime(timestamp_ms/1000) + INTERVAL 2592000 SECOND DELETE;\n" component=clickhouse time="2023-03-17T105829Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.distributed_samples_v2 ON CLUSTER cluster AS signoz_metrics.samples_v2 ENGINE = Distributed(\"cluster\", \"signoz_metrics\", samples_v2, cityHash64(metric_name, fingerprint));\n" component=clickhouse time="2023-03-17T105830Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.samples_v2 ON CLUSTER cluster MODIFY SETTING ttl_only_drop_parts = 1;\n" component=clickhouse time="2023-03-17T105830Z" level=info msg="Executing:\nSET allow_experimental_object_type = 1\n" component=clickhouse time="2023-03-17T105830Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.time_series_v2 ON CLUSTER cluster(\n\t\t\tmetric_name LowCardinality(String),\n\t\t\tfingerprint UInt64 Codec(DoubleDelta, LZ4),\n\t\t\ttimestamp_ms Int64 Codec(DoubleDelta, LZ4),\n\t\t\tlabels String Codec(ZSTD(5))\n\t\t)\n\t\tENGINE = ReplacingMergeTree\n\t\t\tPARTITION BY toDate(timestamp_ms / 1000)\n\t\t\tORDER BY (metric_name, fingerprint)\n\t\t\tTTL toDateTime(timestamp_ms/1000) + INTERVAL 2592000 SECOND DELETE;\n" component=clickhouse time="2023-03-17T105830Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.distributed_time_series_v2 ON CLUSTER cluster AS signoz_metrics.time_series_v2 ENGINE = Distributed(\"cluster\", signoz_metrics, time_series_v2, cityHash64(metric_name, fingerprint));\n" component=clickhouse time="2023-03-17T105830Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.time_series_v2 ON CLUSTER cluster DROP COLUMN IF EXISTS labels_object\n" component=clickhouse time="2023-03-17T105830Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.distributed_time_series_v2 ON CLUSTER cluster DROP COLUMN IF EXISTS labels_object\n" component=clickhouse time="2023-03-17T105830Z" level=info msg="Executing:\nALTER TABLE signoz_metrics.time_series_v2 ON CLUSTER cluster MODIFY SETTING ttl_only_drop_parts = 1;\n" component=clickhouse 2023-03-17T105831.476Z info kube/client.go:101 k8s filtering {"kind": "processor", "name": "k8sattributes", "pipeline": "metrics", "labelSelector": "", "fieldSelector": "spec.nodeName=aks-nodepool1-18518278-vmss00000f"} 2023-03-17T105831.477Z info components/components.go:30 Stability level of component is undefined {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "stability": "Undefined"} 2023-03-17T105831.676Z info clickhousetracesexporter/clickhouse_factory.go:146 Patching views {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"} 2023-03-17T105832.786Z info clickhousetracesexporter/clickhouse_factory.go:116 Running migrations from path: {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "test": "/migrations"} 2023-03-17T105841.160Z info clickhousetracesexporter/clickhouse_factory.go:128 Clickhouse Migrate finished {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces"} Error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": code: 62, message: Syntax error: failed at position 2290 (')') (line 43, col 3): ) ENGINE MergeTree() PARTITION BY toDate(timestamp) ORDER BY (durationNano, timestamp) TTL toDateTime(timestamp) + INTERVAL 604800 SECOND DELETE SETTING. Expected one of: table property (column, index, constraint) declaration, INDEX, CONSTRAINT, PROJECTION, PRIMARY KEY, column declaration, identifier 2023/03/17 105841 application run finished with error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": code: 62, message: Syntax error: failed at position 2290 (')') (line 43, col 3): ) ENGINE MergeTree() PARTITION BY toDate(timestamp) ORDER BY (durationNano, timestamp) TTL toDateTime(timestamp) + INTERVAL 604800 SECOND DELETE SETTING. Expected one of: table property (column, index, constraint) declaration, INDEX, CONSTRAINT, PROJECTION, PRIMARY KEY, column declaration, identifier need help to understand this at the earliest please

What’s the ClickHouse version?

@Srikanth Chekuri - name: clickhouse:19.6 spec: containers: - name: clickhouse-pod image: clickhouse/clickhouse-server:22.3

What version of SigNoz are you running? The current version requires

22.8

@Srikanth Chekuri Signoz below versions I'm using from the latest helm chart This is latest version of signoz

Did you make any changes to charts?

No apart from adding my external clickhouse db details and zookeeper details nothing else @Srikanth Chekuri

That’s what I meant to ask, did you add custom ClickHouse details, Each of our version has some compatible version we provide, for

v0.17.0

it is

22.8

and you are using a lower version we no longer support. Look at the tag here https://github.com/SigNoz/charts/blob/7bf7768df013dd5968c0b80de06ca3c3ab651f70/charts/signoz/values.yaml#L109

So i have to upgrade my clickhouse version? @Srikanth Chekuri

Yes, please use version

22.8.8

or later and it should work.

Ok will check this and will get back thank you

It says migration failed. Share the full logs of the signoz-otel-collector

That is the full logs of both metrics and collector @Srikanth Chekuri

What’s the exact output of

SELECT version()

for ClickHouse. Also when you are using external ClickHouse there should be cluster called

cluster

.

Yes the cluster name is cluster I'll share tbe output as well

It says

signoz_metrics.samples_v2

doesn’t exist, but the line above says otherwise, and I can’t reproduce this locally. We don’t officially support Windows yet if you are running using Windows.

No doing it in kubernetes cluster

You are seeing the

\n

because it is a log message. The actual query runs fine. It will look like this.

CREATE TABLE IF NOT EXISTS signoz_metrics.samples_v2 ON CLUSTER cluster (
			metric_name LowCardinality(String),
			fingerprint UInt64 Codec(DoubleDelta, LZ4),
			timestamp_ms Int64 Codec(DoubleDelta, LZ4),
			value Float64 Codec(Gorilla, LZ4)
		)
		ENGINE = MergeTree
			PARTITION BY toDate(timestamp_ms / 1000)
			ORDER BY (metric_name, fingerprint, timestamp_ms)
			TTL toDateTime(timestamp_ms/1000) + INTERVAL 2592000 SECOND DELETE;

Which is a valid ClickHouse query. You can see the code here https://github.com/SigNoz/signoz-otel-collector/blob/dcf11134a7db5d3f3e55dd14d34d1b1decb77e65/exporter/clickhousemetricsexporter/clickhouse.go#L[…]6

then why few queries are getting executed and few aren't

That’s not clear because your ClickHouse is blackbox to me as I have no idea how it is set up; the queries are run in order; it runs the

time="2023-03-20T01:12:45Z" level=info msg="Executing:\nCREATE TABLE IF NOT EXISTS signoz_metrics.samples_v2 ON CLUSTER cluster (\n\t\t\tmetric_name LowCardinality(String),\n\t\t\tfingerprint UInt64 Codec(DoubleDelta, LZ4),\n\t\t\ttimestamp_ms Int64 Codec(DoubleDelta, LZ4),\n\t\t\tvalue Float64 Codec(Gorilla, LZ4)\n\t\t)\n\t\tENGINE = MergeTree\n\t\t\tPARTITION BY toDate(timestamp_ms / 1000)\n\t\t\tORDER BY (metric_name, fingerprint, timestamp_ms)\n\t\t\tTTL toDateTime(timestamp_ms/1000) + INTERVAL 2592000 SECOND DELETE;\n" component=clickhouse

And proceeds to the next one and throw error

signoz_metrics.samples_v2

doesn’t exist. The prev step didn’t result in an error. Does your clickhouse set up to support the distributed engines and the cluster named

cluster

exist?

Thank you @Srikanth Chekuri for your time. indeed clickhouse version was the problem i deleted entirely everything and recreated everything it started working as expected.

Did you force shutdown it?

normal kubernetes cluster shutdown @Srikanth Chekuri

What was the storage type & class you were using?

azure default storage class, type PVC @Srikanth Chekuri

This could be solved with the init container in k8s, but our charts do not yet support the custom run command. Is the previous data important for you? The immediate fix is to get rid of old data and start fresh with little higher ``max_suspicious_broken_parts_bytes`` config. The long tem fix is support the custom commands for init container in our charts.

ok for now how to add the config and start the cluster? @Srikanth Chekuri also when can we expect full term resolution?

It should be doable with files configuration https://github.com/SigNoz/charts/blob/main/charts/signoz/values.yaml#L306

Hello @Srikanth Chekuri Thanks after adding the line it worked Please let me know the long term fix can be applied. a timeline

What are the resources are given to ClickHouse? Is the CilckHouse cluster running when you saw these errors?

Yes @Srikanth Chekuri clickhouse was running, 2 core and 2 gB of ram

What is the amount of data you are inserting i.e ingestion rate?

for now we are not querying anything data ingestion also let's say with 1k tps or so

The minimum recommended hardware for ClickHouse is 4 core and 16 CPU.

ok will keep that and check and get back @Srikanth Chekuri

You could be a little more explicit when you say you are “unable to see the complete set of data that is pushed”. What data are you pushing? What is the volume? What is missing? What are you trying to do?

We are using go module of otel collector and directly writing logs to otel collector there are 30 applications where logs are being written to signoz otel collector but i'm sometimes able to see 3 components or sometimes 4 and it varies This is frontend UI i'm talking about. @Srikanth Chekuri

there are 30 applications where logs are being written to signoz otel collector

What is the volume of the logs emitted by these applications?

but i’m sometimes able to see 3 components or sometimes 4 and it varies

What component are you referring here?

1. 3000 logs data per second 2. the applications which we are using to push logs

The

Services

tab data comes from the traces. If services it not sending data for the period it will not show up in the list. The items on the list are complete set but the services that sent the data in the selected time range (top-right corner).

ok understood, we assumed that the time interval was on how much time the data should be refreshed hence was getting confused thank you.