This message was deleted SigNoz Community #support

Join Slack

This message was deleted.

# support

Slackbot

07/11/2023, 9:11 PM

This message was deleted.

clickhouse-log-snippets.txt

Prashant Shahi

07/12/2023, 5:40 AM

are you using external clickhouse with SigNoz?

Prashant Shahi

07/12/2023, 5:41 AM

There should be a cluster named:

cluster

Prashant Shahi

07/12/2023, 5:41 AM

which seems to be missing in your case.

Prashant Shahi

07/12/2023, 5:41 AM

Ref: https://signoz.io/docs/operate/clickhouse/external-clickhouse/

07/12/2023, 1:48 PM

Hi @Prashant Shahi I am not using external clickhouse. I am using the clickhouse instance deployed with signoz. It was working with version 0.21

Copy code

┌─cluster─────────────────────────────────────────┐
│ all-replicated                                  │
│ all-sharded                                     │
│ cluster                                         │
│ test_cluster_one_shard_three_replicas_localhost │
│ test_cluster_one_shard_three_replicas_localhost │
│ test_cluster_one_shard_three_replicas_localhost │
│ test_cluster_two_shards                         │
│ test_cluster_two_shards                         │
│ test_cluster_two_shards_internal_replication    │
│ test_cluster_two_shards_internal_replication    │
│ test_cluster_two_shards_localhost               │
│ test_cluster_two_shards_localhost               │
│ test_shard_localhost                            │
│ test_shard_localhost_secure                     │
│ test_unavailable_shard                          │
│ test_unavailable_shard                          │
└─────────────────────────────────────────────────┘

but after upgrading to v0.22 only the

test_*

clusters are available.

07/13/2023, 3:10 PM

helm rollback signoz

to the previous revision (v0.21.0), restored clickhouse to working condition. Not sure what happened during the upgrade. Perhaps I can attempt upgrading to v0.22.0 again and see if it was transient issue.

Prashant Shahi

07/13/2023, 3:27 PM

I tested in latest chart with v0.22

Copy code

SELECT cluster
FROM system.clusters

Query id: 70c515db-ac46-4738-8518-3bf84757e645

┌─cluster─────────────────────────────────────────┐
│ all-replicated                                  │
│ all-sharded                                     │
│ cluster                                         │
│ test_cluster_one_shard_three_replicas_localhost │
│ test_cluster_one_shard_three_replicas_localhost │
│ test_cluster_one_shard_three_replicas_localhost │
│ test_cluster_two_shards                         │
│ test_cluster_two_shards                         │
│ test_cluster_two_shards_internal_replication    │
│ test_cluster_two_shards_internal_replication    │
│ test_cluster_two_shards_localhost               │
│ test_cluster_two_shards_localhost               │
│ test_shard_localhost                            │
│ test_shard_localhost_secure                     │
│ test_unavailable_shard                          │
│ test_unavailable_shard                          │
└─────────────────────────────────────────────────┘

Prashant Shahi

07/13/2023, 3:27 PM

not able to reproduce the issue

Prashant Shahi

07/13/2023, 3:28 PM

did you update the helm repository prior to upgrading?

Copy code

helm repo update

07/13/2023, 3:50 PM

Yes. At the time of the upgrade, chart version 0.18.1 was the latest:

Copy code

signoz/signoz           0.18.1          0.22.0          SigNoz Observability Platform Helm Chart

Resulting in:

Copy code

1. You have just deployed SigNoz cluster:

- frontend version: '0.22.0'
- query-service version: '0.22.0'
- alertmanager version: '0.23.1'
- otel-collector version: '0.79.2'
- otel-collector-metrics version: '0.79.2'

07/13/2023, 6:36 PM

I'll attempt upgrade again and report back.

Prashant Shahi

07/14/2023, 8:22 AM

@Al yes, it will be much appreciated.

07/18/2023, 8:11 PM

@Prashant Shahi Just attempted upgrading to Chart 0.18.2 and the same issue occurred, clickhouse was unable to load 'cluster'.

Copy code

REVISION        UPDATED                         STATUS          CHART           APP VERSION     DESCRIPTION   

14              Tue Jul 18 19:10:43 2023        superseded      signoz-0.18.2   0.22.0          Upgrade complete

The following log entry seems relevant:

Copy code

2023.07.18 19:19:59.669624 [ 240 ] {} <Error> DDLWorker: Cannot parse DDL task query-0000025438: Cannot parse query or obtain cluster info. Will try to send error status: 371
Code: 371. DB::Exception: DDL task query-0000025438 contains current host chi-signoz-clickhouse-cluster-0-0:9000 in cluster cluster, but there is no such cluster here. (INCONSISTENT_CLUSTER_DEFINITION) (version 22.8.8.3 (official build))
2023.07.18 19:19:59.681107 [ 244 ] {} <Information> DDLWorker: Task query-0000025438 is outdated, deleting it
2023.07.18 19:19:59.684970 [ 240 ] {} <Error> DDLWorker: Cannot parse DDL task query-0000025439: Cannot parse query or obtain cluster info. Will try to send error status: 371

I have rolled back to

signoz-0.17.0   0.21.0

and the signoz deployment is functional again.

Prashant Shahi

07/18/2023, 9:12 PM

@Al Latest version is

signoz-0.19.1

. Also, you would be required to run the migration steps: https://signoz.io/docs/operate/migration/upgrade-0.23/

👍 1

07/19/2023, 6:25 PM

Hi @Prashant Shahi upgrade to

signoz-0.19.1

completed successfully!

🙌 1

29 Views

Open in Slack

Previous Next