Hi Team Today I upgraded signoz from 0.44.0 to 0.50.0 and getting below error, Defaulted container "schema-migrator" out of: schema-migrator, my-release-signoz-schema-migrator-init (init), my-release-signoz-schema-migrator-ch-ready (init) {"level":"info","timestamp":"2024-07-22T190403.105Z","caller":"signozschemamigrator/migrate.go:89","msg":"Setting env var SIGNOZ_CLUSTER","component":"migrate cli","cluster-name":"cluster"} {"level":"info","timestamp":"2024-07-22T190403.105Z","caller":"signozschemamigrator/migrate.go:106","msg":"Successfully set env var SIGNOZ_CLUSTER ","component":"migrate cli","cluster-name":"cluster"} {"level":"info","timestamp":"2024-07-22T190403.105Z","caller":"signozschemamigrator/migrate.go:111","msg":"Setting env var SIGNOZ_REPLICATED","component":"migrate cli","replication":false} {"level":"info","timestamp":"2024-07-22T190403.112Z","caller":"migrationmanager/manager.go:76","msg":"Running migrations for all migrators","component":"migrationmanager"} {"level":"info","timestamp":"2024-07-22T190403.112Z","caller":"migrationmanager/manager.go:78","msg":"Running migrations for logs","component":"migrationmanager","migrator":"logs"} {"level":"error","timestamp":"2024-07-22T190403.187Z","caller":"migrationmanager/manager.go:81","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"logs","error":"clickhouse migrate failed to run, error: Dirty database version 12. Fix and force version.","stacktrace":"github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go81\nmain.main\n\t/home/runner/work/signoz otel collector/signoz otel collector/cmd/signozschemamigrator/migrate.go126\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/proc.go:267"} {"level":"fatal","timestamp":"2024-07-22T190403.187Z","caller":"signozschemamigrator/migrate.go:128","msg":"Failed to run migrations","component":"migrate cli","error":"clickhouse migrate failed to run, error: Dirty database version 12. Fix and force version.","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.11/x64/src/runtime/proc.go267"}

cc @nitya-signoz

Hi, please follow this thread there are steps listed to fix the issue https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1721400860382799 . Let me know if you face any issue.

Hi @nitya-signoz Please find the out-put below, chi-my-release-clickhouse-cluster-0-0-0.chi-my-release-clickhouse-cluster-0-0.signoz.svc.cluster.local :) ALTER TABLE signoz_logs.logs ON CLUSTER cluster ADD INDEX IF NOT EXISTS body_idx lower(body) TYPE ngrambf_v1(4, 60000, 5, 0) GRANULARITY 1; ALTER TABLE signoz_logs.logs ON CLUSTER cluster ADD INDEX IF NOT EXISTS body_idx lower(body) TYPE ngrambf_v1(4, 60000, 5, 0) GRANULARITY 1 Query id: 4d685c9a-0742-4a41-beca-fd6a55553e6c Elapsed: 0.001 sec. Received exception: Code: 701. DB:Exception Requested cluster 'cluster' not found. (CLUSTER_DOESNT_EXIST) chi-my-release-clickhouse-cluster-0-0-0.chi-my-release-clickhouse-cluster-0-0.signoz.svc.cluster.local :) ALTER TABLE signoz_logs.logs ADD INDEX IF NOT EXISTS body_idx lower(body) TYPE ngrambf_v1(4, 60000, 5, 0) GRANULARITY 1; ALTER TABLE signoz_logs.logs ADD INDEX IF NOT EXISTS body_idx lower(body) TYPE ngrambf_v1(4, 60000, 5, 0) GRANULARITY 1 Query id: 5ae75d22-7ce6-45d2-95ed-bd8a9328630b Elapsed: 0.001 sec. Received exception: Code: 81. DB:Exception Database signoz_logs does not exist. (UNKNOWN_DATABASE)

Are you using the clickhouse which is shipped with SigNoz or your own deployment ?

clickhouse which is shipped with SigNoz

Databases not being present shouldn’t happen(which means you dont have previous data), also the cluster config should be used. @Prashant Shahi any idea on why this might happen ?

Yes, that's weird behavior. Surprisingly I can still query older logs from signoz console. I tried creating the database manually and running the schema migrator again, it had deleted the databases again.

Surprisingly I can still query older logs from signoz console

It means you are connecting to the wrong clickhouse when you are running the query using console

Not sure how this can happen? I am use Signoz values file template for installation/upgrade. How can he a wrong clickhouse?I can see all my users and other configuration as is.

First of all, are you using the default installation or using multiple shards/replicas?

One change I have made is exposed database using load balancer, type: NodePort instead of type: ClusterIP

> As far as my user details and dashboards are intact, I fine if I lose my logs, traces and metrics. Please let me know if there is any workaround to get Signoz working that way. This you can easily backup by backing up the signoz.db file of query service

I am running it on EKS.

We can do a short call

It would be a great help

pls join above link

Thank you so much @nitya-signoz. Things are up and running now. Great help.

@Vishal Kanani @nitya-signoz Facing the same issue, what turned out to be the root cause for cluster not found error in query service?

Can you explain a bit more on the issue, are you facing issue with migration or facing issue accessing the the clickhouse db