Upgrade to v0.47.0 is failing with signoz-schema-migrator-upgradePod: `clickhouse migrate failed ...
a

Al

over 1 year ago
Upgrade to v0.47.0 is failing with signoz-schema-migrator-upgradePod:
clickhouse migrate failed to run, error: Dirty database version 28. Fix and force version.
Any guidance appreciated thanks
{"level":"info","timestamp":"2024-06-06T13:55:49.546Z","caller":"signozschemamigrator/migrate.go:89","msg":"Setting env var SIGNOZ_CLUSTER","component":"migrate cli","cluster-name":"cluster"}
{"level":"info","timestamp":"2024-06-06T13:55:49.546Z","caller":"signozschemamigrator/migrate.go:106","msg":"Successfully set env var SIGNOZ_CLUSTER ","component":"migrate cli","cluster-name":"cluster"}
{"level":"info","timestamp":"2024-06-06T13:55:49.546Z","caller":"signozschemamigrator/migrate.go:111","msg":"Setting env var SIGNOZ_REPLICATED","component":"migrate cli","replication":false}
{"level":"info","timestamp":"2024-06-06T13:55:49.550Z","caller":"migrationmanager/manager.go:76","msg":"Running migrations for all migrators","component":"migrationmanager"}
{"level":"info","timestamp":"2024-06-06T13:55:49.550Z","caller":"migrationmanager/manager.go:78","msg":"Running migrations for logs","component":"migrationmanager","migrator":"logs"}
{"level":"info","timestamp":"2024-06-06T13:55:49.618Z","caller":"migrationmanager/manager.go:78","msg":"Running migrations for metrics","component":"migrationmanager","migrator":"metrics"}
{"level":"info","timestamp":"2024-06-06T13:55:49.749Z","caller":"migrationmanager/manager.go:78","msg":"Running migrations for traces","component":"migrationmanager","migrator":"traces"}
{"level":"error","timestamp":"2024-06-06T13:55:49.824Z","caller":"migrationmanager/manager.go:81","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"traces","error":"clickhouse migrate failed to run, error: Dirty database version 28. Fix and force version.","stacktrace":"github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:81\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:126\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.10/x64/src/runtime/proc.go:267"}
{"level":"fatal","timestamp":"2024-06-06T13:55:49.825Z","caller":"signozschemamigrator/migrate.go:128","msg":"Failed to run migrations","component":"migrate cli","error":"clickhouse migrate failed to run, error: Dirty database version 28. Fix and force version.","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.10/x64/src/runtime/proc.go:267"}
As a result, signoz-otel-collectors are blocked from starting with:
[2024-06-06 14:33:17] Waiting for job signoz-schema-migrator-upgrade...
[2024-06-06 14:33:19] Waiting for job signoz-schema-migrator-upgrade...
[2024-06-06 14:33:21] Waiting for job signoz-schema-migrator-upgrade...
[2024-06-06 14:33:23] Waiting for job signoz-schema-migrator-upgrade...
[2024-06-06 14:33:25] Waiting for job signoz-schema-migrator-upgrade...
Signoz pods not Running
m

Maitryy

about 2 years ago
Hi Team !! I'm trying to deploy signoz on a minikube using helm on a proxy setup. The status is
platform        chi-my-release-clickhouse-cluster-0-0-0                     1/1     Running                 0                 10h
platform        my-release-clickhouse-operator-657986696-mtgdq              2/2     Running                 0                 12h
platform        my-release-k8s-infra-otel-agent-pvf29                       1/1     Running                 0                 12h
platform        my-release-k8s-infra-otel-deployment-65767679c6-llgmg       1/1     Running                 0                 12h
platform        my-release-signoz-alertmanager-0                            0/1     Init:0/1                0                 9h
platform        my-release-signoz-frontend-5fc8679d4b-zd5c9                 0/1     Init:0/1                0                 15h
platform        my-release-signoz-frontend-775b95894-rl5pm                  0/1     Init:0/1                0                 11h
platform        my-release-signoz-otel-collector-577f7cc9c6-jswbm           0/1     Init:0/1                0                 12h
platform        my-release-signoz-otel-collector-7b7784c866-hr754           0/1     Init:0/1                0                 15h
platform        my-release-signoz-otel-collector-metrics-54d75b67c7-5ccx9   0/1     Init:0/1                0                 12h
platform        my-release-signoz-otel-collector-metrics-7f9fcd767-tqqxv    0/1     Init:0/1                0                 15h
platform        my-release-signoz-query-service-0                           0/1     Init:0/1                0                 10h
platform        my-release-signoz-schema-migrator-56769c434706-mzm2s        0/1     Init:0/1                0                 12h
platform        my-release-zookeeper-0                                      1/1     Running                 0                 15h
In the init container logs i see
wget: bad address 'my-release-signoz-query-service:8080'
waiting for query-service
wget: bad address 'my-release-signoz-query-service:8080'
waiting for query-service

---> init queryservice logs
wget: bad address 'my-release-clickhouse:8123'
waiting for clickhouseDB
wget: bad address 'my-release-clickhouse:8123'
waiting for clickhouseDB
I was thinking it to be a coredns issue coredns logs:
[INFO] 10.244.0.46:45441 - 56699 "AAAA IN my-release-signoz-query-service. udp 49 false 512" SERVFAIL qr,aa,rd,ra 49 0.000118854s
[INFO] 10.244.0.46:45441 - 31615 "A IN my-release-signoz-query-service. udp 49 false 512" SERVFAIL qr,aa,rd,ra 49 0.000033334s
[INFO] 10.244.0.56:54189 - 49629 "A IN my-release-clickhouse. udp 39 false 512" SERVFAIL qr,rd,ra 39 0.025739722s
[INFO] 10.244.0.56:54189 - 54743 "AAAA IN my-release-clickhouse. udp 39 false 512" SERVFAIL qr,rd,ra 39 0.025829025s
[INFO] 10.244.0.48:49914 - 36437 "AAAA IN my-release-clickhouse.platform-1.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd 161 0.000295586s
[INFO] 10.244.0.48:49914 - 43859 "A IN my-release-clickhouse.platform-1.svc.cluster.local. udp 68 false 512" NOERROR qr,aa,rd 134 0.000323042s
[INFO] 10.244.0.43:34472 - 30113 "AAAA IN my-release-signoz-otel-collector.platform-1.svc.cluster.local. udp 90 false 1232" NOERROR qr,aa,rd 172 0.000309202s
[INFO] 10.244.0.43:43698 - 24992 "A IN my-release-signoz-otel-collector.platform-1.svc.cluster.local. udp 90 false 1232" NOERROR qr,aa,rd 156 0.000315525s
Please let me know what the problem is and how to resolve it.