This message was deleted.

Slackbot

02/14/2024, 4:12 PM

This message was deleted.

✅ 1

Srikanth Chekuri

02/14/2024, 4:20 PM

Hello, This is probably happening because if the order of the pod spin up. Can you exec into clickhouse and run

DROP TABLE signoz_metrics.schema_migrations

Wesley Hartford

02/14/2024, 4:43 PM

Thanks, but that didn't seem to help:

Copy code

chi-signoz-clickhouse-cluster-0-0-0.chi-signoz-clickhouse-cluster-0-0.signoz.svc.cluster.local :) DROP TABLE signoz_metrics.schema_migrations

DROP TABLE signoz_metrics.schema_migrations

Query id: f76b3940-d558-4ef0-99eb-4286c4f0fc93

Ok.

0 rows in set. Elapsed: 0.002 sec.

Still getting the same issue:

Copy code

helm upgrade signoz signoz/signoz --namespace signoz --values values.yaml
coalesce.go:237: warning: skipped value for zookeeper.initContainers: Not a table.
Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
	* timed out waiting for the condition

And the pod logs:

Copy code

{"level":"info","timestamp":"2024-02-14T16:41:47.585Z","caller":"signozschemamigrator/migrate.go:83","msg":"Setting env var SIGNOZ_CLUSTER","component":"migrate cli","cluster-name":"cluster"}
{"level":"info","timestamp":"2024-02-14T16:41:47.586Z","caller":"signozschemamigrator/migrate.go:100","msg":"Successfully set env var SIGNOZ_CLUSTER ","component":"migrate cli","cluster-name":"cluster"}
{"level":"info","timestamp":"2024-02-14T16:41:47.591Z","caller":"migrationmanager/manager.go:75","msg":"Running migrations for all migrators","component":"migrationmanager"}
{"level":"info","timestamp":"2024-02-14T16:41:47.591Z","caller":"migrationmanager/manager.go:77","msg":"Running migrations for logs","component":"migrationmanager","migrator":"logs"}
{"level":"info","timestamp":"2024-02-14T16:41:47.717Z","caller":"migrationmanager/manager.go:77","msg":"Running migrations for metrics","component":"migrationmanager","migrator":"metrics"}
{"level":"error","timestamp":"2024-02-14T16:41:47.839Z","caller":"migrationmanager/manager.go:80","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"metrics","error":"clickhouse migrate failed to run, error: Dirty database version 7. Fix and force version.","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate|github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:80\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:108\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250"}
{"level":"fatal","timestamp":"2024-02-14T16:41:47.839Z","caller":"signozschemamigrator/migrate.go:110","msg":"Failed to run migrations","component":"migrate cli","error":"clickhouse migrate failed to run, error: Dirty database version 7. Fix and force version.","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:110\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.14/x64/src/runtime/proc.go:250"}

Srikanth Chekuri

02/14/2024, 4:44 PM

Can you share the clickhouse server version?

Srikanth Chekuri

02/14/2024, 4:44 PM

Do you have more than one clickhouse shard?

Wesley Hartford

02/14/2024, 4:45 PM

I installed yesterday from the helm chart with no major customizations.

Wesley Hartford

02/14/2024, 4:45 PM

Clickhouse image is

<http://docker.io/clickhouse/clickhouse-server:23.11.1-alpine|docker.io/clickhouse/clickhouse-server:23.11.1-alpine>

Srikanth Chekuri

02/14/2024, 4:46 PM

The clickhouse version should be updated to

24.1.2-alpine

Srikanth Chekuri

02/14/2024, 4:46 PM

How did you upgrade from v0.38.2 to v0.39.0

Wesley Hartford

02/14/2024, 4:47 PM

helm

Srikanth Chekuri

02/14/2024, 4:47 PM

Charts version?

Wesley Hartford

02/14/2024, 4:48 PM

Copy code

helm list
NAME  	NAMESPACE	REVISION	UPDATED                               	STATUS	CHART        	APP VERSION
signoz	signoz   	5       	2024-02-14 16:38:45.88264205 +0000 UTC	failed	signoz-0.35.0	0.39.0

Wesley Hartford

02/14/2024, 4:48 PM

Helm is saying app version 0.39.0, but the signoz UI says the current version is 0.38.2

Srikanth Chekuri

02/14/2024, 11:38 PM

The upgrade must have failed.

Pedro Carvalho

02/15/2024, 10:19 AM

I had the same issue right now, trying to upgrade up from 0.36.2. Luckily, it was the test database. I can install chart version 0.39.0 if I do it from scratch, but I'm afraid the upgrade was not a happy path.

Prashant Shahi

02/15/2024, 11:21 AM

Thanks for reporting! yes, there seems to be issue with our schema migrator for the upgrade flow. schema migrator pod is stuck in invalid state however, that shouldn't affect any other running pods.

Prashant Shahi

02/15/2024, 11:21 AM

We are working on the fix for this.

Prashant Shahi

02/15/2024, 1:14 PM

Resolved by this PR: https://github.com/SigNoz/charts/pull/406

Prashant Shahi

02/15/2024, 1:30 PM

Fix is out: signoz-0.35.1

Prashant Shahi

02/15/2024, 1:35 PM

If you had encountered dirty migration issue, you would need to cleanup

schema_migrations

tables prior to the upgrade. One-liner command:

Copy code

kubectl exec -it -n platform chi-my-release-clickhouse-cluster-0-0-0 -- sh -c "clickhouse-client --multiquery 'DROP TABLE signoz_traces.schema_migrations ON CLUSTER cluster; DROP TABLE signoz_metrics.schema_migrations ON CLUSTER cluster; DROP TABLE signoz_logs.schema_migrations ON CLUSTER cluster'"

Prashant Shahi

02/15/2024, 1:36 PM

Update

my-release

and

platform

with your release name and namespace.

Wesley Hartford

02/15/2024, 3:47 PM

Thank you, this resolved my issue.

🙌 1

Prashant Shahi

02/15/2024, 3:54 PM

Glad to know that the issue is resolved 🎉

507 Views

Open in Slack

Previous Next

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.