https://signoz.io logo
#support
Title
# support
m

Mario Giampietri

10/24/2023, 10:11 AM
Hi there. I was migrating a SigNoz installation (k8s / eks via helm) from 0.26.0 to 0.28.0 In a development environment everything went smooth at first attempt running the update via helm. On a production environment (where there's way more data) I get the error below on the migration pod. Being a timeout, I allowed the migration pod to restart and continue but I have the impression it loops and won't get to the bottom of it. I also tried with the instructions as described in the migration guide, that seems to work, but at the subsequent
helm update
I'm back to the issue below. Is there a way to increase that
distributed_ddl_task_timeout
? It's not obvious to me where and how should be set.
Copy code
{"level":"error","timestamp":"2023-10-24T09:53:26.669Z","caller":"migrationmanager/manager.go:79","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"logs","error":"failed to create database, err: code: 159, message: Watching task /clickhouse/signoz-clickhouse/task_queue/ddl/query-0000000251 is executing longer than distributed_ddl_task_timeout (=180) seconds. There are 1 unfinished hosts (0 of them are currently active), they are going to execute the query in background","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate|github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:79\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:83\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.10/x64/src/runtime/proc.go:250"}
{"level":"fatal","timestamp":"2023-10-24T09:53:26.972Z","caller":"signozschemamigrator/migrate.go:85","msg":"Failed to run migrations","component":"migrate cli","error":"failed to create database, err: code: 159, message: Watching task /clickhouse/signoz-clickhouse/task_queue/ddl/query-0000000251 is executing longer than distributed_ddl_task_timeout (=180) seconds. There are 1 unfinished hosts (0 of them are currently active), they are going to execute the query in background","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:85\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.10/x64/src/runtime/proc.go:250"}
Another issue, which started happening after the attempted migrations (see my message above): I can't see my dashboards anymore: • dashboard page returns
422 / Unprocessable entity
, and visually it appears as if there are no dashboards; • as I don't mind recreating them (I use some among the ones available here), if I try to recreate one the operation fails and the page goes 404; The impression is that dashboards are actually there but something is misaligned. Is there a way to reset/clean the dashboard storage?
s

Srikanth Chekuri

10/27/2023, 11:54 AM
Is this still an issue or did you manager to get the migrations work? Ideally, The dashboard should work if the migrations are successful.
m

Mario Giampietri

11/01/2023, 9:37 AM
(I've been out of work a few days) today it worked and dashboard are up again. I attempted several times last week always targeting v0.28.0, was it edited in the meantime?