Hi there I was migrating a SigNoz installation k8s eks via h

Question

Hi there I was migrating a SigNoz installation k8s eks via helm from 0 26 0 to 0 28 0 In a development environment everything went smooth at first attempt running the update via helm On a production environment where there s way more data I get the error below on the migration pod Being a timeout I allowed the migration pod to restart and continue but I have the impression it loops and won t get to the bottom of it I also tried with the instructions as described in the that seems to work but at the subsequent `helm update` I m back to the issue below Is there a way to increase that `distributed ddl task timeout` It s not obvious to me where and how should be set ``` level error timestamp 2023 10 24T09 53 26 669Z caller migrationmanager manager go 79 msg Failed to run migrations for migrator component migrationmanager migrator logs error failed to create database err code 159 message Watching task clickhouse signoz clickhouse task queue ddl query 0000000251 is executing longer than distributed ddl task timeout =180 seconds There are 1 unfinished hosts 0 of them are currently active they are going to execute the query in background stacktrace n t home runner work signoz otel collector signoz otel collector migrationmanager manager go 79 nmain main n t home runner work signoz otel collector signoz otel collector cmd signozschemamigrator migrate go 83 nruntime main n t opt hostedtoolcache go 1 20 10 x64 src runtime proc go 250 level fatal timestamp 2023 10 24T09 53 26 972Z caller signozschemamigrator migrate go 85 msg Failed to run migrations component migrate cli error failed to create database err code 159 message Watching task clickhouse signoz clickhouse task queue ddl query 0000000251 is executing longer than distributed ddl task timeout =180 seconds There are 1 unfinished hosts 0 of them are currently active they are going to execute the query in background stacktrace main main n t home runner work signoz otel collector signoz otel collector cmd signozschemamigrator migrate go 85 nruntime main n t opt hostedtoolcache go 1 20 10 x64 src runtime proc go 250 ```

Mario Giampietri · Answer

Another issue which started happening after the attempted migrations see my message above I can t see my dashboards anymore dashboard page returns `422 Unprocessable entity` and visually it appears as if there are no dashboards as I don t mind recreating them I use some among the ones available <https github com SigNoz dashboards blob main k8s infra metrics kubernetes metrics json|here> if I try to recreate one the operation fails and the page goes 404 The impression is that dashboards are actually there but something is misaligned Is there a way to reset clean the dashboard storage

Srikanth Chekuri · Answer

Is this still an issue or did you manager to get the migrations work Ideally The dashboard should work if the migrations are successful

Mario Giampietri · Answer

I ve been out of work a few days today it worked and dashboard are up again I attempted several times last week always targeting v0 28 0 was it edited in the meantime