We recently upgraded our SigNoz deployment (from version 0.73.1 to 0.94.1) using the Helm chart, strictly following the official upgrade guide. After the upgrade, we faced a critical issue: all of our metrics data, alerts, and dashboards were lost. At first, we suspected this might be related to the ClickHouse version, but that was already patched at the time, so it wasn’t the cause. Upon further investigation, we found that the problem was due to the migration jobs not running successfully during the upgrade process. Unfortunately, we couldn’t identify the exact reason why these jobs failed. This failure led to the loss of historic metrics data and forced us to recreate all alerts and dashboards manually, which was both disruptive and time-consuming. I’d like to know if others upgrading to newer versions have experienced similar migration job failures, and if there are any recommended checks, logs, or steps we can take in future upgrades to ensure the migration runs smoothly and avoids data loss.

The better way would be to use the signoz upgrade guide and upgrade step by step instead of directly upgrading to the latest version