https://signoz.io logo
Join SlackCommunities
Powered by
After updating signoz helm chart from v0.51.0 to v...
# general
d
After updating signoz helm chart from v0.51.0 to v0.54.0, these 2 pods are stuck in Init state
Copy code
NAME                                                 READY   STATUS     RESTARTS      AGE
apm-clickhouse-operator-6fc85bd9b6-698hh             2/2     Running    0             12h
apm-signoz-alertmanager-0                            1/1     Running    0             12h
apm-signoz-frontend-979668b87-p64bz                  1/1     Running    0             12h
apm-signoz-otel-collector-679bc68c99-j5qjr           0/1     Init:0/1   0             11h
apm-signoz-otel-collector-7fd6bb45b8-qbk54           1/1     Running    0             35d
apm-signoz-otel-collector-7fd6bb45b8-xzrpv           1/1     Running    0             35d
apm-signoz-otel-collector-metrics-667b9fc959-84kdd   1/1     Running    0             35d
apm-signoz-otel-collector-metrics-ffc447dc5-nj9dt    0/1     Init:0/1   0             11h
apm-signoz-query-service-0                           1/1     Running    4 (12h ago)   12h
apm-zookeeper-0                                      1/1     Running    0             35d
chi-apm-clickhouse-cluster-0-0-0                     1/1     Running    0             12h
any idea how I should fix this?
@nitya-signoz can you help?
Copy code
{
  "level": "fatal",
  "timestamp": "2024-09-12T12:17:24.387Z",
  "caller": "signozschemamigrator/migrate.go:128",
  "msg": "Failed to run migrations",
  "component": "migrate cli",
  "error": "clickhouse migrate failed to run, error: Dirty database version 8. Fix and force version.",
  "stacktrace": "main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/runtime/proc.go:267"
}
I think this is the issue, the migrator is failing 
apm-signoz-schema-migrator-upgrade-whmtm
n
Please share the entire log
d
Copy code
❯ k logs -n platform --tail 100 -f apm-signoz-schema-migrator-upgrade-whmtm
Defaulted container "schema-migrator" out of: schema-migrator, apm-signoz-schema-migrator-init (init), apm-signoz-schema-migrator-ch-ready (init)
{"level":"info","timestamp":"2024-09-12T12:17:24.270Z","caller":"signozschemamigrator/migrate.go:89","msg":"Setting env var SIGNOZ_CLUSTER","component":"migrate cli","cluster-name":"cluster"}
{"level":"info","timestamp":"2024-09-12T12:17:24.270Z","caller":"signozschemamigrator/migrate.go:106","msg":"Successfully set env var SIGNOZ_CLUSTER ","component":"migrate cli","cluster-name":"cluster"}
{"level":"info","timestamp":"2024-09-12T12:17:24.270Z","caller":"signozschemamigrator/migrate.go:111","msg":"Setting env var SIGNOZ_REPLICATED","component":"migrate cli","replication":false}
{"level":"info","timestamp":"2024-09-12T12:17:24.280Z","caller":"migrationmanager/manager.go:76","msg":"Running migrations for all migrators","component":"migrationmanager"}
{"level":"info","timestamp":"2024-09-12T12:17:24.280Z","caller":"migrationmanager/manager.go:78","msg":"Running migrations for logs","component":"migrationmanager","migrator":"logs"}
{"level":"info","timestamp":"2024-09-12T12:17:24.348Z","caller":"migrationmanager/manager.go:78","msg":"Running migrations for metrics","component":"migrationmanager","migrator":"metrics"}
{"level":"error","timestamp":"2024-09-12T12:17:24.387Z","caller":"migrationmanager/manager.go:81","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"metrics","error":"clickhouse migrate failed to run, error: Dirty database version 8. Fix and force version.","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate|github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:81\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:126\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/runtime/proc.go:267"}
{"level":"fatal","timestamp":"2024-09-12T12:17:24.387Z","caller":"signozschemamigrator/migrate.go:128","msg":"Failed to run migrations","component":"migrate cli","error":"clickhouse migrate failed to run, error: Dirty database version 8. Fix and force version.","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/runtime/proc.go:267"}
Copy code
❯ k get jobs -n platform        
NAME                                 COMPLETIONS   DURATION   AGE
apm-signoz-schema-migrator-upgrade   0/1           4m40s      4m40s
this job is also not getting completed
Copy code
[2024-09-12 12:10:04] Waiting for job apm-signoz-schema-migrator-upgrade...
log from otel collector
n
@Srikanth Chekuri the metrics migration has failed, can you help here ?
d
I followed this document and did some manual changes, but still didn’t work https://signoz.io/docs/userguide/logs_troubleshooting/#v0530-docker-schema-migrator-dirty-database-version
n
Yeah, that doc was for logs, but here the metrics db is causing the issue, will wait for Srikanth to reply
s
Please share logs prior to this that contain the specific error
d
Copy code
{
  "level": "fatal",
  "timestamp": "2024-09-12T12:17:24.387Z",
  "caller": "signozschemamigrator/migrate.go:128",
  "msg": "Failed to run migrations",
  "component": "migrate cli",
  "error": "clickhouse migrate failed to run, error: Dirty database version 8. Fix and force version.",
  "stacktrace": "main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/runtime/proc.go:267"
}
this is the only relevant log I was able to find
s
There would be error logs when it failed for first time. Do you have them?
d
I can try to look for them, if you can tell me a way
s
Do you collect pod logs?
d
not of signoz
s
Then you don't have them. The 8 the version is very simple and should not cause any issues. Run following and try again
Copy code
delete from signoz_metrics.schema_migrations where version=8;
d
worked, thanks for the help!
11 Views
Open in Slack
PreviousNext
Powered by