Why is the

docker-compose.yml

not updated anymore and still stuck in 0.49.1 when the newest version should be 0.53?

Please use the main branch for all stable things. Its updated here https://github.com/SigNoz/signoz/blob/main/deploy/docker/clickhouse-setup/docker-compose.yaml develop might have missed those changes cc @Prashant Shahi

@nitya-signoz Thanks, I'm not even sure why I ended up on the develop branch, was this the default branch for deployments before?

The

develop

branch is the default for the repository, you might have cloned the repo directly from the github. On the docs page we specify the main branch https://signoz.io/docs/install/docker/#install-signoz-using-the-install-script

Now the

otel-collector-migrator

broke 😞 Known issue? Didn't find anything in Github issues.

{"level":"error","timestamp":"2024-09-03T07:37:23.873Z","caller":"migrationmanager/manager.go:81","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"logs","error":"clickhouse migrate failed to run, error: migration failed in line 0: CREATE TABLE IF NOT EXISTS  signoz_logs.logs_v2_resource ON CLUSTER cluster\n(\n    `labels` String CODEC(ZSTD(5)),\n    `fingerprint` String CODEC(ZSTD(1)),\n    `seen_at_ts_bucket_start` Int64 CODEC(Delta(8), ZSTD(1)),\n    INDEX idx_labels lower(labels) TYPE ngrambf_v1(4, 1024, 3, 0) GRANULARITY 1\n)\nENGINE = ReplicatedReplacingMergeTree\nPARTITION BY toDate(seen_at_ts_bucket_start / 1000)\nORDER BY (labels, fingerprint, seen_at_ts_bucket_start)\nTTL toDateTime(seen_at_ts_bucket_start) + INTERVAL 1296000 SECOND + INTERVAL 1800 SECOND DELETE\nSETTINGS ttl_only_drop_parts = 1, index_granularity = 8192; (details: code: 139, message: There was an error on [clickhouse:9000]: Code: 139. DB::Exception: No macro 'shard' in config while processing substitutions in '/clickhouse/tables/{uuid}/{shard}' at '27' or macro is not supported here. (NO_ELEMENTS_IN_CONFIG) (version 24.1.2.5 (official build)))","stacktrace":"github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:81\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:126\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/runtime/proc.go:267"}
{"level":"fatal","timestamp":"2024-09-03T07:37:23.873Z","caller":"signozschemamigrator/migrate.go:128","msg":"Failed to run migrations","component":"migrate cli","error":"clickhouse migrate failed to run, error: migration failed in line 0: CREATE TABLE IF NOT EXISTS  signoz_logs.logs_v2_resource ON CLUSTER cluster\n(\n    `labels` String CODEC(ZSTD(5)),\n    `fingerprint` String CODEC(ZSTD(1)),\n    `seen_at_ts_bucket_start` Int64 CODEC(Delta(8), ZSTD(1)),\n    INDEX idx_labels lower(labels) TYPE ngrambf_v1(4, 1024, 3, 0) GRANULARITY 1\n)\nENGINE = ReplicatedReplacingMergeTree\nPARTITION BY toDate(seen_at_ts_bucket_start / 1000)\nORDER BY (labels, fingerprint, seen_at_ts_bucket_start)\nTTL toDateTime(seen_at_ts_bucket_start) + INTERVAL 1296000 SECOND + INTERVAL 1800 SECOND DELETE\nSETTINGS ttl_only_drop_parts = 1, index_granularity = 8192; (details: code: 139, message: There was an error on [clickhouse:9000]: Code: 139. DB::Exception: No macro 'shard' in config while processing substitutions in '/clickhouse/tables/{uuid}/{shard}' at '27' or macro is not supported here. (NO_ELEMENTS_IN_CONFIG) (version 24.1.2.5 (official build)))","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/runtime/proc.go:267"}
{"level":"info","timestamp":"2024-09-03T07:39:26.408Z","caller":"signozschemamigrator/migrate.go:89","msg":"Setting env var SIGNOZ_CLUSTER","component":"migrate cli","cluster-name":"cluster"}
{"level":"info","timestamp":"2024-09-03T07:39:26.408Z","caller":"signozschemamigrator/migrate.go:106","msg":"Successfully set env var SIGNOZ_CLUSTER ","component":"migrate cli","cluster-name":"cluster"}
{"level":"info","timestamp":"2024-09-03T07:39:26.408Z","caller":"signozschemamigrator/migrate.go:111","msg":"Setting env var SIGNOZ_REPLICATED","component":"migrate cli","replication":false}
{"level":"info","timestamp":"2024-09-03T07:39:26.415Z","caller":"migrationmanager/manager.go:76","msg":"Running migrations for all migrators","component":"migrationmanager"}
{"level":"info","timestamp":"2024-09-03T07:39:26.415Z","caller":"migrationmanager/manager.go:78","msg":"Running migrations for logs","component":"migrationmanager","migrator":"logs"}
{"level":"error","timestamp":"2024-09-03T07:39:26.496Z","caller":"migrationmanager/manager.go:81","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"logs","error":"clickhouse migrate failed to run, error: Dirty database version 14. Fix and force version.","stacktrace":"github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:81\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:126\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/runtime/proc.go:267"}
{"level":"fatal","timestamp":"2024-09-03T07:39:26.496Z","caller":"signozschemamigrator/migrate.go:128","msg":"Failed to run migrations","component":"migrate cli","error":"clickhouse migrate failed to run, error: Dirty database version 14. Fix and force version.","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.13/x64/src/runtime/proc.go:267"}

Can you check if your clickhouse container restarted? and is running

It was running, yes. I moved the Clickhouse data to a backup location, shut all containers down and run

./install.sh

again.

Did it restart ?

Yep

Cool then, Ideally running the migration again after clickhouse restart would have solved the issue.

Tried that actually, maybe didn't shut it down completely before 🤔

So a few more commands you will have to run, I will update the docs on the site for this version. But it’s similar to this https://signoz.io/docs/userguide/logs_troubleshooting/#schema-migrator-dirty-database-version

Ah nice, good documentation