Hi team, we setup on our EKS cluster with 2 clickhouse pod running. And currently we deleted the pvc from 1 pod and tried to rebuild it. the clickhouse pod running on it didn’t have the correct database tables. How can we manually create those missing tables?

Run the schema migrator against the clickhouse instance https://github.com/SigNoz/signoz-otel-collector/blob/main/cmd/signozschemamigrator/main.go. Use the relevant args (same as charts).

go run cmd/signozschemamigrator/main.go sync --dsn=<tcp://host-name-here>:port-here --replication=true|false based on charts config --cluster-name=cluster --up=

Thanks for guide, but your solution will not work because we have 2 replica servers and if try to rebuild one of the replica it will not allow us to create a table that exist on another one as it’s cluster mode. We followed this to recover it: https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication#recovery-after-complete-data-loss

if try to rebuild one of the replica it will not allow us to create a table that exist on another one as it’s cluster mode

care to explain specifically why?

the migration code is trying to create a table on the new replica DB. the create table sql always have the

ON CLUSTER

statement which will try create table on other replica which already exist. And then failed. Maybe you should expose a command line arguments to control if it should include this ON CLUSTER statement or not

> the create table sql always have the

ON CLUSTER

statement which will try create table on other replica which already exist. Did you actually run into this error or are you saying you would get such error? Because every command has IF NOT EXISTS and given that i don't see how that would throw error https://github.com/SigNoz/signoz-otel-collector/blob/cc3d7be214533690eb034410ecdf0[…]a6/cmd/signozschemamigrator/schema_migrator/table_operations.go

I see the error about create schema_migration table.

Even those are managed by same thing ex: https://github.com/SigNoz/signoz-otel-collector/blob/cc3d7be214533690eb034410ecdf0[…]ozschemamigrator/schema_migrator/schema_migrations_v2_tables.go. Can you please share full error?

I used to use the old migrator image

signoz/signoz-schema-migrator:0.102.10

and error table is the old schema_migrations table and not v2.

Ok, the link I pointed to is the latest migrator and it should work fine.

maybe the latest migrator works but I didn’t try it.

Yes, that's the source of confusion. I read your comment that it doesn't work on the migrator linked, and it is as if you already tested and are claiming it doesn't work. It should work without any issues (and let us know if doesn't). Please upgrade. The new migrator is built to solve many issues with the old migrator.

Thanks. we spend the whole day yesterday tried other solutions like I post above. And also met issues about zookeeper. not sure what’s the root cause but we found that we have to cleanup the zookeeper metadata before we run the new replica and then have to manually rebuild the zookeeper metadata