Hello! We're evaluating SigNoz in Kubernetes with the Helm chart provided in the SigNoz website. Our Clickhouse pod restarted last night due to a node restart, but it's been stuck in a loop since then. Any ideas on how to resolve this, and how to stop it from happening in the future?

if you are ok with the data loss please follow this link https://kb.altinity.com/altinity-kb-setup-and-maintenance/suspiciously-many-broken-parts/

@Prashant Shahi should have some idea on this

The linked shared by @panduu Vital is the right one.

that seems to have worked, thanks!

why will this happen and how to avoid this from occurring ? any idea ? @Prashant Shahi

no idea, it keeps happening on my end and I have to add

/var/lib/clickhouse/flags/force_restore_data

every couple of days

@Srikanth Chekuri could you please look into this?

So this keeps happening and I had to keep adding the flag, but I guess it now got bad enough that it actually corrupted something important and I'm unable to bring clickhouse back online.

Run your database on a node where node replacement do not occur relatively often.

so this was mostly solved by upping the default GP2 EBS volume to GP3 as it was due to the default volume provisioned by the helm chart wasn't providing enough IOPS 👍