This message was deleted.
# supports
Slackbot
02/07/2024, 8:32 PMp
panduu Vital
02/08/2024, 4:26 AMif you are ok with the data loss
please follow this link
https://kb.altinity.com/altinity-kb-setup-and-maintenance/suspiciously-many-broken-parts/
👍 1
this 1
n
nitya-signoz
02/08/2024, 5:26 AM@Prashant Shahi should have some idea on this
p
Prashant Shahi
02/08/2024, 12:52 PMThe linked shared by @panduu Vital is the right one.
c
Carlos Martell
02/08/2024, 7:46 PMthat seems to have worked, thanks!
🙌 1
p
panduu Vital
02/20/2024, 5:26 AMwhy will this happen and how to avoid this from occurring ?
any idea ? @Prashant Shahi
c
Carlos Martell
02/26/2024, 5:33 PMno idea, it keeps happening on my end and I have to add
Copy code
/var/lib/clickhouse/flags/force_restore_datap
Prashant Shahi
02/27/2024, 3:50 AM@Srikanth Chekuri could you please look into this?
c
Carlos Martell
03/01/2024, 8:09 PMSo this keeps happening and I had to keep adding the flag, but I guess it now got bad enough that it actually corrupted something important and I'm unable to bring clickhouse back online.
Carlos Martell
03/01/2024, 8:11 PMI just went back and manually deleted anything flagged with "is broken and need manual correction" but I'm unsure why this keeps happening.
Carlos Martell
03/01/2024, 8:12 PMWe mostly run on Spot Nodes, so Kubernetes node replacements do happen relatively often, so I wonder if I need any new flags to make this more resilient? HA?
s
Srikanth Chekuri
03/02/2024, 2:10 AMRun your database on a node where node replacement do not occur relatively often.
➕ 1
c
Carlos Martell
04/01/2024, 7:19 AMso this was mostly solved by upping the default GP2 EBS volume to GP3 as it was due to the default volume provisioned by the helm chart wasn't providing enough IOPS 👍
🙌 1
26 Views