Hi all, Im evaluating signoz hosted for some internal needs. Everything has been going fine until recently. Im on version 0.76.0. I think something may be wrong with my TTL on my data. I recently tried to clean up some sensitive log data (which worked fine) by doing some direct clickhouse queries (im not sure if this is related to my problem) and also setting a short TTL in the UI. Recently it seems that my data set for specific incoming metrics seems to go missing at about 2 weeks (all my dashboards for this particular app show no data on the day it goes missing but seems to start over at some time earlier in the day).
Unfortunately I didn't know how to check the TTL properly in clickhouse until today (after I followed the process for stuck TTL config). Another thing to note is that its not deleting all data. For instance, one metric in one of the time series tables only has 100 rows. And i've been collecting that data successfully for 2 months (at least). But there are 8B rows in that table. Also, FWIW, I have 8B rows in that table.
Is this something I've broken trying to manipulate materialized views? or is this something else. This is on kubernetes (EKS)