Hello, I have question regarding Signoz docker-compose setup (clickhouse) I have 1hour retention for logs/traces/metrics But my storage usage keeps increasing till it hits limits. Retention works because I can see that old logs/metrics/traces are missing (after 1 hour) But something is still taking up disk space. I investigated and it seems like table signoz_metrics.samples_v2 keeps increasing while other tables looks okay. Can you please help me with this? How to prevent signoz from taking up all my disk space? thank you

yeah the default retention is 30 days, but once you update it from the UI it will be updated. Also the change you make in the UI will only be for the newly ingested data, so if you have data for last 5 days which had old retention policy then will be deleted after 30 days, this is done as updating old data is a resource intensive process. on a side note please give more than 4 hour retention, clickhouse does merges every 4 hour. https://clickhouse.com/docs/en/operations/settings/merge-tree-settings#merge_with_ttl_timeout