Installed clickhouse with SigNoz chart. Is there a way to limit clickhouse data persistency? Now clickhouse PV is 100% full which of course brings pain.

@Juha Patrikainen Have you checked this - https://signoz.io/docs/userguide/retention-period/ Though updating retention period is a resource heavy operation - so ensure you have some memory and cpu capacity available before you run this, also this might take some time

Aah, they are empty at the moment

You may also want to upgrade to v0.15.0 where we have upgraded the logic for updating the retention period to make it more robust. In earlier versions, it sometimes causes silent fail issues

@Pranay Would ne nice to be configure retention via Helm chart values. Seems that is not possible at the moment.

Yeah, thats a good idea actually. Can you create an issue for this - https://github.com/SigNoz/signoz/issues/new/choose

Sure, no prob 👍

It would require changes in query-service to support that. With the help of either environment variables or flags.

Setting retention times did not do anything, I guess it could not complete it because PV was already full.

when PV is full, it is recommended to increase PV size prior to updating retention times,

@Prashant Shahi @Pranay I deleted clickhouse PV and created it again, but now SigNoz otel collector gives error on start:

...
2023-02-09T08:12:09.537Z	info	clickhousetracesexporter/clickhouse_factory.go:142	View does not exist, skipping patch	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "table": "dependency_graph_minutes_db_calls_mv"}
2023-02-09T08:12:09.537Z	info	clickhousetracesexporter/clickhouse_factory.go:116	Running migrations from path: 	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "test": "/migrations"}
2023-02-09T08:12:09.546Z	info	clickhousetracesexporter/clickhouse_factory.go:128	Clickhouse Migrate finished	{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "Dirty database version 13. Fix and force version."}
Error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": code: 60, message: Table signoz_traces.distributed_signoz_index_v2 doesn't exist
2023/02/09 08:12:09 application run finished with error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": code: 60, message: Table signoz_traces.distributed_signoz_index_v2 doesn't exist

@Juha Patrikainen I see. That issue could have been resolved by restarting OtelCollector container.

Mostly for people who come across this thread in a search, this is what I do when clickhouse is full; delete all the logs:

kubectl exec chi-signoz-clickhouse-cluster-0-0-0 -n signoz -- clickhouse client -d signoz_logs -q 'truncate table logs'

thanks for sharing @Wesley Hartford