https://signoz.io
Join Slack
<@U03BTNACHNX> or others, how does the retention work in SigNoz? :thinking_face: is Clickhouse expec...
# support
p
@nitya-signoz or others, how does the retention work in SigNoz? 🤔 is Clickhouse expected to automatically optimize the tables periodically, or does SigNoz perform some kind of background job which triggers an optimize call? I am asking because the adjusted retention doesn't seem to take effect on our system
we managed to clean up data manually by connecting to the DB using 
clickhouse-client
, something like this:
Copy code
SELECT count(*)
FROM signoz_spans

Query id: c6949b98-39b1-4517-a844-38cf42b9af03

   ┌──count()─┐
1. │ 18280880 │ -- 18.28 million
   └──────────┘

1 row in set. Elapsed: 0.003 sec. 

clickhouse :) OPTIMIZE TABLE signoz_spans FINAL

OPTIMIZE TABLE signoz_spans FINAL

Query id: 20c96a08-9583-4172-9b78-a44db1aa808d

Ok.

0 rows in set. Elapsed: 213.808 sec. 

clickhouse :) select count(*) from signoz_spans

SELECT count(*)
FROM signoz_spans

Query id: 314776ce-3bd3-4c1f-9478-f5b341a74629

   ┌──count()─┐
1. │ 17510478 │ -- 17.51 million
   └──────────┘

1 row in set. Elapsed: 0.002 sec.
After running the 
OPTIMIZE
command, we could see in the UI that the retention period hade come into effect; old data has been removed now. Why doesn't this happen automatically for us?
If there's a background job running for this, how often does it run?
n
When you update retention settings, the changes are applied to the new data that is ingested. Old data will have the old retention settings. This is done because updating the retention setttings of existing data is an resource intensive task.
p
ok, big thanks for clarifying that! 🙇
would it even be worth adding to the retention-period SigNoz docs? 🤔 might not be obvious to everyone not familiar with the Clickhouse DB from before.
n
Ahh right, thanks for bringing it up. Will update the docs.
4 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by