Hi, I have 2 problems with our Signoz that need help: 1. The traces retention period process is stuck while I change the setting, is there any way to force stop and reset the retention period? 2. I tried to update Signoz from 0.11.4 to 0.12.0 running using docker-compose.yaml file, but I got an error on otel-collector. "Error: cannot build pipelines: failed to create "clickhouselogsexporter" exporter, in pipeline "logs": cannot configure clickhouse logs exporter: failed to create database, err: code: 170, message: Requested cluster 'cluster' not found. Do I miss any steps for upgrading Signoz?

@Prashant Shahi should be able to help for point 2

thank you @Pranay, Hi @Prashant Shahi I appreciate if you could help.

1. The traces retention period process is stuck while I change the setting, is there any way to force stop and reset the retention period?

Time to enabling TTL depends on amount of data due to limitations of clickhouseDB. How long has been since the TTL operation stuck? If query-service container was not "restarted", TTL operation should either success/fail. If you query-service container was "restarted" after initiation of TTL operation, your checks might not be in-sync with the actual TTL being set in ClickHouse.

It's been a while, and I don't remember exactly how long, but I just notice the storage usage of ClickhouseDB getting bigger, and when I open the settings page, I saw the changes of the retention period still ongoing. I already restart all the container, but the process not stopped.

There are multiple changes in

docker-compose.yaml

like Zookeeper dependency,

clickhouse-cluster.xml

volume mounting, etc. Make sure the YAML is in-sync if not identical from the one in

develop

. https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/docker-compose.yaml

Hi @Prashant Shahi it's my bad, I forgot to add the clickhouse-cluster.xml. now it's successful to upgrade to v0.12.0. thank you.

@moronmon How long has it been since you update retention period? And how much traces data do you have?

I forgot exactly, I think it's passed 1 month. for now the data is almost 120GB

@moronmon TTL takes very long as data size becomes huge. For now you can delete TTL status entries which might have become stale after you restarted query service. Are you using docker or k8s?

I run it using docker, could you help how to delete the TTL?

Connect to query-service

docker exec -it query-service sh

Run the following:

# install sqlite
apk update
apk add sqlite

# open sqlite with signoz.db
sqlite3 /var/lib/signoz/signoz.db

# (sqlite shell) check existing ttl status
select * from ttl_status;

# (sqlite shell) delete all ttl status
DELETE from ttl_status where 1=1

# (sqlite shell) verify ttl status deletion
select * from ttl_status;

Thank you @Vishal Sharma it works. thank you @Prashant Shahi for your help

👍

Clickhouse is using 600GB+ and I have also deleted ttl_status from query service. Still I'm unable to update retention Days.