Title
s
sarthak gupta
05/03/2023, 9:46 AMhi , is there a way or some configuration tag in clickhouse-storage.xml to write a rule to move data on s3 based on time period eg : 1 month and empty the clickhouse tables so that it can ingest data periodically without any space issues .
v
Vishal Sharma
05/03/2023, 11:26 AMYou can set retention based on time period like move to s3 by following this doc: https://signoz.io/docs/userguide/retention-period/#configuring-cold-storage---amazon-s3
Even in s3 the performance drop is around 30% so you can access s3 data like normal data (with some performance drop)
s
sarthak gupta
05/03/2023, 11:54 AMthanks vishal for the response , so you are saying , suppose if i retain after 7 days data into s3 by configuring then , after that clickhouse will automatically query old data from s3 , actually i also had that doubt that if i need to write some code/query using s3 table engine to get visuals from data stored in s3 .
v
Vishal Sharma
05/03/2023, 11:56 AMso you are saying , suppose if i retain after 7 days data into s3 by configuring then , after that clickhouse will automatically query old data from s3Yes
s
sarthak gupta
05/03/2023, 11:56 AMthanks bro for clearification
v
Vishal Sharma
05/03/2023, 11:57 AMactually i also had that doubt that if i need to write some code/query using s3 table engine to get visuals from data stored in s3No, SigNoz handles everything for you
s
sarthak gupta
05/03/2023, 11:58 AMactually i want to deeply understand about clickhouse working and storage pipeline , that's why i m exploring so that i can handle all issues whatever comes while going into production experiment .
so acc to you , clickhouse official documentation is enough right ?
v
Vishal Sharma
05/03/2023, 11:59 AMYes, you can follow this guide to connect to clickhouse: https://signoz.io/docs/operate/clickhouse/connect-to-clickhouse/
s
sarthak gupta
05/03/2023, 11:59 AMokay
one more thing , i think i should discuss is that , i m thinking to use ec2 over elastic load balancing for replication but will be keeping vertically scalable rather than sharding after estimating the current scale i need to handle, will it be the good option in terms of best practices .
v
Vishal Sharma
05/03/2023, 12:30 PMyes it should be fine
s
sarthak gupta
05/03/2023, 12:57 PMyeah , actually i m deploying it into production as a experiment for application performance analysis
so i was exploring some open source apm , i found this suitable in terms of features , usecases coverage , technical architecture wise
so i started reading more about observebility , opentelem and signoz in detail so that i can setup completely infrastructure in scalable way so as to remove dependency on saas based apm's
now i understood its working , deployed on stage and now trying to get proper grip on all components including best deployment stretagy