has anybody have experience with Signoz ingesting 500 gb per day ? what hardware you will need for that ?

you'll definitely want to use a clustered clickhouse with a multi-disk setup where you keep N days on SSD and the rest in S3 for the tail data. Most queries I find are the last 5 minutes and rarely do people look at the last week or month

sure, but is there a documentation for that ?

yeah there aren't extensive docs for this, the closest is this: https://signoz.io/docs/operate/clickhouse/distributed-clickhouse/

@Garegin Tamrazyan How long do you want to retain the data? Are you just interested in APM metrics or do you also want to retain the traces longer?

just APM, , need to retain for shorter period

@Garegin Tamrazyan so you want to just retain the APM metrics. So when you say 500 GB per day, will it just be APM or other data?

APM

Can you share more details on how your APM data will be 500 GB? Like how many approximate timeseries and samples and How did you get the estimate of 500 GB?

i see the data in our NewRelic

Do you also use tracing NewRelic provides or is it just APM metrics?

no tracing

Ok, I don't think you need a big resource if you just want to use APM metrics. Set up the SigNoz or decent machine with a disk capacity of 1TB (The APM metrics are stored in optimised storage format and storage should be very less) and 8 CPU machine with 16 GB RAM is a good start and then depending on the usage you can try to optimize. When you start sending the data make sure you get rid of the tracing pipeline https://github.com/SigNoz/signoz/blob/085cf34a49434e90f8ec50428a52017e14a985a3/deploy/docker/clickhouse-setup/otel-collector-config.yaml#L156-L159 so that you don't end up writing tracing data (You can always enable it back when needed).

i don’t need distributed setup for Clickhouse

In general, you should scale vertically as much as possible so the above suggestion is good enough. You mentioned you only want APM metrics so I don't think you need distributed setup yet. I'd start with the above setup and iterate on it if needed.

sure , thanks

Yes, the old EOL versions are not supported and 4 is really old.

so we have no option as a workaround, right ?

Yeah, I am not familiar with any workaround for that.