This message was deleted.

I'm going to look with the team at what's the max scale we're being used at @Christian Theilemann, will look at what data we can share. In general, SigNoz is using Clickhouse as its datastore, they've written a bit about optimizing performance for log storage, I should ping Dale to see if he ever wrote the promised follow-up

Hi @Christian Theilemann , it is difficult to do exact comparison but we have tested with 5TB/day and 1M active timeseries per minute.

have you done something like search/query for a random string (like

foo

) on the entire data (and not just a pre-filtered dataset or specific column) of the last 2 days - which would, in the worst case - scan about 10TB of data? I know that things like that are very problematic in loki and some other systems (in loki this would almost certainly time out), but in humio they optimize queries like these via bloom filters.

bloom filters are already in place. https://github.com/SigNoz/signoz-otel-collector/blob/main/migrationmanager/migrators/logs/migrations/000001_init_db.up.sql#L19 https://clickhouse.com/docs/en/optimize/skipping-indexes#bloom-filter-types