Hey guys recently we moved our signoz instance from docker-compose to kubernetes for better scalability and reliability - we are facing issues where the signoz instance is crashing with the otel-collector instance queue getting full and the otel-collector runs out of memory. We have 10 replicas of the otel-collector and still facing issues - is this a problem with the clickhouse database not being able to handle the load. If this the case how do I scale the database to handle this load

are you adding batch processors before sending insert queries to clickhouse ? if yes try increasing the batch size and the timeout

by default doesn't the otel-collector do batch processing? how do I increase the batch size and the timeout? Srikanth had suggested that wouldn't be the problem

Can you try increasing the batch timeout to a higher number, clickhouse is complaining because too many small insert requests are coming up. @Srikanth Chekuri @Prashant Shahi any more ideas ?

I've changed the timeout to 10s now and the batch size is 50k which I think should be good enough

For anyone reading this, it was the disk causing slower merges.