error	clickhousetracesexporter/writer.go:419	Could not write a batch of spans to model table: {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "code: 252, message: Too many parts (3003 with average size of 3.22 MiB) in table 'signoz_traces.signoz_spans (10128c23-7659-4528-91e0-0979de9fe932)'. Merges are processing significantly slower than inserts"}

getting the above error continuously

Update your batch processor to send larger batches https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1715765813511469?thread_ts=1715752545.824699&cid=C01HWQ1R0BC

current timeout is 10s and batch size is 50k and still facing the same issue and my otel collector pods count is 10

Why do you have 10 otel collectors ?

there are around 200 services running and send traces to signoz

I guess you should start with less number of collector then scale vertically and then move to vertical scaling. Atleast for the collectors that are writing to clickhouse, this will ensure bigger batches are written to clickhouse instead of small batches from multiple collectors. @Prashant Shahi any more suggestions on this ?

ok @nitya-signoz Thanks ... @Prashant Shahi please suggest more if any.

Most likely (I am very certain) the ten collector is not really the issue. Please look at your disk metrics. What is the throughput/iops and the read/write latency. If the disk is at bottleneck then then merging become slower leading to too many parts.