Slackbot
12/05/2022, 1:53 PMPranay
Ankit Nayan
Fernando Bellincanta
12/05/2022, 1:57 PMAnkit Nayan
Ankit Nayan
Fernando Bellincanta
12/05/2022, 1:58 PMFernando Bellincanta
12/05/2022, 1:58 PMAnkit Nayan
Ankit Nayan
Ankit Nayan
Fernando Bellincanta
12/05/2022, 2:02 PMAnkit Nayan
Ankit Nayan
Ankit Nayan
Fernando Bellincanta
12/05/2022, 2:04 PMAnkit Nayan
Fernando Bellincanta
12/05/2022, 2:05 PMCode: 252. DB::Exception: Too many parts (318). Merges are processing significantly slower than inserts. (TOO_MANY_PARTS), Stack trace (when copying this message, always include the lines below):
this happensAnkit Nayan
Fernando Bellincanta
12/05/2022, 2:06 PMAnkit Nayan
Fernando Bellincanta
12/05/2022, 2:07 PMAnkit Nayan
Ankit Nayan
Ankit Nayan
Code: 252. DB::Exception: Too many parts (318). Merges are processing significantly slower than inserts. (TOO_MANY_PARTS), Stack trace (when copying this message, always include the lines below):
the above error is probably due to too many inserts per secAnkit Nayan
Fernando Bellincanta
12/05/2022, 2:12 PMFernando Bellincanta
12/05/2022, 2:13 PMAnkit Nayan
Fernando Bellincanta
12/05/2022, 2:16 PMAnkit Nayan
Ankit Nayan
Fernando Bellincanta
12/05/2022, 2:17 PMFernando Bellincanta
12/05/2022, 2:17 PMAnkit Nayan
Ankit Nayan
Fernando Bellincanta
12/05/2022, 2:18 PMAnkit Nayan
Fernando Bellincanta
12/05/2022, 2:19 PMFernando Bellincanta
12/05/2022, 2:19 PM<Information> signoz_metrics.time_series_v2 (97ae47da-a264-4e46-8a1b-a97c6fc892f9): Delaying inserting block by 4.168693834703354 ms. because there are 180 parts
should I tune something?Ankit Nayan
Fernando Bellincanta
12/05/2022, 2:22 PM<Error> executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket (10.252.1.5:39574). (NETWORK_ERROR) (version 22.8.8.3 (official build)) (from 10.252.1.5:39574) (in query: SELECT metric_name, fingerprint, timestamp_ms, value FROM signoz_metrics.samples_v2 WHERE metric_name = 'system_cpu_time' AND fingerprint IN (SELECT DISTINCT fingerprint FROM signoz_metrics.time_series_v2 WHERE metric_name = 'system_cpu_time' AND JSONExtractString(labels, 'host_name') = 'backyardbrands-us-east-2' AND JSONExtractString(labels, 'state') = 'idle' AND JSONExtractString(labels, '__name__') = 'system_cpu_time') AND timestamp_ms >= 1670249736000 AND timestamp_ms <= 1670250036000 ORDER BY fingerprint, timestamp_ms;), Stack trace (when copying this message, always include the lines below):
Ankit Nayan
Srikanth Chekuri
12/05/2022, 2:25 PMFernando Bellincanta
12/05/2022, 2:26 PMSrikanth Chekuri
12/05/2022, 2:26 PMSrikanth Chekuri
12/05/2022, 2:27 PMFernando Bellincanta
12/05/2022, 2:27 PMFernando Bellincanta
12/05/2022, 2:28 PMSrikanth Chekuri
12/05/2022, 2:28 PMFernando Bellincanta
12/05/2022, 2:28 PMresources:
requests:
cpu: 1
memory: 2Gi
Srikanth Chekuri
12/05/2022, 2:28 PMSrikanth Chekuri
12/05/2022, 2:29 PMAnkit Nayan
Fernando Bellincanta
12/05/2022, 2:29 PMFernando Bellincanta
12/05/2022, 2:30 PMFernando Bellincanta
12/05/2022, 2:30 PMSrikanth Chekuri
12/05/2022, 2:31 PMAnkit Nayan
resources
has 2 parts requests and limits. Does the autoscaler work on the requested resource?Ankit Nayan
Fernando Bellincanta
12/05/2022, 2:33 PMLucas Carlos
12/05/2022, 2:33 PMLucas Carlos
12/05/2022, 2:33 PMAnkit Nayan
Lucas Carlos
12/05/2022, 2:34 PMLucas Carlos
12/05/2022, 2:34 PMLucas Carlos
12/05/2022, 2:35 PMAnkit Nayan
Srikanth Chekuri
12/05/2022, 2:37 PMPrashant Shahi
12/05/2022, 2:39 PM@Prashant Shahi where is the equivalent setting at helm chart for https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/clickhouse-config.xml#LL273C6-L273C28It would be this:
clickhouse:
profile:
default/max_concurrent_queries: 200
Lucas Carlos
12/05/2022, 2:42 PMAnkit Nayan
Srikanth Chekuri
12/05/2022, 2:46 PMwe have set it to 1kThis is going to result in adverse effects overall. The general rule of thumb for ClickHouse is a few large batch inserts over multiple small inserts. The collector with 2/4CPUs and 4/8GB memory will be able to take your workload and not overload the DB.