Getting the below error once I configure "move to ...
# support
s
Getting the below error once I configure "move to s3" for traces.
2023.11.06 05:10:51.620992 [ 340 ] {968d81b9-a08d-4114-bda2-17458fc94cf0} <Error> TCPHandler: Code: 341. DB::Exception: There was an error on [chi-uat-signoz-release-clickhouse-cluster-0-0:9000]: Code: 341. DB::Exception: Exception happened during execution of mutation 'mutation_2386.txt' with part '20231101_1235_1383_11_2385' reason: 'Code: 241. DB::Exception: Memory limit (total) exceeded: would use 4.07 GiB (attempt to allocate chunk of 5147400 bytes), maximum: 3.60 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero.: While executing TTL. (MEMORY_LIMIT_EXCEEDED) (version 23.7.3.14 (official build))'. This error maybe retryable or not. In case of unretryable error, mutation can be killed with KILL MUTATION query. (UNFINISHED) (version 23.7.3.14 (official build)). (UNFINISHED), Stack trace (when copying this message, always include the lines below):
For logs and metrics working fine
Screenshot 2023-11-06 at 10.57.58 AM.png
@nitya-signoz @Prashant Shahi Any idea?
I recently upgraded to latest version as well. I also found that few users experienced the same in past.
p
@Vishal Sharma can you please look into this?
s
@Vishal Sharma Below are the clickhouse resource details.
clickhouse: resources: requests: cpu: 100m memory: 200Mi limits: cpu: 2000m memory: 4Gi
v
@sudhanshu dev Can you temporarily increase memory of clickhouse? When you configure retention changes then it tries to move/delete lot of data and it can cause such memory errors.
s
Yes I just deployed after increasing it
Let me test and let u know
Ya it worked now
v
@sudhanshu dev Can you please share how many spans are there?
s
After increasing
v
How much did you increase?
s
How can I check span?
Helm
v
Check usage explorer for 1 day data
s
Increase the resources
In CH deployment
v
Also how much did you increase?
s
By 2 Gb
Earlier it was 4
Now 6
And total span is 331087
Screenshot 2023-11-06 at 11.32.48 AM.png
v
Thanks for sharing
s
Thanks to resolve the issue