https://signoz.io logo
#support
Title
# support
m

Muhammed Hussein Karimi

02/19/2023, 8:22 AM
Hi, I have increased the size of ClickHouse, but some of the charts are not loading correctly, is there any extra step todo?
These are the possible states, Clickhouse layout is 1 replica and 4 shards
Also for other charts I'm getting diffrent response for each refresh
@Pranay
v

Vishal Sharma

02/20/2023, 4:11 AM
@Muhammed Hussein Karimi On refresh time period changes so that might be reason. @Srikanth Chekuri are we aware of any metrics issues after increasing shards?
m

Muhammed Hussein Karimi

02/20/2023, 2:29 PM
there is no increase in shards, but shards are restarted in a bad order
s

Srikanth Chekuri

02/20/2023, 3:54 PM
How many shards were up and able to reachable when you saw this behaviour?
m

Muhammed Hussein Karimi

02/20/2023, 5:13 PM
I have the total of 4 shards, all of them were reachable
I'm having this problem right now
s

Srikanth Chekuri

02/20/2023, 5:13 PM
Can you share the logs of each shard process?
m

Muhammed Hussein Karimi

02/20/2023, 5:15 PM
yes
log.txt
there is no error log as I see
s

Srikanth Chekuri

02/20/2023, 5:18 PM
What’s the output logs for other shards?
m

Muhammed Hussein Karimi

02/20/2023, 5:19 PM
I put all of them in that file
search for "k logs"
s

Srikanth Chekuri

02/20/2023, 5:22 PM
Each shard runs as a separate pod?
m

Muhammed Hussein Karimi

02/20/2023, 5:22 PM
yes
using signoz chart (with altinity operator)
s

Srikanth Chekuri

02/20/2023, 5:23 PM
And the shared file is only for one pod
chi-signoz-clickhouse-cluster-0-0-0
right?
m

Muhammed Hussein Karimi

02/20/2023, 5:24 PM
no
search for "k logs" and you see there is other pods
s

Srikanth Chekuri

02/20/2023, 5:29 PM
Ah I see
Is this a frequent issue or happening occasionally? I am not sure what might be the reason here.
m

Muhammed Hussein Karimi

02/20/2023, 5:37 PM
Let me checkout signoz right now
that's heppening right now
image.png,image.png,image.png
s

Srikanth Chekuri

02/20/2023, 5:41 PM
Hmm, interesting which version are you on? We might have to debug this carefully.
m

Muhammed Hussein Karimi

02/20/2023, 5:41 PM
v0.16.0
I think It will be fixed by just a rebalance but I don't know how to do it in ClickHouse
s

Srikanth Chekuri

02/20/2023, 5:44 PM
That’s a somewhat manual process, as it doesn’t automatically take care of it. We have some logic to avoid writing the duplicate time series, so that could also be the reason.
m

Muhammed Hussein Karimi

02/20/2023, 5:44 PM
Oh
s

Srikanth Chekuri

02/20/2023, 5:46 PM
I will try to reproduce and see what we can do about this.
m

Muhammed Hussein Karimi

02/20/2023, 5:54 PM
Thanks, Let me know If I can help about with reproducing problem
4 Views