Title
m
Muhammed Hussein Karimi
02/19/2023, 8:22 AMHi,
I have increased the size of ClickHouse, but some of the charts are not loading correctly, is there any extra step todo?
These are the possible states, Clickhouse layout is 1 replica and 4 shards
Also for other charts I'm getting diffrent response for each refresh
@Pranay
v
Vishal Sharma
02/20/2023, 4:11 AM@Muhammed Hussein Karimi On refresh time period changes so that might be reason.
@Srikanth Chekuri are we aware of any metrics issues after increasing shards?
m
Muhammed Hussein Karimi
02/20/2023, 2:29 PMthere is no increase in shards, but shards are restarted in a bad order
s
Srikanth Chekuri
02/20/2023, 3:54 PMHow many shards were up and able to reachable when you saw this behaviour?
m
Muhammed Hussein Karimi
02/20/2023, 5:13 PMI have the total of 4 shards, all of them were reachable
I'm having this problem right now
s
Srikanth Chekuri
02/20/2023, 5:13 PMCan you share the logs of each shard process?
m
Muhammed Hussein Karimi
02/20/2023, 5:15 PMyes
log.txt
there is no error log as I see
s
Srikanth Chekuri
02/20/2023, 5:18 PMWhat’s the output logs for other shards?
m
Muhammed Hussein Karimi
02/20/2023, 5:19 PMI put all of them in that file
search for "k logs"
s
Srikanth Chekuri
02/20/2023, 5:22 PMEach shard runs as a separate pod?
m
Muhammed Hussein Karimi
02/20/2023, 5:22 PMyes
using signoz chart (with altinity operator)
s
Srikanth Chekuri
02/20/2023, 5:23 PMAnd the shared file is only for one pod
chi-signoz-clickhouse-cluster-0-0-0m
Muhammed Hussein Karimi
02/20/2023, 5:24 PMno
search for "k logs" and you see there is other pods
s
Srikanth Chekuri
02/20/2023, 5:29 PMAh I see
Is this a frequent issue or happening occasionally? I am not sure what might be the reason here.
m
Muhammed Hussein Karimi
02/20/2023, 5:37 PMLet me checkout signoz right now
that's heppening right now
image.png,image.png,image.png
s
Srikanth Chekuri
02/20/2023, 5:41 PMHmm, interesting which version are you on? We might have to debug this carefully.
m
Muhammed Hussein Karimi
02/20/2023, 5:41 PMv0.16.0
I think It will be fixed by just a rebalance
but I don't know how to do it in ClickHouse
s
Srikanth Chekuri
02/20/2023, 5:44 PMThat’s a somewhat manual process, as it doesn’t automatically take care of it. We have some logic to avoid writing the duplicate time series, so that could also be the reason.
m
Muhammed Hussein Karimi
02/20/2023, 5:44 PMOh
s
Srikanth Chekuri
02/20/2023, 5:46 PMI will try to reproduce and see what we can do about this.
m
Muhammed Hussein Karimi
02/20/2023, 5:54 PMThanks, Let me know If I can help about with reproducing problem