This message was deleted.
# supports
Slackbot
02/19/2023, 8:22 AMThis message was deleted.
m
Muhammed Hussein Karimi
02/19/2023, 8:26 AMThese are the possible states, Clickhouse layout is 1 replica and 4 shards
Muhammed Hussein Karimi
02/19/2023, 8:27 AMAlso for other charts I'm getting diffrent response for each refresh
Muhammed Hussein Karimi
02/19/2023, 3:04 PM@Pranay
v
Vishal Sharma
02/20/2023, 4:11 AM@Muhammed Hussein Karimi On refresh time period changes so that might be reason.
@Srikanth Chekuri are we aware of any metrics issues after increasing shards?
m
Muhammed Hussein Karimi
02/20/2023, 2:29 PMthere is no increase in shards, but shards are restarted in a bad order
s
Srikanth Chekuri
02/20/2023, 3:54 PMHow many shards were up and able to reachable when you saw this behaviour?
m
Muhammed Hussein Karimi
02/20/2023, 5:13 PMI have the total of 4 shards, all of them were reachable
Muhammed Hussein Karimi
02/20/2023, 5:13 PMI'm having this problem right now
s
Srikanth Chekuri
02/20/2023, 5:13 PMCan you share the logs of each shard process?
m
Muhammed Hussein Karimi
02/20/2023, 5:15 PMyes
Muhammed Hussein Karimi
02/20/2023, 5:17 PMthere is no error log as I see
s
Srikanth Chekuri
02/20/2023, 5:18 PMWhat’s the output logs for other shards?
m
Muhammed Hussein Karimi
02/20/2023, 5:19 PMI put all of them in that file
Muhammed Hussein Karimi
02/20/2023, 5:19 PMsearch for "k logs"
s
Srikanth Chekuri
02/20/2023, 5:22 PMEach shard runs as a separate pod?
m
Muhammed Hussein Karimi
02/20/2023, 5:22 PMyes
Muhammed Hussein Karimi
02/20/2023, 5:23 PMusing signoz chart (with altinity operator)
s
Srikanth Chekuri
02/20/2023, 5:23 PMAnd the shared file is only for one pod
chi-signoz-clickhouse-cluster-0-0-0m
Muhammed Hussein Karimi
02/20/2023, 5:24 PMno
Muhammed Hussein Karimi
02/20/2023, 5:24 PMsearch for "k logs" and you see there is other pods
s
Srikanth Chekuri
02/20/2023, 5:29 PMAh I see
✅ 1
Srikanth Chekuri
02/20/2023, 5:36 PMIs this a frequent issue or happening occasionally? I am not sure what might be the reason here.
m
Muhammed Hussein Karimi
02/20/2023, 5:37 PMLet me checkout signoz right now
Muhammed Hussein Karimi
02/20/2023, 5:38 PMthat's heppening right now
Muhammed Hussein Karimi
02/20/2023, 5:39 PMmessage has been deleted
s
Srikanth Chekuri
02/20/2023, 5:41 PMHmm, interesting which version are you on? We might have to debug this carefully.
m
Muhammed Hussein Karimi
02/20/2023, 5:41 PMv0.16.0
Muhammed Hussein Karimi
02/20/2023, 5:42 PMI think It will be fixed by just a rebalance
but I don't know how to do it in ClickHouse
s
Srikanth Chekuri
02/20/2023, 5:44 PMThat’s a somewhat manual process, as it doesn’t automatically take care of it. We have some logic to avoid writing the duplicate time series, so that could also be the reason.
m
Muhammed Hussein Karimi
02/20/2023, 5:44 PMOh
s
Srikanth Chekuri
02/20/2023, 5:46 PMI will try to reproduce and see what we can do about this.
m
Muhammed Hussein Karimi
02/20/2023, 5:54 PMThanks, Let me know If I can help about with reproducing problem
7 Views