Hi Team,
query service
alert manager
zookeeper is ...
# supports
surya prakash
09/06/2023, 7:45 AMHi Team,
query service
alert manager
zookeeper is not working
Can't find the reason can anyone help me with this issue
p
Prashant Shahi
09/07/2023, 7:41 AMThat’s strange.
ClickHouse in cluster mode requires zookeeper pod to be ready.
Can you share
kubectl pod describes
surya prakash
09/07/2023, 7:45 AMkubectl describe my-release-zookeeper-0 pod
error: the server doesn't have a resource type "my-release-zookeeper-0"
p
Prashant Shahi
09/07/2023, 7:56 AMcomplete command:
Copy code
kubectl -n platform describe pod/my-release-zookeeper-0s
surya prakash
09/07/2023, 7:57 AMWarning FailedScheduling 4m40s (x228 over 4h12m) default-scheduler 0/4 nodes are available: 4 node(s) had volume node affinity conflict.
surya prakash
09/07/2023, 7:57 AMbut it works properly till last week
surya prakash
09/07/2023, 7:58 AMwhy suddenly get this error
p
Prashant Shahi
09/07/2023, 9:35 AMvolume node affinity conflictPrashant Shahi
09/07/2023, 9:39 AMIt is an issue seen in kubernetes cluster with nodes from multiple zones/regions.
I had encountered this before as well. I resolved it by setting
nodeAffinityPrashant Shahi
09/07/2023, 9:41 AMif you do not care about data loss, you can go about deleting the PVC and restart statefulset pod(s).
It should spawn the new PVCs in same zone as that of the pods.
s
surya prakash
09/07/2023, 9:46 AMNo, i need data
surya prakash
09/08/2023, 10:33 AM@Prashant Shahi, Since zookeeper, alert manager and query service pvc where facing this issue, i won't face any metric data loss after deleting the affected pvc.
thanks
p
Prashant Shahi
09/08/2023, 12:49 PMSince zookeeper, alert manager and query service pvc where facing this issue, i won't face any metric data loss after deleting the affected pvc.actually SQLite would be removed since it is attached to
query-servicePrashant Shahi
09/08/2023, 12:49 PMThat would mean data related to user credentials, alerts, and dashboards would be affected.
s
surya prakash
09/08/2023, 12:50 PMYes i was supposed to sign up again
surya prakash
09/08/2023, 12:51 PMAnd recreated the dashboard using config json
p
Prashant Shahi
09/08/2023, 12:51 PMyes, that is correct
Prashant Shahi
09/08/2023, 12:51 PMbtw, which cloud vendor are you using for managing K8s cluster?
s
surya prakash
09/08/2023, 12:52 PMIn what frequency do you collect metrics and persist to db
surya prakash
09/08/2023, 12:52 PMAws
p
Prashant Shahi
09/08/2023, 2:15 PMIn what frequency do you collect metrics and persist to dbdepends on your collection interval. By default it should be 30s for Hostmetrics and K8s Metrics.
5 Views