https://signoz.io logo
#support
Title
# support
b

Bartosz Kościug

06/13/2022, 7:47 PM
Hi all, I am trying to run signoz with the help of helm chart on AWS, but I ended up with the issue with chi-signoz-cluster. I see that there is some problem with clickhouse running and thats why signoz pods don't want to up. Could you help me please?
p

Prashant Shahi

06/13/2022, 8:08 PM
Hey @Bartosz Kościug 👋 Can you share logs of pod
chi-signoz-cluster
?
b

Bartosz Kościug

06/13/2022, 8:28 PM
This is also interesting, no logs in this pod:
ubuntu@LAPTOP-96CDV04K:/tmp$ kubectl describe pod chi-signoz-cluster-0-0-0 -n signoz Name: chi-signoz-cluster-0-0-0 Namespace: signoz Priority: 0 Node: ip-172-16-2-170.eu-west-2.compute.internal/172.16.2.170 Start Time: Mon, 13 Jun 2022 222702 +0200 Labels: app.kubernetes.io/managed-by=Helm clickhouse.altinity.com/app=chop clickhouse.altinity.com/chi=signoz clickhouse.altinity.com/cluster=cluster clickhouse.altinity.com/namespace=signoz clickhouse.altinity.com/replica=0 clickhouse.altinity.com/settings-version=208b41823c98541c9c4f31abd14664612d884c1a clickhouse.altinity.com/shard=0 clickhouse.altinity.com/zookeeper-version=6a636eeea8e6d68b04c95ee6df9054070080b0cf controller-revision-hash=chi-signoz-cluster-0-0-7d94859f9 instana-autotrace-applied=true instana-autotrace-transformation=DefaultPodTransformation instana-autotrace-version=1.226.5 statefulset.kubernetes.io/pod-name=chi-signoz-cluster-0-0-0 Annotations: kubernetes.io/psp: eks.privileged meta.helm.sh/release-name: test-signoz meta.helm.sh/release-namespace: signoz Status: Running IP: 172.16.2.141 IPs: IP: 172.16.2.141 Controlled By: StatefulSet/chi-signoz-cluster-0-0 Init Containers: instana-instrumentation-init: Container ID: docker://a0e3e610a8d0b36fbaf1da3ec0eafcffcff330c90516da782bf38944b80cb974 Image: icr.io/instana/instrumentation@sha256:8292ed9460e2da6e96ffa6d5ca547b4fbf2f05bc793f99ce9e68e5c3e97d91cc Image ID: docker-pullable://icr.io/instana/instrumentation@sha256:8292ed9460e2da6e96ffa6d5ca547b4fbf2f05bc793f99ce9e68e5c3e97d91cc Port: <none> Host Port: <none> State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 13 Jun 2022 222712 +0200 Finished: Mon, 13 Jun 2022 222714 +0200 Ready: True Restart Count: 0 Limits: cpu: 250m memory: 64Mi Requests: cpu: 150m memory: 16Mi Environment: INSTANA_AGENT_HOST: (v1:status.hostIP) Mounts: /opt/instana/instrumentation from instana-instrumentation-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-82khn (ro) Containers: clickhouse: Container ID: docker://52e6086a7239da42be1b60cd8c67a58cdc2da392ef5f789e8d86de94dec66ec8 Image: docker.io/yandex/clickhouse-server:21.12.3.32 Image ID: docker-pullable://yandex/clickhouse-server@sha256:7d58af905fef789288892f1eac0d570ac23ef4c9b326baf684f202ac2894af09 Ports: 8123/TCP, 9000/TCP, 9009/TCP, 9000/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP Command: /bin/bash -c /usr/bin/clickhouse-server --config-file=/etc/clickhouse-server/config.xml State: Running Started: Mon, 13 Jun 2022 222911 +0200 Last State: Terminated Reason: Error Exit Code: 137 Started: Mon, 13 Jun 2022 222714 +0200 Finished: Mon, 13 Jun 2022 222911 +0200 Ready: False Restart Count: 1 Liveness: http-get http//http/ping delay=60s timeout=1s period=3s #success=1 #failure=10 Readiness: http-get http//http/ping delay=10s timeout=1s period=3s #success=1 #failure=3 Environment: LD_PRELOAD: /opt/instana/instrumentation/libinstana_init/libinstana_init.so ACE_ENABLE_OPEN_TRACING: true MQ_ENABLE_OPEN_TRACING: true INSTANA_AGENT_HOST: (v1:status.hostIP) Mounts: /etc/clickhouse-server/conf.d/ from chi-signoz-deploy-confd-cluster-0-0 (rw) /etc/clickhouse-server/config.d/ from chi-signoz-common-configd (rw) /etc/clickhouse-server/users.d/ from chi-signoz-common-usersd (rw) /opt/instana/instrumentation from instana-instrumentation-volume (rw) /var/lib/clickhouse from data-volumeclaim-template (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-82khn (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: data-volumeclaim-template: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-volumeclaim-template-chi-signoz-cluster-0-0-0 ReadOnly: false chi-signoz-common-configd: Type: ConfigMap (a volume populated by a ConfigMap) Name: chi-signoz-common-configd Optional: false chi-signoz-common-usersd: Type: ConfigMap (a volume populated by a ConfigMap) Name: chi-signoz-common-usersd Optional: false chi-signoz-deploy-confd-cluster-0-0: Type: ConfigMap (a volume populated by a ConfigMap) Name: chi-signoz-deploy-confd-cluster-0-0 Optional: false kube-api-access-82khn: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true instana-instrumentation-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m20s (x3 over 2m26s) default-scheduler 0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims. Normal Scheduled 2m16s default-scheduler Successfully assigned signoz/chi-signoz-cluster-0-0-0 to ip-172-16-2-170.eu-west-2.compute.internal Normal SuccessfulAttachVolume 2m13s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-31926244-242e-471b-8968-2daa06f0c5ee" Normal Pulled 2m6s kubelet Container image "icr.io/instana/instrumentation@sha256:8292ed9460e2da6e96ffa6d5ca547b4fbf2f05bc793f99ce9e68e5c3e97d91cc" already present on machine Normal Created 2m6s kubelet Created container instana-instrumentation-init Normal Started 2m6s kubelet Started container instana-instrumentation-init Normal Pulled 2m4s kubelet Container image "docker.io/yandex/clickhouse-server:21.12.3.32" already present on machine Normal Created 2m4s kubelet Created container clickhouse Normal Started 2m4s kubelet Started container clickhouse Warning Unhealthy 64s (x17 over 112s) kubelet Readiness probe failed: Get "http://172.16.2.141:8123/ping": dial tcp 172.16.2.1418123 connect: connection refused Warning Unhealthy 61s (x2 over 64s) kubelet Liveness probe failed: Get "http://172.16.2.141:8123/ping": dial tcp 172.16.2.1418123 connect: connection refused
p

Prashant Shahi

06/14/2022, 6:15 AM
@Bartosz Kościug I tested in new EKS cluster. Same works for me. Exit Code 137 is standard for OOMKilled. Are you sure have allocated enough resources for the cluster?
b

Bartosz Kościug

06/14/2022, 12:03 PM
what you mean in the new EKS cluster?
p

Prashant Shahi

06/14/2022, 12:05 PM
newly created EKS cluster*
98 Views