👋 Hello, team I tried installing Signoz on k3s cluster using helm. However all the pods are not coming up in a running state. Below is the output of ,kubectl -n platform get pods command : NAME READY STATUS RESTARTS AGE my-release-k8s-infra-otel-agent-q449t 1/1 Running 3 (8m39s ago) 23h my-release-clickhouse-operator-5457b49dfc-2wpkp 2/2 Running 5 (8m39s ago) 23h my-release-signoz-frontend-86699c44c5-64kdg 0/1 Init:0/1 3 23h my-release-signoz-query-service-0 0/1 Init:0/1 3 23h my-release-signoz-otel-collector-fd6b4899-zbcsv 0/1 Init:0/1 3 23h my-release-signoz-otel-collector-metrics-7594f556c9-7vj9r 0/1 Init:0/1 3 23h my-release-k8s-infra-otel-deployment-6669899f75-xdlfq 1/1 Running 4 (8m39s ago) 23h my-release-zookeeper-0 1/1 Running 0 23h my-release-signoz-alertmanager-0 0/1 Pending 0 23h chi-my-release-clickhouse-cluster-0-0-0 0/1 Pending 0 23h Can you share some clue on what is that I am missing here ? Thanks !

Use

kubectl describe

to get the detail. This is generic “my pod is stuck” issue.

@Srikanth Chekuri is right. Try

kubectl describe

on the chi pods and perhaps also related resources like PVCs of clickhouse. Complete command:

kubectl describe -n platform pod/chi-my-release-clickhouse-cluster-0-0-0

hi @Prashant Shahi please see the output below : kubectl describe -n platform pod/chi-my-release-clickhouse-cluster-0-0-0 Name: chi-my-release-clickhouse-cluster-0-0-0 Namespace: platform Priority: 0 Service Account: my-release-clickhouse Node: nroy-virtual-machine/192.168.163.128 Start Time: Mon, 13 Mar 2023 224947 +0530 Labels: app.kubernetes.io/component=clickhouse app.kubernetes.io/instance=my-release app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=clickhouse app.kubernetes.io/version=22.8.8 clickhouse.altinity.com/app=chop clickhouse.altinity.com/chi=my-release-clickhouse clickhouse.altinity.com/cluster=cluster clickhouse.altinity.com/namespace=platform clickhouse.altinity.com/ready=yes clickhouse.altinity.com/replica=0 clickhouse.altinity.com/settings-version=a0b5649f5ea9121accf6ecc528db9f761f7f1768 clickhouse.altinity.com/shard=0 clickhouse.altinity.com/zookeeper-version=98145c0055cd1bf605e2ba1ed27b9c43f4de32c7 controller-revision-hash=chi-my-release-clickhouse-cluster-0-0-558ffc5f76 helm.sh/chart=clickhouse-23.8.8 statefulset.kubernetes.io/pod-name=chi-my-release-clickhouse-cluster-0-0-0 Annotations: meta.helm.sh/release-name: my-release meta.helm.sh/release-namespace: platform signoz.io/path: /metrics signoz.io/port: 9363 signoz.io/scrape: true Status: Pending IP: 10.42.0.2 IPs: IP: 10.42.0.2 Controlled By: StatefulSet/chi-my-release-clickhouse-cluster-0-0 Init Containers: my-release-clickhouse-init: Container ID: docker://d996ccf8646d640434122161142e69dab8081227344442eda2a6ec35b71fc691 Image: docker.io/busybox:1.35 Image ID: docker-pullable://busybox@sha256:f75aadb4c50f4fe0e790e5e081de3df4153a5adbe77a176205763d9808e3c12a Port: <none> Host Port: <none> Command: sh -c set -x wget -O /tmp/histogramQuantile https://github.com/SigNoz/signoz/raw/develop/deploy/docker/clickhouse-setup/user_scripts/histogramQuantile mv /tmp/histogramQuantile /var/lib/clickhouse/user_scripts/histogramQuantile chmod +x /var/lib/clickhouse/user_scripts/histogramQuantile State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 13 Mar 2023 225052 +0530 Finished: Mon, 13 Mar 2023 225106 +0530 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/lib/clickhouse/user_scripts from shared-binary-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-g4s9x (ro) Containers: clickhouse: Container ID: Image: docker.io/clickhouse/clickhouse-server:22.8.8-alpine Image ID: Ports: 8123/TCP, 9000/TCP, 9009/TCP, 9000/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP Command: /bin/bash -c /usr/bin/clickhouse-server --config-file=/etc/clickhouse-server/config.xml State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Requests: cpu: 100m memory: 200Mi Liveness: http-get http//http/ping delay=60s timeout=1s period=3s #success=1 #failure=10 Readiness: http-get http//http/ping delay=10s timeout=1s period=3s #success=1 #failure=3 Environment: <none> Mounts: /etc/clickhouse-server/conf.d/ from chi-my-release-clickhouse-deploy-confd-cluster-0-0 (rw) /etc/clickhouse-server/config.d/ from chi-my-release-clickhouse-common-configd (rw) /etc/clickhouse-server/functions from custom-functions-volume (rw) /etc/clickhouse-server/users.d/ from chi-my-release-clickhouse-common-usersd (rw) /var/lib/clickhouse from data-volumeclaim-template (rw) /var/lib/clickhouse/user_scripts from shared-binary-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-g4s9x (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: data-volumeclaim-template: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-volumeclaim-template-chi-my-release-clickhouse-cluster-0-0-0 ReadOnly: false shared-binary-volume: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> custom-functions-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: my-release-clickhouse-custom-functions Optional: false chi-my-release-clickhouse-common-configd: Type: ConfigMap (a volume populated by a ConfigMap) Name: chi-my-release-clickhouse-common-configd Optional: false chi-my-release-clickhouse-common-usersd: Type: ConfigMap (a volume populated by a ConfigMap) Name: chi-my-release-clickhouse-common-usersd Optional: false chi-my-release-clickhouse-deploy-confd-cluster-0-0: Type: ConfigMap (a volume populated by a ConfigMap) Name: chi-my-release-clickhouse-deploy-confd-cluster-0-0 Optional: false kube-api-access-g4s9x: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 44m default-scheduler Successfully assigned platform/chi-my-release-clickhouse-cluster-0-0-0 to nroy-virtual-machine Normal Pulled 43m kubelet Container image "docker.io/busybox:1.35" already present on machine Normal Created 43m kubelet Created container my-release-clickhouse-init Normal Started 43m kubelet Started container my-release-clickhouse-init Normal Pulling 35m (x4 over 42m) kubelet Pulling image "docker.io/clickhouse/clickhouse-server:22.8.8-alpine" Warning Failed 33m (x4 over 40m) kubelet Error: ErrImagePull Warning Failed 32m (x6 over 40m) kubelet Error: ImagePullBackOff Warning Failed 17m (x7 over 40m) kubelet Failed to pull image "docker.io/clickhouse/clickhouse-server:22.8.8-alpine": rpc error: code = Unknown desc = context deadline exceeded Normal BackOff 3m35s (x84 over 40m) kubelet Back-off pulling image "docker.io/clickhouse/clickhouse-server:22.8.8-alpine"