https://signoz.io logo
#support
Title
# support
s

surya prakash

11/27/2023, 5:54 AM
Hi Team, I installed signoz in our test environment by default it took 20GB Now it is almost full, how to increase disk size and how to delete old data?
helm -n platform upgrade my-release signoz/signoz \ --set clickhouse.persistence.size=25Gi WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml coalesce.go175 warning: skipped value for zookeeper.initContainers: Not a table. Error: UPGRADE FAILED: cannot patch "my-release-zookeeper" with kind StatefulSet: StatefulSet.apps "my-release-zookeeper" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-alertmanager" with kind StatefulSet: StatefulSet.apps "my-release-signoz-alertmanager" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-query-service" with kind StatefulSet: StatefulSet.apps "my-release-signoz-query-service" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
p

Pranay

11/27/2023, 6:48 AM
Is increasing PV size not working?
s

surya prakash

11/27/2023, 6:48 AM
Yes getting this error
@Pranay
Hello team is there anyone looking on it?!
Hi team Please help me to fix this issue
Team anyone looking into it
p

Prashant Shahi

11/28/2023, 5:10 AM
@surya prakash Increasing persistence of ClickHouse should be straightforward, just updating the persistence size and
helm upgrade
. Rest is handled by ClickHouse Operator.
For other statefulset, I don't think you will generally need to update from the default ones. But in case you do, it would be additional work for each statefulset. •
kubectl delete sts --cascade=orphan statefulset-name
• edit pvc size with
kubectl edit
• followed by
helm upgrade
s

surya prakash

11/28/2023, 5:22 AM
I want to increase clickhouse pvc size only
@Prashant Shahi how to do it?
p

Prashant Shahi

11/28/2023, 5:30 AM
from your previously passed override values YAML only increase clickhouse persistence size, followed by
helm upgrade
from error logs, it looks like you are updating zookeeper and alertmanager statefulset as well
s

surya prakash

11/28/2023, 5:34 AM
okay i will check now
from error logs, it looks like you are updating zookeeper and alertmanager statefulset as well how to avoid zookeeper and alertmanager?
p

Prashant Shahi

11/28/2023, 5:48 AM
to get user supplied values:
Copy code
helm get values -n platform my-release
From the user supplied YAML, only update clickhouse persistence size
s

surya prakash

11/28/2023, 5:49 AM
helm get values -n platform my-release WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml USER-SUPPLIED VALUES: clickhouse: persistence: size: 25Gi
p

Prashant Shahi

11/28/2023, 5:50 AM
save the yaml after
clickhouse:
to the end of output as
override-values.yaml
update the
clickhouse.persistence.size
to
100Gi
or desired value in the file.
s

surya prakash

11/28/2023, 5:51 AM
thank you
in the mean time i tried to update to latest version 0.34.3
after updating i get this error
before that i can access signoz
p

Prashant Shahi

11/28/2023, 5:56 AM
This means query-service is not ready yet.
can you check query-service pod status or logs?
s

surya prakash

11/28/2023, 6:02 AM
it's running
now its working
helm -n platform upgrade my-release -f override-values.yaml signoz/signoz WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml coalesce.go175 warning: skipped value for zookeeper.initContainers: Not a table. Error: UPGRADE FAILED: cannot patch "my-release-zookeeper" with kind StatefulSet: StatefulSet.apps "my-release-zookeeper" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-alertmanager" with kind StatefulSet: StatefulSet.apps "my-release-signoz-alertmanager" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-query-service" with kind StatefulSet: StatefulSet.apps "my-release-signoz-query-service" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
cat override-values.yaml clickhouse: persistence: size: 50Gi
same error @Prashant Shahi
p

Prashant Shahi

11/28/2023, 6:19 AM
@surya prakash which signoz chart version are you upgrading from?
^ can you retry this cmd and share output?
s

surya prakash

11/28/2023, 6:22 AM
helm get values -n platform my-release WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml USER-SUPPLIED VALUES: clickhouse: persistence: size: 50Gi
which signoz chart version are you upgrading from? 0.34.3
p

Prashant Shahi

11/28/2023, 6:23 AM
Copy code
kubectl get pods,chi,pvc -n platform
^ share output of this
s

surya prakash

11/28/2023, 6:24 AM
kubectl get pods,chi,pvc -n platform NAME READY STATUS RESTARTS AGE pod/chi-my-release-clickhouse-cluster-0-0-0 1/1 Running 0 11m pod/my-release-clickhouse-operator-5bdff9845f-zfpdr 2/2 Running 0 32m pod/my-release-k8s-infra-otel-agent-gxwnw 1/1 Running 0 31m pod/my-release-k8s-infra-otel-agent-h7j68 1/1 Running 0 30m pod/my-release-k8s-infra-otel-agent-hxw8x 1/1 Running 0 30m pod/my-release-k8s-infra-otel-agent-kdzzr 1/1 Running 0 31m pod/my-release-k8s-infra-otel-agent-kg2mv 1/1 Running 0 29m pod/my-release-k8s-infra-otel-agent-lcjts 1/1 Running 0 29m pod/my-release-k8s-infra-otel-agent-nsjvm 1/1 Running 0 32m pod/my-release-k8s-infra-otel-agent-ptvz8 1/1 Running 0 32m pod/my-release-k8s-infra-otel-agent-xht2q 1/1 Running 0 32m pod/my-release-k8s-infra-otel-deployment-c64db8949-ttgmz 1/1 Running 0 32m pod/my-release-signoz-alertmanager-0 1/1 Running 0 32m pod/my-release-signoz-frontend-76658dfdc5-dpwdn 1/1 Running 0 32m pod/my-release-signoz-otel-collector-646b96fc59-csdb9 1/1 Running 1 (31m ago) 32m pod/my-release-signoz-otel-collector-metrics-665ddb9588-hh25c 1/1 Running 0 32m pod/my-release-signoz-query-service-0 1/1 Running 0 32m pod/my-release-signoz-schema-migrator-56769c434706-9pq5r 0/1 Completed 0 32m pod/my-release-zookeeper-0 1/1 Running 0 2d10h NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE clickhouseinstallation.clickhouse.altinity.com/my-release-clickhouse 1 1 Completed 121d NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/data-my-release-zookeeper-0 Bound pvc-451d0fb7-a1e7-4931-8104-aee42d5ae243 8Gi RWO gp2-resizable 121d persistentvolumeclaim/data-volumeclaim-template-chi-my-release-clickhouse-cluster-0-0-0 Bound pvc-7dbfe87a-ce3d-4df6-b0c8-d1246b0d4da9 50Gi RWO gp2-resizable 121d persistentvolumeclaim/signoz-db-my-release-signoz-query-service-0 Bound pvc-e69696db-9db7-4363-9075-eef8db310e82 1Gi RWO gp2-resizable 121d persistentvolumeclaim/storage-my-release-signoz-alertmanager-0 Bound pvc-13be7533-d8f9-427d-a013-941eb7a945ee 1Gi RWO gp2-resizable 121d
p

Prashant Shahi

11/28/2023, 6:25 AM
It is running fine with 50Gi ClickHouse storage, when are you seeing the issue?
for every
helm upgrade
command, you will need to include the override-values.yaml.
s

surya prakash

11/28/2023, 6:32 AM
oh ok thanks yes as of now ok with 50Gi
7 Views