https://signoz.io logo
Join Slack
Powered by
Hi Team, I installed signoz in our test environmen...
# support
s
Hi Team, I installed signoz in our test environment by default it took 20GB Now it is almost full, how to increase disk size and how to delete old data?
helm -n platform upgrade my-release signoz/signoz \ --set clickhouse.persistence.size=25Gi WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml coalesce.go175 warning: skipped value for zookeeper.initContainers: Not a table. Error: UPGRADE FAILED: cannot patch "my-release-zookeeper" with kind StatefulSet: StatefulSet.apps "my-release-zookeeper" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-alertmanager" with kind StatefulSet: StatefulSet.apps "my-release-signoz-alertmanager" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-query-service" with kind StatefulSet: StatefulSet.apps "my-release-signoz-query-service" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
p
Is increasing PV size not working?
s
Yes getting this error
@Pranay
Hello team is there anyone looking on it?!
Hi team Please help me to fix this issue
Team anyone looking into it
p
@surya prakash Increasing persistence of ClickHouse should be straightforward, just updating the persistence size and 
helm upgrade
. Rest is handled by ClickHouse Operator.
For other statefulset, I don't think you will generally need to update from the default ones. But in case you do, it would be additional work for each statefulset. • 
kubectl delete sts --cascade=orphan statefulset-name
• edit pvc size with 
kubectl edit
• followed by 
helm upgrade
s
I want to increase clickhouse pvc size only
@Prashant Shahi how to do it?
p
from your previously passed override values YAML only increase clickhouse persistence size, followed by 
helm upgrade
from error logs, it looks like you are updating zookeeper and alertmanager statefulset as well
s
okay i will check now
from error logs, it looks like you are updating zookeeper and alertmanager statefulset as well how to avoid zookeeper and alertmanager?
p
to get user supplied values:
Copy code
helm get values -n platform my-release
From the user supplied YAML, only update clickhouse persistence size
s
helm get values -n platform my-release WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml USER-SUPPLIED VALUES: clickhouse: persistence: size: 25Gi
p
save the yaml after 
clickhouse:
to the end of output as 
override-values.yaml
update the 
clickhouse.persistence.size
to 
100Gi
or desired value in the file.
s
thank you
in the mean time i tried to update to latest version 0.34.3
after updating i get this error
before that i can access signoz
p
This means query-service is not ready yet.
can you check query-service pod status or logs?
s
it's running
now its working
helm -n platform upgrade my-release -f override-values.yaml signoz/signoz WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml coalesce.go175 warning: skipped value for zookeeper.initContainers: Not a table. Error: UPGRADE FAILED: cannot patch "my-release-zookeeper" with kind StatefulSet: StatefulSet.apps "my-release-zookeeper" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-alertmanager" with kind StatefulSet: StatefulSet.apps "my-release-signoz-alertmanager" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-query-service" with kind StatefulSet: StatefulSet.apps "my-release-signoz-query-service" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
cat override-values.yaml clickhouse: persistence: size: 50Gi
same error @Prashant Shahi
p
@surya prakash which signoz chart version are you upgrading from?
^ can you retry this cmd and share output?
s
helm get values -n platform my-release WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml USER-SUPPLIED VALUES: clickhouse: persistence: size: 50Gi
which signoz chart version are you upgrading from? 0.34.3
p
Copy code
kubectl get pods,chi,pvc -n platform
^ share output of this
s
kubectl get pods,chi,pvc -n platform NAME READY STATUS RESTARTS AGE pod/chi-my-release-clickhouse-cluster-0-0-0 1/1 Running 0 11m pod/my-release-clickhouse-operator-5bdff9845f-zfpdr 2/2 Running 0 32m pod/my-release-k8s-infra-otel-agent-gxwnw 1/1 Running 0 31m pod/my-release-k8s-infra-otel-agent-h7j68 1/1 Running 0 30m pod/my-release-k8s-infra-otel-agent-hxw8x 1/1 Running 0 30m pod/my-release-k8s-infra-otel-agent-kdzzr 1/1 Running 0 31m pod/my-release-k8s-infra-otel-agent-kg2mv 1/1 Running 0 29m pod/my-release-k8s-infra-otel-agent-lcjts 1/1 Running 0 29m pod/my-release-k8s-infra-otel-agent-nsjvm 1/1 Running 0 32m pod/my-release-k8s-infra-otel-agent-ptvz8 1/1 Running 0 32m pod/my-release-k8s-infra-otel-agent-xht2q 1/1 Running 0 32m pod/my-release-k8s-infra-otel-deployment-c64db8949-ttgmz 1/1 Running 0 32m pod/my-release-signoz-alertmanager-0 1/1 Running 0 32m pod/my-release-signoz-frontend-76658dfdc5-dpwdn 1/1 Running 0 32m pod/my-release-signoz-otel-collector-646b96fc59-csdb9 1/1 Running 1 (31m ago) 32m pod/my-release-signoz-otel-collector-metrics-665ddb9588-hh25c 1/1 Running 0 32m pod/my-release-signoz-query-service-0 1/1 Running 0 32m pod/my-release-signoz-schema-migrator-56769c434706-9pq5r 0/1 Completed 0 32m pod/my-release-zookeeper-0 1/1 Running 0 2d10h NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE clickhouseinstallation.clickhouse.altinity.com/my-release-clickhouse 1 1 Completed 121d NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/data-my-release-zookeeper-0 Bound pvc-451d0fb7-a1e7-4931-8104-aee42d5ae243 8Gi RWO gp2-resizable 121d persistentvolumeclaim/data-volumeclaim-template-chi-my-release-clickhouse-cluster-0-0-0 Bound pvc-7dbfe87a-ce3d-4df6-b0c8-d1246b0d4da9 50Gi RWO gp2-resizable 121d persistentvolumeclaim/signoz-db-my-release-signoz-query-service-0 Bound pvc-e69696db-9db7-4363-9075-eef8db310e82 1Gi RWO gp2-resizable 121d persistentvolumeclaim/storage-my-release-signoz-alertmanager-0 Bound pvc-13be7533-d8f9-427d-a013-941eb7a945ee 1Gi RWO gp2-resizable 121d
p
It is running fine with 50Gi ClickHouse storage, when are you seeing the issue?
for every 
helm upgrade
command, you will need to include the override-values.yaml.
s
oh ok thanks yes as of now ok with 50Gi
30 Views
Open in Slack
PreviousNext
Powered by