This message was deleted.

Slackbot

11/27/2023, 5:54 AM

This message was deleted.

✅ 1

surya prakash

11/27/2023, 6:27 AM

helm -n platform upgrade my-release signoz/signoz \ --set clickhouse.persistence.size=25Gi WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml coalesce.go175 warning: skipped value for zookeeper.initContainers: Not a table. Error: UPGRADE FAILED: cannot patch "my-release-zookeeper" with kind StatefulSet: StatefulSet.apps "my-release-zookeeper" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-alertmanager" with kind StatefulSet: StatefulSet.apps "my-release-signoz-alertmanager" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-query-service" with kind StatefulSet: StatefulSet.apps "my-release-signoz-query-service" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Pranay

11/27/2023, 6:48 AM

Is increasing PV size not working?

surya prakash

11/27/2023, 6:48 AM

Yes getting this error

surya prakash

11/27/2023, 7:18 AM

@Pranay

surya prakash

11/27/2023, 8:41 AM

Hello team is there anyone looking on it?!

surya prakash

11/28/2023, 4:35 AM

Hi team Please help me to fix this issue

surya prakash

11/28/2023, 4:35 AM

Team anyone looking into it

Prashant Shahi

11/28/2023, 5:10 AM

@surya prakash Increasing persistence of ClickHouse should be straightforward, just updating the persistence size and

helm upgrade

. Rest is handled by ClickHouse Operator.

Prashant Shahi

11/28/2023, 5:11 AM

For other statefulset, I don't think you will generally need to update from the default ones. But in case you do, it would be additional work for each statefulset. •

kubectl delete sts --cascade=orphan statefulset-name

• edit pvc size with

kubectl edit

• followed by

helm upgrade

surya prakash

11/28/2023, 5:22 AM

I want to increase clickhouse pvc size only

surya prakash

11/28/2023, 5:22 AM

@Prashant Shahi how to do it?

Prashant Shahi

11/28/2023, 5:30 AM

from your previously passed override values YAML only increase clickhouse persistence size, followed by

helm upgrade

Prashant Shahi

11/28/2023, 5:31 AM

from error logs, it looks like you are updating zookeeper and alertmanager statefulset as well

surya prakash

11/28/2023, 5:34 AM

okay i will check now

surya prakash

11/28/2023, 5:36 AM

from error logs, it looks like you are updating zookeeper and alertmanager statefulset as well how to avoid zookeeper and alertmanager?

Prashant Shahi

11/28/2023, 5:48 AM

to get user supplied values:

Copy code

helm get values -n platform my-release

Prashant Shahi

11/28/2023, 5:49 AM

From the user supplied YAML, only update clickhouse persistence size

surya prakash

11/28/2023, 5:49 AM

helm get values -n platform my-release WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml USER-SUPPLIED VALUES: clickhouse: persistence: size: 25Gi

Prashant Shahi

11/28/2023, 5:50 AM

save the yaml after

clickhouse:

to the end of output as

override-values.yaml

Prashant Shahi

11/28/2023, 5:50 AM

update the

clickhouse.persistence.size

100Gi

or desired value in the file.

✅ 1

surya prakash

11/28/2023, 5:51 AM

thank you

✌️ 1

surya prakash

11/28/2023, 5:53 AM

in the mean time i tried to update to latest version 0.34.3

surya prakash

11/28/2023, 5:53 AM

after updating i get this error

surya prakash

11/28/2023, 5:53 AM

before that i can access signoz

Prashant Shahi

11/28/2023, 5:56 AM

This means query-service is not ready yet.

Prashant Shahi

11/28/2023, 5:57 AM

can you check query-service pod status or logs?

✅ 1

surya prakash

11/28/2023, 6:02 AM

it's running

surya prakash

11/28/2023, 6:03 AM

now its working

🎉 1

surya prakash

11/28/2023, 6:12 AM

helm -n platform upgrade my-release -f override-values.yaml signoz/signoz WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /location/updatedkc.yaml WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /location/updatedkc.yaml coalesce.go175 warning: skipped value for zookeeper.initContainers: Not a table. Error: UPGRADE FAILED: cannot patch "my-release-zookeeper" with kind StatefulSet: StatefulSet.apps "my-release-zookeeper" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-alertmanager" with kind StatefulSet: StatefulSet.apps "my-release-signoz-alertmanager" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden && cannot patch "my-release-signoz-query-service" with kind StatefulSet: StatefulSet.apps "my-release-signoz-query-service" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

surya prakash

11/28/2023, 6:13 AM

cat override-values.yaml clickhouse: persistence: size: 50Gi

surya prakash

11/28/2023, 6:15 AM

same error @Prashant Shahi

Prashant Shahi

11/28/2023, 6:19 AM

@surya prakash which signoz chart version are you upgrading from?

Prashant Shahi

11/28/2023, 6:21 AM

Also, is that all you have in values files? https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1701150514937819?thread_ts=1701064444.702959&cid=C01HWQ1R0BC

Prashant Shahi

11/28/2023, 6:22 AM

^ can you retry this cmd and share output?

surya prakash

11/28/2023, 6:22 AM

surya prakash

11/28/2023, 6:23 AM

which signoz chart version are you upgrading from? 0.34.3

Prashant Shahi

11/28/2023, 6:23 AM

Copy code

kubectl get pods,chi,pvc -n platform

Prashant Shahi

11/28/2023, 6:23 AM

^ share output of this

surya prakash

11/28/2023, 6:24 AM

kubectl get pods,chi,pvc -n platform NAME READY STATUS RESTARTS AGE pod/chi-my-release-clickhouse-cluster-0-0-0 1/1 Running 0 11m pod/my-release-clickhouse-operator-5bdff9845f-zfpdr 2/2 Running 0 32m pod/my-release-k8s-infra-otel-agent-gxwnw 1/1 Running 0 31m pod/my-release-k8s-infra-otel-agent-h7j68 1/1 Running 0 30m pod/my-release-k8s-infra-otel-agent-hxw8x 1/1 Running 0 30m pod/my-release-k8s-infra-otel-agent-kdzzr 1/1 Running 0 31m pod/my-release-k8s-infra-otel-agent-kg2mv 1/1 Running 0 29m pod/my-release-k8s-infra-otel-agent-lcjts 1/1 Running 0 29m pod/my-release-k8s-infra-otel-agent-nsjvm 1/1 Running 0 32m pod/my-release-k8s-infra-otel-agent-ptvz8 1/1 Running 0 32m pod/my-release-k8s-infra-otel-agent-xht2q 1/1 Running 0 32m pod/my-release-k8s-infra-otel-deployment-c64db8949-ttgmz 1/1 Running 0 32m pod/my-release-signoz-alertmanager-0 1/1 Running 0 32m pod/my-release-signoz-frontend-76658dfdc5-dpwdn 1/1 Running 0 32m pod/my-release-signoz-otel-collector-646b96fc59-csdb9 1/1 Running 1 (31m ago) 32m pod/my-release-signoz-otel-collector-metrics-665ddb9588-hh25c 1/1 Running 0 32m pod/my-release-signoz-query-service-0 1/1 Running 0 32m pod/my-release-signoz-schema-migrator-56769c434706-9pq5r 0/1 Completed 0 32m pod/my-release-zookeeper-0 1/1 Running 0 2d10h NAME CLUSTERS HOSTS STATUS HOSTS-COMPLETED AGE clickhouseinstallation.clickhouse.altinity.com/my-release-clickhouse 1 1 Completed 121d NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/data-my-release-zookeeper-0 Bound pvc-451d0fb7-a1e7-4931-8104-aee42d5ae243 8Gi RWO gp2-resizable 121d persistentvolumeclaim/data-volumeclaim-template-chi-my-release-clickhouse-cluster-0-0-0 Bound pvc-7dbfe87a-ce3d-4df6-b0c8-d1246b0d4da9 50Gi RWO gp2-resizable 121d persistentvolumeclaim/signoz-db-my-release-signoz-query-service-0 Bound pvc-e69696db-9db7-4363-9075-eef8db310e82 1Gi RWO gp2-resizable 121d persistentvolumeclaim/storage-my-release-signoz-alertmanager-0 Bound pvc-13be7533-d8f9-427d-a013-941eb7a945ee 1Gi RWO gp2-resizable 121d

Prashant Shahi

11/28/2023, 6:25 AM

It is running fine with 50Gi ClickHouse storage, when are you seeing the issue?

Prashant Shahi

11/28/2023, 6:31 AM

for every

helm upgrade

command, you will need to include the override-values.yaml.

surya prakash

11/28/2023, 6:32 AM

oh ok thanks yes as of now ok with 50Gi

64 Views

Open in Slack

Previous Next

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.