Hello SigNoz team! Is it possible to upgrade a standard k8s helm deployment w/ single clickhouse cluster to a distributed clickhouse as mentioned in here in the doc? If i run helm upgrade with new values, the schema migrator is stuck waiting for new shards/replicas, but the operator isn't spawning any new clickhouse pods.

Yes it should be possible. Check the operator logs to understand why more pods are not coming up

hmm so something's not right... 1. we have authentication errors

Authentication failed: password is incorrect, or there is no user with such name. (AUTHENTICATION_FAILED)

2. then it says

Failed to reconcile objects

3.

processed replicas: 0