Hi. We are playing around with external clickhouse setup (every other part of signoz runs on k8s), latest version of chart (0.56.2, signoz 0.58.2). When we use the default clickhouse user with empty password: externalClickhouse: host: 10.10.10.10 user: default password: "" everything works correctly, signoz deployment finishes, schema-migrator completes. But when we setup a password for the default user: user: default password: yyyy or password: "yyyy" or password: 'yyyy' schema migrator seems to complete part of migrations but then fails and goes into CrashLoopBack and timeouts with this error:

Error: failed to run squashed migrations: failed to run squashed migrations

code: 516, message: Received from 10.10.10.10:9000. DB::Exception: default: Authentication failed: password is incorrect, or there is no user with such name.

If we change the user, delete the deployment and helm install fresh - same situation, same error. We use the default clickhouse ports (8123, 9000) so we don't setup them in the yaml file. Connecting locally or externally to clickhouse instance on 10.10.10.10 with the default user or the other user and providing yyyy password - works without any issues. Any idea what we are doing incorrectly?

Run helm template and see what is the clickhouse endpoint generated for

dsn

@Srikanth Chekuri Not sure if I did this correctly but after saving the template into file and then doing grep for dsn I get:

dsn: tcp://${env:CLICKHOUSE_USER}:${env:CLICKHOUSE_PASSWORD}@${env:CLICKHOUSE_HOST}:${env:CLICKHOUSE_PORT}/${env:CLICKHOUSE_LOG_DATABASE}

- "--dsn"

- "--dsn

Checking schema-migrator pod config seems that those values are setup correctly (we are using rancher for "k8s gui" and this time admin user just like signoz with the exact password, also not working):

Hi @WD, does the user and password remain as default admin even when you use externalClickHouse?

From what I remember setting empty password on any account (default, admin) works, setting password results in errors. I will check it again when I'll be able to run the SigNoz test deployment again.