Gutemberg Ribeiro
06/02/2025, 8:54 PM{"L":"info","timestamp":"2025-06-02T20:47:22.555Z","C":"signozschemamigrator/main.go:91","M":"Running migrations in sync mode","dsn":"<tcp://posta:postapw@clickhouse-posta.clickhouse.svc.cluster.local:9000>","replication":false,"cluster-name":"cluster"}
{"L":"info","timestamp":"2025-06-02T20:47:22.555Z","C":"signozschemamigrator/main.go:104","M":"Up migrations","versions":[]}
{"L":"info","timestamp":"2025-06-02T20:47:22.555Z","C":"signozschemamigrator/main.go:117","M":"Down migrations","versions":[]}
{"L":"info","timestamp":"2025-06-02T20:47:22.555Z","C":"signozschemamigrator/main.go:127","M":"Parsed DSN","optsError":"json: unsupported type: func(context.Context, string) (net.Conn, error)"}
{"L":"info","timestamp":"2025-06-02T20:47:22.556Z","C":"signozschemamigrator/main.go:133","M":"Opened connection"}
Error: failed to bootstrap migrations: failed to create dbs
failed to create dbs
code: 516, message: posta: Authentication failed: password is incorrect, or there is no user with such name.
Usage:
signoz-schema-migrator sync [flags]
Flags:
--down string Down migrations to run, comma separated. Must provide down migrations explicitly to run
-h, --help help for sync
--up string Up migrations to run, comma separated. Leave empty to run all up migrations
Global Flags:
--cluster-name string Cluster name to use while running migrations (default "cluster")
--dev Development mode
--dsn string Clickhouse DSN
--replication Enable replication
The values.yaml we're using for test is as follow:
global:
storageClass: vsan
clickhouse:
enabled: false
externalClickhouse:
host: clickhouse-posta.clickhouse.svc.cluster.local
cluster: cluster
user: "posta"
password: "postapw"
secure: false
httpPort: 8123
tcpPort: 9000
I can confirm that the clickhouse user can connect from anywhere, create databases, tables, etc. I event tried pre-create the databases but no luck.
Can someone shed a light what are we missing on this?
Thank you!Gil
06/04/2025, 7:56 AMGutemberg Ribeiro
06/04/2025, 10:30 AMGutemberg Ribeiro
06/04/2025, 10:30 AMGil
06/04/2025, 12:57 PMGutemberg Ribeiro
06/04/2025, 12:57 PMGutemberg Ribeiro
06/04/2025, 12:57 PMGil
06/04/2025, 1:06 PMcat Dockerfile
# use a minimal debian image
FROM debian:bookworm-slim
# define arguments and default values
ARG TARGETOS TARGETARCH
ARG USER_UID=10001
# create a non-root user for running the migrator
USER ${USER_UID}
# copy the binaries from the multi-stage build
COPY .build/${TARGETOS}-${TARGETARCH}/signoz-schema-migrator /signoz-schema-migrator
# run the binary as the entrypoint and pass the default dsn as a flag
ENTRYPOINT [ "/signoz-schema-migrator" ]
CMD ["--dsn", "<tcp://localhost:9000>"]
So the script is creating a non root user to runGutemberg Ribeiro
06/04/2025, 1:07 PMGutemberg Ribeiro
06/04/2025, 1:08 PMGil
06/04/2025, 1:09 PMGil
06/04/2025, 1:09 PMGutemberg Ribeiro
06/04/2025, 1:10 PMGutemberg Ribeiro
06/04/2025, 1:11 PMGil
06/04/2025, 1:20 PMGutemberg Ribeiro
06/04/2025, 1:22 PMGutemberg Ribeiro
06/04/2025, 1:22 PMGil
06/04/2025, 1:23 PMGil
06/04/2025, 1:23 PMGil
06/04/2025, 1:26 PMGutemberg Ribeiro
06/04/2025, 1:26 PMGil
06/04/2025, 1:27 PMGutemberg Ribeiro
06/04/2025, 1:27 PMNagesh Bansal
06/04/2025, 1:44 PMGutemberg Ribeiro
06/04/2025, 1:44 PMGutemberg Ribeiro
06/04/2025, 1:46 PMclickhouse/clickhouse-server:23.8
Gutemberg Ribeiro
06/04/2025, 1:46 PM- signoz version: 'v0.85.3'
- otel-collector version: 'v0.111.42'
Nagesh Bansal
06/04/2025, 1:49 PM24.1.2
version of clickhouse?
Also, you must whitelist the IP address range used for your environment (eg, Kubernetes nodes IPs) manually in the Signoz values chart.
clickhouse:
allowedNetworkIps:
- "192.173.0.0/16"
Gutemberg Ribeiro
06/04/2025, 1:50 PMNagesh Bansal
06/04/2025, 1:56 PMGutemberg Ribeiro
06/04/2025, 1:56 PMGutemberg Ribeiro
06/04/2025, 1:56 PMapiVersion: "<http://clickhouse.altinity.com/v1|clickhouse.altinity.com/v1>"
kind: "ClickHouseInstallation"
metadata:
name: "posta"
spec:
defaults:
templates:
dataVolumeClaimTemplate: default
podTemplate: clickhouse:19.6
configuration:
users:
posta/password: "postapw"
posta/profile: default
posta/grants/query:
- "GRANT ALL ON *.*"
zookeeper:
nodes:
- host: zookeeper.zoons.svc.cluster.local
port: 2181
clusters:
- name: cluster
layout:
shardsCount: 1
replicasCount: 3
templates:
volumeClaimTemplates:
- name: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
podTemplates:
- name: clickhouse:19.6
spec:
containers:
- name: clickhouse-pod
image: clickhouse/clickhouse-server:23.8
Srikanth Chekuri
06/04/2025, 3:13 PMSrikanth Chekuri
06/04/2025, 3:14 PMGil
06/04/2025, 3:24 PMSrikanth Chekuri
06/04/2025, 3:26 PMGil
06/04/2025, 3:27 PMSrikanth Chekuri
06/04/2025, 3:30 PMSrikanth Chekuri
06/04/2025, 3:45 PMGutemberg Ribeiro
06/04/2025, 3:46 PMGutemberg Ribeiro
06/04/2025, 3:46 PMposta
user?Srikanth Chekuri
06/04/2025, 3:46 PMGutemberg Ribeiro
06/04/2025, 3:46 PMClickhouseInstallation
resource itself.Srikanth Chekuri
06/04/2025, 3:48 PM/etc/clickhouse-server/users.d
clickhouse server?Gutemberg Ribeiro
06/04/2025, 3:49 PMGutemberg Ribeiro
06/04/2025, 3:49 PMGutemberg Ribeiro
06/04/2025, 3:50 PM# cat chop-generated-users.xml
<clickhouse>
<users>
<clickhouse_operator>
<access_management>1</access_management>
<named_collection_control>1</named_collection_control>
<networks>
<ip>10.42.1.116</ip>
</networks>
<password_sha256_hex>716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448</password_sha256_hex>
<profile>clickhouse_operator</profile>
<show_named_collections>1</show_named_collections>
<show_named_collections_secrets>1</show_named_collections_secrets>
</clickhouse_operator>
<default>
<networks>
<host_regexp>(chi-posta-[^.]+\d+-\d+|clickhouse\-posta)\.clickhouse\.svc\.cluster\.local$</host_regexp>
<ip>::1</ip>
<ip>127.0.0.1</ip>
<ip>10.42.1.157</ip>
<ip>10.42.1.158</ip>
<ip>10.42.1.159</ip>
</networks>
<profile>default</profile>
<quota>default</quota>
</default>
<posta>
<grants>
<query>GRANT ALL ON *.*</query>
</grants>
<networks>
<host_regexp>(chi-posta-[^.]+\d+-\d+|clickhouse\-posta)\.clickhouse\.svc\.cluster\.local$</host_regexp>
<ip>::1</ip>
<ip>127.0.0.1</ip>
</networks>
<password_sha256_hex>0753ebbfaa69a64a7a6006a33afb6d81b222970a2fd3bd586fb1eb0b37854482</password_sha256_hex>
<profile>default</profile>
<quota>default</quota>
</posta>
</users>
</clickhouse>
Gutemberg Ribeiro
06/04/2025, 3:51 PM# cat users.xml
<clickhouse>
<!-- See also the files in users.d directory where the settings can be overridden. -->
<!-- Profiles of settings. -->
<profiles>
<!-- Default settings. -->
<default>
</default>
<!-- Profile that allows only read queries. -->
<readonly>
<readonly>1</readonly>
</readonly>
</profiles>
<!-- Users and ACL. -->
<users>
<!-- If user name was not specified, 'default' user is used. -->
<default>
<!-- See also the files in users.d directory where the password can be overridden.
Password could be specified in plaintext or in SHA256 (in hex format).
If you want to specify password in plaintext (not recommended), place it in 'password' element.
Example: <password>qwerty</password>.
Password could be empty.
If you want to specify SHA256, place it in 'password_sha256_hex' element.
Example: <password_sha256_hex>65e84be33532fb784c48129675f9eff3a682b27168c0ea744b2cf58ee02337c5</password_sha256_hex>
Restrictions of SHA256: impossibility to connect to ClickHouse using MySQL JS client (as of July 2019).
If you want to specify double SHA1, place it in 'password_double_sha1_hex' element.
Example: <password_double_sha1_hex>e395796d6546b1b65db9d665cd43f0e858dd4303</password_double_sha1_hex>
If you want to specify a previously defined LDAP server (see 'ldap_servers' in the main config) for authentication,
place its name in 'server' element inside 'ldap' element.
Example: <ldap><server>my_ldap_server</server></ldap>
If you want to authenticate the user via Kerberos (assuming Kerberos is enabled, see 'kerberos' in the main config),
place 'kerberos' element instead of 'password' (and similar) elements.
The name part of the canonical principal name of the initiator must match the user name for authentication to succeed.
You can also place 'realm' element inside 'kerberos' element to further restrict authentication to only those requests
whose initiator's realm matches it.
Example: <kerberos />
Example: <kerberos><realm><http://EXAMPLE.COM|EXAMPLE.COM></realm></kerberos>
How to generate decent password:
Execute: PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha256sum | tr -d '-'
In first line will be password and in second - corresponding SHA256.
How to generate double SHA1:
Execute: PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha1sum | tr -d '-' | xxd -r -p | sha1sum | tr -d '-'
In first line will be password and in second - corresponding double SHA1.
-->
<password></password>
<!-- List of networks with open access.
To open access from everywhere, specify:
<ip>::/0</ip>
To open access only from localhost, specify:
<ip>::1</ip>
<ip>127.0.0.1</ip>
Each element of list has one of the following forms:
<ip> IP-address or network mask. Examples: 213.180.204.3 or 10.0.0.1/8 or 10.0.0.1/255.255.255.0
2a02:6b8::3 or 2a02:6b8::3/64 or 2a02:6b8::3/ffff:ffff:ffff:ffff::.
<host> Hostname. Example: <http://server01.clickhouse.com|server01.clickhouse.com>.
To check access, DNS query is performed, and all received addresses compared to peer address.
<host_regexp> Regular expression for host names. Example, ^server\d\d-\d\d-\d\.clickhouse\.com$
To check access, DNS PTR query is performed for peer address and then regexp is applied.
Then, for result of PTR query, another DNS query is performed and all received addresses compared to peer address.
Strongly recommended that regexp is ends with $
All results of DNS requests are cached till server restart.
-->
<networks>
<ip>::/0</ip>
</networks>
<!-- Settings profile for user. -->
<profile>default</profile>
<!-- Quota for user. -->
<quota>default</quota>
<!-- User can create other users and grant rights to them. -->
<!-- <access_management>1</access_management> -->
</default>
</users>
<!-- Quotas. -->
<quotas>
<!-- Name of quota. -->
<default>
<!-- Limits for time interval. You could specify many intervals with different limits. -->
<interval>
<!-- Length of interval. -->
<duration>3600</duration>
<!-- No limits. Just calculate resource usage for time interval. -->
<queries>0</queries>
<errors>0</errors>
<result_rows>0</result_rows>
<read_rows>0</read_rows>
<execution_time>0</execution_time>
</interval>
</default>
</quotas>
</clickhouse>
Gutemberg Ribeiro
06/04/2025, 3:51 PMGutemberg Ribeiro
06/04/2025, 3:51 PMGutemberg Ribeiro
06/04/2025, 3:52 PMSrikanth Chekuri
06/04/2025, 3:55 PM<posta>
<networks>
<host_regexp>(chi-posta-[^.]+\d+-\d+|clickhouse\-posta)\.clickhouse\.svc\.cluster\.local$</host_regexp>
<ip>::1</ip>
<ip>127.0.0.1</ip>
</networks>
<!-- ... -->
</posta>
I can upgrade it. But by default the chart whitelist all RFC private subnets CIDRs. As I mentioned, I can connect on other machines on the network. Even remotely from my machine over VPN. It is not a networking issue.Something doesn't add up because posta user has such a restrictive access.
Gutemberg Ribeiro
06/04/2025, 3:56 PMGutemberg Ribeiro
06/04/2025, 3:56 PMSrikanth Chekuri
06/04/2025, 3:57 PMlocalhost:50372
?Gutemberg Ribeiro
06/04/2025, 3:57 PMGutemberg Ribeiro
06/04/2025, 3:57 PMSrikanth Chekuri
06/04/2025, 3:58 PMGutemberg Ribeiro
06/04/2025, 3:58 PMGutemberg Ribeiro
06/04/2025, 3:58 PMGutemberg Ribeiro
06/04/2025, 3:58 PMGutemberg Ribeiro
06/04/2025, 3:58 PMSrikanth Chekuri
06/04/2025, 4:00 PMGutemberg Ribeiro
06/04/2025, 4:01 PMGutemberg Ribeiro
06/04/2025, 4:01 PMGutemberg Ribeiro
06/04/2025, 4:01 PMSrikanth Chekuri
06/04/2025, 4:02 PMGutemberg Ribeiro
06/04/2025, 4:02 PMSrikanth Chekuri
06/04/2025, 4:04 PM<posta>
<grants>
<query>GRANT ALL ON *.*</query>
</grants>
<networks>
<host_regexp>(chi-posta-[^.]+\d+-\d+|clickhouse\-posta)\.clickhouse\.svc\.cluster\.local$</host_regexp>
<ip>::1</ip>
<ip>127.0.0.1</ip>
</networks>
<password_sha256_hex>0753ebbfaa69a64a7a6006a33afb6d81b222970a2fd3bd586fb1eb0b37854482</password_sha256_hex>
<profile>default</profile>
<quota>default</quota>
</posta>
This ClickHouse user has a very restrictive access. With this user, only it's own localhost or other shards can connectGutemberg Ribeiro
06/04/2025, 4:04 PMGutemberg Ribeiro
06/04/2025, 4:04 PMGutemberg Ribeiro
06/04/2025, 4:04 PMSrikanth Chekuri
06/04/2025, 4:07 PMGutemberg Ribeiro
06/04/2025, 4:12 PM