https://signoz.io logo
Join Slack
Powered by
Hi guys. I have deployed Signoz to Kubernetes and ...
# general
m
Hi guys. I have deployed Signoz to Kubernetes and I have run into signoz-schema-migrator-sync-init-8nqkq pod error 
{"L":"info","timestamp":"2024-12-03T21:24:16.080Z","C":"schema_migrator/manager.go:395","M":"Mutation details","database":"signoz_logs","table":"schema_migrations_v2","command":"UPDATE status = 'failed', error = 'failed to wait for mutations\\nbackoff stopped', updated_at = '2024-12-03 21:14:50' WHERE migration_id = 1","mutation_id":"0000000007","latest_fail_reason":"Code: 198. DB::NetException: Not found address of host: core2-clicksignoz-prod-n2. (DNS_ERROR) (version 24.11.1.2557 (official build))"}
I see that it tries to connect to 
core2-clicksignoz-prod-n2
. It is a hostname of one of external clickhouse nodes, BUT I defined IP address in helm chart.
Copy code
...
spec:
  containers:
  - args:
    - sync
    - --dsn
    - tcp://$(CLICKHOUSE_USER):$(CLICKHOUSE_PASSWORD)@10.xx.x.220:9001
    - --replication
    - --up=
    env:
    - name: CLICKHOUSE_HOST
      value: 10.xx.x.220
    - name: CLICKHOUSE_PORT
      value: "9001"
    - name: CLICKHOUSE_HTTP_PORT
      value: "8124"
    - name: CLICKHOUSE_CLUSTER
      value: cluster
...
How to resolve this problem? Thank you in advance
s
The migrator inspects the info from 
system.clusters
table and ensures no node has active mutations running before running a migration, in this case, it found the address 
core2-clicksignoz-prod-n2
, it should be accessible for the migrator to run successfully.
m
But I have already checked system.clusters table and there is no any hostnames. Let me show you
Copy code
SELECT *
FROM system.clusters
FORMAT vertical

Query id: 291cd44a-16db-4fb3-8243-70e58f6e496c

Row 1:
──────
cluster:                 cluster
shard_num:               1
shard_weight:            1
internal_replication:    1
replica_num:             1
host_name:               10.xx.x.221
host_address:            10.xx.x.221
port:                    9000
is_local:                1
user:                    default
default_database:        
errors_count:            0
slowdowns_count:         0
estimated_recovery_time: 0
database_shard_name:     
database_replica_name:   
is_active:               ᴺᵁᴸᴸ
replication_lag:         ᴺᵁᴸᴸ
recovery_time:           ᴺᵁᴸᴸ

Row 2:
──────
cluster:                 cluster
shard_num:               1
shard_weight:            1
internal_replication:    1
replica_num:             2
host_name:               10.xx.x.222
host_address:            10.xx.x.222
port:                    9000
is_local:                0
user:                    default
default_database:        
errors_count:            0
slowdowns_count:         0
estimated_recovery_time: 0
database_shard_name:     
database_replica_name:   
is_active:               ᴺᵁᴸᴸ
replication_lag:         ᴺᵁᴸᴸ
recovery_time:           ᴺᵁᴸᴸ

2 rows in set. Elapsed: 0.001 sec.
50 Views
Open in Slack
PreviousNext
Powered by