Just updated to signoz to 0.77.0 using helm chart, and had an issue with the

signoz-schema-migrator-sync-init

Job. It seems to have an issue with connecting to Zookeeper.

│ {"L":"info","timestamp":"<>","C":"signozschemamigrator/main.go:104","M":"Up migrations","versions":[]}
│ {"L":"info","timestamp":"<>","C":"signozschemamigrator/main.go:117","M":"Down migrations","versions":[]}
│ {"L":"info","timestamp":"<>","C":"signozschemamigrator/main.go:127","M":"Parsed DSN","optsError":"json: unsupported type: func(context.Context, string) (net.Conn, error)"
│ }
│ {"L":"info","timestamp":"<>","C":"signozschemamigrator/main.go:133","M":"Opened connection"}
│ Error: failed to bootstrap migrations: failed to create dbs
│ failed to create dbs
│ code: 999, message: All connection tries failed while connecting to ZooKeeper. nodes: <IPS>
│ Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: while receiving handshake from ZooKeeper. (CANNOT_READ_ALL_DATA) (version 24.... (official build)), <IP>
│ Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: while receiving handshake from ZooKeeper. (CANNOT_READ_ALL_DATA) (version 24.... (official build)), <IP>
│ Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: while receiving handshake from ZooKeeper. (CANNOT_READ_ALL_DATA) (version 24.... (official build)), <IP>
│ Code: 33. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 4.: while receiving handshake from ZooKeeper. (CANNOT_READ_ALL_DATA) (version 24.... (official build)), <IP>

Zookeeper and Clickhouse are hosted externally, and this is the first time I've seen Signoz interact with Zookeeper. I have not even provided any zookeeper address on the helm chart, so I can only assume that the IPS were collected from Clickhouse. I think this issue might be on our end, the machines with ZK might not be accepting any inbound connections except from the CH hosts and the metrics scrapper, but I'm putting this out there in either case.

The error messages is from clickhouse itself, just that the migrator is printing them as it received from clickhouse. Something is wrong with your clickhouse and zookeeper connection.

@nitya-signoz Thank you for the clarification. This is good to know.