Hi Team, We are getting the following error while ...
# support
v
Hi Team, We are getting the following error while upgrading signoz cluster form 0.31.0 to 0.36.0 (we are using external clickhouse) We are able to telnet to the external clickhouse and the port is also valid. Any suggestions to debug this?
Copy code
{"level":"error","timestamp":"2023-12-27T09:31:27.509Z","caller":"basemigrator/migrator.go:27","msg":"Failed to create clickhouse connection","migrator":"logs","error":"failed to parse dsn: parse \"<tcp://k8s-clickhou-clickhou-4ef2ed6df1-95b9e93260ea37d4.elb.ap-south-1.amazonaws.com>:%!s(float64=9000)?username=signoz&password={{redacted}}\": invalid port \":%!s(float64=9000)\" after host","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/migrationmanager/migrators/basemigrator.New|github.com/SigNoz/signoz-otel-collector/migrationmanager/migrators/basemigrator.New>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/migrators/basemigrator/migrator.go:27\ngithub.com/SigNoz/signoz-otel-collector/migrationmanager.createNewMigrator\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:57\ngithub.com/SigNoz/signoz-otel-collector/migrationmanager.New\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:30\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:102\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.12/x64/src/runtime/proc.go:250"}
{"level":"error","timestamp":"2023-12-27T09:31:27.509Z","caller":"migrationmanager/manager.go:59","msg":"Failed to create base migrator","migrator":"logs","error":"failed to parse dsn: parse \"<tcp://k8s-clickhou-clickhou-4ef2ed6df1-95b9e93260ea37d4.elb.ap-south-1.amazonaws.com>:%!s(float64=9000)?username=signoz&password={{redacted}}\": invalid port \":%!s(float64=9000)\" after host","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/migrationmanager.createNewMigrator|github.com/SigNoz/signoz-otel-collector/migrationmanager.createNewMigrator>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:59\ngithub.com/SigNoz/signoz-otel-collector/migrationmanager.New\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:30\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:102\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.12/x64/src/runtime/proc.go:250"}
{"level":"error","timestamp":"2023-12-27T09:31:27.509Z","caller":"migrationmanager/manager.go:32","msg":"Failed to create logs migrator","component":"migrationmanager","error":"failed to parse dsn: parse \"<tcp://k8s-clickhou-clickhou-4ef2ed6df1-95b9e93260ea37d4.elb.ap-south-1.amazonaws.com>:%!s(float64=9000)?username=signoz&password={{redacted}}\": invalid port \":%!s(float64=9000)\" after host","stacktrace":"<http://github.com/SigNoz/signoz-otel-collector/migrationmanager.New|github.com/SigNoz/signoz-otel-collector/migrationmanager.New>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:32\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:102\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.12/x64/src/runtime/proc.go:250"}
{"level":"fatal","timestamp":"2023-12-27T09:31:27.509Z","caller":"signozschemamigrator/migrate.go:104","msg":"Failed to create migration manager","component":"migrate cli","error":"failed to parse dsn: parse \"<tcp://k8s-clickhou-clickhou-4ef2ed6df1-95b9e93260ea37d4.elb.ap-south-1.amazonaws.com>:%!s(float64=9000)?username=signoz&password={{redacted}}\": invalid port \":%!s(float64=9000)\" after host","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:104\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.12/x64/src/runtime/proc.go:250"}
d
@Vishnu Teja Vallala the port is getting configured as
%!s(float64=9000)
, can you check where does this happen?
this seems to be the error
invalid port \":%!s(float64=9000)\" after host
s
@Vishnu Teja Vallala pass port as string instead in the values
v
Thanks @Srikanth Chekuri @dhawal will try this 😃
Thanks @Srikanth Chekuri @dhawal This worked. The documentation should be updated for this i guess. https://signoz.io/docs/operate/clickhouse/external-clickhouse/ Also I faced one more issue where the signozspanmetrics/prometheus was replaced with signozspanmetrics/cumulative. We had this in our override_values.yaml, i replaced it and it worked. But should these changes be also published in migration guide so that the upgrade would be easier. Please let me know if this is already published.
s
The documentation should be updated for this i guess.
Could you send a PR for this? This is the repo https://github.com/SigNoz/signoz.io
Also I faced one more issue where the signozspanmetrics/prometheus was replaced with signozspanmetrics/cumulative. We had this in our override_values.yaml, i replaced it and it worked.
But should these changes be also published in migration guide so that the upgrade would be easier.
Please let me know if this is already published.
This was an overlooked aspect from our side. We didn't think about the override values situation. We will take care of such details next time. Thanks for pointing it out.
v
Sure @Srikanth Chekuri here is the PR for the same https://github.com/SigNoz/signoz.io/pull/1422