This message was deleted.
# supports
Slackbot
04/21/2023, 11:20 AMThis message was deleted.
✅ 1
s
Syed Muhammad Hassan
04/21/2023, 1:45 PM@Goutham Sridhar Please check your clickhouse server. It is stuck in init pods and init pods tries to make connection with clickhouse server. If it is successfull it will proceed otherwise it will stuck in init state
a
Anil Kumar Bandrapalli
04/21/2023, 2:04 PM@Syed Muhammad Hassan, my clickhouse server is running my-release-clickhouse-operator-9fc495f79-6dls
g
Goutham Sridhar
04/21/2023, 2:04 PM@Syed Muhammad Hassan click house operator server is running fine and I have checked logs.
Copy code
I0421 13:58:09.204356 1 rest_server.go:38] Starting metrics exporter at ':8888/metrics'
I0421 13:58:09.219345 1 exporter.go:318] Add explicitly found CHI tempsignoz/my-release-clickhouse with 1 hosts
I0421 13:58:09.219370 1 exporter.go:158] Added ClickHouseInstallation (tempsignoz/my-release-clickhouse): including hostnames into Exporter Copy code
I0421 13:58:06.978004 1 controller.go:497] Run():ClickHouseInstallation controller: starting workers number: 11
I0421 13:58:06.978031 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 1 out of 11
I0421 13:58:06.978052 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 2 out of 11
I0421 13:58:06.978067 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 3 out of 11
I0421 13:58:06.978088 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 4 out of 11
I0421 13:58:06.978110 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 5 out of 11
I0421 13:58:06.978161 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 6 out of 11
I0421 13:58:06.978182 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 7 out of 11
I0421 13:58:06.978198 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 8 out of 11
I0421 13:58:06.978322 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 9 out of 11
I0421 13:58:06.978343 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 10 out of 11
I0421 13:58:06.978360 1 controller.go:499] Run():ClickHouseInstallation controller: starting worker 11 out of 11
I0421 13:58:06.978388 1 controller.go:509] Run():ClickHouseInstallation controller: workers starteds
Syed Muhammad Hassan
04/21/2023, 2:05 PMcheck the init container logs, it will tell you what it is trying to do
a
Anil Kumar Bandrapalli
04/21/2023, 2:09 PMAll init pods are showing same error
Copy code
Error from server (BadRequest): container "my-release-signoz-query-service" in pod "my-release-signoz-query-service-0" is waiting to start: PodInitializingAnil Kumar Bandrapalli
04/21/2023, 2:18 PMBut i could see some errors from k8s otel agent
Copy code
2023-04-21T14:12:13.539Z error exporterhelper/queued_retry.go:175 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "max elapsed time expired rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.90.13.186:4317: connect: connection refused\"", "dropped_items": 1056}Anil Kumar Bandrapalli
04/21/2023, 2:42 PM@Syed Muhammad Hassan, can we get any update on this ?
Anil Kumar Bandrapalli
04/21/2023, 2:50 PMunable to setup this signoz. kindly help us out
s
Srikanth Chekuri
04/21/2023, 3:23 PMWhat does the
describe podg
Goutham Sridhar
04/21/2023, 4:14 PMAll the pods are showing container is created and started. But pod is still initializing
s
Syed Muhammad Hassan
04/21/2023, 8:26 PMI think there is problem with clickhouse server.
Copy code
connect: connection refuseda
Anil Kumar Bandrapalli
04/21/2023, 9:05 PMhow we gonna solve it
Anil Kumar Bandrapalli
04/21/2023, 9:06 PMwould you please help us in this ?
Anil Kumar Bandrapalli
04/24/2023, 7:16 AM@Srikanth Chekuri / @Syed Muhammad Hassan, could you please help me on this ?
p
Prashant Shahi
04/25/2023, 8:26 AM@Goutham Sridhar It looks like there is no clickhouse instance pod (chi-pod).
can you run
helm upgradePrashant Shahi
04/25/2023, 8:27 AMit should spin up the chi pod, if that was terminated due to some reason.
n
Nond H
06/22/2023, 8:09 PM@Goutham Sridhar Did you figure out a solution? If so, can you please lmk?
a
Anil Kumar Bandrapalli
06/30/2023, 8:43 AMafter upgrading helm , we are able to run the signoz.
27 Views