Hii eveyone There s a problem happening every now and then w SigNoz Community #general

Hii eveyone, There's a problem happening every no...

Ritek Saxena

07/25/2022, 10:28 AM

Hii eveyone, There's a problem happening every now and then with me, Whenever I use docker compose to run the signoz service, one of the containers is flagged as unhealthy and the service doesn't start. Sometimes it is the alert manager, sometimes it is otel metrics collector. I am not able to understand the problem with it. I even reinstalled docker and signoz both, it worked once and then the problem started to occur again. Please help

✅ 1

Ritek Saxena

07/25/2022, 10:44 AM

SS also attached for reference

Prashant Shahi

07/25/2022, 10:46 AM

Hi @Ritek Saxena! Can you share logs of unhealthy containers?

Ritek Saxena

07/25/2022, 10:53 AM

This time it is for the alert manager container

Prashant Shahi

07/25/2022, 10:53 AM

looks likely related to clickhouse

Prashant Shahi

07/25/2022, 10:54 AM

query-service, otel-collector, otel-collector-metrics are dependent on clickhouse

Prashant Shahi

07/25/2022, 10:54 AM

alertmanager is dependent on query-service

Prashant Shahi

07/25/2022, 10:54 AM

can you share clickhouse logs?

Ritek Saxena

07/25/2022, 10:55 AM

Sure I will just search how to do that ! and then I will share.

Prashant Shahi

07/25/2022, 10:56 AM

Copy code

docker logs --tail=100 clickhouse-setup_clickhouse_1

Ritek Saxena

07/25/2022, 10:58 AM

Thanks for the command

Prashant Shahi

07/25/2022, 11:00 AM

OpenSSL

crt

issues.. Those config should have been ignore if not enabled.

Prashant Shahi

07/25/2022, 11:02 AM

anyways, can you comment out the following lines from your local cloned repo and retry? https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/clickhouse-config.xml#L239-L240

Ritek Saxena

07/25/2022, 11:03 AM

Sure I will try

Ritek Saxena

07/25/2022, 11:09 AM

Still the same container unhealthy error

Ritek Saxena

07/25/2022, 11:10 AM

This time for the otel collector

Prashant Shahi

07/25/2022, 11:10 AM

Did you bring down the cluster as the config won't be updated if same container is used?

Ritek Saxena

07/25/2022, 11:11 AM

I killed all the containers and then restarted.

Prashant Shahi

07/25/2022, 11:12 AM

okay.. can you share clickhouse logs now?

Ritek Saxena

07/25/2022, 11:12 AM

Sure

Ritek Saxena

07/25/2022, 11:13 AM

Prashant Shahi

07/25/2022, 11:14 AM

Config is not updated

Prashant Shahi

07/25/2022, 11:14 AM

Still same error

Ritek Saxena

07/25/2022, 11:14 AM

I'll check once again

Prashant Shahi

07/25/2022, 11:15 AM

https://signoz.io/docs/operate/docker-standalone/#uninstall

Ritek Saxena

07/25/2022, 11:24 AM

Here's the ss of the config file and I have took down the cluster as well but still the same error is there for the alert manager container

Ritek Saxena

07/25/2022, 11:25 AM

Prashant Shahi

07/25/2022, 11:35 AM

okay, lastly clickhouse and alertmanager logs?

Ritek Saxena

07/25/2022, 11:36 AM

Sure

Ritek Saxena

07/25/2022, 11:40 AM

Prashant Shahi

07/25/2022, 11:42 AM

logs seems to be incomplete

Ritek Saxena

07/25/2022, 11:42 AM

If you're looking for error It is not there this time

Ritek Saxena

07/25/2022, 11:42 AM

Idk how

Ritek Saxena

07/25/2022, 11:43 AM

I will once again run the logs command

Ritek Saxena

07/25/2022, 11:44 AM

I confirm these are the complete logs and no error message is there in the clickhouse logs

Prashant Shahi

07/25/2022, 12:36 PM

Let's catch up over DM to get this resolved

5 Views

Open in Slack

Previous Next