Hello, gonna dig around deep soon, but wanted to get this started. I am self-hosting signoz on an ec2 instance. checking out the repo via git, and running install script as part of the userdata. Things going mostly well so far, just running the docker standalone setup on the instance. It runs great consistently for about 10 hours, then overnight seeing something killing the frontend, zookeeper, collector, alertsmanager, and clickhouse containers all at once.

sudo docker-compose -f ./docker/clickhouse-setup/docker-compose.yaml ps -a
NAME                    IMAGE                                        COMMAND                  SERVICE                         CREATED        STATUS                          PORTS
otel-migrator-async     signoz/signoz-schema-migrator:0.111.21       "/signoz-schema-migr…"   otel-collector-migrator-async   17 hours ago   Exited (0) 17 hours ago         
otel-migrator-sync      signoz/signoz-schema-migrator:0.111.21       "/signoz-schema-migr…"   otel-collector-migrator-sync    17 hours ago   Exited (0) 17 hours ago         
signoz-alertmanager     signoz/alertmanager:0.23.7                   "/bin/alertmanager -…"   alertmanager                    17 hours ago   Exited (0) 8 hours ago          
signoz-clickhouse       clickhouse/clickhouse-server:24.1.2-alpine   "/entrypoint.sh"         clickhouse                      17 hours ago   Exited (0) 8 hours ago          
signoz-frontend         signoz/frontend:0.66.0                       "nginx -g 'daemon of…"   frontend                        17 hours ago   Exited (0) 8 hours ago          
signoz-logspout         gliderlabs/logspout:v3.2.14                  "/bin/logspout syslo…"   logspout                        17 hours ago   Restarting (1) 51 seconds ago   
signoz-otel-collector   signoz/signoz-otel-collector:0.111.21        "/signoz-collector -…"   otel-collector                  17 hours ago   Exited (0) 8 hours ago          
signoz-query-service    signoz/query-service:0.66.0                  "./query-service -co…"   query-service                   17 hours ago   Restarting (1) 36 seconds ago   
signoz-zookeeper-1      bitnami/zookeeper:3.7.1                      "/opt/bitnami/script…"   zookeeper-1                     17 hours ago   Exited (143) 8 hours ago

could use some assistance tracking down what is tipping the app over first, and what I need to do to resolve it and keep it healthy. Examining each service with

sudo docker logs signoz-<exited-container>

and not finding much more than the equivalent of each service got some type of Sigkill. Any others trying to run Signoz in this style of setup seen this before, Or which service is most likely to have originiated the Kill command and I should dig deeper there? Any help would be greatly appreciated!

Did you check if the Docker daemon restarted? It depends on how it's being run. On a machine with systemd it'd be

systemctl status docker

i did not, I will go try to find and report back, yes it should be ubuntu

With the default Docker daemon configuration, running containers will be stopped when the daemon is stopped. So when I see multiple containers ending at the same exact time, it's usually the daemon getting restarted.

yeah okay, if it happens again organically, i will look out for that first