following error displayed on the screen,... :la...
# general
j
following error displayed on the screen,... 🟡 Starting the SigNoz containers. It may take a few minutes ... [+] Building 0.0s (0/0) docker:default [+] Running 10/10 ✔️ Container hotrod Started 0.1s ✔️ Container signoz-zookeeper-1 Started 0.1s ✔️ Container load-hotrod Started 0.1s ✔️ Container signoz-clickhouse Healthy 0.1s ✘ Container otel-migrator service "otel-collector-migrator" didn't complete successfully: exit 1 0.1s ✔️ Container signoz-query-service Created 0.1s ✔️ Container signoz-alertmanager Created 0.1s ✔️ Container signoz-otel-collector Created 0.1s ✔️ Container signoz-logspout Created 0.1s ✔️ Container signoz-frontend Created 0.1s service "otel-collector-migrator" didn't complete successfully: exit 1 Waiting for all containers to start. This check will timeout in 1 seconds ....
a
hi, you might find more information about the issues with --> docker logs
Maybe could be a bad configuration on the
otel-collector-config.yaml
j
thanks for the update , I checked the log for the continer otel-collector there are some error
a
maybe can help
j
I checked the archive messages, My signoz folder in /home/ubuntu location , As a newbie I can't understand how to move out from the installation issues.
p
✘ Container otel-migrator service "otel-collector-migrator" didn't complete successfully: exit 1 0.1s
@Jyothish can you share the
otel-migrator
container logs?
Also, do share output of this command:
docker ps -a
@Angeles MP The solution in the shared link is for WSL Ubuntu (Windows), where the cross-platform volume mounting is not possible.
signoz on my ubuntu Ec2 instance
Since, the user mentioned EC2 instance, I don't think that would solve this problem.
j
Hi . Here is the result of docker ps -a
root@ip-172-31-14-88:/home/ubuntu/signoz/deploy# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 76644b11bc27 signoz/frontend:0.45.0 "nginx -g 'daemon of…" 2 minutes ago Created signoz-frontend 234c98a5742e gliderlabs/logspout:v3.2.14 "/bin/logspout syslo…" 2 minutes ago Created signoz-logspout 9ac00f2c3d09 signoz/signoz-otel-collector:0.88.22 "/signoz-collector -…" 2 minutes ago Created signoz-otel-collector 2c26e58999d2 signoz/alertmanager:0.23.5 "/bin/alertmanager -…" 2 minutes ago Created signoz-alertmanager 0824efe4361c signoz/query-service:0.45.0 "./query-service -co…" 2 minutes ago Created signoz-query-service 133abc47b884 signoz/signoz-schema-migrator:0.88.22 "/signoz-schema-migr…" 2 minutes ago Exited (1) 2 minutes ago otel-migrator 805dcf25f35a clickhouse/clickhouse-server:24.1.2-alpine "/entrypoint.sh" 2 minutes ago Up 2 minutes (healthy) 0.0.0.0:8123->8123/tcp, :::8123->8123/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 0.0.0.0:9181->9181/tcp, :::9181->9181/tcp, 9009/tcp signoz-clickhouse 207c6794e8a3 bitnami/zookeeper:3.7.1 "/opt/bitnami/script…" 2 minutes ago Exited (3) 2 minutes ago signoz-zookeeper-1 ecc6853455ee signoz/locust:1.2.3 "/docker-entrypoint.…" 2 minutes ago Up 2 minutes 5557-5558/tcp, 8089/tcp load-hotrod 12af72dd3bdb jaegertracing/example-hotrod:1.30 "/go/bin/hotrod-linu…" 2 minutes ago Up 2 minutes 8080-8083/tcp hotrod root@ip-172-31-14-88:/home/ubuntu/signoz/deploy#
p
Looks like zookeeper container has exited. Can you share its logs?
Copy code
docker logs signoz-zookeeper-1
j
please have a look.
root@ip-172-31-14-88:/home/ubuntu/signoz/deploy# docker logs 133abc47b884 {"level":"info","timestamp":"2024-05-09T100328.732Z","caller":"signozschemamigrator/migrate.go:89","msg":"Setting env var SIGNOZ_CLUSTER","component":"migrate cli","cluster-name":"cluster"} {"level":"info","timestamp":"2024-05-09T100328.732Z","caller":"signozschemamigrator/migrate.go:106","msg":"Successfully set env var SIGNOZ_CLUSTER ","component":"migrate cli","cluster-name":"cluster"} {"level":"info","timestamp":"2024-05-09T100328.732Z","caller":"signozschemamigrator/migrate.go:111","msg":"Setting env var SIGNOZ_REPLICATED","component":"migrate cli","replication":false} {"level":"info","timestamp":"2024-05-09T100328.738Z","caller":"migrationmanager/manager.go:76","msg":"Running migrations for all migrators","component":"migrationmanager"} {"level":"info","timestamp":"2024-05-09T100328.739Z","caller":"migrationmanager/manager.go:78","msg":"Running migrations for logs","component":"migrationmanager","migrator":"logs"} {"level":"error","timestamp":"2024-05-09T100328.743Z","caller":"migrationmanager/manager.go:81","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"logs","error":"failed to create database, err: code: 999, message: Cannot use any of provided ZooKeeper nodes","stacktrace":"github.com/SigNoz/signoz-otel-collector/migrationmanager.(*MigrationManager).Migrate\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go81\nmain.main\n\t/home/runner/work/signoz otel collector/signoz otel collector/cmd/signozschemamigrator/migrate.go126\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.9/x64/src/runtime/proc.go:267"} {"level":"fatal","timestamp":"2024-05-09T100328.743Z","caller":"signozschemamigrator/migrate.go:128","msg":"Failed to run migrations","component":"migrate cli","error":"failed to create database, err: code: 999, message: Cannot use any of provided ZooKeeper nodes","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go128\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.9/x64/src/runtime/proc.go267"} root@ip-172-31-14-88:/home/ubuntu/signoz/deploy#
gnoz/deploy# docker logs signoz-zookeeper-1 zookeeper 100256.94 zookeeper 100256.95 Welcome to the Bitnami zookeeper container zookeeper 100256.95 Subscribe to project updates by watching https://github.com/bitnami/containers zookeeper 100256.95 Submit issues and feature requests at https://github.com/bitnami/containers/issues zookeeper 100256.95 zookeeper 100256.95 INFO ==> * Starting ZooKeeper setup * zookeeper 100256.99 WARN ==> You have set the environment variable ALLOW_ANONYMOUS_LOGIN=yes. For safety reasons, do not use this flag in a production environment. zookeeper 100257.09 INFO ==> Initializing ZooKeeper... zookeeper 100257.09 INFO ==> No injected configuration file found, creating default config files... zookeeper 100257.20 INFO ==> No additional servers were specified. ZooKeeper will run in standalone mode... zookeeper 100257.22 INFO ==> Deploying ZooKeeper from scratch... zookeeper 100257.23 INFO ==> * ZooKeeper setup finished! * zookeeper 100257.25 INFO ==> * Starting ZooKeeper * /opt/bitnami/java/bin/java ZooKeeper JMX enabled by default Using config: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg mkdir: cannot create directory '/bitnami/zookeeper': Permission denied 2024-05-09 100258,662 [myid:] - INFO [main:QuorumPeerConfig@174] - Reading configuration from: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg 2024-05-09 100258,679 [myid:] - INFO [main:QuorumPeerConfig@444] - clientPortAddress is 0.0.0.0:2181 2024-05-09 100258,681 [myid:] - INFO [main:QuorumPeerConfig@448] - secureClientPort is not set 2024-05-09 100258,681 [myid:] - INFO [main:QuorumPeerConfig@464] - observerMasterPort is not set 2024-05-09 100258,681 [myid:] - INFO [main:QuorumPeerConfig@481] - metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider 2024-05-09 100258,683 [myid:] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2024-05-09 100258,685 [myid:] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 1 2024-05-09 100258,686 [myid:] - WARN [main:QuorumPeerMain@139] - Either no config or no quorum defined in config, running in standalone mode 2024-05-09 100258,688 [myid:] - INFO [main:ManagedUtil@46] - Log4j 1.2 jmx support not found; jmx disabled. 2024-05-09 100258,691 [myid:] - INFO [main:QuorumPeerConfig@174] - Reading configuration from: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg 2024-05-09 100258,692 [myid:] - INFO [main:QuorumPeerConfig@444] - clientPortAddress is 0.0.0.0:2181 2024-05-09 100258,692 [myid:] - INFO [main:QuorumPeerConfig@448] - secureClientPort is not set 2024-05-09 100258,693 [myid:] - INFO [main:QuorumPeerConfig@464] - observerMasterPort is not set 2024-05-09 100258,693 [myid:] - INFO [main:QuorumPeerConfig@481] - metricsProvider.className is org.apache.zookeeper.metrics.impl.DefaultMetricsProvider 2024-05-09 100258,688 [myid:] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@139] - Purge task started. 2024-05-09 100258,693 [myid:] - INFO [main:ZooKeeperServerMain@123] - Starting server 2024-05-09 100258,742 [myid:] - INFO [PurgeTask:FileTxnSnapLog@124] - zookeeper.snapshot.trust.empty : false 2024-05-09 100258,744 [myid:] - ERROR [PurgeTask:DatadirCleanupManager$PurgeTask@143] - Error occurred while purging. org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory /bitnami/zookeeper/data/version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:136) at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:80) at org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:141) at java.base/java.util.TimerThread.mainLoop(Timer.java:556) at java.base/java.util.TimerThread.run(Timer.java:506) 2024-05-09 100258,748 [myid:] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@145] - Purge task completed. 2024-05-09 100258,749 [myid:] - INFO [main:ServerMetrics@62] - ServerMetrics initialized with provider org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@17a7f733 2024-05-09 100258,752 [myid:] - INFO [main:DigestAuthenticationProvider@47] - ACL digest algorithm is: SHA1 2024-05-09 100258,752 [myid:] - INFO [main:DigestAuthenticationProvider@61] - zookeeper.DigestAuthenticationProvider.enabled = true 2024-05-09 100258,753 [myid:] - INFO [main:FileTxnSnapLog@124] - zookeeper.snapshot.trust.empty : false 2024-05-09 100258,753 [myid:] - ERROR [main:ZooKeeperServerMain@81] - Unable to access datadir, exiting abnormally org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Unable to create data directory /bitnami/zookeeper/data/version-2 at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:136) at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:139) at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:113) at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:68) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:141) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91) Unable to access datadir, exiting abnormally 2024-05-09 100258,755 [myid:] - INFO [main:ZKAuditProvider@42] - ZooKeeper audit is disabled. 2024-05-09 100258,757 [myid:] - ERROR [main:ServiceUtils@48] - Exiting JVM with code 3 root@ip-172-31-14-88:/home/ubuntu/signoz/deploy#
p
File permissions for the zookeeper data directory is not right.
Copy code
ls -al deploy/docker/clickhouse-setup/data/zookeeper-1
j
drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 10:14 data
So I need to change it to root.? Because I am running the install.sh as a root user in the instance
a
Ohh my apologies @Jyothish did not rely well the cause. Are you still facing this issue?
j
yes, I think I have some issue with the Ec2 instance.
the permission issue. of the folder
p
Try updating the owner.
The issue could happen when you switch the user back and fourth.
Since we have data mounting from the host machine path, the file ownership and permissions become important factors. You could actually opt for the solution suggested by @Angeles MP in that case. 😅 Your data will be persisted in Docker volume instead of volume mounted from host machine.
596 Views