https://signoz.io logo
Join Slack
Powered by
I have accessed the container and tried to create ...
# general
j
I have accessed the container and tried to create the symlink manually, and indeed I get the same error in the command line:
Copy code
# ln -s /var/lib/clickhouse/store/f2e/f2e32bce-a837-45d6-b2e3-2bcea83785d6 /var/lib/clickhouse/metadata/system
ln: failed to create symbolic link '/var/lib/clickhouse/metadata/system': Operation not permitted
a
and do you have sudo permissions?
did you use 
./install.sh
to run
j
I am running 
ln -s
as root.
I didn’t run install. I git cloned signoz and then 
docker-compose -f ./docker/clickhouse-setup/docker-compose.yaml up -d
At first I had an issue with the clickhouse healthcheck, but fixed it running the service with UID:GID 1000:
Copy code
$ vi docker-compose.yaml

	services:
	  clickhouse:
	      image: yandex/clickhouse-server
	      user: "1000:1000"                  # run clickhouse with user 1000, which is the "docker" user in the Docker host HyperkitVM, which has write permission.
a
what does the otel-collector logs tell you?
installing clickhouse should have been a breeze
j
Yeah, it worked after addressing the UID with which clickhouse runs.
Checking otel-collector logs…
2021-11-19T134201.514Z error clickhouseexporter/writer.go:97 Could not write a batch of spans {“kind”: “exporter”, “name”: “clickhouse”, “error”: “code: 243, message: Cannot reserve 1.00 MiB, not enough space”} 2021-11-19T134201.515238127Z github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhouseexporter.(*SpanWriter).backgroundWriter 2021-11-19T134201.515260110Z /Users/ankitnayan/go/src/github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhouseexporter/writer.go:97
\/Users/ankitnayan … seems familiar, lol
So the “_Cannot reserve 1.00 MiB, not enough space”_ message seems to be what clickhouse is responding.
a
enough disk space?
j
Ah sorry, was responding in the other thread. Yeah enough.
Copy code
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          17G  4.7G   11G  31% /
tmpfs            64M     0   64M   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1        17G  4.7G   11G  31% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           2.0G     0  2.0G   0% /proc/asound
tmpfs           2.0G     0  2.0G   0% /proc/acpi
tmpfs           2.0G     0  2.0G   0% /proc/scsi
tmpfs           2.0G     0  2.0G   0% /sys/firmware
Inodes are all 1%
I think I could try dropping the minikube cluster and recreating with a bit more of RAM (currently 4 GB) and disk (currently 20 GB) to see if it solves it.
Hmm.. don’t think it’ll help:
Copy code
$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          31        24        3.795GB   436.4MB (11%)
Containers      32        21        206.8MB   74.23MB (35%)
Local Volumes   3         3         111.6MB   0B (0%)
Build Cache     0         0         0B        0B
a
can you try running the usual way -> using 
./install.sh
just to see if things work
j
I was able to run it.. but I had to tweak it a little bit. As I said, I am not using Docker.app (desktop) but Minikube (with driver Hyperkit).
The changes I did to the script were: 1. commented out line 226 (skip starting of Docker.app, and just assume the command ‘docker system info’ works, which is) 2. remove all attempts to sudo docker-composer, as in a Homebrew based installation of Minikube the whole thing runs under my local user.
(just removed the ‘sudo’ bits)
With that, all services started, despite the error at the end:
Copy code
Waiting for all containers to start. This check will timeout in 1 seconds ....

+++++++++++ ERROR ++++++++++++++++++++++
🔴 The containers didn't seem to start correctly. Please run the following command to check containers that may have errored out:

docker-compose -f docker/clickhouse-setup/docker-compose.yaml ps -a
Please read our troubleshooting guide <https://signoz.io/docs/deployment/docker/#troubleshooting-of-common-issues>
or reach us on SigNoz for support <https://join.slack.com/t/signoz-community/shared_invite/zt-lrjknbbp-J_mI13rlw8pGF4EWBnorJA>
Doing that check manually says otherwise though:
Copy code
$ docker-compose -f docker/clickhouse-setup/docker-compose.yaml ps -a
                  Name                                 Command                  State                                                                                    Ports
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
clickhouse-setup_clickhouse_1               /entrypoint.sh                   Up (healthy)   0.0.0.0:8123->8123/tcp, 0.0.0.0:9001->9000/tcp, 9009/tcp
clickhouse-setup_otel-collector-metrics_1   /otelcontribcol --config=/ ...   Up             4317/tcp, 55679/tcp, 55680/tcp
clickhouse-setup_otel-collector_1           /otelcontribcol --config=/ ...   Up             0.0.0.0:49180->13133/tcp, 0.0.0.0:14268->14268/tcp, 0.0.0.0:1777->1777/tcp, 0.0.0.0:4317->4317/tcp, 0.0.0.0:49179->55678/tcp, 0.0.0.0:55679->55679/tcp,
                                                                                            0.0.0.0:55680->55680/tcp, 0.0.0.0:55681->55681/tcp, 0.0.0.0:8887->8888/tcp, 0.0.0.0:8889->8889/tcp
frontend                                    nginx -g daemon off;             Up             0.0.0.0:3000->3000/tcp, 80/tcp
hotrod                                      /go/bin/hotrod-linux all         Up             0.0.0.0:9000->8080/tcp, 8081/tcp, 8082/tcp, 8083/tcp
load-hotrod                                 /docker-entrypoint.sh            Up             5557/tcp, 5558/tcp, 0.0.0.0:8089->8089/tcp
query-service                               ./query-service -config=/r ...   Up             0.0.0.0:8080->8080/tcp
Now, same as before, I can access the frontend but nothing there, and I see the same errors in the collector.
This link may be useful, regarding Mac OS X and Minikube as a replacement for Docker desktop: https://itnext.io/goodbye-docker-desktop-hello-minikube-3649f2a1c469
I have the feeling that this all might be related with Minikube/Hyperkit and its way to handle permissions on mounts/volumes.
What beats me is the error when creating that symlink:
Copy code
# ln -s /var/lib/clickhouse/store/f2e/f2e32bce-a837-45d6-b2e3-2bcea83785d6 /var/lib/clickhouse/metadata/system
ln: failed to create symbolic link '/var/lib/clickhouse/metadata/system': Operation not permitted
/var/lib/clickhouse/metadata/
is not a mounted volume AFAIK
Ah! 
/var/lib/clickhouse/
is a volume indeed:
Copy code
services:
  clickhouse:
      image: yandex/clickhouse-server
      user: "1000:1000"
      expose:
        - 8123
        - 9000
      ports:
        - 9001:9000
        - 8123:8123
      volumes:
        - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
        - ./docker-entrypoint-initdb.d/init-db.sql:/docker-entrypoint-initdb.d/init-db.sql
        - ./data/clickhouse/:/var/lib/clickhouse/
I’ll see if I can handle this issue with the Minikube guys. I’ll let you know if I find something.
👍 2
p
cool - that would be great
10 Views
Open in Slack
PreviousNext
Powered by