https://signoz.io logo
Join Slack
Powered by
Hi, We are facing some issues while trying to use...
# support
d
Hi, We are facing some issues while trying to use network_mode: host docker-compose file has multiple interdependent services We're getting error due to clickhouse services on which other services are dependent. Like query service is unable to connect to some internal db and throwing error clickhouseReader/reader.go:90 error connecting to primary db: dial tcp: lookup clickhouse on 127.0.0.5353 server misbehaving We checked clickhouse service logs and it says address already in use. <Warning> Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9009 (version 21.12.3.32 (official build)) Similarly for 8123 and 9009, but we checked that these ports are being used by the given service itself.
#support
we are using network mode host in docker compose file for all the services. as we are getting issues in using IP generated by docker container which is not in our allowed IP ranges.
p
@Danish Ahmad Ansari Can you share your docker compose file? Not very sure on how things change with network_mode though
d
version: "2.4" services: clickhouse: image: yandex/clickhouse-server:21.12.3.32 # ports: # - "9000:9000" # - "8123:8123" volumes: - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml - ./data/clickhouse/:/var/lib/clickhouse/ restart: on-failure logging: options: max-size: 50m max-file: "3" healthcheck: # "clickhouse", "client", "-u ${CLICKHOUSE_USER}", "--password ${CLICKHOUSE_PASSWORD}", "-q 'SELECT 1'" test: ["CMD", "wget", "--spider", "-q", "localhost:8123/ping"] interval: 30s timeout: 5s retries: 3 network_mode: host alertmanager: image: signoz/alertmanager:0.6.1 volumes: - ./data/alertmanager:/data depends_on: query-service: condition: service_healthy restart: on-failure command: - --queryService.url=http://query-service:8080 - --storage.path=/data network_mode: host # Notes for Maintainers/Contributors who will change Line Numbers of Frontend & Query-Section. Please Update Line Numbers in 
./scripts/commentLinesForSetup.sh
& 
./CONTRIBUTING.md
query-service: image: signoz/query-service:0.7.5 container_name: query-service command: ["-config=/root/config/prometheus.yml"] volumes: - ./prometheus.yml:/root/config/prometheus.yml - ../dashboards:/root/config/dashboards - ./data/signoz/:/var/lib/signoz/ environment: - ClickHouseUrl=tcp://clickhouse:9000 - STORAGE=clickhouse - GODEBUG=netdns=go - TELEMETRY_ENABLED=true - DEPLOYMENT_TYPE=docker-standalone-amd restart: on-failure healthcheck: test: ["CMD", "wget", "--spider", "-q", "localhost:8080/api/v1/version"] interval: 30s timeout: 5s retries: 3 depends_on: clickhouse: condition: service_healthy network_mode: host frontend: image: signoz/frontend:0.7.5 container_name: frontend restart: on-failure depends_on: - alertmanager - query-service ports: - "3022:3301" volumes: - ../common/nginx-config.conf:/etc/nginx/conf.d/default.conf network_mode: host otel-collector: image: signoz/otelcontribcol:0.43.0 command: ["--config=/etc/otel-collector-config.yaml"] volumes: - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml ports: - "3023:4317" # OTLP gRPC receiver - "3024:4318" # OTLP HTTP receiver # - "8889:8889" # Prometheus metrics exposed by the agent # - "13133" # health_check # - "14268:14268" # Jaeger receiver # - "55678:55678" # OpenCensus receiver # - "55679:55679" # zpages extension # - "55680:55680" # OTLP gRPC legacy receiver # - "55681:55681" # OTLP HTTP legacy receiver mem_limit: 2000m restart: on-failure depends_on: clickhouse: condition: service_healthy network_mode: host otel-collector-metrics: image: signoz/otelcontribcol:0.43.0 command: ["--config=/etc/otel-collector-metrics-config.yaml"] volumes: - ./otel-collector-metrics-config.yaml:/etc/otel-collector-metrics-config.yaml restart: on-failure depends_on: clickhouse: condition: service_healthy network_mode: host
This is the error we get when you use our own allowed IP with network mode host. Poco::Exception. Code: 1000, e.code() = 0, Not found: user_files_path (version 21.12.3.32 (official build)) Poco::Exception. Code: 1000, e.code() = 0, Not found: logger.log (version 21.12.3.32 (official build)) Poco::Exception. Code: 1000, e.code() = 0, Not found: logger.errorlog (version 21.12.3.32 (official build)) Processing configuration file '/etc/clickhouse-server/config.xml'. Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'. Include not found: clickhouse_remote_servers Include not found: clickhouse_compression Logging information to console 2022.05.10 091030.510276 [ 1 ] {} <Information> SentryWriter: Sending crash reports is disabled 2022.05.10 091030.600292 [ 1 ] {} <Information> : Starting ClickHouse 21.12.3.32 with revision 54457, build id: FA4A7F489F3FF6E3, PID 1 2022.05.10 091030.600422 [ 1 ] {} <Information> Application: starting up 2022.05.10 091030.600472 [ 1 ] {} <Information> Application: OS name: Linux, version: 5.13.0-1025-oracle, architecture: x86_64 2022.05.10 091030.815839 [ 1 ] {} <Warning> Application: Calculated checksum of the binary: 5BEBF5792A40F7E345921EDA3698245B. There is no information about the reference checksum. Include not found: clickhouse_remote_servers Include not found: clickhouse_compression 2022.05.10 091030.823762 [ 1 ] {} <Information> Application: Setting max_server_memory_usage was set to 13.18 GiB (14.64 GiB available * 0.90 max_server_memory_usage_to_ram_ratio) 2022.05.10 091030.832642 [ 1 ] {} <Information> Application: Uncompressed cache size was lowered to 7.32 GiB because the system has low amount of memory 2022.05.10 091030.835096 [ 1 ] {} <Information> Context: Initialized background executor for merges and mutations with num_threads=16, num_tasks=32 2022.05.10 091030.835910 [ 1 ] {} <Information> Context: Initialized background executor for move operations with num_threads=8, num_tasks=8 2022.05.10 091030.836690 [ 1 ] {} <Information> Context: Initialized background executor for fetches with num_threads=8, num_tasks=8 2022.05.10 091030.838409 [ 1 ] {} <Information> Context: Initialized background executor for common operations (e.g. clearing old parts) with num_threads=8, num_tasks=8 2022.05.10 091030.839891 [ 1 ] {} <Information> Application: Loading user defined objects from /var/lib/clickhouse/ 2022.05.10 091030.839978 [ 1 ] {} <Information> Application: Loading metadata from /var/lib/clickhouse/ 2022.05.10 091030.842340 [ 1 ] {} <Information> DatabaseAtomic (system): Metadata processed, database system has 1 tables and 0 dictionaries in total. 2022.05.10 091030.842548 [ 1 ] {} <Information> TablesLoader: Parsed metadata of 1 tables in 1 databases in 0.002090954 sec 2022.05.10 091030.842753 [ 1 ] {} <Information> TablesLoader: Loading 1 tables with 0 dependency level 2022.05.10 091030.881294 [ 1 ] {} <Information> DatabaseCatalog: Found 0 partially dropped tables. Will load them and retry removal. 2022.05.10 091030.884045 [ 1 ] {} <Information> DatabaseAtomic (default): Metadata processed, database default has 3 tables and 0 dictionaries in total. 2022.05.10 091030.884710 [ 1 ] {} <Information> DatabaseAtomic (signoz_metrics): Metadata processed, database signoz_metrics has 2 tables and 0 dictionaries in total. 2022.05.10 091030.884743 [ 1 ] {} <Information> TablesLoader: Parsed metadata of 5 tables in 2 databases in 0.00279007 sec 2022.05.10 091030.884753 [ 1 ] {} <Information> TablesLoader: Loading 5 tables with 0 dependency level 2022.05.10 091030.960156 [ 1 ] {} <Information> DatabaseAtomic (default): Starting up tables. 2022.05.10 091030.960424 [ 90 ] {} <Information> BackgroundSchedulePool/BgSchPool: Create BackgroundSchedulePool with 128 threads 2022.05.10 091030.980685 [ 1 ] {} <Information> DatabaseAtomic (signoz_metrics): Starting up tables. 2022.05.10 091030.981606 [ 1 ] {} <Information> DatabaseAtomic (system): Starting up tables. 2022.05.10 091030.985939 [ 1 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_priority' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems. 2022.05.10 091031.000198 [ 1 ] {} <Information> Application: Listening for http//[]8123 2022.05.10 091031.000618 [ 1 ] {} <Information> Application: Listening for connections with native protocol (tcp): [:]9000 2022.05.10 091031.001012 [ 1 ] {} <Information> Application: Listening for replica communication (interserver): http//[]9009 2022.05.10 091031.001849 [ 1 ] {} <Warning> Application: Listen [0.0.0.0]:8123 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:8123 (version 21.12.3.32 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host> 2022.05.10 091031.002280 [ 1 ] {} <Warning> Application: Listen [0.0.0.0]:9000 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9000 (version 21.12.3.32 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host> 2022.05.10 091031.002698 [ 1 ] {} <Warning> Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9009 (version 21.12.3.32 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host> 2022.05.10 091031.004903 [ 1 ] {} <Information> DNSCacheUpdater: Update period 15 seconds 2022.05.10 091031.005240 [ 1 ] {} <Information> Application: Available RAM: 14.64 GiB; physical cores: 1; logical cores: 2. 2022.05.10 091031.006527 [ 1 ] {} <Information> Application: Ready for connections.
@Ankit Nayan @Ashu
345 Views
Open in Slack
PreviousNext
Powered by