Hi everyone I just discovered SigNoz I m trying to deploy it

Question

Hi everyone I just discovered SigNoz I m trying to deploy it via Kubernetes by following the tutorial with Helm Some pods are crashing because of an unsuccessful DB access Logs from the pod my release signoz otel collector ddb76f957 kxjp9 ```time= 2022 09 01T12 49 09Z level=info msg= Executing nCREATE DATABASE IF NOT EXISTS signoz metrics n component=clickhouse Error cannot build pipelines failed to create clickhouselogsexporter exporter in pipeline logs cannot configure clickhouse logs exporter code 516 message admin Authentication failed password is incorrect or there is no user with such name```

Prashant Shahi · Answer

Hey < Valentin Lorand> wave That usually happens due to clickhouse IP whitelisting Could you please share the output of the following ```kubectl n platform get pods o=wide```

Valentin Lorand · Answer

```NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES chi signoz cluster 0 0 0 1 1 Running 0 14m 100 64 0 55 scw k8s vld default 368a16ee0c6b403eb3ee422bcb lt none gt lt none gt clickhouse operator 6c966f59cd dk7pv 2 2 Running 0 14m 100 64 0 197 scw k8s vld default 368a16ee0c6b403eb3ee422bcb lt none gt lt none gt my release signoz alertmanager 0 0 1 Pending 0 14m lt none gt lt none gt lt none gt lt none gt my release signoz frontend cf9b8d8c9 rkj7n 0 1 Init 0 1 0 14m 100 64 0 145 scw k8s vld default 368a16ee0c6b403eb3ee422bcb lt none gt lt none gt my release signoz otel collector ddb76f957 kg7lk 0 1 CrashLoopBackOff 3 26s ago 74s 100 64 0 106 scw k8s vld default 368a16ee0c6b403eb3ee422bcb lt none gt lt none gt my release signoz otel collector metrics 5dd74686c 9g9sc 0 1 CrashLoopBackOff 6 2m55s ago 14m 100 64 0 119 scw k8s vld default 368a16ee0c6b403eb3ee422bcb lt none gt lt none gt my release signoz query service 0 0 1 CrashLoopBackOff 7 19s ago 14m 100 64 0 98 scw k8s vld default 368a16ee0c6b403eb3ee422bcb lt none gt lt none gt my release zookeeper 0 1 1 Running 0 14m 100 64 0 19 scw k8s vld default 368a16ee0c6b403eb3ee422bcb lt none gt lt none gt ```

Valentin Lorand · Answer

Indeed it s seems that it s a IP access problem More logs here ``` chi signoz cluster 0 0 0 2022 09 01 13 37 57 003959 11 lt Error gt Access user directories from ffff 100 64 0 119 user admin Authentication failed Code 195 DB Exception Connections from ffff 100 64 0 119 are not allowed IP ADDRESS NOT ALLOWED Stack trace when copying this message always include the lines below ``` ``` chi signoz cluster 0 0 0 2022 09 01 13 37 57 003780 11 lt Warning gt AddressPatterns Failed to check if the allowed client hosts contain address ffff 100 64 0 119 DB Exception Cannot getnameinfo ffff 100 64 0 119 Name or service not known code = 198```

Prashant Shahi · Answer

You can either whitelist all IPv4 IPv6 sources or allow SigNoz the pod IP address `100 64 0 0 16` <https github com SigNoz charts blob main charts signoz values yaml L117 L120>

Prashant Shahi · Answer

Include the above IP range in `clickhouse allowedNetworkIps` list

Prashant Shahi · Answer

To learn more about it <https clickhouse com docs en operations settings settings users user namenetworks>

Valentin Lorand · Answer

Mmmh it s seems that By default anything within a private network will be allowed by clickhouse thinking face Another strange thing I don t know if it s linked ``` my release signoz otel collector metrics 54d4778cbf b2qs4 my release signoz otel collector metrics init wget bad address my release clickhouse 8123 my release signoz otel collector metrics 54d4778cbf b2qs4 my release signoz otel collector metrics init waiting for clickhouseDB```

Prashant Shahi · Answer

yes signoz clickhouse chart by default allows following IPs ``` 10 0 0 0 8 172 16 0 0 12 192 168 0 0 16 ```

Benjamin Carriou · Answer

` my release signoz otel collector 5cgbv my release signoz otel collector init waiting for clickhouseDB` ` my release signoz query service 0 my release signoz query service init wget bad address my release clickhouse 8123 ` ` my release signoz query service 0 my release signoz query service init waiting for clickhouseDB` ` my release signoz otel collector 5cgbv my release signoz otel collector init wget bad address my release clickhouse 8123 ` ` my release signoz otel collector 5cgbv my release signoz otel collector init waiting for clickhouseDB` ` my release signoz otel collector metrics 54d4778cbf p6hp2 my release signoz otel collector metrics init wget bad address my release clickhouse 8123 ` ` my release signoz otel collector metrics 54d4778cbf p6hp2 my release signoz otel collector metrics init waiting for clickhouseDB` ` my release signoz query service 0 my release signoz query service init wget bad address my release clickhouse 8123 ` ` my release signoz query service 0 my release signoz query service init waiting for clickhouseDB`

Benjamin Carriou · Answer

Hi I had ` 0 0 0 0 0 ` in the allowedNetworksIps array

Benjamin Carriou · Answer

hmmmm maybe a serviceAccount problem `I0901 14 48 48 489450 1 labeler go 80 OPERATOR POD NAMESPACE=platform OPERATOR POD NAME=my release clickhouse operator 5cbd5c88d5 6frhb` `E0901 14 48 48 576474 1 labeler go 209 labelDeployment platform my release clickhouse operator ERROR get Deployment platform my release clickhouse operator` `E0901 14 48 48 576707 1 controller go 488 Run ERROR label objects will retry Err deployments apps my release clickhouse operator is forbidden User system serviceaccount platform my release clickhouse operator cannot get resource deployments in API group apps in the namespace platform ` `I0901 14 48 53 577790 1 labeler go 80 OPERATOR POD NAMESPACE=platform OPERATOR POD NAME=my release clickhouse operator 5cbd5c88d5 6frhb` `E0901 14 48 53 688019 1 labeler go 209 labelDeployment platform my release clickhouse operator ERROR get Deployment platform my release clickhouse operator` `E0901 14 48 53 688047 1 controller go 488 Run ERROR label objects will retry Err deployments apps my release clickhouse operator is forbidden User system serviceaccount platform my release clickhouse operator cannot get resource deployments in API group apps in the namespace platform `

Prashant Shahi · Answer

I don t think it is related to the previous issue It was IP whitelisting issue before The one above seems to be with RBAC

Prashant Shahi · Answer

Could you please share more details about your K8s cluster How did you set it up And how many nodes and their types

Prashant Shahi · Answer

it works without any issue for EKS kind and k3s

Prashant Shahi · Answer

< Benjamin Carriou> If you are using any updated `override values yaml` do share that as well

Prashant Shahi · Answer

gt deployments apps my release clickhouse operator is forbidden I was able to reproduce the error above Creating PR to fix it in some time

Benjamin Carriou · Answer

Ok thx Do you know the fix To try it on my cluster

Prashant Shahi · Answer

< Benjamin Carriou> It is out now However the error above is unrelated to the IP whitelist issue

Prashant Shahi · Answer

< Benjamin Carriou> < Valentin Lorand> IP whitelisting issue is resolved in the latest charts release Now by default we whitelist all private IP ranges from the reserved IPv4 list