This message was deleted.
s
This message was deleted.
โœ… 1
p
Hey @Rahul Tiwari! Are you following the documentation steps correctly? You would have to create
override-values.yaml
with the YAML present in the steps. https://signoz.io/docs/install/kubernetes/aws/#chart-configuration
r
@Prashant Shahi i have created the override-values.yaml file and the issue is resolved. Thank you.
๐Ÿ‘ 1
p
kube config directory permissions seems to be insecure. you can resolve with this:
Copy code
sudo chmod 600 ~/.kube/config
r
@Prashant Shahi thank you for that also
@Prashant Shahi unable to access signoz tool.
^C[ec2-user@ip-10-0-4-191 ~]kubectl get pods -n platform -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES chi-signoz-cluster-0-0-0 1/1 Running 0 12m 10.0.2.209 ip-10-0-2-165.us-west-2.compute.internal <none> <none> clickhouse-operator-766b6b8947-26zlj 2/2 Running 0 12m 10.0.2.33 ip-10-0-2-165.us-west-2.compute.internal <none> <none> my-release-signoz-alertmanager-0 1/1 Running 0 12m 10.0.1.91 ip-10-0-1-48.us-west-2.compute.internal <none> <none> my-release-signoz-frontend-7f484ffb5-7g4jx 1/1 Running 0 12m 10.0.1.17 ip-10-0-1-153.us-west-2.compute.internal <none> <none> my-release-signoz-otel-collector-797fc4cdb6-k8626 1/1 Running 0 12m 10.0.3.86 ip-10-0-3-245.us-west-2.compute.internal <none> <none> my-release-signoz-otel-collector-metrics-6f5c849cc9-jsvkt 1/1 Running 0 12m 10.0.3.138 ip-10-0-3-245.us-west-2.compute.internal <none> <none> my-release-signoz-query-service-0 1/1 Running 0 12m 10.0.1.34 ip-10-0-1-234.us-west-2.compute.internal <none> <none> my-release-zookeeper-0 1/1 Running 0 12m 10.0.1.33 ip-10-0-1-48.us-west-2.compute.internal <none> <none> [ec2-user@ip-10-0-4-191 ~]$ kubectl --namespace platform port-forward svc/$SERVICE_NAME 3301:3301 Forwarding from 127.0.0.1:3301 -> 3301 Forwarding from [:1]3301 -> 3301
Getting
This site canโ€™t be reached http://10.0.1.17:3301/ is unreachable.ERR_ADDRESS_UNREACHABLE
p
you will need to run the following to expose the frontend UI locally:
Copy code
export SERVICE_NAME=$(kubectl get svc --namespace platform -l "<http://app.kubernetes.io/component=frontend|app.kubernetes.io/component=frontend>" -o jsonpath="{.items[0].metadata.name}")

kubectl --namespace platform port-forward svc/$SERVICE_NAME 3301:3301
you will be able to access it in your local machine: localhost:3301
r
@Prashant Shahi i run the above but still unable to access signoz dashboard
i tried with localhost:3301 and also <my-release-signoz-frontend-7f484ffb5-7g4jx:3301> but still not working
p
when you use
port-forward
, frontend UI will be tunnelled to
localhost
in the machine where you execute that command.
You could execute the following command to get a load balancer from AWS:
Copy code
helm upgrade --install -n platform my-release signoz/signoz \
--set frontend.serviceType="LoadBalancer"
r
i am using aws eks and i have install signoz in one of the amazon linux jumpbox machine
p
After executing above commands:
Copy code
kubectl -n platform get svc  -l "<http://app.kubernetes.io/component=frontend|app.kubernetes.io/component=frontend>"
You should see External IP
r
external IP is showing as empty
[ec2-user@ip-10-0-4-191 ~]$ helm upgrade --install -n platform my-release signoz/signoz --set frontend.serviceType="LoadBalancer" Release "my-release" has been upgraded. Happy Helming! NAME: my-release LAST DEPLOYED: Mon Jun 13 125918 2022 NAMESPACE: platform STATUS: deployed REVISION: 3 NOTES: 1. You have just deployed SigNoz cluster: - frontend version: '0.8.1' - query-service version: '0.8.1' - alertmanager version: '0.23.0-0.1' - otel-collector version: '0.45.1-0.2' - otel-collector-metrics version: '0.45.1-0.2' 2. Get the application URL by running these commands: export POD_NAME=$(kubectl get pods --namespace platform -l "app.kubernetes.io/name=signoz,app.kubernetes.io/instance=my-release,app.kubernetes.io/component=frontend" -o jsonpath="{.items[0].metadata.name}") echo "Visit http://127.0.0.1:3301 to use your application" kubectl --namespace platform port-forward $POD_NAME 3301:3301 If you have any ideas, questions, or any feedback, please share on our Github Discussions: https://github.com/SigNoz/signoz/discussions/713 [ec2-user@ip-10-0-4-191 ~]$ kubectl -n platform get svc -l "app.kubernetes.io/component=frontend" NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-release-signoz-frontend ClusterIP 172.20.18.56 <none> 3301/TCP 28m [ec2-user@ip-10-0-4-191 ~]$
@Prashant Shahi even i tried helm upgrade --install -n platform my-release signoz/signoz --set my-release-signoz-frontend.serviceType="LoadBalancer"
p
can you try this again?
Copy code
kubectl -n platform get svc  -l "<http://app.kubernetes.io/component=frontend|app.kubernetes.io/component=frontend>"
your frontend service type is still
ClusterIP
r
but still external ip is empty and svc is of type CLUSTERIP
p
it should be updated to
LoadBalancer
helm upgrade --install -n platform my-release signoz/signoz --set my-release-signoz-frontend.serviceType="LoadBalancer"
This is incorrect. => You should use this updated command:
Copy code
helm upgrade --install -n platform my-release signoz/signoz \
--set frontend.service.type="LoadBalancer"
^^ updated
r
no it is still showing type as CLUSTERIP only
[ec2-user@ip-10-0-4-191 ~]$ helm upgrade --install -n platform my-release signoz/signoz \
--set frontend.serviceType="LoadBalancer"
Release "my-release" has been upgraded. Happy Helming! NAME: my-release LAST DEPLOYED: Mon Jun 13 131157 2022 NAMESPACE: platform STATUS: deployed REVISION: 5 NOTES: 1. You have just deployed SigNoz cluster: - frontend version: '0.8.1' - query-service version: '0.8.1' - alertmanager version: '0.23.0-0.1' - otel-collector version: '0.45.1-0.2' - otel-collector-metrics version: '0.45.1-0.2' 2. Get the application URL by running these commands: export POD_NAME=$(kubectl get pods --namespace platform -l "app.kubernetes.io/name=signoz,app.kubernetes.io/instance=my-release,app.kubernetes.io/component=frontend" -o jsonpath="{.items[0].metadata.name}") echo "Visit http://127.0.0.1:3301 to use your application" kubectl --namespace platform port-forward $POD_NAME 3301:3301 If you have any ideas, questions, or any feedback, please share on our Github Discussions: https://github.com/SigNoz/signoz/discussions/713 [ec2-user@ip-10-0-4-191 ~]$ kubectl get svc -n platform NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE chi-signoz-cluster-0-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 39m clickhouse-operator-metrics ClusterIP 172.20.132.206 <none> 8888/TCP 41m my-release-clickhouse ClusterIP 172.20.83.105 <none> 8123/TCP,9000/TCP 40m my-release-signoz-alertmanager ClusterIP 172.20.237.241 <none> 9093/TCP 41m my-release-signoz-alertmanager-headless ClusterIP None <none> 9093/TCP 41m my-release-signoz-frontend ClusterIP 172.20.18.56 <none> 3301/TCP 41m my-release-signoz-otel-collector ClusterIP 172.20.157.129 <none> 4317/TCP,4318/TCP,55680/TCP,55681/TCP,14250/TCP,14268/TCP,9411/TCP,8888/TCP,8889/TCP 41m my-release-signoz-otel-collector-metrics ClusterIP 172.20.33.70 <none> 4317/TCP,4318/TCP,55680/TCP,55681/TCP,14250/TCP,14268/TCP,9411/TCP,8888/TCP 41m my-release-signoz-query-service ClusterIP 172.20.24.176 <none> 8080/TCP,8085/TCP 41m my-release-zookeeper ClusterIP 172.20.216.19 <none> 2181/TCP,2888/TCP,3888/TCP 41m my-release-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 41m [ec2-user@ip-10-0-4-191 ~]$
p
@Rahul Tiwari can you run the updated command?
r
@Prashant Shahi should i take yaml file of frontend svc and edit it to loadbalancer
p
My typo.. Turns out it is
frontend.service.type
and not
frontend.serviceType
r
ok
yes now i can see the svc as type loadbalancer
how to access the signoz dashboard now
podip:3301 ?
p
Copy code
kubectl -n platform get svc  -l "<http://app.kubernetes.io/component=frontend|app.kubernetes.io/component=frontend>"
External IP
r
i tried with externalip:3301
but unable to access
do i also need to do portforwarding? i think no
@Prashant Shahi the external ip name which i got is one of the load balancer of type classic from AWS
p
Copy code
http://<load-balancer-address>:3301
this one works.. I just confirmed on my eks cluster.
r
@Prashant Shahi it is not working for me, can you send me the service yaml file for exporting the loadbalancer
@Prashant Shahi also i tried to change load balancer listener from TCP to HTTP for port 3301 and 32114 in aws console
p
can you send me output of the following?
Copy code
kubectl -n platform get pods,svc
r
[ec2-user@ip-10-0-4-191 ~]$ kubectl -n platform get pods,svc NAME READY STATUS RESTARTS AGE pod/chi-signoz-cluster-0-0-0 1/1 Running 0 131m pod/clickhouse-operator-766b6b8947-26zlj 2/2 Running 0 156m pod/my-release-signoz-alertmanager-0 1/1 Running 0 156m pod/my-release-signoz-frontend-7f484ffb5-7g4jx 1/1 Running 0 156m pod/my-release-signoz-otel-collector-797fc4cdb6-k8626 1/1 Running 0 156m pod/my-release-signoz-otel-collector-metrics-6f5c849cc9-jsvkt 1/1 Running 0 156m pod/my-release-signoz-query-service-0 1/1 Running 0 156m pod/my-release-zookeeper-0 1/1 Running 0 156m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/chi-signoz-cluster-0-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 155m service/clickhouse-operator-metrics ClusterIP 172.20.132.206 <none> 8888/TCP 156m service/my-release-clickhouse ClusterIP 172.20.83.105 <none> 8123/TCP,9000/TCP 156m service/my-release-signoz-alertmanager ClusterIP 172.20.237.241 <none> 9093/TCP 156m service/my-release-signoz-alertmanager-headless ClusterIP None <none> 9093/TCP 156m service/my-release-signoz-frontend LoadBalancer 172.20.18.56 a38e116d2074a41c9a2233850ce5ed9a-210884774.us-west-2.elb.amazonaws.com 3301:32114/TCP 74m service/my-release-signoz-otel-collector ClusterIP 172.20.157.129 <none> 4317/TCP,4318/TCP,55680/TCP,55681/TCP,14250/TCP,14268/TCP,9411/TCP,8888/TCP,8889/TCP 156m service/my-release-signoz-otel-collector-metrics ClusterIP 172.20.33.70 <none> 4317/TCP,4318/TCP,55680/TCP,55681/TCP,14250/TCP,14268/TCP,9411/TCP,8888/TCP 156m service/my-release-signoz-query-service ClusterIP 172.20.24.176 <none> 8080/TCP,8085/TCP 156m service/my-release-zookeeper ClusterIP 172.20.216.19 <none> 2181/TCP,2888/TCP,3888/TCP 156m service/my-release-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 156m service/signoz-service LoadBalancer 172.20.166.140 ae38856dfa8794c63b0de5a85c71c588-1600739069.us-west-2.elb.amazonaws.com 3301:31320/TCP 27m [ec2-user@ip-10-0-4-191 ~]$
@Prashant Shahi pls ignore service/signoz-service as i have manually created it
p
cluster looks healthy.. Should have worked.
Can you share output of this?
Copy code
kubectl describe -n platform service/my-release-signoz-frontend
and this?
Copy code
kubectl -n platform get pods -o wide
r
[ec2-user@ip-10-0-4-191 ~]$ kubectl describe svc my-release-signoz-frontend -n platform Name: my-release-signoz-frontend Namespace: platform Labels: app.kubernetes.io/component=frontend app.kubernetes.io/instance=my-release app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=signoz app.kubernetes.io/version=0.8.1 helm.sh/chart=signoz-0.0.17 release=my-release Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"meta.helm.sh/release-name":"my-release","meta.helm.sh/release-namespace":"... meta.helm.sh/release-name: my-release meta.helm.sh/release-namespace: platform Selector: app.kubernetes.io/component=frontend,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=signoz Type: LoadBalancer IP: 172.20.18.56 LoadBalancer Ingress: a38e116d2074a41c9a2233850ce5ed9a-210884774.us-west-2.elb.amazonaws.com Port: http 3301/TCP TargetPort: 3301/TCP NodePort: http 32114/TCP Endpoints: 10.0.1.17:3301 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal EnsuringLoadBalancer 29m (x3 over 85m) service-controller Ensuring load balancer Normal EnsuredLoadBalancer 29m (x3 over 85m) service-controller Ensured load balancer [ec2-user@ip-10-0-4-191 ~]$
p
output for this one?
Copy code
kubectl -n platform get pods -o wide
r
[ec2-user@ip-10-0-4-191 ~]$ kubectl -n platform get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES chi-signoz-cluster-0-0-0 1/1 Running 0 162m 10.0.2.241 ip-10-0-2-165.us-west-2.compute.internal <none> <none> clickhouse-operator-766b6b8947-26zlj 2/2 Running 0 3h7m 10.0.2.33 ip-10-0-2-165.us-west-2.compute.internal <none> <none> my-release-signoz-alertmanager-0 1/1 Running 0 3h7m 10.0.1.91 ip-10-0-1-48.us-west-2.compute.internal <none> <none> my-release-signoz-frontend-7f484ffb5-7g4jx 1/1 Running 0 3h7m 10.0.1.17 ip-10-0-1-153.us-west-2.compute.internal <none> <none> my-release-signoz-otel-collector-797fc4cdb6-k8626 1/1 Running 0 3h7m 10.0.3.86 ip-10-0-3-245.us-west-2.compute.internal <none> <none> my-release-signoz-otel-collector-metrics-6f5c849cc9-jsvkt 1/1 Running 0 3h7m 10.0.3.138 ip-10-0-3-245.us-west-2.compute.internal <none> <none> my-release-signoz-query-service-0 1/1 Running 0 3h7m 10.0.1.34 ip-10-0-1-234.us-west-2.compute.internal <none> <none> my-release-zookeeper-0 1/1 Running 0 3h7m 10.0.1.33 ip-10-0-1-48.us-west-2.compute.internal <none> <none> [ec2-user@ip-10-0-4-191 ~]$
p
So, the
IP
of frontend pod should be part of frontend svc
Endpoints
r
yes
both are same
endpoints of the svc and my-release-signoz-frontend-7f484ffb5-7g4jx pod IP
p
debugging is not gonna be easy.. as the issue seems to be from your eks cluster.. as it works fine on a fresh eks cluster. it could be version mismatch issue, EKS nodes issue, AWS LoadBalancer side issue, or something entirely different.
can you try this?
Copy code
helm upgrade --install -n platform my-release signoz/signoz \
--set frontend.service.type="ClusterIP"
after couple of minutes:
Copy code
helm upgrade --install -n platform my-release signoz/signoz \
--set frontend.service.type="LoadBalancer"
if this doesn't resolve, I can investigate your cluster-info dump.
you can share that over DM
r
For doing this I need to delete that SVC naa
p
service/signoz-service? why do you even have that ๐Ÿค”
also, did you
edit
or
update
anything else?
r
No i edited only that frontend svc
Although I deleted that signoz-service which I created manually
[ec2-user@ip-10-0-4-191 ~]$ kubectl -n platform get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE chi-signoz-cluster-0-0 ClusterIP None <none> 8123/TCP,9000/TCP,9009/TCP 3h21m clickhouse-operator-metrics ClusterIP 172.20.132.206 <none> 8888/TCP 3h22m my-release-clickhouse ClusterIP 172.20.83.105 <none> 8123/TCP,9000/TCP 3h22m my-release-signoz-alertmanager ClusterIP 172.20.237.241 <none> 9093/TCP 3h22m my-release-signoz-alertmanager-headless ClusterIP None <none> 9093/TCP 3h22m my-release-signoz-frontend LoadBalancer 172.20.18.56 a38e116d2074a41c9a2233850ce5ed9a-210884774.us-west-2.elb.amazonaws.com 3301:32114/TCP 121m my-release-signoz-otel-collector ClusterIP 172.20.157.129 <none> 4317/TCP,4318/TCP,55680/TCP,55681/TCP,14250/TCP,14268/TCP,9411/TCP,8888/TCP,8889/TCP 3h22m my-release-signoz-otel-collector-metrics ClusterIP 172.20.33.70 <none> 4317/TCP,4318/TCP,55680/TCP,55681/TCP,14250/TCP,14268/TCP,9411/TCP,8888/TCP 3h22m my-release-signoz-query-service ClusterIP 172.20.24.176 <none> 8080/TCP,8085/TCP 3h22m my-release-zookeeper ClusterIP 172.20.216.19 <none> 2181/TCP,2888/TCP,3888/TCP 3h22m my-release-zookeeper-headless ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 3h22m [ec2-user@ip-10-0-4-191 ~]$ helm upgrade --install -n platform my-release signoz/signoz --set frontend.service.type="ClusterIP" Error: UPGRADE FAILED: cannot patch "my-release-signoz-frontend" with kind Service: Service "my-release-signoz-frontend" is invalid: spec.ports[0].nodePort: Forbidden: may not be used when
type
is 'ClusterIP' [ec2-user@ip-10-0-4-191 ~]$
do i need to delete first my-release-signoz-frontend svc
p
you can delete, if you had modified it directly before
r
ok
Done
[ec2-user@ip-10-0-4-191 ~]$ kubectl describe svc my-release-signoz-frontend -n platform Name: my-release-signoz-frontend Namespace: platform Labels: app.kubernetes.io/component=frontend app.kubernetes.io/instance=my-release app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=signoz app.kubernetes.io/version=0.8.1 helm.sh/chart=signoz-0.0.17 release=my-release Annotations: meta.helm.sh/release-name: my-release meta.helm.sh/release-namespace: platform Selector: app.kubernetes.io/component=frontend,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=signoz Type: LoadBalancer IP: 172.20.151.195 LoadBalancer Ingress: a9a156a770920446dad8bbe1eb7d59fd-1724916561.us-west-2.elb.amazonaws.com Port: http 3301/TCP TargetPort: http/TCP NodePort: http 31287/TCP Endpoints: 10.0.1.17:3301 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Type 72s service-controller ClusterIP -> LoadBalancer Normal EnsuringLoadBalancer 72s service-controller Ensuring load balancer Normal EnsuredLoadBalancer 70s service-controller Ensured load balancer [ec2-user@ip-10-0-4-191 ~]$
@Prashant Shahi still it is not working
@Prashant Shahi can you send me the command for taking cluster dump
@Prashant Shahi i have taken the dump, how should i share it to you, i cannot find any attachment link.
p
can you sent it over DM?
r
ok