https://signoz.io logo
#support
Title
# support
s

Srinjoy Choudhury

03/16/2022, 1:58 PM
Hello Team, My name is Srinjoy , i had setup a new signoz deployment using the default values and helm on my kubernetes cluster next i deployed the otel collector agent to collect metrics from pods and nodes following https://signoz.io/docs/tutorial/kubernetes-infra-metrics/ But i could not see the metrics anywhere on the frontend ui, Also plot metric in signoz ui instruction doc is empty https://signoz.io/docs/userguide/metrics-dashboard/ 1. I checked all signoz pods are up and running on my cluster 2. I added the endpoint for my signoz application in the collector agent yaml 3. I also checked that each node is running a otel collector agent Am i missing something.
@User
p

Prashant Shahi

03/16/2022, 2:09 PM
Hey @User! You would have to plot for the graph manually using PromQL from Dashboard. The instructions to create dashboard was migrated to the following: https://signoz.io/docs/userguide/dashboards/
we will update our docs shortly to either update the URL or 301 redirect to it.
Yes, it should be running otel collector agent in each Node as it is a DaemonSet. It collects host metrics and kubelet stats from each node and forwards to SigNoz otlp receiver.
s

Srinjoy Choudhury

03/16/2022, 2:32 PM
hey @User yes i get that , one more thing can you tell me how can i say if the otel agent is able to communicate to the collector any log line on the agent or collector which would suggest the same
p

Prashant Shahi

03/16/2022, 2:34 PM
I don't think it does. @User can confirm the same
@User you can verify if any of the following metrics is visible from the Dashboard. https://signoz.io/docs/tutorial/kubernetes-infra-metrics/#list-of-metrics-from-kubernetes-receiver
s

Srinjoy Choudhury

03/16/2022, 3:10 PM
@User no i cant see any data on the dashboards i checked for a few different metrics none of them seem to have any data
p

Prashant Shahi

03/16/2022, 3:17 PM
did you update
IP of machine hosting SigNoz
with accessible OtelCollector IP/address?
you can run troubleshoot to check if your SigNoz cluster is accessible. Also, refer to the IP grid in the end of the page. https://signoz.io/docs/install/troubleshooting/#kubernetes
s

Srinjoy Choudhury

03/16/2022, 4:06 PM
@User yes i had updated the value of the ip and also i am able to run the troubleshoot pod and it worked
@User any updates
p

Prashant Shahi

03/17/2022, 10:17 AM
Hey @User! Last month, I had tested the k8s infra metrics with SigNoz
0.6.x
and I can confirm that it worked.
I just tested with
0.7.1
signoz and
0.43.0
otel collector. And I can confirm that I able to reproduce your issue.
@User I am pushing changes to the repo with the fix.
s

Srinjoy Choudhury

03/17/2022, 10:57 AM
@User just curious to know what was the issue
p

Prashant Shahi

03/17/2022, 10:58 AM
otel collector version mismatch between
otel-collector-k8s
and signoz receiver otel collector.
🙌 1
@User changes have been merged. https://github.com/SigNoz/otel-collector-k8s/pull/1 Can you please sync the changes from the main branch and deploy on your cluster? Do let me know how it goes.
s

Srinjoy Choudhury

03/17/2022, 11:15 AM
yes thanks @User
@User working now able to get the metrics
🎉 2
p

Prashant Shahi

03/17/2022, 11:31 AM
That's great!
a

Ankit Nayan

03/17/2022, 11:36 AM
@User I would suggest to change the scrape interval to
60s
at https://github.com/SigNoz/otel-collector-k8s/blob/main/agent/infra-metrics.yaml#L20 and reapply config and restart pod. Will help in reducing amount of data. @User I would suggest to make this change in repo also.
1
p

Prashant Shahi

03/17/2022, 11:37 AM
a

Ankit Nayan

03/17/2022, 11:39 AM
Yes 💯
3 Views