https://signoz.io logo
Join Slack
Powered by
Hi team! I've just installed signoz in k8s cluster...
# support
a
Hi team! I've just installed signoz in k8s cluster via helm. I can see my java app activities in services section. But there is no any logs in Logs section (No data). In documentation I've read that by default it will automatically start collecting all the pod logs. But in my case not. Any sugestions ?
n
Are you using EKS ?
a
Not EKS. k8s was deployed in bare metal
n
By default this is the path that is checked for pod logs https://github.com/SigNoz/charts/blob/cc57a6dd458e167a4137e729f6d7b29b9667ccca/charts/k8s-infra/values.yaml#L85. 
/var/log/pods/*/*/*.log
@Prashant Shahi any idea if there can be some path issue while running k8s on bare metal ?
p
@Ayan Khamza which container runtime tool are you using for your bare metal cluster?
anyways, updating the above mentioned path to the logs path should do it.
a
Container runtime tool is Docker v20.10.11 Signoz installed with default parameters also with this path for logs. I checked path 
/var/log/pods/*/*/*.log
in worker nodes for logs, there it's ok. Logs of pods are exists.
But still No Data. Should I scale to more replicas for each worker node deployment of signoz-k8s-infra-otel-deployment. To locate in each node to scrape logs maybe ?
p
@Ayan Khamza k8s-infra-otel-agent is the component responsible for scraping logs from the path above.
it is a daemonset, so each node should have one instance of the pod running which is collecting logs.
Logs of the otelAgent pod(s) and clickhouse could help.
a
Copy code
kubectl get pod -n apm-signoz
NAME                                                READY   STATUS    RESTARTS   AGE
chi-signoz-clickhouse-cluster-0-0-0                 1/1     Running   0          8d
signoz-alertmanager-0                               1/1     Running   0          8d
signoz-clickhouse-operator-6b9db9c58d-mvp4t         2/2     Running   0          8d
signoz-frontend-7b77897bd-j6vfc                     1/1     Running   0          8d
signoz-k8s-infra-otel-deployment-5896477d95-55ngp   1/1     Running   0          8d
signoz-k8s-infra-otel-deployment-5896477d95-8p6w7   1/1     Running   0          7d20h
signoz-otel-collector-5c7448c944-6vg28              1/1     Running   0          7d17h
signoz-otel-collector-5c7448c944-dz5sv              1/1     Running   0          7d17h
signoz-otel-collector-5c7448c944-p4b2b              1/1     Running   0          8d
signoz-otel-collector-5c7448c944-vdjsf              1/1     Running   0          7d17h
signoz-otel-collector-metrics-5c8cb64bdf-m5mmn      1/1     Running   0          8d
signoz-query-service-0                              1/1     Running   0          8d
signoz-zookeeper-0                                  1/1     Running   0          8d
Thank's for reply. But I have't any k8s-infra-otel-agent. When I applyed via helm only this kind of resources were installed.
I found that Daemonset of k8s-infra-otel-agent is not Running :) The reason: 
daemonset-controller  Error creating: pods "signoz-k8s-infra-otel-agent-" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used
I should configure pod security policy for access host path for agent pods
Thanks for your help, you helped a lot!
3 Views
Open in Slack
PreviousNext
Powered by