https://signoz.io
Join Slack
This message was deleted.
# support
s
This message was deleted.
n
Are you using EKS ?
a
Not EKS. k8s was deployed in bare metal
n
By default this is the path that is checked for pod logs https://github.com/SigNoz/charts/blob/cc57a6dd458e167a4137e729f6d7b29b9667ccca/charts/k8s-infra/values.yaml#L85. 
/var/log/pods/*/*/*.log
@Prashant Shahi any idea if there can be some path issue while running k8s on bare metal ?
p
@Ayan Khamza which container runtime tool are you using for your bare metal cluster?
anyways, updating the above mentioned path to the logs path should do it.
a
Container runtime tool is Docker v20.10.11 Signoz installed with default parameters also with this path for logs. I checked path 
/var/log/pods/*/*/*.log
in worker nodes for logs, there it's ok. Logs of pods are exists.
But still No Data. Should I scale to more replicas for each worker node deployment of signoz-k8s-infra-otel-deployment. To locate in each node to scrape logs maybe ?
p
@Ayan Khamza k8s-infra-otel-agent is the component responsible for scraping logs from the path above.
it is a daemonset, so each node should have one instance of the pod running which is collecting logs.
Logs of the otelAgent pod(s) and clickhouse could help.
a
Copy code
kubectl get pod -n apm-signoz
NAME                                                READY   STATUS    RESTARTS   AGE
chi-signoz-clickhouse-cluster-0-0-0                 1/1     Running   0          8d
signoz-alertmanager-0                               1/1     Running   0          8d
signoz-clickhouse-operator-6b9db9c58d-mvp4t         2/2     Running   0          8d
signoz-frontend-7b77897bd-j6vfc                     1/1     Running   0          8d
signoz-k8s-infra-otel-deployment-5896477d95-55ngp   1/1     Running   0          8d
signoz-k8s-infra-otel-deployment-5896477d95-8p6w7   1/1     Running   0          7d20h
signoz-otel-collector-5c7448c944-6vg28              1/1     Running   0          7d17h
signoz-otel-collector-5c7448c944-dz5sv              1/1     Running   0          7d17h
signoz-otel-collector-5c7448c944-p4b2b              1/1     Running   0          8d
signoz-otel-collector-5c7448c944-vdjsf              1/1     Running   0          7d17h
signoz-otel-collector-metrics-5c8cb64bdf-m5mmn      1/1     Running   0          8d
signoz-query-service-0                              1/1     Running   0          8d
signoz-zookeeper-0                                  1/1     Running   0          8d
Thank's for reply. But I have't any k8s-infra-otel-agent. When I applyed via helm only this kind of resources were installed.
I found that Daemonset of k8s-infra-otel-agent is not Running :) The reason: 
daemonset-controller  Error creating: pods "signoz-k8s-infra-otel-agent-" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used
I should configure pod security policy for access host path for agent pods
Thanks for your help, you helped a lot!
30 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by