This message was deleted SigNoz Community #support

Join Slack Communities

This message was deleted.

# support

Slackbot

11/17/2023, 1:34 AM

This message was deleted.

✅ 1

Nicolas Rakover

11/17/2023, 1:40 AM

One interesting note, I've been playing around with the

k8s-infra

preset to try and hone in on the problem. I've hard-coded an expected log file path in the

include

attribute, and when I tail the logs of the otel agent pod, I do see it detect the log file I care about:

Nicolas Rakover

11/17/2023, 1:40 AM

Copy code

2023-11-17T01:18:40.789Z	info	fileconsumer/file.go:194	Started watching file from end. To read preexisting logs, configure the argument 'start_at' to 'beginning'	{"kind": "receiver", "name": "filelog/k8s", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/solver_service-00030-deployment-7bc9bb776d-5nfwl_87c60b29-9487-44c3-a209-767ad8410eb9/retrieval-pod/0.log"}

Nicolas Rakover

11/17/2023, 1:40 AM

Moreover, I've confirmed that the pod in question is producing logs after the agent spins up and detects the log file

Nicolas Rakover

11/17/2023, 1:40 AM

yet no logs make it over to SigNoz 😭

nitya-signoz

11/17/2023, 5:49 AM

Are you using EKS ?

Nicolas Rakover

11/17/2023, 5:40 PM

No, but I'm on Oracle Cloud's managed K8s offering (OKE)

Nicolas Rakover

11/17/2023, 10:41 PM

@nitya-signoz any idea what might be going wrong or how to debug further?

nitya-signoz

11/18/2023, 4:15 AM

@Prashant Shahi might be able to help here.

Nicolas Rakover

11/21/2023, 11:48 PM

@Prashant Shahi any suggestions for next steps here?

Prashant Shahi

11/26/2023, 9:52 AM

@Nicolai Rybnikar In Oracle Cloud, it must be using CRI-O instead of Docker. You will need to use crio log path instead. Update include configuration under

presets.logsCollection

Nicolas Rakover

11/27/2023, 5:30 PM

@Prashant Shahi could you elaborate on this a bit more or point me to docs? I have ssh'ed into a box on the cluster and do in fact see log files matching the default pattern.

Nicolas Rakover

11/27/2023, 6:28 PM

Moreover, as of a recent upgrade to my charts, I'm seeing the

filelog/k8s

receiver fail to read from the log files it finds with a

permission denied

error:

Nicolas Rakover

11/27/2023, 6:28 PM

Copy code

2023-11-27T18:27:06.058Z	error	fileconsumer/file.go:190	Failed to open file	{"kind": "receiver", "name": "filelog/k8s", "data_type": "logs", "component": "fileconsumer", "error": "open /var/log/pods/solver_auth-00005-deployment-b89f8d64b-k2ghs_65c16b35-5412-466b-aa54-fced56e5cc37/auth-pod/0.log: permission denied"}
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).makeFingerprint|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).makeFingerprint>
	<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:190|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:190>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).makeReader|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).makeReader>
	<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:226|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:226>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume>
	<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:140|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:140>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).poll|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).poll>
	<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:130|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:130>
<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).startPoller.func1|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).startPoller.func1>
	<http://github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:100|github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza@v0.88.0/fileconsumer/file.go:100>

Nicolas Rakover

11/27/2023, 6:29 PM

☝️ that's from tailing the logs on one of the OTel Agent pods via

kubectl logs -f

Nicolas Rakover

11/27/2023, 10:29 PM

@Prashant Shahi if I

kubectl patch

the OTel Agent Daemonset's

securityContext

runAsUser: 0

, it starts to work. However, 1. I'm currently unable to customize the

securityContext

via the values.yaml file (it doesn't seem to update the daemonset) 2. I'm wondering if there's a more surgical configuration change I should be making instead from a least-privilege standpoint

Nicolas Rakover

11/27/2023, 10:31 PM

Do you have any guidance on the above?

Prashant Shahi

11/28/2023, 5:27 AM

@Nicolas Rakover Running with correct permission would be needed. Running as root i.e.

should do the job.

Prashant Shahi

11/28/2023, 5:27 AM

1. I'm currently unable to customize the
securityContext
via the values.yaml file (it doesn't seem to update the daemonset)

looks like a bug in Otel-Agent. Fixing this right away.

Prashant Shahi

11/28/2023, 5:28 AM

> 2. I'm wondering if there's a more surgical configuration change I should be making instead from a least-privilege standpoint can you try to exec into the otel-agent pod or k8s node and find the log file permissions? You can run the otel-agent with same user and permissions.

Prashant Shahi

11/28/2023, 7:55 AM

Fix for

is out here: https://github.com/SigNoz/charts/pull/349

🙌 1

Nicolas Rakover

11/28/2023, 5:19 PM

Ok yea, the permissions require root 😞

Copy code

-rw-------. 1 root root

😞 1

Nicolas Rakover

11/28/2023, 5:20 PM

Thanks for looking into this, will await the chart fix

Prashant Shahi

11/28/2023, 5:41 PM

The fix is out in

k8s-infra-0.10.1

Prashant Shahi

11/28/2023, 5:42 PM

That will be part of tomorrow's signoz release.

🙌 1

55 Views

Open in Slack

Previous Next