Hi Team, I need support in filtering logs in Kubernetes. i want to send logs to Signoz only from one namespace.. i tried filtering in processors but didn't work. can any one pls help me with this.

@Manikanta Reddy P Can you share some sample logs format, and the config you can in otel processor

otel-agent-config.yaml: |-

exporters:

otlp:

endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}

headers:

signoz-access-token: Bearer ${SIGNOZ_API_KEY}

tls:

cert_file: ${OTEL_SECRETS_PATH}/cert.pem

insecure: ${OTEL_EXPORTER_OTLP_INSECURE}

insecure_skip_verify: ${OTEL_EXPORTER_OTLP_INSECURE_SKIP_VERIFY}

key_file: ${OTEL_SECRETS_PATH}/key.pem

extensions:

health_check:

endpoint: 0.0.0.0:13133

pprof:

endpoint: localhost:1777

zpages:

endpoint: localhost:55679

processors:

batch:

send_batch_size: 10000

timeout: 200ms

k8sattributes:

extract:

metadata:

- k8s.namespace.name

- k8s.pod.name

- k8s.pod.uid

- k8s.pod.start_time

- k8s.deployment.name

- k8s.node.name

passthrough: true

pod_association:

- from: resource_attribute

name: k8s.pod.ip

- from: resource_attribute

name: k8s.pod.uid

- from: connection

resourcedetection:

detectors:

- env

- system

system:

hostname_sources:

- os

timeout: 2s

receivers:

filelog/k8s:

exclude:

- /var/log/pods/kube-system_*.log

- /var/log/pods/*_hotrod*_*/*/*.log

- /var/log/pods/*_locust*_*/*/*.log

include:

- /var/log/pods/*/*/*.log

include_file_name: false

include_file_path: true

operators:

- id: get-format

routes:

- expr: body matches "^\\{"

output: parser-docker

- expr: body matches "^[^ Z]+ "

output: parser-crio

- expr: body matches "^[^ Z]+Z"

output: parser-containerd

type: router

- id: parser-crio

output: extract_metadata_from_filepath

regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$

timestamp:

layout: "2006-01-02T15:04:05.000000000-07:00"

layout_type: gotime

parse_from: attributes.time

type: regex_parser

- id: parser-containerd

output: extract_metadata_from_filepath

regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$

timestamp:

layout: '%Y-%m-%dT%H:%M:%S.%LZ'

parse_from: attributes.time

type: regex_parser

- id: parser-docker

output: extract_metadata_from_filepath

timestamp:

layout: '%Y-%m-%dT%H:%M:%S.%LZ'

parse_from: attributes.time

type: json_parser

- id: extract_metadata_from_filepath

parse_from: attributes["log.file.path"]

regex: ^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$

type: regex_parser

- from: attributes.stream

to: attributes["log.iostream"]

type: move

- from: attributes.container_name

to: resource["k8s.container.name"]

type: move

- from: attributes.namespace

to: resource["k8s.namespace.name"]

type: move

- from: attributes.pod_name

to: resource["k8s.pod.name"]

type: move

- from: attributes.restart_count

to: resource["k8s.container.restart_count"]

type: move

- from: attributes.uid

to: resource["k8s.pod.uid"]

type: move

- from: attributes.log

to: body

type: move

start_at: beginning

hostmetrics:

collection_interval: 30s

scrapers:

cpu: {}

disk: {}

filesystem: {}

load: {}

memory: {}

network: {}

kubeletstats:

auth_type: serviceAccount

collection_interval: 30s

endpoint: ${K8S_NODE_NAME}:10250

extra_metadata_labels: []

insecure_skip_verify: true

metric_groups:

- container

- pod

- node

- volume

otlp:

protocols:

grpc:

endpoint: 0.0.0.0:4317

max_recv_msg_size_mib: 4

http:

endpoint: 0.0.0.0:4318

prometheus:

config:

global:

scrape_interval: 60s

scrape_configs:

- job_name: otel-agent

static_configs:

- targets:

- ${MY_POD_IP}:8888

service:

extensions:

- health_check

- zpages

pipelines:

logs:

exporters:

- otlp

processors:

- k8sattributes

- batch

receivers:

- otlp

- filelog/k8s

metrics:

exporters:

- otlp

processors:

- k8sattributes

- batch

receivers:

- otlp

metrics/generic:

exporters:

- otlp

processors:

- k8sattributes

- resourcedetection

- batch

receivers:

- prometheus

- hostmetrics

- kubeletstats

traces:

exporters:

- otlp

processors:

- k8sattributes

- batch

receivers:

- otlp

telemetry:

metrics:

address: 0.0.0.0:8888

@Manikanta Reddy P there is an option to blacklist namespaces https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml#L77

i will try this.. thanks @Ankit Nayan

ok

yes..

cc: @Prashant Shahi

@Manikanta Reddy P I suppose you meant whitelisting. At the moment, we only have option for blacklisting. Could you please raise an issue for the same? So that, we can prioritise it up. I see that

include

option helped. The file format is

/var/log/pods/<namespace_name>_<pod_name>_<pod_uid>/<container_name>/<run_id>.log

For time-being, you can keep using

/var/log/pods/<namespace_name>_*/*/*.log

in

include

@Prashant Shahi my use-case is , i wanted to capture only logs from 1 or 2 namespaces all other namespaces logs i don’t want to store.. i used blacklist of namespaces but still i can see logs from backlisted namespaces. then i used “include” list that worked for me..

thanks.. 👍