https://signoz.io logo
#general
Title
# general
m

Manikanta Reddy P

01/07/2023, 5:38 PM
Hi Team, I need support in filtering logs in Kubernetes. i want to send logs to Signoz only from one namespace.. i tried filtering in processors but didn't work. can any one pls help me with this.
p

Pranay

01/07/2023, 5:47 PM
@Manikanta Reddy P Can you share some sample logs format, and the config you can in otel processor
m

Manikanta Reddy P

01/07/2023, 6:15 PM
otel-agent-config.yaml: |-
exporters:
otlp:
endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}
headers:
signoz-access-token: Bearer ${SIGNOZ_API_KEY}
tls:
cert_file: ${OTEL_SECRETS_PATH}/cert.pem
insecure: ${OTEL_EXPORTER_OTLP_INSECURE}
insecure_skip_verify: ${OTEL_EXPORTER_OTLP_INSECURE_SKIP_VERIFY}
key_file: ${OTEL_SECRETS_PATH}/key.pem
extensions:
health_check:
endpoint: 0.0.0.0:13133
pprof:
endpoint: localhost:1777
zpages:
endpoint: localhost:55679
processors:
batch:
send_batch_size: 10000
timeout: 200ms
k8sattributes:
extract:
metadata:
- k8s.namespace.name
- k8s.pod.name
- k8s.pod.uid
- k8s.pod.start_time
- k8s.deployment.name
- k8s.node.name
passthrough: true
pod_association:
- from: resource_attribute
name: k8s.pod.ip
- from: resource_attribute
name: k8s.pod.uid
- from: connection
resourcedetection:
detectors:
- env
- system
system:
hostname_sources:
- os
timeout: 2s
receivers:
filelog/k8s:
exclude:
- /var/log/pods/kube-system_*.log
- /var/log/pods/*_hotrod*_*/*/*.log
- /var/log/pods/*_locust*_*/*/*.log
include:
- /var/log/pods/*/*/*.log
include_file_name: false
include_file_path: true
operators:
- id: get-format
routes:
- expr: body matches "^\\{"
output: parser-docker
- expr: body matches "^[^ Z]+ "
output: parser-crio
- expr: body matches "^[^ Z]+Z"
output: parser-containerd
type: router
- id: parser-crio
output: extract_metadata_from_filepath
regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
timestamp:
layout: "2006-01-02T15:04:05.000000000-07:00"
layout_type: gotime
parse_from: attributes.time
type: regex_parser
- id: parser-containerd
output: extract_metadata_from_filepath
regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.time
type: regex_parser
- id: parser-docker
output: extract_metadata_from_filepath
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.time
type: json_parser
- id: extract_metadata_from_filepath
parse_from: attributes["log.file.path"]
regex: ^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$
type: regex_parser
- from: attributes.stream
to: attributes["log.iostream"]
type: move
- from: attributes.container_name
to: resource["k8s.container.name"]
type: move
- from: attributes.namespace
to: resource["k8s.namespace.name"]
type: move
- from: attributes.pod_name
to: resource["k8s.pod.name"]
type: move
- from: attributes.restart_count
to: resource["k8s.container.restart_count"]
type: move
- from: attributes.uid
to: resource["k8s.pod.uid"]
type: move
- from: attributes.log
to: body
type: move
start_at: beginning
hostmetrics:
collection_interval: 30s
scrapers:
cpu: {}
disk: {}
filesystem: {}
load: {}
memory: {}
network: {}
kubeletstats:
auth_type: serviceAccount
collection_interval: 30s
endpoint: ${K8S_NODE_NAME}:10250
extra_metadata_labels: []
insecure_skip_verify: true
metric_groups:
- container
- pod
- node
- volume
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
max_recv_msg_size_mib: 4
http:
endpoint: 0.0.0.0:4318
prometheus:
config:
global:
scrape_interval: 60s
scrape_configs:
- job_name: otel-agent
static_configs:
- targets:
- ${MY_POD_IP}:8888
service:
extensions:
- health_check
- zpages
pipelines:
logs:
exporters:
- otlp
processors:
- k8sattributes
- batch
receivers:
- otlp
- filelog/k8s
metrics:
exporters:
- otlp
processors:
- k8sattributes
- batch
receivers:
- otlp
metrics/generic:
exporters:
- otlp
processors:
- k8sattributes
- resourcedetection
- batch
receivers:
- prometheus
- hostmetrics
- kubeletstats
traces:
exporters:
- otlp
processors:
- k8sattributes
- batch
receivers:
- otlp
telemetry:
metrics:
address: 0.0.0.0:8888
Hi Above is my otel agent YAML. by my logs are general logs. i want to filter logs from only one Kubernetes namespace.. we have 10 namespaces but we need to send logs to SIGNOZ only from one namespace(defulat) and ignore all other namespace logs.. (MongoDB, cert-manager, kube system etc)
@Pranay please let me know if you need further details
a

Ankit Nayan

01/08/2023, 4:41 AM
@Manikanta Reddy P there is an option to blacklist namespaces https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml#L77
m

Manikanta Reddy P

01/09/2023, 8:40 AM
i will try this.. thanks @Ankit Nayan
@Ankit Nayan blacklist namespaces didn’t help. but below line helped me https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml#L70
a

Ankit Nayan

01/09/2023, 11:30 AM
ok
blacklisting does not work?
m

Manikanta Reddy P

01/09/2023, 11:31 AM
yes..
a

Ankit Nayan

01/09/2023, 11:56 AM
cc: @Prashant Shahi
p

Prashant Shahi

01/09/2023, 12:56 PM
@Manikanta Reddy P I suppose you meant whitelisting. At the moment, we only have option for blacklisting. Could you please raise an issue for the same? So that, we can prioritise it up. I see that
include
option helped. The file format is
/var/log/pods/<namespace_name>_<pod_name>_<pod_uid>/<container_name>/<run_id>.log
For time-being, you can keep using
/var/log/pods/<namespace_name>_*/*/*.log
in
include
m

Manikanta Reddy P

01/09/2023, 2:09 PM
@Prashant Shahi my use-case is , i wanted to capture only logs from 1 or 2 namespaces all other namespaces logs i don’t want to store.. i used blacklist of namespaces but still i can see logs from backlisted namespaces. then i used “include” list that worked for me..
@Prashant Shahi can you please share link the link to raise the bug i am new this
a

Ankit Nayan

01/09/2023, 2:13 PM
m

Manikanta Reddy P

01/09/2023, 2:36 PM
thanks.. 👍
8 Views