Apoorva
10/24/2022, 10:03 AM2022-10-24T09:49:29.995Z error helper/transformer.go:110 Failed to process entry {"kind": "receiver", "name": "filelog/k8s", "pipeline": "logs", "operator_id": "parser-nucash-regex", "operator_type": "regex_parser", "error": "regex pattern does not match", "action": "send", "entry": {"observed_timestamp":"2022-10-24T09:49:29.995336396Z","timestamp":"2022-10-24T09:49:29.908076398Z","body":"{\"component\":\"aaa\",\"env\":\"dev\",\"file\":\"/go/src/bitbucket.org/nucashin/sample/main.go:73\",\"func\":\"main.main\",\"level\":\"info\",\"msg\":\"Something\",\"service\":\"aaa\",\"time\":\"2022-10-24T09:49:29Z\"}\n","attributes":{"k8s.container.name":"sample","k8s.container.restart_count":"0","k8s.namespace.name":"services","k8s.pod.name":"sample-6c879cc484-6vmbr","k8s.pod.uid":"94b40e31-b854-4af4-9a83-f4daadadd951","log.file.path":"/var/log/pods/services_sample-6c879cc484-6vmbr_94b40e31-b854-4af4-9a83-f4daadadd951/sample/0.log","log.iostream":"stderr","time":"2022-10-24T09:49:29.908076398Z"},"severity":0,"scope_name":""}}
Getting this error in k8s-agent below is the updated configmap for k8s-agent, any idea why ..
- from: attributes.restart_count
to: attributes["k8s.container.restart_count"]
type: move
- from: attributes.uid
to: attributes["k8s.pod.uid"]
type: move
- from: attributes.log
to: body
type: move
- id: parser-nucash-regex
parse_from: body
regex: ^"(?P<nucash_json>{.*})\\n"
type: regex_parser
- id: parser-nucash-json
parse_from: attributes.nucash_json
type: json_parser
Pranay
10/24/2022, 2:18 PMnitya-signoz
10/25/2022, 5:36 AMApoorva
10/25/2022, 5:38 AM"body":"{\"component\":\"aaa\",\"env\":\"dev\",\"file\":\"/go/src/bitbucket.org/nucashin/sample/main.go:73\",\"func\":\"main.main\",\"level\":\"info\",\"msg\":\"Something\",\"service\":\"aaa\",\"time\":\"2022-10-24T09:49:29Z\"}\n"
I am trying to get the fields of this json to be displayed and to be filtered in dasboard based on env, service, components, log level etc @nitya-signoznitya-signoz
10/25/2022, 5:50 AMApoorva
10/25/2022, 6:21 AM- id: parser-nucash-regex
parse_from: body
regex: ^"(?P<nucash_json>{.*})\\n"
type: regex_parser
- id: parser-nucash-json
parse_from: attributes.nucash_json
type: json_parser
nitya-signoz
10/25/2022, 7:59 AM{
"component": "aaa",
"env": "dev",
"file": "/go/src/bitbucket.org/nucashin/sample/main.go:73",
"func": "main.main",
"level": "info",
"msg": "Something",
"service": "aaa",
"time": "2022-10-24T09:49:29Z"
}
• Now you want these keys/fields to be parsed so that you can filter them on the UI.
For this, you can directly use the JSON parser and add the following operator towards the end of your config file
- id: parser-nucash-json
parse_from: body
type: json_parser
This will convert all the keys to fields. If you don’t want some of these keys to be present as fields you can use the remove operatorfilelog/services:
include:
# Include logs from all container in services namespace
- /var/log/pods/services/*/*.log
If you decide going with this approach, do disable this namespace from the normal k8s log receiver.
filelog/k8s:
include:
# Include logs from all container
- /var/log/pods/*/*/*.log
exclude:
# exclude services logs
- /var/log/pods/services/*/*.log
# Exclude logs from all containers from kube-system namespace
- /var/log/pods/kube-system_*/*/*.log
# Exclude logs from all hotrod containers
- /var/log/pods/*_hotrod-*/*/*.log
- /var/log/pods/*_locust-*_*/*/*.log
Apoorva
10/25/2022, 11:27 AM