```2022 10 24T09 49 29 995Z error helper transformer go 110 SigNoz Community #support

```2022-10-24T09:49:29.995Z error helper/transform...

Apoorva

10/24/2022, 10:03 AM

Copy code

2022-10-24T09:49:29.995Z	error	helper/transformer.go:110	Failed to process entry	{"kind": "receiver", "name": "filelog/k8s", "pipeline": "logs", "operator_id": "parser-nucash-regex", "operator_type": "regex_parser", "error": "regex pattern does not match", "action": "send", "entry": {"observed_timestamp":"2022-10-24T09:49:29.995336396Z","timestamp":"2022-10-24T09:49:29.908076398Z","body":"{\"component\":\"aaa\",\"env\":\"dev\",\"file\":\"/go/src/bitbucket.org/nucashin/sample/main.go:73\",\"func\":\"main.main\",\"level\":\"info\",\"msg\":\"Something\",\"service\":\"aaa\",\"time\":\"2022-10-24T09:49:29Z\"}\n","attributes":{"k8s.container.name":"sample","k8s.container.restart_count":"0","k8s.namespace.name":"services","k8s.pod.name":"sample-6c879cc484-6vmbr","k8s.pod.uid":"94b40e31-b854-4af4-9a83-f4daadadd951","log.file.path":"/var/log/pods/services_sample-6c879cc484-6vmbr_94b40e31-b854-4af4-9a83-f4daadadd951/sample/0.log","log.iostream":"stderr","time":"2022-10-24T09:49:29.908076398Z"},"severity":0,"scope_name":""}}

Getting this error in k8s-agent below is the updated configmap for k8s-agent, any idea why ..

Copy code

- from: attributes.restart_count
          to: attributes["k8s.container.restart_count"]
          type: move
        - from: attributes.uid
          to: attributes["k8s.pod.uid"]
          type: move
        - from: attributes.log
          to: body
          type: move
        - id: parser-nucash-regex
          parse_from: body
          regex: ^"(?P<nucash_json>{.*})\\n"
          type: regex_parser
        - id: parser-nucash-json
          parse_from: attributes.nucash_json
          type: json_parser

Pranay

10/24/2022, 2:18 PM

@nitya-signoz may have more insights on this

nitya-signoz

10/25/2022, 5:36 AM

Hi @Apoorva, can you please explain what you are trying to achieve here? I can see that you are trying to parse something out from the logs, but some explanation will be great. It will help us to suggest a proper configuration for parsing and help you debug the issue.

Apoorva

10/25/2022, 5:38 AM

Copy code

"body":"{\"component\":\"aaa\",\"env\":\"dev\",\"file\":\"/go/src/bitbucket.org/nucashin/sample/main.go:73\",\"func\":\"main.main\",\"level\":\"info\",\"msg\":\"Something\",\"service\":\"aaa\",\"time\":\"2022-10-24T09:49:29Z\"}\n"

I am trying to get the fields of this json to be displayed and to be filtered in dasboard based on env, service, components, log level etc @nitya-signoz

nitya-signoz

10/25/2022, 5:50 AM

Got it, can you please share the number of applications that emit the same kind of logs, because this is a parsing usecase is specific to an application and it will require error handling or some kind of separation via routing.

Apoorva

10/25/2022, 6:21 AM

All the application in "services" namespace send me same kind of logs @nitya-signoz

Apoorva

10/25/2022, 6:26 AM

With the below configmap I was trying to parse any json if present in the body, it is giving me regex parsing error, not sure why

Copy code

- id: parser-nucash-regex
          parse_from: body
          regex: ^"(?P<nucash_json>{.*})\\n"
          type: regex_parser
        - id: parser-nucash-json
          parse_from: attributes.nucash_json
          type: json_parser

nitya-signoz

10/25/2022, 7:59 AM

Sure, getting back on this.

nitya-signoz

10/25/2022, 11:09 AM

@Apoorva you don’t need to use a regex parser. This is what I understand. • Your applications emit logs in JSON format something like this.

Copy code

{
  "component": "aaa",
  "env": "dev",
  "file": "/go/src/bitbucket.org/nucashin/sample/main.go:73",
  "func": "main.main",
  "level": "info",
  "msg": "Something",
  "service": "aaa",
  "time": "2022-10-24T09:49:29Z"
}

• Now you want these keys/fields to be parsed so that you can filter them on the UI. For this, you can directly use the JSON parser and add the following operator towards the end of your config file

Copy code

- id: parser-nucash-json
        parse_from: body
        type: json_parser

This will convert all the keys to fields. If you don’t want some of these keys to be present as fields you can use the remove operator

nitya-signoz

10/25/2022, 11:13 AM

I would also recommend you to setup a new receiver for this namespace as you are modifying the parsing rules for specifiec namespace. for this you can create a new filelog receiver with and configure it only for that namespace where you can copy the config of normal k8s receiver and add your logic.

Copy code

filelog/services:
          include:
            # Include logs from all container in services namespace
            - /var/log/pods/services/*/*.log

If you decide going with this approach, do disable this namespace from the normal k8s log receiver.

Copy code

filelog/k8s:
          include:
            # Include logs from all container
            - /var/log/pods/*/*/*.log
          exclude:
            # exclude services logs
            - /var/log/pods/services/*/*.log
            # Exclude logs from all containers from kube-system namespace
            - /var/log/pods/kube-system_*/*/*.log
            # Exclude logs from all hotrod containers
            - /var/log/pods/*_hotrod-*/*/*.log
            - /var/log/pods/*_locust-*_*/*/*.log

Apoorva

10/25/2022, 11:27 AM

@nitya-signoz Thanks a lot, this works for me

2 Views

Open in Slack

Previous Next