hey hey me again slightly smiling face Once again with logs SigNoz Community #support

Join Slack

hey hey, me again :slightly_smiling_face: Once aga...

# support

Hannes Benson

08/11/2023, 5:47 AM

hey hey, me again 🙂 Once again with logs and regex 🙂 🧵

Hannes Benson

08/11/2023, 5:47 AM

I have a log entry that looks like this:

Copy code

"1622 <190>1 2023-08-11T05:46:08.075562+00:00 d.7c9c4867-7cdb-4a9a-b4b8-cb877ed3084c app web.1 - - {\"severity_text\":\"INFO\",\"time\":\"2023-08-11T05:46:08.075+00:00\",\"message\":\"[9e330e7e-0e64-432e-a141-51d61aa1e667]   Parameters: {\\\"operationName\\\"=\\u003e\\\"GetListOptions\\\", \\\"variables\\\"=\\u003e{\\\"page\\\"=\\u003e1, \\\"perPage\\\"=\\u003e50, \\\"filter\\\"=\\u003e{\\\"term\\\"=\\u003e\\\"\\\", \\\"usageModes\\\"=\\u003e[]}}, \\\"query\\\"=\\u003e\\\"query GetListOptions($page: Int, $perPage: Int, $filter: ListFiltering) {\\\\n  lists(page: $page, perPage: $perPage, filter: $filter) {\\\\n    entries {\\\\n      ...ListOption\\\\n      __typename\\\\n    }\\\\n    __typename\\\\n  }\\\\n}\\\\n\\\\nfragment ListOption on List {\\\\n  id\\\\n  name\\\\n  usageMode\\\\n  creator {\\\\n    firstName\\\\n    lastName\\\\n    email\\\\n    __typename\\\\n  }\\\\n  project {\\\\n    title\\\\n    __typename\\\\n  }\\\\n  __typename\\\\n}\\\", \\\"graphql\\\"=\\u003e{\\\"operationName\\\"=\\u003e\\\"GetListOptions\\\", \\\"variables\\\"=\\u003e{\\\"page\\\"=\\u003e1, \\\"perPage\\\"=\\u003e50, \\\"filter\\\"=\\u003e{\\\"term\\\"=\\u003e\\\"\\\", \\\"usageModes\\\"=\\u003e[]}}, \\\"query\\\"=\\u003e\\\"query GetListOptions($page: Int, $perPage: Int, $filter: ListFiltering) {\\\\n  lists(page: $page, perPage: $perPage, filter: $filter) {\\\\n    entries {\\\\n      ...ListOption\\\\n      __typename\\\\n    }\\\\n    __typename\\\\n  }\\\\n}\\\\n\\\\nfragment ListOption on List {\\\\n  id\\\\n  name\\\\n  usageMode\\\\n  creator {\\\\n    firstName\\\\n    lastName\\\\n    email\\\\n    __typename\\\\n  }\\\\n  project {\\\\n    title\\\\n    __typename\\\\n  }\\\\n  __typename\\\\n}\\\"}}\",\"operation\":\"GraphqlController#execute\",\"span_id\":\"7b9a038f77454b7a\",\"trace_id\":\"60c75f5a12c3c9aba3c066473ed69800\"}"

Hannes Benson

08/11/2023, 5:47 AM

I'd like to regex out the json

Hannes Benson

08/11/2023, 5:47 AM

before parsing the json

Hannes Benson

08/11/2023, 5:47 AM

I have the following processor set up

Hannes Benson

08/11/2023, 5:48 AM

Copy code

operators:
      - type: regex_parser
        regex: '{(?P<body>.*)}'
        parse_from: body
        if: "body matches '^{.*}$'"
      - type: json_parser
        parse_to: attributes
        if: "body matches '^{.*}$'"
      - type: severity_parser
        if: "'severity_text' in attributes"
        parse_from: attributes.severity_text
        output: remove_severity_text
      - type: remove
        id: remove_severity_text
        if: "'severity_text' in attributes"
        field: attributes.severity_text

Hannes Benson

08/11/2023, 7:07 AM

Copy code

logstransform/heroku_parser:
    operators:
      - type: regex_parser
        regex: '(?P<body>{.*})'
        parse_from: body
        parse_to: attributes
        if: "body matches '{.*}$'"
      - type: json_parser
        parse_from: attributes.body
        parse_to: attributes
        if: "body matches '{.*}$'"
      - type: remove
        id: remove_parsed_string
        if: "'body' in attributes"
        field: attributes.body
      - type: severity_parser
        if: "'severity_text' in attributes"
        parse_from: attributes.severity_text
        output: remove_severity_text
      - type: remove
        id: remove_severity_text
        if: "'severity_text' in attributes"
        field: attributes.severity_text

has resolved the issues, but now I'm not getting

trace_id

span_id

😕

Hannes Benson

08/11/2023, 7:57 AM

Managed to fix it 🙂

Hannes Benson

08/11/2023, 7:57 AM

Copy code

logstransform/heroku_parser:
    operators:
      - type: regex_parser
        regex: '(?P<body>{.*})'
        parse_from: body
        parse_to: attributes
        if: "body matches '{.*}$'"
      - type: json_parser
        parse_from: attributes.body
        parse_to: attributes
        if: "body matches '{.*}$'"
      - type: severity_parser
        if: "'severity_text' in attributes"
        parse_from: attributes.severity_text
        output: remove_severity_text
      - type: remove
        id: remove_severity_text
        if: "'severity_text' in attributes"
        field: attributes.severity_text
      - type: trace_parser
        if: '"trace_id" in attributes or "span_id" in attributes'
        trace_id:
          parse_from: attributes.trace_id
        span_id:
          parse_from: attributes.span_id
        output: remove_trace_id
      - id: remove_trace_id
        type: remove
        if: '"trace_id" in attributes'
        field: attributes.trace_id
        output: remove_span_id
      - id: remove_span_id
        type: remove
        if: '"span_id" in attributes'
        field: attributes.span_id
      - type: remove
        id: remove_parsed_string
        if: "'body' in attributes"
        field: attributes.body

For future reference to anyone possibly trying to do the same thing

nitya-signoz

08/11/2023, 11:06 AM

Great, let me know if you need more help . If this is a generic heroku parser that you have created would love to have a blog from you. @Ashu do reach out if Hannes is interested.

Ashu

08/16/2023, 3:36 AM

Thanks @nitya-signoz Will reach out to @Hannes Benson for documenting this.

Open in Slack

Previous Next