https://signoz.io logo
#support
Title
# support
n

Noor Ali

10/06/2023, 4:13 PM
Is this common error in signoz when testing local host log file: 2023-10-06T155005.809Z error helper/transformer.go:99 Failed to process entry {"kind": "receiver", "name": "filelog", "data_type": "logs", "operator_id": "remove", "operator_type": "remove", "error": "remove: field does not exist", "action": "send", "entry": {"observed_timestamp":"2023-10-06T155005.809132405Z","timestamp":"0001-01-01T000000Z","body":"at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:282)","attributes":{"log.file.name":"startServer.log"},"severity":0,"scope_name":""}}
s

Srikanth Chekuri

10/06/2023, 10:06 PM
You can put a condition to avoid this error,
if 'key' in attributes
or something like that
n

Noor Ali

10/07/2023, 1:31 AM
Srikanth one more question this is my operator filelog: include: [ /tmp/startServer.log ] start_at: beginning operators: - type: regex_parser regex: (\D+)(\S+)([0-9]).([0-9])+(\S+)(\D)+(\W+)(\S{8})(\D+\D+)([A-Z]).([0-9]\w+:\D+) timestamp: layout: '%Y-%m-%d,%H:%M:%S %z' parse_from: attributes.timestamp_field - type: remove field: attributes.timestamp when restart the collector I get the following error 2023/10/07 012857 application run finished with error: failed to build pipelines: failed to create "filelog" receiver for data type "logs": no named capture groups in regex pattern. This regex parser tested ok in regex101 is this parser allowed in signoz
what is best regex pattern is allowed by signoz to run without giving the following error Error: failed to build pipelines: failed to create "filelog" receiver for data type "logs": no named capture groups in regex pattern 2023/10/09 162351 application run finished with error: failed to build pipelines: failed to create "filelog" receiver for data type "logs": no named capture groups in regex pattern
s

Srikanth Chekuri

10/10/2023, 12:11 AM
@nitya-signoz might be able to help
n

Noor Ali

10/10/2023, 3:03 AM
Hello Nitya I am currently using this - type: regex_parser regex: (\S+\d+\D+)(\S+\d+\d+)([0-9]\D+\W+\S+)+([a-z]+\W+\S+\D+) timestamp: parse_from: attributes.timestamp_field layout: '%Y-%m-%d,%H:%M:%S %z' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp
I am getting this error filelog" receiver for data type "logs": no named capture groups in regex pattern
Per Signoz all parse are test in regex101
n

nitya-signoz

10/10/2023, 5:31 AM
Since you are extracting
timestamp_field
and
body
form the regex those named group should be present in your regex
(\S+\d+\D+)(\S+\d+\d+)([0-9]\D+\W+\S+)+([a-z]+\W+\S+\D+)
https://github.com/StefanSchroeder/Golang-Regex-Tutorial/blob/master/01-chapter2.markdown#named-matches
n

Noor Ali

10/10/2023, 4:26 PM
Just wondering getting closer I made few changes I got this error 2023/10/10 162041 application run finished with error: failed to build pipelines: failed to create "filelog" receiver for data type "logs": compiling regex: error parsing regexp: invalid or unsupported Perl syntax:
(?<
n

nitya-signoz

10/12/2023, 5:07 AM
Can you choose golang as the language in regex101 and test your regex out ?
n

Noor Ali

10/12/2023, 12:00 PM
I currently have it setup this way filelog: include: [ /tmp/startServer.log ] start_at: beginning operators: - type: regex_parser regex: (\S+\d+\D+)(\S+\d+\d+)([0-9]\D+\W+\S+)+([a-z]+\W+\S+\D+)(?P<body>.*)|(?<timestamp>\S+)| parseTimestamp(field=timestamp) timestamp: parse_from: attributes.timestamp_field layout: '%Y-%m-%d,%H:%M:%S %z' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp tcplog/docker: listen_address: "0.0.0.0:2255" operators: - type: regex_parser regex: '^\[(?P<time>\d+\/\d+\/\d+ \d{2}\d+\d{2}:\d{3} UTC)\](?P<body>.*)' timestamp: parse_from: attributes.timestamp layout: '%Y-%m-%dT%H:%M:%S.%LZ' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp
Hello Nitya please review my word doc let me what else I need to do make it work for us thanks
I need to know why it is not working for me if I am able to make this work I will be moving forward to deploy it next. Thanks
n

nitya-signoz

10/12/2023, 4:37 PM
can you share your config ? the regex in doc and the config doesn’t match. Also provide some sample log lines for testing it out.
n

Noor Ali

10/12/2023, 4:38 PM
eceivers: filelog: include: [ /tmp/startServer.log ] start_at: beginning operators: - type: regex_parser regex: layout: '%Y-%m-%d,%H:%M:%S.%z' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp tcplog/docker: listen_address: "0.0.0.0:2255" operators: - type: regex_parser regex: '^\[(?P<time>\d+\/\d+\/\d+ \d{2}\d+\d{2}:\d{3} UTC)\](?P<body>.*)' timestamp: parse_from: attributes.timestamp layout: '%Y-%m-%dT%H:%M:%S.%LZ' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp
This is what we or I am trying to parse 4/28/21 122354:157 UTC] 00000001 ManagerAdmin I TRAS0017I: The startup trace state is *=info. [4/28/21 122354:302 UTC] 00000001 AdminTool A ADMU0128I: Starting tool with the Custom01 profile [4/28/21 122354:305 UTC] 00000001 AdminTool A ADMU3100I: Reading configuration for server: igawas02 [4/28/21 122354:321 UTC] 00000001 ImplFactory W WSVR0072W: Ignoring undeclared override of interface, com.ibm.websphere.cluster.topography.DescriptionManager, with implementation, com.ibm.ws.cluster.propagation.bulletinboard.BBDescriptionManager [4/28/21 122354:574 UTC] 00000001 ModelMgr I WSVR0801I: Initializing all server configuration models
I have been trying to use(?P<timestamp>, <?(message>), (?P,<body>) and even <?P<Date>) but has been showing up with error
my last error now is this 2023/10/12 135349 application run finished with error: failed to get config: cannot resolve the configuration: cannot retrieve the configuration: yaml: line 8: mapping values are not allowed in this context
I need to know what I am doing in correct so I can educate my team member in the future thank you so much for helping me on this
n

nitya-signoz

10/12/2023, 4:47 PM
• The regex pattern is not present and it should be enclosed in quotes • The regex is not correct it doesn’t work (golang as language)
n

Noor Ali

10/12/2023, 4:57 PM
do you mean like this
I have this now it is matching now in regex
cape character 2023/10/12 171057 application run finished with error: failed to get config: cannot resolve the configuration: cannot retrieve the configuration: yaml: line 7: found unknown escape character
Please do tell me where is the issue in my config now receivers: filelog: include: [ /tmp/startServer.log ] start_at: beginning operators: - type: regex_parser regex: "(\S+\d+\D+)(S\+\d+\d+)[0-9]\D+\W+\S+([a-z]+\W+\S+\D+)(?P<bod>.*)>.*|(?P<timestamp>\S+)|parseTimestamp(field=timestamp)" layout: '%Y-%m-%d,%H:%M:%S.%z' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp tcplog/docker: listen_address: "0.0.0.0:2255" operators: - type: regex_parser regex: '^\[(?P<time>\d+\/\d+\/\d+ \d{2}\d+\d{2}:\d{3} UTC)\](?P<body>.*)' timestamp: parse_from: attributes.timestamp layout: '%Y-%m-%dT%H:%M:%S.%LZ' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp
2023/10/12 171502 application run finished with error: failed to get config: cannot resolve the configuration: cannot retrieve the configuration: yaml: line 5: did not find expected key with this config operators: - type: regex_parser regex: '(\S+\d+\D+)(S\+\d+\d+)[0-9]\D+\W+\S+([a-z]+\W+\S+\D+)(?P<bod>.*)>.*|(?P<timestamp>\S+)|parseTimestamp(field=timestamp)' layout: '%Y-%m-%d,%H:%M:%S.%z' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp
Does it have to do with my pass 30 day testing with my log in id just wondering that is all