https://signoz.io logo
#support
Title
# support
r

Rahul v

11/28/2022, 9:09 AM
Hi all, I am trying to test parsing a logfile with entries like below but nothings comes up in the UI under logs or elsewhere any suggestions on where I might be going wrong
Copy code
2022-11-25 22:55:47,673 - DEBUG - ca770 output {"timestamp": 1669377347.7902305, "Type": 2, "event": 7 }
2022-11-25 22:55:47,689 - INFO - Latency:-0.101148 seconds
otel collector config:
Copy code
filelog/cav:
    include: [  "/testlog/*.log" ]
    start_at: beginning
    include_file_path: true
    include_file_name: false
    operators:
    - type: regex_parser
      regex: '^(?P<log_timestamp>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}[,.]\d{3})\s+-\s+(?P<log_entry_type>\w+)\s+-\s+(?P<log_msg>.*)$'
p

Palash Gupta

11/28/2022, 9:10 AM
@nitya-signoz
n

nitya-signoz

11/28/2022, 9:13 AM
From the operators that you have used, the timestamp is parsed by the regex, but you will have to use timestamp parser to populate the value eg:- https://github.com/SigNoz/nginx-logs-parsing/blob/c14a2178a050293a50549b79991ed3858e295c78/clickhouse-setup/otel-collector-config.yaml#L34 Right now your timestamp is getting set to epoch 0.
r

Rahul v

11/28/2022, 9:15 AM
Thanks for the quick response, sure I will chain more operators to get the final output but from the first transformation, shouldn't I see something in the UI? I was expecting to see 3 attributes with string values named: • log_timestamp • log_entry_type • log_msg
n

nitya-signoz

11/28/2022, 9:16 AM
Are you able to see the attributes on the left side in interesting fields?
r

Rahul v

11/28/2022, 9:17 AM
yes I do
n

nitya-signoz

11/28/2022, 9:18 AM
If your log is ingested you will be able to see logs when you change the time range in top right to something like Dec 1969 to current. As epoch 0 corresponds to 1 January 1970
r

Rahul v

11/28/2022, 9:22 AM
ok, I set the time frame to what you suggested, but still no log entries what container logs should I look at to confirm log entries were consumed?
do the otel-collector logs show this info? the last log entry says:
Copy code
2022-11-28T09:03:39.192Z        info    fileconsumer/file.go:159        Started watching file   {"kind": "receiver", "name": "filelog/cav", "pipeline": "logs", "component": "fileconsumer", "path": "/testlog/interface.mod.log"}
n

nitya-signoz

11/28/2022, 9:26 AM
You can exec into the clickhouse pod and check
Copy code
docker exec -it clickhouse-setup-clickhouse-1 /bin/bash

clickhouse client

select * from signoz_logs.logs limit 5;
You should see the logs, would suggest you to use the time parser after that.
r

Rahul v

11/28/2022, 9:27 AM
ok, let me try
Thanks @nitya-signoz parsing the timestamp made the logs show up, they were ingested but not showing up in the UI. This did the trick
Copy code
- type: regex_parser
      regex: '^(?P<log_timestamp>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}[,.]\d{3})\s+-\s+(?P<log_entry_type>\w+)\s+-\s+(?P<log_msg>.*)$'
      timestamp:
        parse_from: attributes.log_timestamp
        layout_type: strptime
        layout: '%Y-%m-%d %H:%M:%S,%f'
looks like the first operation requires the timestamp, will remember that, cheers
2 Views