Hi all, I am trying to test parsing a logfile with entries like below but nothings comes up in the UI under logs or elsewhere any suggestions on where I might be going wrong

2022-11-25 22:55:47,673 - DEBUG - ca770 output {"timestamp": 1669377347.7902305, "Type": 2, "event": 7 }
2022-11-25 22:55:47,689 - INFO - Latency:-0.101148 seconds

otel collector config:

filelog/cav:
    include: [  "/testlog/*.log" ]
    start_at: beginning
    include_file_path: true
    include_file_name: false
    operators:
    - type: regex_parser
      regex: '^(?P<log_timestamp>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}[,.]\d{3})\s+-\s+(?P<log_entry_type>\w+)\s+-\s+(?P<log_msg>.*)$'

From the operators that you have used, the timestamp is parsed by the regex, but you will have to use timestamp parser to populate the value eg:- https://github.com/SigNoz/nginx-logs-parsing/blob/c14a2178a050293a50549b79991ed3858e295c78/clickhouse-setup/otel-collector-config.yaml#L34 Right now your timestamp is getting set to epoch 0.

Thanks for the quick response, sure I will chain more operators to get the final output but from the first transformation, shouldn't I see something in the UI? I was expecting to see 3 attributes with string values named: • log_timestamp • log_entry_type • log_msg

Are you able to see the attributes on the left side in interesting fields?

yes I do

If your log is ingested you will be able to see logs when you change the time range in top right to something like Dec 1969 to current. As epoch 0 corresponds to 1 January 1970

ok, I set the time frame to what you suggested, but still no log entries what container logs should I look at to confirm log entries were consumed?

You can exec into the clickhouse pod and check

docker exec -it clickhouse-setup-clickhouse-1 /bin/bash

clickhouse client

select * from signoz_logs.logs limit 5;

ok, let me try