hey all having some difficulty getting off the ground fresh SigNoz Community #support

hey all. having some difficulty getting off the g...

George Larson

11/29/2022, 6:10 AM

hey all. having some difficulty getting off the ground. fresh install of ubuntu 20.04. as proof of concept, i was just trying to send in logs from the host system. followed the tutorial (https://signoz.io/docs/userguide/collecting_syslogs/) UFW is disabled. receiver is open (checked w/

telnet

). can see traffic in

tcpdump

there are entries in

signoz_logs.logs

i tried changing to

rfc5424

but it did not seem to help

signoz-logs.logs.txt

Pranay

11/29/2022, 6:11 AM

@nitya-signoz Do you have more insights here?

nitya-signoz

11/29/2022, 6:22 AM

I am not able to get the problem you are facing, are you saying that the right logs are not getting ingested to signoz ?

George Larson

11/29/2022, 6:29 AM

unless i'm misunderstanding what i am seeing, i'm not getting the host system's syslog info all the way to the SigNoz UI i could try removing hotrod and commenting out filelog/dockercontainers to validate?

nitya-signoz

11/29/2022, 6:30 AM

yes, due to hotrod a lot of dummy data is generated, please try removing it and disabling docker container logs for testing. https://signoz.io/docs/userguide/collect_docker_logs/#disable-automatic-container-log-collection

George Larson

11/29/2022, 6:50 AM

i took out the receiver i deleted the hotrod lines. i ran

install.sh

and it seemed to finish too quickly so then, i ran

docker compose up -d

clickhouse-setup

and it looked better now, there seems to be nothing new making it to the UI. last entry was about 10 minutes ago. 'Go Live' just waits a bit and then goes inactive

George Larson

11/29/2022, 6:52 AM

Copy code

SELECT *
FROM signoz_logs.logs
LIMIT 5

Query id: 1bebce37-e014-4a72-a5ef-92be77714264

Row 1:
──────
timestamp:                0
observed_timestamp:       1669702624575937327
id:                       2ID40HwEbb4bhF9XGP4ZT1zOlFW
trace_id:
span_id:
trace_flags:              0
severity_text:
severity_number:          0
body:                     <6>Nov 29 01:16:34 logs kernel: [86477.545087] br-c1648d5fe1e3: port 4(veth54bac04) entered disabled state
resources_string_key:     []
resources_string_value:   []
attributes_string_key:    []
attributes_string_value:  []
attributes_int64_key:     []
attributes_int64_value:   []
attributes_float64_key:   []
attributes_float64_value: []

Row 2:
──────
timestamp:                0
observed_timestamp:       1669702624595970218
id:                       2ID40HwEbb4bhF9XGP4ZT1zOlFX
trace_id:
span_id:
trace_flags:              0
severity_text:
severity_number:          0
body:                     <6>Nov 29 01:16:34 logs kernel: [86477.545144] veth4a56195: renamed from eth0
resources_string_key:     []
resources_string_value:   []
attributes_string_key:    []
attributes_string_value:  []
attributes_int64_key:     []
attributes_int64_value:   []
attributes_float64_key:   []
attributes_float64_value: []

Row 3:
──────
timestamp:                0
observed_timestamp:       1669702624595998210
id:                       2ID40HwEbb4bhF9XGP4ZT1zOlFY
trace_id:
span_id:
trace_flags:              0
severity_text:
severity_number:          0
body:                     <43>Nov 29 01:16:34 logs rsyslogd: omfwd: TCPSendBuf error -2027, destruct TCP Connection to 0.0.0.0:54527 [v8.2001.0 try <https://www.rsyslog.com/e/2027> ]
resources_string_key:     []
resources_string_value:   []
attributes_string_key:    []
attributes_string_value:  []
attributes_int64_key:     []
attributes_int64_value:   []
attributes_float64_key:   []
attributes_float64_value: []

Row 4:
──────
timestamp:                0
observed_timestamp:       1669702624596011445
id:                       2ID40HwEbb4bhF9XGP4ZT1zOlFZ
trace_id:
span_id:
trace_flags:              0
severity_text:
severity_number:          0
body:                     <14>Nov 29 01:16:34 logs networkd-dispatcher[20727]: WARNING:Unknown index 27 seen, reloading interface list
resources_string_key:     []
resources_string_value:   []
attributes_string_key:    []
attributes_string_value:  []
attributes_int64_key:     []
attributes_int64_value:   []
attributes_float64_key:   []
attributes_float64_value: []

Row 5:
──────
timestamp:                0
observed_timestamp:       1669702624596018698
id:                       2ID40HwEbb4bhF9XGP4ZT1zOlFa
trace_id:
span_id:
trace_flags:              0
severity_text:
severity_number:          0
body:                     <30>Nov 29 01:16:34 logs systemd-udevd[1613907]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
resources_string_key:     []
resources_string_value:   []
attributes_string_key:    []
attributes_string_value:  []
attributes_int64_key:     []
attributes_int64_value:   []
attributes_float64_key:   []
attributes_float64_value: []

5 rows in set. Elapsed: 0.003 sec.

nitya-signoz

11/29/2022, 6:52 AM

Ahh so here timestamp is not getting parsed because of which it is getting set to 0 and is not rendered in the UI.

nitya-signoz

11/29/2022, 6:53 AM

In these results timestamp was getting parsed, did you change anything in the config for syslog . https://signoz-community.slack.com/files/U04D04UBXC1/F04CUMF0PMH/signoz-logs.logs.txt

signoz-logs.logs.txt

George Larson

11/29/2022, 6:54 AM

i only added the line from the tutorial. the rest came with the VPS but, y'know, things can vary quite a bit between VPS providers

nitya-signoz

11/29/2022, 6:56 AM

So one thing you can do now is parse the timestamp manually using regex parser, since the timestamp is present in the body. https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/time_parser.md https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/regex_parser.md

George Larson

11/29/2022, 6:56 AM

this one VPS is from 'https://letbox.com/'

George Larson

11/29/2022, 6:57 AM

it was pretty inexpensive. 😂

nitya-signoz

11/29/2022, 6:58 AM

Here is an example of parsing the datetime manually https://github.com/SigNoz/nginx-logs-parsing/blob/c14a2178a050293a50549b79991ed3858e295c78/clickhouse-setup/otel-collector-config.yaml#L34

George Larson

11/29/2022, 7:12 AM

thanks! visually, the date format looks identical to logs from other boxes that i checked i hesitate to parse the datetime manually bc i don't want to cause problems when i start to import logs from other boxen. so, i'm gonna deploy a clean image on this VPS and try again. just in case i messed with something i don't remember doing it's about 2a in my tz and Letbox is not speedy abt anything so this may become a tomorrow thing. but, once i get it working, i'll report back so that you know what it took thanks for help!!

George Larson

11/29/2022, 8:45 AM

fresh install. i see timestamp, nothing @ ui:

Copy code

SELECT *
FROM signoz_logs.logs
LIMIT 5

Query id: fc1f2556-9259-414d-8407-3df972d89484

Row 1:
──────
timestamp:                1669692727000000000
observed_timestamp:       1669710878066903337
id:                       2IDKn4q1XZQkyIZWg8QRJW8Kc3x
trace_id:
span_id:
trace_flags:              0
severity_text:            info
severity_number:          9
body:                     [origin software="rsyslogd" swVersion="8.2001.0" x-pid="19422" x-info="<https://www.rsyslog.com>"] exiting on signal 15.
resources_string_key:     []
resources_string_value:   []
attributes_string_key:    ['hostname','appname']
attributes_string_value:  ['logs','rsyslogd']
attributes_int64_key:     ['priority','facility']
attributes_int64_value:   [46,5]
attributes_float64_key:   []
attributes_float64_value: []

Row 2:
──────
timestamp:                1669692727000000000
observed_timestamp:       1669710878066955495
id:                       2IDKn4q1XZQkyIZWg8QRJW8Kc3y
trace_id:
span_id:
trace_flags:              0
severity_text:            info
severity_number:          9
body:                     rsyslog.service: Succeeded.
resources_string_key:     []
resources_string_value:   []
attributes_string_key:    ['proc_id','hostname','appname']
attributes_string_value:  ['1','logs','systemd']
attributes_int64_key:     ['facility','priority']
attributes_int64_value:   [3,30]
attributes_float64_key:   []
attributes_float64_value: []

nitya-signoz

11/29/2022, 8:47 AM

Interesting, this is not expected. Can you change the time range on the top right to a larger duration?

George Larson

11/29/2022, 8:48 AM

oooooh! that worked!

nitya-signoz

11/29/2022, 8:50 AM

Interesting, can you compare the time on the machine which is generating logs, there might be a mismatch.

George Larson

11/29/2022, 8:50 AM

stamps don't match but it's def progress!

George Larson

11/29/2022, 8:51 AM

oh. lols

nitya-signoz

11/29/2022, 8:52 AM

Cool, can you please create an issue for this, might help others as well. You can add some sample logs from the above screenshot as well.

George Larson

11/29/2022, 8:52 AM

Copy code

❯ timedatectl
               Local time: Tue 2022-11-29 03:52:11 EST
           Universal time: Tue 2022-11-29 08:52:11 UTC
                 RTC time: Tue 2022-11-29 08:52:12
                Time zone: America/New_York (EST, -0500)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

George Larson

11/29/2022, 8:52 AM

yes. will do

George Larson

11/29/2022, 8:53 AM

after that, i'm gonna get a few hours of sleep before my meetings begin. lol. thanks for help!

nitya-signoz

11/29/2022, 8:54 AM

Haha cool. 🙂

George Larson

11/29/2022, 9:17 AM

i'll test more tomorrow but i'm betting on a classic layer 8 error. i am nearly 100% certain that i set the tz with

Copy code

timedatectl set-timezone America/New_York

but that i did not restart the machine or even the rsyslog.service i could be wrong because it would seem that the delta is still there even after rebooting. i'll test this theory tomorrow and add any useful findings to the ticket okay! last time, for real this time. thanks for all the help, i'm gonna take a power-nap 🙂

67 Views

Open in Slack

Previous Next