https://signoz.io logo
Join SlackCommunities
Powered by
I have an issue that has me stumped. Self-hosted S...
# support
b
I have an issue that has me stumped. Self-hosted Signoz with Docker. All the default settings except I added a syslog receiver in otel-collector-config.yaml (and added the appropriate ports in docker-compose-minimal.yaml). UDP 514, and TCP 601. I also made sure to add syslog to the "logs" pipeline. I can see the syslog traffic making it to the host machine by running a tcpdump. And by running this command: "docker exec -it signoz-otel-collector nc -u -l -p 514" I can see the syslog traffic making it into the signoz-otel-collector container. So why isn't it showing up in Signoz?
Relevant sections of otel-collector-config.yaml:
Copy code
receivers:
  syslog:
    udp:
      listen_address: "0.0.0.0:514"
    tcp:
      listen_address: "0.0.0.0:601"
    protocol: rfc3164
    location: America/Chicago
    operators:
      - type: move
        from: attributes.message
        to: body
-----
Copy code
logs:
      receivers: [otlp, tcplog/docker, syslog]
      processors: [batch]
      exporters: [clickhouselogsexporter]
And here is the otel-collector section of my docker-compose-minimal.yaml:
Copy code
otel-collector:
    image: signoz/signoz-otel-collector:${OTELCOL_TAG:-0.111.14}
    container_name: signoz-otel-collector
    command:
      [
        "--config=/etc/otel-collector-config.yaml",
        "--manager-config=/etc/manager-config.yaml",
        "--copy-path=/var/tmp/collector-config.yaml",
        "--feature-gates=-pkg.translator.prometheus.NormalizeName"
      ]
    user: root # required for reading docker container logs
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
      - ./otel-collector-opamp-config.yaml:/etc/manager-config.yaml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /:/hostfs:ro
    environment:
      - OTEL_RESOURCE_ATTRIBUTES=host.name=signoz-host,os.type=linux
      - LOW_CARDINAL_EXCEPTION_GROUPING=false
    ports:
      # - "1777:1777"     # pprof extension
      - "4317:4317" # OTLP gRPC receiver
      - "4318:4318" # OTLP HTTP receiver
      # - "8888:8888"     # OtelCollector internal metrics
      # - "8889:8889"     # signoz spanmetrics exposed by the agent
      # - "9411:9411"     # Zipkin port
      # - "13133:13133"   # health check extension
      # - "14250:14250"   # Jaeger gRPC
      # - "14268:14268"   # Jaeger thrift HTTP
      # - "55678:55678"   # OpenCensus receiver
      # - "55679:55679"   # zPages extension
      - "514:514/udp"   #standard syslog
      - "601:601/tcp"   #cisco standard tcp syslog
    restart: unless-stopped
    depends_on:
      clickhouse:
        condition: service_healthy
      otel-collector-migrator-sync:
        condition: service_completed_successfully
      query-service:
        condition: service_healthy
s
Do you see any error logs in collector logs?
b
No errors.
s
b
I can't seem to add the debug exporter, it causes the collector to crash on startup:
Copy code
{
  "level": "error",
  "timestamp": "2024-12-10T20:21:43.198Z",
  "caller": "opamp/server_client.go:268",
  "msg": "Collector failed for restart during rollback",
  "component": "opamp-server-client",
  "error": "failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):\n\nerror decoding 'exporters': unknown type: \"debug\" for id: \"debug\" (valid values: [clickhousemetricswrite clickhousemetricswritev2 clickhousetraces syslog carbon cassandra clickhouselogsexporter prometheusremotewrite logging otlphttp alertmanager awss3 kafka zipkin file googlecloudpubsub rabbitmq prometheus pulsar signozkafkaexporter otlp awskinesis loadbalancing opencensus])",
  "stacktrace": "<http://github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload|github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload>\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:268\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).applyRemoteConfig\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:173\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).Apply\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:159\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:209\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/opamp-go@v0.5.0/client/internal/clientcommon.go:197"
}
s
Use 
logging
b
Like this? (I just un-commented the line that was already there.)
Copy code
exporters:
  logging: {}
If so, it didn't give me any new information when I check the logs. 😕
After some Googling I changed it to this:
Copy code
exporters:
  logging:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200
But that still didn't give me more info when I check the logs.
I don't know if this helps or not, but I have udp/514 and tcp/601 set up in my syslog receiver. The 601 traffic is making it all the way through and showing up in Signoz. It's just the 514 traffic that doesn't. (They both use the same exporters and service pipelines.)
s
Not sure what could be the issue
b
Thanks for trying. And something to make this even stranger is that it was working for a while, until the (DHCP) address of my host machine changed. I updated the IP address in all the devices that send syslog (and I can see it being received) but still SigNoz can no longer see it.
Just in case anybody has a similar issue and finds this. I found the issue. I think this device is doesn't properly conform to rfc3164. I changed the device to send it's syslog on udp/415 and added
Copy code
udplog:
    listen_address: "0.0.0.0:415"
And now the logs are showing up. I think it's actually better to send the logs from this device to a separate port since it's my firewall logs and I want to add a regex to pull src_ip, dst_ip, rule_name, etc.
42 Views
Open in Slack
PreviousNext
Powered by