https://signoz.io logo
Title
c

Chủ Huỳnh Minh

10/10/2022, 5:02 AM
Hi Team, I'm having issue with sending logs from logstash to signoz. I followed this guide: https://signoz.io/docs/userguide/logstash_to_signoz/ but in signoz dashboard there's still no logs at all.
Here's all my configurations: • otel-collector-config.yaml
receivers:
tcplog/logstash:
max_log_size: 1MiB
listen_address: "0.0.0.0:2255"
attributes: {}
resource: {}
add_attributes: false
operators: []
------
service:
...
logs:
receivers: [otlp, filelog/dockercontainers, tcplog/logstash]
processors: [ batch ]
exporters: [  clickhouselogsexporter ]
• In logstash config:
output {
tcp {
codec => json_lines # this is required otherwise it will send eveything in a single line
host => "my-otelcollector-host-private-IP"
port => 2255
}
}
I also mapped port 2255 -> 2255 for otel-collector in docker-compose as well
p

Pranay

10/10/2022, 6:13 AM
@nitya-signoz do you have insights on this
n

nitya-signoz

10/10/2022, 6:15 AM
Hi, @Chủ Huỳnh Minh can you share if you are able to see any error logs in otel collector or logstash?
c

Chủ Huỳnh Minh

10/10/2022, 10:19 AM
Hi @nitya-signoz. I only see error logs in logstash. Its something like this: `tcp output exception {:host=>"10.100.6.91", :port=>54525, :exception=>#<EOFError: End of file reached>, :backtrace=&gt;["org/jruby/RubyIO.java:3021:in `sysread'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-tcp-6.0.2/lib/logstash/outputs/tcp.rb:161:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-json_lines-3.1.0/lib/logstash/codecs/json_lines.rb:67:in `encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:48:in `block in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:65:in `time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:64:in `time'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-tcp-6.0.2/lib/logstash/outputs/tcp.rb:209:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `block in multi_receive'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:105:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:143:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300:in
block in start_workers'"]}
My input format in case you want to know:
{
"@timestamp":"$time_iso8601",
"hostname":"$hostname",
"source":"$server_addr",
"client":"$remote_addr",
...
"request_length":$request_length,
"protocol":"$server_protocol",
"upstreamhost":"$upstream_addr",
}
I also used netstat and telnet to verify that the logstash receiver is listening on port 2255 as well so i don't think its firewall related
n

nitya-signoz

10/10/2022, 10:27 AM
Can you share your logstash version so that I can try to replicate this issue, once I get some bandwidth.
c

Chủ Huỳnh Minh

10/11/2022, 2:30 AM
i'm using logstash 7.8.1 and newest signoz
Hi, I've just tested again and see this error in otel collector:
2022-10-11T03:13:02.876Z        error   tcp/tcp.go:282  Scanner error   {"kind": "receiver", "name": "tcplog/logstash", "pipeline": "logs", "operator_id": "tcp_input", "operator_type": "tcp_input", "error": "bufio.Scanner: token too long"}
I tried to increase max_log_size to 10MiB in otel collector config but it doesnt seem to fix it (I dont see any error logs at all)