I am trying to get logs from logstash to Signoz. L...
# supportk
Krishnadas K P
07/26/2023, 8:16 AMI am trying to get logs from logstash to Signoz. Logstash is installed directly in the ubuntu VM and signoz is installed using docker in the same VM. But I am unable to get any logs in Signoz via open telemetry collector.
This is my logstash config
input {file {path => "/home/iislogs.log"start_position => "beginning"}}output {tcp {port => 2255host => "localhost"}}p
Prashant Shahi
07/26/2023, 8:40 AMDid you restart otel-collector properly after config change and exposed ports?
k
Krishnadas K P
07/26/2023, 8:42 AMYes. I had restarted logstash and also docker compose down and up all containers
Krishnadas K P
07/26/2023, 8:46 AMAre we supposed to see any logs in otel collector when it is receiving data from a receiver ? Because I am not seeing any logs for that yet
Krishnadas K P
07/26/2023, 9:18 AM@Prashant Shahi I have set logstash to send data to localhost:2255 and set signoz to receive data at 0.0.0.0:2255.. Is this correct ? I can see in logstash logs that it is opening connection correctly. But no movement in signoz otel collector logs
Krishnadas K P
07/26/2023, 9:56 AM@Prashant Shahi I have also tried to test logs from mounted volume as per https://signoz.io/docs/userguide/collect_logs_from_file/
This is not showing up in Signoz UI either.
I can see the following log in otel collector container. But nothing more.
2023-07-26T094823.883Z info fileconsumer/file.go:192 Started watching file {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/tmp/python.log"}
Krishnadas K P
07/26/2023, 10:57 AMCuriously enough the docker container logs are working fine and getting exported to clickhouse logs and visualized in signoz UI. Both logstash and filelog receivers that I added arent working
Krishnadas K P
07/26/2023, 1:15 PM@Prashant Shahi, @nitya-signoz Any idea on this please ?
n
nitya-signoz
07/26/2023, 3:29 PM• For file log receiver, are new logs getting written to the file, if not then you will have to use the
start_atk
Krishnadas K P
07/26/2023, 3:50 PMThanks for the reply @nitya-signoz.
1. Yes. New logs are getting written. I am using the same tutorial of python logging as given in Signoz docs
2. Yes. I have exposed the 2255 port on otel collector. Logstash log snippet is attached.
n
nitya-signoz
07/26/2023, 3:52 PM• Can you check if you have mounted your file properly ?
• Also please share your otel-collector config
k
Krishnadas K P
07/26/2023, 3:57 PM1. Yes I can see the file after exec ing into the docker container.
2. Attached.
n
nitya-signoz
07/26/2023, 3:59 PMThis seems correct, what environment are you running this ? linux ?
k
Krishnadas K P
07/26/2023, 4:00 PMYes. In Ubuntu 20.04.6 LTS (Focal Fossa)
Krishnadas K P
07/26/2023, 4:48 PMAny idea @nitya-signoz ? I would have thought the filelog receiver would have been fairly simple to work with :(
p
Prashant Shahi
07/27/2023, 1:28 AM@Krishnadas K P updating content of already mounted otelcol-config file is not often updated in the restarted container.
hard remove of the otel-collector container is required for that.
Copy code
docker stop <otelcollector-container>
docker rm <otelcollector-container>
docker-compose -f ...
OR
./install.shk
Krishnadas K P
07/27/2023, 1:40 AMI was always doing docker-compose down and then docker-compose up -d.
Funnily enough, I removed the existing docker container logs receiver from the otel-collector-config and just added the filelog receiver for my logs folder and it started working. I will play around a bit and confirm what changed
6 Views