Hi team, I will be needing help to export logs of a node application to Signoz. I have a self hosted Signoz on a kubernetes cluster and a node application running on a different VM. My node application write application logs in a log file in the VM and the log file is getting rotated on daily basis. I am using the docker container available to configure signoz otel collector and export the logs from the log file. The setup works fine on the first day of implementation, but as soon as new log file is created then logs are not getting exported to Signoz. Please help me out to find a solution for this. I am attaching configuration for otel collector and the command used to up the docker container.

receivers:
  filelog:
    include: [/tmp/*.*]
    start_at: beginning
processors:
  attributes/nodeApplication:
    actions:
      - key: node_application
        value: "my-node-app"
        action: insert
  attributes/nodeApplicationEnvironment:
    actions:
      - key: node_application_environment
        value: "Dev"
        action: insert
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s
exporters:
  otlphttp:
    endpoint: <https://signals-collector-http.qualix.ai>
service:
  pipelines:
    logs:
      receivers: [filelog]
      processors:
        [
          attributes/nodeApplication,
          attributes/nodeApplicationEnvironment,
          batch,
        ]
      exporters: [otlphttp]

docker run -d --name signoz-host-otel-collector --user root -v $(pwd)/logs:/tmp:ro -v $(pwd)/otel-collector-config.yaml:/etc/otel/config.yaml signoz/signoz-otel-collector:0.88.11

A new file is getting generated using this format YYYY-MM-DD.log on daily basis in the logs directory of the node application. And the otel collector configuration is present at root directory of the node application.

can you check if the file the new file is accessbile from inside the otel-collector container by manually exec into it. because the filelog receiver should automatically read those files https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/filelogreceiver/README.md#log-rotation

Yeah I have verified this as well and found that new log file is getting mounted in the desired directory of docker container but logs are not getting exported from log file. This is the code which creates log rotation for node application

const bunyan = require('bunyan');
const RotatingFileStream = require('bunyan-rotating-file-stream');

const buny = bunyan.createLogger({
    name: process.env.APP_NAME || 'nodelib',
    level: "debug",
    serializers: bunyan.stdSerializers,
    streams: [{
        type: 'raw',
        stream: new RotatingFileStream({
            path: "./logs/%Y-%m-%d.log",
            period: '1d',   // daily rotation
            rotateExisting: true,
            totalFiles: 30,        // keep 30 back copies
            threshold: '256m',      // Rotate log files larger than 256 megabytes
            totalSize: '1g',      // Don't keep more than 1g of archived log files
            gzip: true,
            startNewFile: false,
        })
    }]
});

module.exports = buny;