@Vishal Sharma Thanks a lot for the hints and sorry for not replying for so long! I'd like to concentrate on the logs for now since this is most important to me - error tracking later.
I got my hands dirty with fluentd and the fluent forward receiver now. I configured everything according to
https://signoz.io/docs/userguide/fluentd_to_signoz/ but the logs don't seem to appear in Signoz still.
I have a docker-compose installation of signoz running and retrieved the ip of the clickhouse-setup-otel-collector container (in my case 172.19.0.6). This I added to the fluentd config of my rails application (see below). I also adjusted the docker-compose.yaml to open the 24224 port as described in the mentioned article (and configured the receiver, the exporter and the log-service).
When I curl 172.19.0.6:24224 from the machine where I have signoz docker-compose and my rails application docker-compose running, I can see this, so it seems to be ok:
curl: (52) Empty reply from server
Though when my application logs something and I see it in the docker logs of the fluentd service that I have there e.g. a line like the one below, I would expect it to appear in signoz too but it doesn't. I know that the format needs to be improved but I would expect at least anything to appear.
Any hints how to move on from here? Thanks a lot!
Log line, made visible with docker compose logs fluentd (inside the folder of my rails app):
cfa-fluentd-1 | 2023-08-02 07
0903.000000000 +0000 click-for-all-development: {"messages":"Started GET \"/\" for 31.16.250.201 at 2023-08-02 07
0902 +0000\nCannot render console from 31.16.250.201! Allowed networks: 127.0.0.0/127.255.255.255, ::1\nProcessing by WelcomeController#index as HTML\n Rendered welcome/index.html.erb within layouts/application (Duration: 4.6ms | Allocations: 1418)\nImportmap skipped missing path: turbo.min.js\nImportmap skipped missing path: turbo.min.js\n Rendered layout layouts/application.html.erb (Duration: 99.4ms | Allocations: 31977)\nCompleted 200 OK in 116ms (Views: 102.9ms | ActiveRecord: 0.0ms | Allocations: 35811)\n\n","info":"WARN"}
fluentd-config in the rails app
<source>
@type forward
port 24227 # don't worry about this, that's just where my rails app sends the logs to fluentd
</source>
<match>
@type copy
<store>
@type forward
send_timeout 60s
<server>
name signo # name chosen on purpose to reduce naming duplications and speed up trouble shooting
host 172.19.0.6
port 24224
</server>
</store>
<store>
@type stdout
</store>
</match>