Hi team, we are sending logs from a remote VM to s...
# support
s
Hi team, we are sending logs from a remote VM to self-hosted signoz (Signoz otel collector )in our cluster via otel collector running in VM receiver filelog and otlp exporter. We want to have this setup in production. but we have reliability concerns like if the VM otel collector is down or signoz otel collector is down. we will miss logs. Signoz team please help with a detailed explanation for this usecase this would be really helpful and it is only blocker for us. @Nitish @prasanth
n
Interesting! Curious to know why did you self-host signoz rather than using SigNoz cloud?
n
You will miss some logs if your VM otel collector is down and logs are being read from end of file, but if the signoz-otel-collector is down and comes back in some amount of time it should be fine as there are retry mechanisms in VM otel collector.
Incase you are using OTEL SDK’s in your applications that should be fine as that also has retry mechanisms built into it.
s
@Nitish We have a compliance issue with sending any data outside the cluster, we were using Elastic Cloud earlier. now have to move to a self-hosted solution.
We are not using OTEL SDK instead we are reading logs from files, does the VM OTEL collector maintain any pointer to read data from where it is left in the file. Also how reliable is the retry mechanism I am not able to find any documentation about it.
n
maintain any pointer to read data from where it is left
No it doesn’t maintain any pointer. you can configure retry mechanism and queue here, it depends on how much resources you can allocate to make it retry-able here are the docs https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md
s
thanks @nitya-signoz, checking above
This will work in our setup
n
Good to hear