Hey Team, i was looking to altogether replace our grafana elastic and prometheus setup with signoz and was trying to do the poc for metrics. We run a signoz in docker standalone mode in a dedicated vm, and have our backend stack on k8s. Currently we have installed grafana-agent in the k8s which scrapes metric endpoints exposed by different services internally and sends them to prometheus via remote write. I wanted to get those metrics in signoz as well, but all the docs i scoured only have info about letting signoz scrape the metrics by defining jobs and scraping endpoints. As you can see its a tad bit difficult for me to do a poc with this if i wanna expose all those endpoints internal to the k8s cluster to where signoz is running. Is their any other way of sending those metrics from k8s to signoz via some remote write option. I might be missing something simple, would really appreciate if someone could point me in the right direction

Hi there Dipen these are great questions. It seems like there are two options here, one which will definitely work, one which I'm less sure about: starting with the less certain option: I'm not sure if there's a way to get the Grafana agent to send data into an OpenTelemetry collector. That would certainly be convenient, but it doesn't look like it's a thing. And also isn't really what you'd asked for The next option would be to have some OpenTelemetry collectors running where you've currently got the Grafana agent, in a config something like this. (for the large Otel collector at the center, that would be the SigNoz collector running there)

I wanted to get those metrics in signoz as well, but all the docs i scoured only have info about letting signoz scrape the metrics by defining jobs and scraping endpoints. As you can see its a tad bit difficult for me to do a poc with this if i wanna expose all those endpoints internal to the k8s cluster to where signoz is running. Is their any other way of sending those metrics from k8s

@Dipen Bhatt Please use the target allocator https://github.com/open-telemetry/opentelemetry-operator#target-allocator. It will do the job of scraping the targets and exporting the data to otlp destination (SigNoz).