Have you guys thought of leveraging Kuberhealthy to support Synthetic Checks? I think that may be a great way to add synthetics to the platform. https://github.com/kuberhealthy/kuberhealthy < This is one thing that I know my company heavily uses within New Relic/DataDog that Signoz doesn’t support

interesting @Blake Romano Didn't know about the project Does your company uses Kuberhealthy currently for Synthetic checks?

It is a tool we might use as we explore maybe moving away from the big vendors

what do you mean by SigNoz leveraging it since we don't run ourselves any k8s clusters. It looks like a tool a SRE would use in their stack.

heavily uses within New Relic/DataDog that Signoz doesn’t support

Can you elaborate this? What is not supported by SigNoz? do you want to receive the metrics exposed by kuberhealthy. You can always configure this to include more scrape targets here https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/otel-collector-metrics-config.yaml#L10-L14.

What I am suggesting is allowing teams to make synthetics by allowing kuberhealthy to create checks. New Relic/DataDog have this concept of Synthetic Checks. https://docs.datadoghq.com/synthetics/ where you can create scripts, it can check if a website is up, or if a script passes or not. If it is possible to integrate that directly within the Signoz UI that would be ideal. Also the ability to create custom synthetic scripts within the UI. I don't disagree that for now it's possible to send the prometheus metrics to Otel and then query and create alerts from that but it'd be nice for there to be a way for Signoz to create those synthetics via kuberhealthy or some other method for us and having it built within the UI

What I am suggesting is allowing teams to make synthetics by allowing kuberhealthy to create checks.

That's why I suggested being able to leverage it.

I am still curious to understand your line of thought, especially in the context of kuberhealthy and SigNoz. Are you asking for a feature where SigNoz creates

KuberhealthyCheck

resources in your cluster? What I understood from this graphic and where SigNoz fits in this is, you as an end user would create a CRD the checker pod runs periodically. And the responses are exposed in multiple ways json, influx, prom (which one can configure with scrape job).

Yeah I would see a case where I as a user could write a script in node.js. Signoz could deploy that script as a docker image as a Checker CRD. Then the results Signoz with Otel would recieve back and there'd be nice graphics similar to DataDog surrounding that

Synthetic monitoring as a product vertical is something that adds value but in the context of kuberhealthy, Even if we find a way to integrate with it, I doubt if practitioners would allow SigNoz to create a custom resources in their k8s clusters because it requires admin privileges.

I'm not super specific about exact implementation.

So you are just interested in synthetic monitoring support in SigNoz but since you mentioned about kuberhealthy I was talking about it all along how you could already make use of it with scrape job and et al...

You could use a concept like this https://github.com/backstage/backstage/tree/master/packages/backend-tasks for creating those synthetic tasks without new infrastructure.