if anyone else has similar problem 🙂 Signoz works like a charm , but be careful with your node taints and add tolerations to agent block in helm chart.
SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.