Hey folks. Needed some pointers with alerting. I've set up the SigNoz agent to send cluster metrics from a bunch of clusters to SigNoz, and the idea is to fire an alert whenever a particular Pod's status is not running(so the k8s_pod_phase label is != 2). Now I need the alert message to also contain the k8s_cluster_name label, so I know which cluster that Pod is breaking on. How would I go about that? I've tried using $labels but that doesn't seem to work/maybe I'm doing it wrong.
Appreciate the help, thanks!