Kumaran Ilangovan

    Kumaran Ilangovan

    5 months ago
    Hi @User - I created a webhook and defined a basic rule. But the alert is not triggered anytime - not sure what I am missing here. Attached screenshot of rule definition and logs from alert-manager. I dont find any triggered alerts, so far. I kept value to greater than -1 - so that i can see some alert atleast. But till now, no luck. Note - the alert is in firing status. Please do the needful. Thanks!
    Prashant Shahi

    Prashant Shahi

    5 months ago
    Hi @User! We encountered this issue yesterday and currently investigating it. This issue is happening on k8s. Using the Docker, alerts works seemlessly. We will be creating a fix for this soon.
    Kumaran Ilangovan

    Kumaran Ilangovan

    5 months ago
    Thanks @User - will wait for k8s fix.
    Hi @User - meanwhile - can you send me a sample body which I will receive in webhook. Mainly wanted to look at json data available as part of the signoz alerts. Thanks!
    Prashant Shahi

    Prashant Shahi

    5 months ago
    @User SigNoz v0.7.5 is released and along with that chart signoz-0.0.12 is also out with the alerts fix. Refer here to upgrade:https://signoz.io/docs/operate/kubernetes/#upgrade
    Kumaran Ilangovan

    Kumaran Ilangovan

    5 months ago
    thanks @User - installed latest chart. Now I am getting a new error while accessing metrics, traces or any page. But Alerts page is fine - able to add/delete my alerts. IP mentioned in the error (refer screenshot) doesnt seem to be my cluster IP. Please help me in troubleshooting this. Note: Agents are ingesting - and no error seen at agent side. All pods after chart install are in running status. Thanks!
    Also, these are my pods running - not sure whether clickhouse will be running as separate pod or not.
    Prashant Shahi

    Prashant Shahi

    5 months ago
    It looks like clickhouse pod is not up anymore. Can you check if clickhouse statefulset is running?
    Kumaran Ilangovan

    Kumaran Ilangovan

    5 months ago
    Hi @User - found the problem. Yes, Chi cluster was not initiated earlier - but not sure about real reason. Now, I enabled nameOverride and fullnameOverride fields in chart (signoz service)- and its working fine. Hope this is not fix - but not sure too 🙂 Able to connect fine now. Thanks!