Hi Team, I'm new to SigNoz and currently exploring exception-based alerts using ClickHouse queries, but I'm encountering an issue. I would really appreciate your support. Thank you in advance! Here's the additional environment configuration I have made for email for signoz service in docker-compose,

- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST=<http://scluster.stablehost.com:587|scluster.stablehost.com:587>
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__USERNAME=name@website.com
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__PASSWORD=password      
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__FROM=support@website.com
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__REQUIRE__TLS=false
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__TLS__INSECURE__SKIP__VERIFY=true

Issue: I'm able to receive the test email from the alert channel configuration, but not from test notifications or actual alerts. When I trigger a test notification, I see a success message and an entry with the status "Firing" in the Triggered Alerts tab. However, I encounter an error in the SigNoz service. Error:

2025-05-12 16:30:49 {"level":"info","timestamp":"2025-05-12T11:00:49.950Z","logger":"<http://go.signoz.io/pkg/http/middleware|go.signoz.io/pkg/http/middleware>","caller":"middleware/logging.go:88","msg":"::RECEIVED-REQUEST::","client.address":"172.18.0.1:35196","user_agent.original":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36","server.address":"localhost","server.port":"8080","http.request.size":0,"http.route":"/api/v1/rules","http.response.status_code":200,"http.server.request.duration":0.015405533}
2025-05-12 16:31:24 {"level":"info","timestamp":"2025-05-12T11:01:24.860Z","caller":"rules/threshold_rule.go:109","msg":"prepareQueryRange","ts":1747047684854,"evalWindow":300000,"evalDelay":120000}
2025-05-12 16:31:24 {"level":"info","timestamp":"2025-05-12T11:01:24.908Z","caller":"utils/time.go:17","msg":"Elapsed time","func_name":"GetTimeSeriesResultV3","duration":0.047875804,"query":"SELECT\n    toStartOfMinute(min_timestamp) AS minute,\n    count() AS trace_count\nFROM\n(\n    SELECT\n        trace_id,\n        min(timestamp) AS min_timestamp,\n        groupArray(`resource_string_service$$name`) AS services\n    --WHERE timestamp BETWEEN toDateTime(1747047240) AND toDateTime(1747047540)\n    FROM signoz_traces.signoz_index_v3\n    GROUP BY trace_id\n)\nWHERE (has(services, 'service-one') = 1) AND (has(services, 'debezium-connect') = 1) AND (has(services, 'service-two') = 0)\nGROUP BY minute\nORDER BY minute","source":"alerts","client":"query-service","alertID":"0196a520-e782-78bc-b30c-49f40e68395e"}
2025-05-12 16:31:24 {"level":"info","timestamp":"2025-05-12T11:01:24.908Z","caller":"rules/threshold_rule.go:483","msg":"number of alerts found","name":"Missing service alert","count":1}
2025-05-12 16:31:49 {"level":"info","timestamp":"2025-05-12T11:01:49.906Z","logger":"<http://go.signoz.io/pkg/http/middleware|go.signoz.io/pkg/http/middleware>","caller":"middleware/logging.go:88","msg":"::RECEIVED-REQUEST::","client.address":"172.18.0.1:35196","user_agent.original":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36","server.address":"localhost","server.port":"8080","http.request.size":0,"http.route":"/api/v1/rules","http.response.status_code":200,"http.server.request.duration":0.00284774}
2025-05-12 16:31:54 {"timestamp":"2025-05-12T11:01:54.927349855Z","level":"WARN","code":{"function":"<http://github.com/prometheus/alertmanager/notify.RetryStage.exec|github.com/prometheus/alertmanager/notify.RetryStage.exec>","file":"/home/runner/go/pkg/mod/github.com/prometheus/alertmanager@v0.28.0/notify/notify.go","line":866},"msg":"Notify attempt failed, will retry later","logger":"<http://github.com/SigNoz/signoz/pkg/alertmanager/signozalertmanager|github.com/SigNoz/signoz/pkg/alertmanager/signozalertmanager>","pkg":"<http://go.signoz.io/pkg/alertmanager/alertmanagerserver|go.signoz.io/pkg/alertmanager/alertmanagerserver>","component":"dispatcher","receiver":"Pruthvi NB Email","integration":"email[0]","aggrGroup":"{}/{ruleId=~\"-1|0196a520-e782-78bc-b30c-49f40e68395e\"}:{alertname=\"Missing service alert\"}","attempts":1,"err":"establish connection to server: dial tcp [::1]:25: connect: connection refused"}

Hey @Yash Z what's the version of signoz you're using

Hey @Nagesh Bansal, Thanks for the reply, I'm using the latest version and to confirm on reachability, as mentioned, I'm able to receive the test email from the alert channel configuration.

How are you setting up the envs in the docker-compose file?

@Nagesh Bansal I have added 6 new environment variable for email under signoz service in docker-compose file, as mentioned. I have just attached the docker-compose file for the reference.

Hey @Yash Z, try with the setting up the variable:

SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__TLS__CONFIG_INSECURE__SKIP__VERIFY: true

@Nagesh Bansal I tried adding this variable, but it's still failing with the same error.

this one right:

"function":"<http://github.com/prometheus/alertmanager/notify.RetryStage.exec|github.com/prometheus/alertmanager/notify.RetryStage.exec>","file":"/home/runner/go/pkg/mod/github.com/prometheus/alertmanager@v0.28.0/notify/notify.go","line":866},"msg":"Notify attempt failed, will retry later","logger":"<http://github.com/SigNoz/signoz/pkg/alertmanager/signozalertmanager|github.com/SigNoz/signoz/pkg/alertmanager/signozalertmanager>","pkg":"<http://go.signoz.io/pkg/alertmanager/alertmanagerserver|go.signoz.io/pkg/alertmanager/alertmanagerserver>","component":"dispatcher","receiver":"Pruthvi NB Email","integration":"email[0]","aggrGroup":"{}/{ruleId=~\"-1|0196a520-e782-78bc-b30c-49f40e68395e\"}:{alertname=\"Missing service alert\"}","attempts":1,"err":"establish connection to server: dial tcp [::1]:25: connect: connection refused"}

yes

Hey @Yash Z could you please check with your smtp provider once, we didn't came across any issue with the configuration

yes @Nagesh Bansal, checking on that as well. Also, I'm still little confused that how I'm able to receive the test email from the alert channel configuration, but not from test notifications.