Hi Team,
I'm new to SigNoz and currently exploring exception-based alerts using ClickHouse queries, but I'm encountering an issue. I would really appreciate your support. Thank you in advance!
Here's the additional environment configuration I have made for email for signoz service in docker-compose,
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST=<http://scluster.stablehost.com:587|scluster.stablehost.com:587>
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__USERNAME=name@website.com
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__PASSWORD=password
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__FROM=support@website.com
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__REQUIRE__TLS=false
- SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__TLS__INSECURE__SKIP__VERIFY=true
Issue: I'm able to receive the test email from the alert channel configuration, but not from test notifications or actual alerts. When I trigger a test notification, I see a success message and an entry with the status "Firing" in the
Triggered Alerts tab. However, I encounter an error in the SigNoz service.
Error:
2025-05-12 16:30:49 {"level":"info","timestamp":"2025-05-12T11:00:49.950Z","logger":"<http://go.signoz.io/pkg/http/middleware|go.signoz.io/pkg/http/middleware>","caller":"middleware/logging.go:88","msg":"::RECEIVED-REQUEST::","client.address":"172.18.0.1:35196","user_agent.original":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36","server.address":"localhost","server.port":"8080","http.request.size":0,"http.route":"/api/v1/rules","http.response.status_code":200,"http.server.request.duration":0.015405533}
2025-05-12 16:31:24 {"level":"info","timestamp":"2025-05-12T11:01:24.860Z","caller":"rules/threshold_rule.go:109","msg":"prepareQueryRange","ts":1747047684854,"evalWindow":300000,"evalDelay":120000}
2025-05-12 16:31:24 {"level":"info","timestamp":"2025-05-12T11:01:24.908Z","caller":"utils/time.go:17","msg":"Elapsed time","func_name":"GetTimeSeriesResultV3","duration":0.047875804,"query":"SELECT\n toStartOfMinute(min_timestamp) AS minute,\n count() AS trace_count\nFROM\n(\n SELECT\n trace_id,\n min(timestamp) AS min_timestamp,\n groupArray(`resource_string_service$$name`) AS services\n --WHERE timestamp BETWEEN toDateTime(1747047240) AND toDateTime(1747047540)\n FROM signoz_traces.signoz_index_v3\n GROUP BY trace_id\n)\nWHERE (has(services, 'service-one') = 1) AND (has(services, 'debezium-connect') = 1) AND (has(services, 'service-two') = 0)\nGROUP BY minute\nORDER BY minute","source":"alerts","client":"query-service","alertID":"0196a520-e782-78bc-b30c-49f40e68395e"}
2025-05-12 16:31:24 {"level":"info","timestamp":"2025-05-12T11:01:24.908Z","caller":"rules/threshold_rule.go:483","msg":"number of alerts found","name":"Missing service alert","count":1}
2025-05-12 16:31:49 {"level":"info","timestamp":"2025-05-12T11:01:49.906Z","logger":"<http://go.signoz.io/pkg/http/middleware|go.signoz.io/pkg/http/middleware>","caller":"middleware/logging.go:88","msg":"::RECEIVED-REQUEST::","client.address":"172.18.0.1:35196","user_agent.original":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36","server.address":"localhost","server.port":"8080","http.request.size":0,"http.route":"/api/v1/rules","http.response.status_code":200,"http.server.request.duration":0.00284774}
2025-05-12 16:31:54 {"timestamp":"2025-05-12T11:01:54.927349855Z","level":"WARN","code":{"function":"<http://github.com/prometheus/alertmanager/notify.RetryStage.exec|github.com/prometheus/alertmanager/notify.RetryStage.exec>","file":"/home/runner/go/pkg/mod/github.com/prometheus/alertmanager@v0.28.0/notify/notify.go","line":866},"msg":"Notify attempt failed, will retry later","logger":"<http://github.com/SigNoz/signoz/pkg/alertmanager/signozalertmanager|github.com/SigNoz/signoz/pkg/alertmanager/signozalertmanager>","pkg":"<http://go.signoz.io/pkg/alertmanager/alertmanagerserver|go.signoz.io/pkg/alertmanager/alertmanagerserver>","component":"dispatcher","receiver":"Pruthvi NB Email","integration":"email[0]","aggrGroup":"{}/{ruleId=~\"-1|0196a520-e782-78bc-b30c-49f40e68395e\"}:{alertname=\"Missing service alert\"}","attempts":1,"err":"establish connection to server: dial tcp [::1]:25: connect: connection refused"}
