Trying to set up a webhook for Alerting Channels with: https://signoz.io/docs/alerts-management/notification-channel/webhook/ What did I do wrong? I went to this website: https://webhook.site/ I pasted the given webhook URL in the right place. But afterwards, I'm not able to test or save my configuration. I'm on Kubernetes with Signoz installed using this manifest with a custom namespace.

Share the logs of alertmanager and query-service pods at the time of testing.

I don't know how, but now it's working well with the webhook.site URL. I don't really know where that came from, but it's good. I'm trying it with the Google APIs chat URL, but that doesn't seem supported: URL like:

<https://chat.googleapis.com/v1/spaces/>...

Alertmanager log:

level=error ts=2024-07-30T07:10:48.103Z caller=api.go:808 component=api version=v1 msg="API error" err="server_error: unexpected status code 400: <https://chat.googleapis.com/v1/spaces/>...

I really think this error corresponds to this one:

{
  "error": {
    "code": 400,
    "message": "Message cannot be empty.",
    "status": "INVALID_ARGUMENT"
  }
}

It's because you have to send the message through JSON in the body of the POST request, like: Body:

{
  "text": "{\"receiver\":\"test-google tchat fux events\",\"status\":\"firing\",\"alerts\":[{\"status\":\"firing\",\"labels\":{\"alertname\":\"Test Alert (google tchat fux events)\",\"severity\":\"critical\"},\"annotations\":{\"description\":\"Test alert fired from SigNoz dashboard\",\"message\":\"Test alert fired from SigNoz dashboard\",\"summary\":\"Test alert fired from SigNoz dashboard\"},\"startsAt\":\"0001-01-01T00:00:00Z\",\"endsAt\":\"0001-01-01T00:00:00Z\",\"generatorURL\":\"\",\"fingerprint\":\"9e56996e970e12c6\"}],\"groupLabels\":{\"alertname\":\"Test Alert (google tchat fux events)\",\"severity\":\"info\"},\"commonLabels\":{\"alertname\":\"Test Alert (google tchat fux events)\",\"severity\":\"critical\"},\"commonAnnotations\":{\"description\":\"Test alert fired from SigNoz dashboard\",\"message\":\"Test alert fired from SigNoz dashboard\",\"summary\":\"Test alert fired from SigNoz dashboard\"},\"externalURL\":\"<http://signoz.io>\",\"version\":\"4\",\"groupKey\":\"1\",\"truncatedAlerts\":0}"
}

But since it's a Google Chat, it's a good idea to send JSON data. I think the actual body is something like this:

{
  "receiver": "test-google tchat fux events",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "Test Alert (google tchat fux events)",
        "severity": "critical"
      },
      "annotations": {
        "description": "Test alert fired from SigNoz dashboard",
        "message": "Test alert fired from SigNoz dashboard",
        "summary": "Test alert fired from SigNoz dashboard"
      },
      "startsAt": "0001-01-01T00:00:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "",
      "fingerprint": "9e56996e970e12c6"
    }
  ],
  "groupLabels": {
    "alertname": "Test Alert (google tchat fux events)",
    "severity": "info"
  },
  "commonLabels": {
    "alertname": "Test Alert (google tchat fux events)",
    "severity": "critical"
  },
  "commonAnnotations": {
    "description": "Test alert fired from SigNoz dashboard",
    "message": "Test alert fired from SigNoz dashboard",
    "summary": "Test alert fired from SigNoz dashboard"
  },
  "externalURL": "<http://signoz.io>",
  "version": "4",
  "groupKey": "1",
  "truncatedAlerts": 0
}

Gchat is not currently supported