This message was deleted.

@User version of SigNoz?

I am using v0.7.5

Hi @User - got any troubleshoot steps i should follow?

@User you should see alerts logs in query-service. From the logs, you should be able to know triggered alerts. I tested the webhook alert, it worked fine for me. I was able to pass username/password as well. Is your webhook channel configured appropriately? Is your webhook accessible from SigNoz cluster? Is it accepting POST data from the endpoint? Here is a sample webhook data:

{
  "receiver": "webhook-tester",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "SigNozOtelMissing",
        "severity": "critical"
      },
      "annotations": {
        "description": "SigNoz OtelCollector is not up VALUE = 1 LABELS = map[]",
        "summary": "SigNoz Otel Missing"
      },
      "startsAt": "2022-04-11T20:40:59.657116411Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "<http://my-release-signoz-query-service-0:3301/graph?g0.expr=absent%28up%29>&g0.tab=1",
      "fingerprint": "fac4e16c7c178c32"
    }
  ],
  "groupLabels": {
    "alertname": "SigNozOtelMissing"
  },
  "commonLabels": {
    "alertname": "SigNozOtelMissing",
    "severity": "critical"
  },
  "commonAnnotations": {
    "description": "SigNoz OtelCollector is not up VALUE = 1 LABELS = map[]",
    "summary": "SigNoz Otel Missing"
  },
  "externalURL": "<http://my-release-signoz-alertmanager-0:9093>",
  "version": "4",
  "groupKey": "{}/{}:{alertname=\"SigNozOtelMissing\"}",
  "truncatedAlerts": 0
}

Header:

{ "authorization":"Basic dXNlcjpwYXNzd29yZA==","content-type":"application/json", ... }

Hi @User Is your webhook channel configured appropriately? -- Yes in Signoz i created a channel - and ensured its not authorized. Is your webhook accessible from SigNoz cluster? -- Yes my webhook is public - and its consuming requests from other systems. Is it accepting POST data from the endpoint? -- Yes, POST data is accepted. Whether any Bearer Token header or empty token be sent in Header - when sending webhook data? Also, in query-service logs - do we have any keyword search for confirming alerts? From Query-service - I notice this log frequently.

2022-04-12T04:22:02.424Z        INFO    app/server.go:155       /api/v1/version timeTaken: 18.7µs
2022-04-12T04:22:02.424Z        INFO    app/server.go:155       /api/v1/version timeTaken: 21.901µs
time="2022-04-12T04:22:03Z" level=warning msg="Ignoring hint {StepMs:0 Func:rate StartMs:1649737263891 EndMs:1649737323891} for query [1649737263891,1649737323891,{span_kind=\"SPAN_KIND_SERVER\",__name__=\"signoz_latency_count\"}]." component=clickhouse

My rule definition is,

alert: Sample RPS
        expr: sum(rate(signoz_latency_count{span_kind="SPAN_KIND_SERVER"}[2m])) by (service_name) > 1
        for: 0m
        labels:
            severity: warning
        annotations:
            summary: High RPS of Applications
            description: "RPS is > 1
             VALUE = {{ $value }}
             LABELS = {{ $labels }}"

Whether any Bearer Token header or empty token be sent in Header - when sending webhook data?

If username/password are set in webhook alert channel, it includes HTTP basic authentication header.

Ok @User - from log shared above - do you find any issues? or anyway to confirm - whether webhook requests are posted from query-service?

@User please post alertmanager logs too.

Hi @Ankit Nayan - I added already in first post - added again here.

I don't think alertmanager is receiving alerts

No @Ankit Nayan - I can share the helm. Updated query-service, frontend and collector as nodeport.

@User that looks fine to me. Do you have more than one channels configured? If that's the case, the alert will be sent to the latest channel.

I have 2 webhooks created - but to same destination... I created 2nd only after first one didnt respond any alerts. Will keep only one webhook for now.

@User are you using latest helm chart

signoz-0.0.12

? you can follow the steps below to test the alert: • Configure alert webhook channel. Make sure it accepts POST method and accessible • Create SigNozOtelMissing alert

alert: SigNozOtelMissing
expr: absent(up{hostname="",job="otel-collector"})
for: 0m
labels:
    severity: warning
annotations:
    summary: SigNoz OtelCollector missing
    description: "SigNoz OtelCollector up status metrics disappeared\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

• Set replicas for

Otel-Collector

and

Otel-Collector-Metrics

to 0

# set replicas to 0
kubectl -n platform edit deployment.apps/my-release-signoz-otel-collector

# set replicas to 0
kubectl -n platform edit deployment.apps/my-release-signoz-otel-collector-metrics

• wait for around 5 minutes for the trigger

Sure @User - let me enable log level in alertmanager and check. Thanks!