This message was deleted.

Slackbot

02/28/2023, 12:18 PM

This message was deleted.

Harshith.R.S

02/28/2023, 12:22 PM

the query I'm writing for this is

Copy code

select
toStartOfInterval(timestamp, INTERVAL 2 MINUTE) AS interval,
serviceName,
((select count() as value from signoz_traces.signoz_index_v2
where httpCode>='100' AND httpCode<='499')*100.0
/
(select count() as value from signoz_traces.signoz_index_v2
where httpCode>='100' AND httpCode<='599')*1.0)
FROM signoz_traces.distributed_signoz_index_v2
where
serviceName = 'frontend' AND
timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}
GROUP BY (serviceName, interval);

Amol Umbark

02/28/2023, 1:15 PM

hey @Harshith.R.S which version are you on? are you on docker or k8s ?

Amol Umbark

02/28/2023, 1:15 PM

I will try to reproduce and get back

Harshith.R.S

02/28/2023, 1:23 PM

docker setup v1.16.1

👍 1

Amol Umbark

03/02/2023, 5:35 AM

hey @Harshith.R.S can you try sending a test notification on this form. I would suggest picking very high threshold so the below is going to always work

Harshith.R.S

03/02/2023, 7:44 AM

I've set the threshold to 80 the alert is still firing

Amol Umbark

03/03/2023, 8:20 AM

Hey @Harshith.R.S I could not reproduce the error. Can you try the following query. I just changed interval to 5 minute instead of 2. Also change "in total" to at least once.

Copy code

select
toStartOfInterval(timestamp, INTERVAL 5 MINUTE) AS interval,
serviceName,
((select count() as value from signoz_traces.signoz_index_v2
where httpCode>='100' AND httpCode<='499')*100.0
/
(select count() as value from signoz_traces.signoz_index_v2
where httpCode>='100' AND httpCode<='599')*1.0)
FROM signoz_traces.distributed_signoz_index_v2
where
serviceName = 'frontend' AND
timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}
GROUP BY (serviceName, interval);

Also do these steps to get more clarity: 1. Get a new webhook url from webhook.site 2. Define a channel for the webhook url 3. When defining alert add webhook channel in the list of preferred channels 4. Send a test notification to see the observed value in webhook.site. Please share the results.

Amol Umbark

03/03/2023, 8:26 AM

Also would like to see the output of the query from clickhouse cli. more instructions on setting up cli here

Harshith.R.S

03/03/2023, 8:34 AM

I don't know why it's not working for me

Amol Umbark

03/03/2023, 8:39 AM

can we do a quick huddle?

Harshith.R.S

03/03/2023, 9:11 AM

Copy code

2023-03-03T09:10:36.514Z        ERROR   alertManager/notifier.go:232    alertmanager%!(EXTRA string=<http://alertmanager:9093/api/v1/alerts>, string=count, int=2, string=msg, string=Error calling alert API, string=err, *errors.errorString=bad response status 400 Bad Request)
<http://go.signoz.io/signoz/pkg/query-service/integrations/alertManager.(*Notifier).sendAll.func1|go.signoz.io/signoz/pkg/query-service/integrations/alertManager.(*Notifier).sendAll.func1>
        /go/src/github.com/signoz/signoz/pkg/query-service/integrations/alertManager/notifier.go:232
2023-03-03T09:10:36.514Z        WARN    alertManager/notifier.go:136    msg: dropped alerts      count:2
<http://go.signoz.io/signoz/pkg/query-service/integrations/alertManager.(*Notifier).Run|go.signoz.io/signoz/pkg/query-service/integrations/alertManager.(*Notifier).Run>

Amol Umbark

03/03/2023, 9:30 AM

can you pls share log of alert manager

Amol Umbark

03/03/2023, 9:30 AM

same liek query service you will find a container for alert manager

Harshith.R.S

03/03/2023, 9:34 AM

Copy code

level=error ts=2023-03-03T09:33:36.516Z caller=api.go:808 component=api version=v1 msg="API error" err="bad_data: \"divide(multiply(_subquery310, 100.), _subquery311)\" is not a valid label name"

Harshith.R.S

03/03/2023, 12:05 PM

I completely rewrote the query

Copy code

SELECT
subq1.interval,
subq1.value * 100.0 / subq2.value AS success_rate
FROM (
SELECT
toStartOfInterval(timestamp, INTERVAL 2 MINUTE) AS interval,
COUNT() AS value
FROM signoz_traces.signoz_index_v2
WHERE
httpCode >= '100' AND
httpCode <= '499' AND
timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}
GROUP BY interval
) AS subq1
JOIN (
SELECT
toStartOfInterval(timestamp, INTERVAL 2 MINUTE) AS interval,
COUNT() AS value
FROM signoz_traces.signoz_index_v2
WHERE
httpCode >= '100' AND
httpCode <= '599' AND
timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}
GROUP BY interval
) AS subq2
ON subq1.interval = subq2.interval;

now after updating it is still firing the alert I've attached the necessary screencaps

Harshith.R.S

03/03/2023, 12:11 PM

it says the value went down to 0 but as you can see in graph the value stays at 100 does not fluctuate

Amol Umbark

03/03/2023, 12:17 PM

can try running the query from log in the clickhouse cli

Amol Umbark

03/03/2023, 12:20 PM

in your alert definition can you change the option in total to at least once

Harshith.R.S

03/03/2023, 12:25 PM

alert fired even after adding atleast once, trying cli rn

Harshith.R.S

03/03/2023, 12:28 PM

Copy code

Error: No such container: clickhouse-setup_clickhouse_1

getting this edit: ok so container name given in the docs is wrong now I'm able to access cli edit2: it needs values in place of {{.start_datetime}} and {{.end_datetime}}

Amol Umbark

03/03/2023, 1:40 PM

the query must have a value alias

Harshith.R.S

03/03/2023, 1:41 PM

It does

Amol Umbark

03/03/2023, 1:41 PM

also the interval must be alias as ‘ts’

Harshith.R.S

03/03/2023, 1:42 PM

I'm getting the graph without that know?

Amol Umbark

03/03/2023, 1:43 PM

yeah interval is fine alias

Amol Umbark

03/03/2023, 1:44 PM

did you get results in clickhouse cli

Harshith.R.S

03/03/2023, 1:47 PM

Sorry I logged off rn I'll be able to send you the results tomorrow

4 Views

Open in Slack

Previous Next

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.