https://signoz.io logo
Title
a

Apoorva

11/13/2022, 6:46 AM
Hi, Is there a way to write alerts/notifications on 1. p99 latencies 2. rate of errors thrown 3. rate of logs being thrown based on specific keys, like container name, log level etc Apart from promql, Raw clickhouse queries from your own codebase will also work for me
p

Pranay

11/14/2022, 2:36 AM
@Ankit Nayan do you have more insights on this
a

Apoorva

11/14/2022, 5:29 PM
Reminder
a

Ankit Nayan

11/14/2022, 5:54 PM
@Apoorva 1. Application latency. You can add more filters or remove operation filter. Below is just an example promql
histogram_quantile(0.99, sum(rate(signoz_latency_bucket{service_name="frontend", operation="HTTP GET /dispatch"}[5m])) by (le))
2. Error percentage of application. Below example for
redis
service in hotrod example
max(sum(rate(signoz_calls_total{service_name="redis", status_code="STATUS_CODE_ERROR"}[5m]) OR rate(signoz_calls_total{service_name="redis", http_status_code=~"5.."}[5m]))*100/sum(rate(signoz_calls_total{service_name="redis"}[5m]))) < 1000 OR vector(0)
3. Yet to come. Development for enabling clickhouse query based alerts in progress https://github.com/SigNoz/signoz/issues/1699