Hui Cong Cai, Kobe
11/07/2024, 11:25 AMSELECT toStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 86400 SECOND) AS ts, toFloat64(count(distinct(attributes_string_value[indexOf(attributes_string_key, 'zzz')]))) as value from signoz_logs.distributed_logs where (timestamp >= 1729738800000000000 AND timestamp <= 1729825200000000000) AND `attribute_string_event` = 'yyy' AND `resource_string_service$$name` = 'xxx' AND not has(attributes_string_key, 'error') AND has(attributes_string_key, 'zzz') group by ts order by value DESC
result
{
"labels": {},
"labelsArray": null,
"values": [
{
"timestamp": 1729728000000,
"value": "549"
},
{
"timestamp": 1729814400000,
"value": "63"
}
]
}
The toStartOfInterval function will split data into two buckets by UTC 0h. It make sense when viewing in dashboard. But the alert rules are executed at random time. This make cause some noise.
Here's an example.
I want to compare if week1 has a more than 30% increase over week0. The 24 hours count in week1 is 490 + 10 = 500. And the 24 hours count in week0 is 500 + 5 = 505. The increase percentage should be (500-505)/505 * 100% .
But actually , I got two group of increase percentage.
1. (490-500)/500 * 100%
2. (10-5)/5 * 100%
The second value will trigger a alert. Because 100% is bigger than the threshold 30%.
Are there a better option to calculate week-over-week growth rate to trigger a alert? It's logs, not metrics.
Thanks for your help!Srikanth Chekuri
11/08/2024, 4:05 PMHui Cong Cai, Kobe
11/13/2024, 7:23 AM