https://signoz.io
Join Slack
This message was deleted.
# support
s
This message was deleted.
s
SigNoz doesn’t support the delta temporality, yet
it makes it impossible to create an query that can alarm if there are x amount of errors in time interval y.
You can use rate to achieve this.
To be more explicit. The Rate of change will give you the value for 30s seconds interval. You will get the absolute value when you multiply this result by 30. IIUC, you wanted this absolute value and set alert. There is a formula tab in the builder where you can write the expression 
A*30
where the 
A
is the rate query.
j
Thanks! Makes sense; played with rate before but just assumed it should be the raw difference and was confused at the number it gave because the interval length isnt specified Can we assume cumulative aggregation would be a robust measurement for the following scenarios? • multiple services pushing the same metrics with frequent restarts (rate will be negative if a service restarts?). How do we take the rate per service and sum them (SUM_RATE)? • export interval is greater than alarm interval (is rate correctly zero on no-exports?) Thanks again for your help 🙂
s
The counter resets are not handled correctly, so that can be a problem. Yes, if there is no change in the counter for an interval the rate will be zero
However, the promql eval takes the resets into account. If you could share the error counter metric you created I would share the expression by service for this. It would roughly look like following 
sum by (service_name) (rate(your_metric_name_here{your_filters}[5m]))
j
your example should be helpful enough to get there if i need it, It will only affect dashboard graphs a litte since I only alert above a threshold just confirming that besides this SUM_RATE in query builder should work out of the box correctly with identical horizontally scaled services? aka each service counter is tracked separately as a time series and the sum of differences of those will be calculated. or since they are identical they are pooled in same time series?
s
aka each service counter is tracked separately as a time series and the sum of differences of those will be calculated
This is correct.
since they are identical they are pooled in same time series?
How are they identical? There should be some identifier that differentiates each scaled instance and that should come from the instrumentation.
Unfortunately SDK doesn’t do this today, since you mentioned Python SDK here is the issue tracker issue https://github.com/open-telemetry/opentelemetry-python/issues/2113
You should work around this by creating your own unique ID until sdk supports it.
j
cool, so I must manually set this then from ptyhon sdk?
s
Yes
👍 1
7 Views
Open in Slack
PreviousNext
Powered by

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.

Powered by