This message was deleted.
# supports
Slackbot
02/06/2023, 9:46 AMThis message was deleted.
s
Srikanth Chekuri
02/06/2023, 10:59 AMSigNoz doesn’t support the delta temporality, yet
it makes it impossible to create an query that can alarm if there are x amount of errors in time interval y.You can use rate to achieve this.
Srikanth Chekuri
02/06/2023, 12:59 PMTo be more explicit. The Rate of change will give you the value for 30s seconds interval. You will get the absolute value when you multiply this result by 30. IIUC, you wanted this absolute value and set alert. There is a formula tab in the builder where you can write the expression
A*30Aj
James Henrich
02/06/2023, 7:26 PMThanks! Makes sense; played with rate before but just assumed it should be the raw difference and was confused at the number it gave because the interval length isnt specified
Can we assume cumulative aggregation would be a robust measurement for the following scenarios?
• multiple services pushing the same metrics with frequent restarts (rate will be negative if a service restarts?). How do we take the rate per service and sum them (SUM_RATE)?
• export interval is greater than alarm interval (is rate correctly zero on no-exports?)
Thanks again for your help 🙂
s
Srikanth Chekuri
02/07/2023, 2:17 AMThe counter resets are not handled correctly, so that can be a problem. Yes, if there is no change in the counter for an interval the rate will be zero
Srikanth Chekuri
02/07/2023, 2:32 AMHowever, the promql eval takes the resets into account. If you could share the error counter metric you created I would share the expression by service for this. It would roughly look like following
sum by (service_name) (rate(your_metric_name_here{your_filters}[5m]))j
James Henrich
02/07/2023, 4:59 AMyour example should be helpful enough to get there if i need it, It will only affect dashboard graphs a litte since I only alert above a threshold
just confirming that besides this SUM_RATE in query builder should work out of the box correctly with identical horizontally scaled services? aka each service counter is tracked separately as a time series and the sum of differences of those will be calculated. or since they are identical they are pooled in same time series?
s
Srikanth Chekuri
02/07/2023, 5:00 AMaka each service counter is tracked separately as a time series and the sum of differences of those will be calculatedThis is correct.
Srikanth Chekuri
02/07/2023, 5:01 AMsince they are identical they are pooled in same time series?How are they identical? There should be some identifier that differentiates each scaled instance and that should come from the instrumentation.
Srikanth Chekuri
02/07/2023, 5:02 AMUnfortunately SDK doesn’t do this today, since you mentioned Python SDK here is the issue tracker issue https://github.com/open-telemetry/opentelemetry-python/issues/2113
Srikanth Chekuri
02/07/2023, 5:03 AMYou should work around this by creating your own unique ID until sdk supports it.
j
James Henrich
02/07/2023, 5:03 AMcool, so I must manually set this then from ptyhon sdk?
s
Srikanth Chekuri
02/07/2023, 5:04 AMYes
👍 1
8 Views