I am ingesting data in signoz on an hourly basis. I've setup an alert based on the data. When the condition is satisfied, the alert should get fired immediately . But the alert that i am getting is on an average 25mins late from the time of the ingestion. Why is that

As you can see, i have set up an alert if the count is more than 30. So ideally whenever the count goes above 30, it should fire an alert instantly. But when the value is ingested and it is above 30, I am getting an alert around 25-30 mins late. So if the ingestion happens at 12:30pm, I get an alert for that at 12:51pm (for eg) I have attached screenshot of the alert setup as well as a delayed alert. Please check and let me know if it is an configuration issue or something else

Please zoom in and show the data for +- 2h near the alert time

image.png

Is your data ingested in real time?

We're using putty to ingest the data so there is a timzone diff of 5:30 hours

Is there a delay in sending the data to SigNoz, one case could be that data is ingested at 9:50 for the period of 9:30 afterwards.

No but I can see the data at 9:30 as soon as its ingested

I am not sure what might be the reason but there shouldn't be this much delay with real-time data ingestion. This is most likely some data related issue and not a logical bug in code or configuration related. Next time when it happens, please share the full logs of query-service.