James Henrich
04/11/2023, 3:35 AMVishal Sharma
04/11/2023, 5:11 AMJames Henrich
04/11/2023, 11:15 AMVishal Sharma
04/11/2023, 11:31 AMJames Henrich
04/11/2023, 11:33 AM{
"start": 1681421689000,
"end": 1681422589000,
"step": 60,
"variables": {
"SIGNOZ_START_TIME": 1681421689000,
"SIGNOZ_END_TIME": 1681422589000
},
"dataSource": 1,
"compositeMetricQuery": {
"queryType": 1,
"panelType": 1,
"builderQueries": {
"A": {
"queryName": "A",
"aggregateOperator": 18,
"metricName": "scalper_service_down",
"tagFilters": {
"items": [],
"op": "AND"
},
"groupBy": [
"profile"
],
"expression": "A",
"disabled": false
}
}
}
}
resp:
{
"status": "success",
"data": {
"resultType": "matrix",
"result": null
}
}
query service logs from request:
2023-04-13T20:52:16.269Z INFO app/server.go:236 /api/v1/version timeTaken: 155.978µs
2023-04-13T20:52:16.271Z INFO app/server.go:236 /api/v1/featureFlags timeTaken: 269.238µs
2023-04-13T20:52:16.323Z INFO app/server.go:236 /api/v1/configs timeTaken: 45.034907ms
2023-04-13T20:52:16.449Z INFO app/server.go:236 /api/v1/dashboards/{uuid} timeTaken: 1.375588ms
2023-04-13T20:52:16.640Z INFO api/metrics.go:21 CustomMetricsFunction feature is not enabled in this plan
2023-04-13T20:52:16.640Z INFO clickhouseReader/reader.go:3071 Executing metric result query: SELECT profile, ts, sum(value) as value FROM (SELECT profile, ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, profile, toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT JSONExtractString(labels, 'profile') as profile, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'scalper_service_down') as filtered_time_series USING fingerprint WHERE metric_name = 'scalper_service_down' AND timestamp_ms >= 1681421689000 AND timestamp_ms <= 1681422540000 GROUP BY fingerprint, profile,ts ORDER BY fingerprint, profile, ts) OFFSET 1) GROUP BY profile,ts ORDER BY profile, ts
so im noticing the following: "CustomMetricsFunction feature is not enabled in this plan"
Looking at the plans, nothing about this is mentioned that posting custom metrics premium feature. Did something recently change?Vishal Sharma
04/14/2023, 4:19 AMSrikanth Chekuri
04/14/2023, 4:41 AMJames Henrich
04/14/2023, 7:38 AMSrikanth Chekuri
04/14/2023, 7:40 AMAlerts are working properly and metrics are being stored, its just that the data on the dashboard has been broken for about a week now.Are these alerts on the same data or different metrics?
James Henrich
04/14/2023, 7:40 AMSrikanth Chekuri
04/14/2023, 7:44 AMJames Henrich
04/14/2023, 7:49 AMSrikanth Chekuri
04/14/2023, 7:54 AMand metrics are being storedYou mentioned metrics are being stored. So by that statement I assumed you have confirmed that data exists in DB.
My frontend dashboard stopped displaying all metric graphs / trace graphs after a certain point in timeYou mentioned this in your original post. I suspect your applications have stopped sending data or there is some issue from SDK when sending data.
I am not sure how alerts can be triggered properly on all the metrics I am posting without being stored properly (edited)Alerts use the exact same queries and database. It would be very surprising for me if the same kind of query works in an alert but not in the dashboard. Alerts don’t break if the query returns no data. Can you share some of these alerts where dashboard doesn’t work but alert works.
James Henrich
04/14/2023, 8:06 AM2023-04-14T08:00:10.196Z DEBUG rules/thresholdRule.go:625 ruleid:%!(EXTRA string=14, string= runQueries:, map[string]string=map[A:SELECT profile, ts, sum(value) as value FROM (SELECT profile, ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, profile, toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 30 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT JSONExtractString(labels, 'profile') as profile, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'scalper_service_down' AND JSONExtractString(labels, 'profile') NOT IN ['gerald']) as filtered_time_series USING fingerprint WHERE metric_name = 'scalper_service_down' AND timestamp_ms >= 1681458310194 AND timestamp_ms <= 1681459210194 GROUP BY fingerprint, profile,ts ORDER BY fingerprint, profile, ts) OFFSET 1) GROUP BY profile,ts ORDER BY profile, ts])
2023-04-14T08:00:10.196Z DEBUG rules/thresholdRule.go:643 ruleId: %!(EXTRA string=14, string= result query label:, string=A)
2023-04-14T08:00:10.229Z DEBUG rules/thresholdRule.go:525 ruleid:%!(EXTRA string=14, string= resultmap(potential alerts):, int=5)
2023-04-14T08:00:10.229Z DEBUG rules/thresholdRule.go:326 target:%!(EXTRA float64=0, float64=0.001)
2023-04-14T08:00:10.229Z DEBUG rules/thresholdRule.go:326 target:%!(EXTRA float64=0, float64=0.001)
2023-04-14T08:00:10.229Z DEBUG rules/thresholdRule.go:326 target:%!(EXTRA float64=0, float64=0.001)
2023-04-14T08:00:10.229Z DEBUG rules/thresholdRule.go:326 target:%!(EXTRA float64=1, float64=0.001)
2023-04-14T08:00:10.229Z DEBUG rules/thresholdRule.go:326 target:%!(EXTRA float64=0, float64=0.001)
2023-04-14T08:00:10.229Z DEBUG rules/thresholdRule.go:534 ruleid:%!(EXTRA string=14, string= result (found alerts):, int=1)
2023-04-14T08:00:10.229Z INFO rules/thresholdRule.go:740 rule:Gerald Scalper Service Down alerts found: 1
2023-04-14T08:00:10.229Z INFO rules/thresholdRule.go:295 msg:sending alerts rule:Gerald Scalper Service Down
And correct, I did not verify directly as I'm not too comfortable with clickhouse. It was just a strong assumption based on alert behaviour.Srikanth Chekuri
04/14/2023, 8:09 AMan example of logs from query-service for an alert currently being fired (“Gerald Scalper Service Down” -> metrc: scalper_service_down) with no metric data in dashboard:You are not seeing data because the service is down and not sending any data?
James Henrich
04/14/2023, 8:11 AMSrikanth Chekuri
04/14/2023, 8:13 AMI am very confident metrics are being sent properly as these alerts are correct and in this example the scalper service is in fact downI was asking for a microservice which has alerts setup and working but the dashboard not working? Is there a such service you can share? How are you confirming the metrics are being sent from these microserivices?
James Henrich
04/14/2023, 8:20 AMSrikanth Chekuri
04/14/2023, 8:25 AMTraces
tab for the service?James Henrich
04/14/2023, 8:28 AMSrikanth Chekuri
04/14/2023, 8:30 AMNo, they’ve existed before. Nothing under services tab either.This is high likely because there is some issue sending data to SigNoz as you mentioned collector is healthy and has no error logs. Can you select longer time range and check if it shows the services? Can you exec into ClickHouse and run some queries that I can share to confirm if there is any data received in the last X minutes/hours/days?
James Henrich
04/14/2023, 8:34 AMSrikanth Chekuri
04/14/2023, 8:37 AMJames Henrich
04/14/2023, 8:42 AM