I am running into an issue with metrics. I am running a rate over a 30 minute window but I am only getting maybe 5 measurements returned of varying intervals (i.e. between two points is 2 minutes and the next two points is 10 minutes). I have a dashboard using the previous sum_rate functions that are returning a full 30 minutes worth of data at 1 minute intervals.

@Srikanth Chekuri can you look into this?

That shouldn't be the case. The queries prepared should be the same. Please share the logs from the query service when you run the v3 and v4.

v3 Query:

SELECT service_name,
       ts,
       If((value - lagInFrame(value, 1, 0) OVER rate_window) < 0, nan,
          If((ts - lagInFrame(ts, 1, toDate('1970-01-01')) OVER rate_window) >= 86400, nan,
             (value - lagInFrame(value, 1, 0) OVER rate_window) /
             (ts - lagInFrame(ts, 1, toDate('1970-01-01')) OVER rate_window))) as value
FROM (SELECT service_name,
             toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts,
             sum(value)                                                                    as value
      FROM signoz_metrics.distributed_samples_v2
               INNER JOIN (SELECT JSONExtractString(labels, 'service_name') as service_name, fingerprint
                           FROM signoz_metrics.time_series_v2
                           WHERE metric_name = 'seis_calls'
                             AND temporality IN ['Cumulative', 'Unspecified']
                             AND has(JSONExtractKeys(labels), 'http_method')
                             AND JSONExtractString(labels, 'span_kind') IN ['SPAN_KIND_SERVER','SPAN_KIND_INTERNAL']
                             AND JSONExtractString(labels, 'deployment_environment') = 'production'
                             AND JSONExtractString(labels, 'service_name') IN
                                 ['seis-gateway_seis-gateway','gateway']) as filtered_time_series USING fingerprint
      WHERE metric_name = 'seis_calls'
        AND timestamp_ms >= 1712637000000
        AND timestamp_ms < 1712637960000
      GROUP BY GROUPING SETS ( (service_name, ts), (service_name) )
      ORDER BY service_name ASC, ts)
WINDOW rate_window as (PARTITION BY service_name ORDER BY service_name, ts)

v4 query

SELECT service_name,
       toStartOfInterval(toDateTime(intDiv(unix_milli, 1000)), INTERVAL 60 SECOND) as ts,
       sum(value) / 60                                                             as value
FROM signoz_metrics.distributed_samples_v4
         INNER JOIN (SELECT DISTINCT JSONExtractString(labels, 'service_name') as service_name, fingerprint
                     FROM signoz_metrics.time_series_v4
                     WHERE metric_name = 'seis_calls'
                       AND temporality = 'Delta'
                       AND unix_milli >= 1712631600000
                       AND unix_milli < 1712635800000
                       AND JSONExtractString(labels, 'service_name') IN ['seis-gateway_seis-gateway','gateway']
                       AND has(JSONExtractKeys(labels), 'http_method')
                       AND JSONExtractString(labels, 'deployment_environment') = 'production'
                       AND JSONExtractString(labels, 'span_kind') = 'SPAN_KIND_SERVER') as filtered_time_series
                    USING fingerprint
WHERE metric_name = 'seis_calls'
  AND unix_milli >= 1712633940000
  AND unix_milli < 1712635800000
GROUP BY GROUPING SETS ( (service_name, ts), (service_name) )

You are sending delta temporality data for

seis_calls

. Please make sure you continuously send data with the same temporality everywhere.

Can I use cumulative? In v3 it is using cumulative.

Either send all cumulative or all delta for a single metric; do not send a mix of both. If a delta temporality exists for a metric, the query service will default to it because it's faster and cleaner.

So I changed it to stop sending both...only delta...but the v3 query is still using the cumulative temporality. Should that be expected?

Yes, the v3 will eventually be deprecated. meanwhile you can pass the flag

--prefer-delta=true

for query service to use delta for v3 also.