https://signoz.io logo
Join Slack
Powered by
This message was deleted.
# support
s
This message was deleted.
s
Where should I debug?
n
@Srikanth Chekuri
s
@Srikanth Chekuri Is this issue related to opentelemerty configuration ?
s
Yes, all these come from the otel-collector-metrics, is that pod running?
s
yes
where we deploy signoz?
Below is the list of pods
s
Can you share the logs of the query-service by loading the screen again?
s
ok
logs generated for database calls
Copy code
2023-02-28T06:00:59.271Z	DEBUG	clickhouseReader/reader.go:1867
		SELECT
			quantile(0.5)(durationNano) as p50,
			quantile(0.95)(durationNano) as p95,
			quantile(0.99)(durationNano) as p99,
			COUNT(*) as numCalls,
			name
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND timestamp>= @start AND timestamp<= @end GROUP BY name ORDER BY p99 DESC LIMIT 10
2023-02-28T06:00:59.271Z	INFO	app/server.go:236	/api/v1/service/top_operations	timeTaken: 14.483965ms
2023-02-28T06:00:59.276Z	DEBUG	clickhouseReader/reader.go:869
		SELECT
			toStartOfInterval(timestamp, INTERVAL @interval minute) as time,
			quantile(0.99)(durationNano) as p99,
			quantile(0.95)(durationNano) as p95,
			quantile(0.50)(durationNano) as p50,
			count(*) as numCalls
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND name In [@names] AND timestamp>= @start AND timestamp<= @end GROUP BY time ORDER BY time DESC
2023-02-28T06:00:59.287Z	DEBUG	clickhouseReader/reader.go:897
		SELECT
			toStartOfInterval(timestamp, INTERVAL @interval minute) as time,
			count(*) as numErrors
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND name In [@names] AND timestamp>= @start AND timestamp<= @end AND statusCode=2 GROUP BY time ORDER BY time DESC
2023-02-28T06:00:59.287Z	INFO	app/server.go:236	/api/v1/service/overview	timeTaken: 31.303781ms
2023-02-28T06:00:59.340Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:00:59.340Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:00:59.340Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.ts as ts, A.value / B.value as value FROM (SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_db_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_db_latency_sum' AND timestamp_ms >= 1677562259000 AND timestamp_ms <= 1677564000000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as A  GLOBAL INNER JOIN(SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_db_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_db_latency_count' AND timestamp_ms >= 1677562259000 AND timestamp_ms <= 1677564000000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as B  ON A.ts = B.ts
2023-02-28T06:00:59.340Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT db_system,  ts, sum(value) as value FROM (SELECT db_system,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, db_system,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'db_system') as db_system, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_db_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_db_latency_count' AND timestamp_ms >= 1677562259000 AND timestamp_ms <= 1677564000000 GROUP BY fingerprint, db_system,ts ORDER BY fingerprint, db_system,  ts) OFFSET 1) GROUP BY db_system,ts ORDER BY db_system,  ts
2023-02-28T06:00:59.401Z	INFO	utils/time.go:12	GetMetricResult took 60.660535ms

2023-02-28T06:00:59.401Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 61.073809ms
2023-02-28T06:00:59.424Z	INFO	app/server.go:236	/api/v1/configs	timeTaken: 197.318715ms
2023-02-28T06:00:59.459Z	INFO	utils/time.go:12	GetMetricResult took 118.686484ms

2023-02-28T06:00:59.459Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 119.0515ms
2023-02-28T06:01:00.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 13.678µs
2023-02-28T06:01:00.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 9.285µs
2023-02-28T06:01:10.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 11.66µs
2023-02-28T06:01:10.982Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 5.334µs
logs for external calls
Copy code
2023-02-28T06:02:40.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 11.692µs
2023-02-28T06:02:40.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 9.867µs
2023-02-28T06:02:47.164Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:47.164Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:47.164Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.address as address, A.ts as ts, A.value / B.value as value FROM (SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as A  GLOBAL INNER JOIN(SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as B  ON A.address = B.address AND A.ts = B.ts
2023-02-28T06:02:47.164Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.address as address, A.ts as ts, A.value * 100 / B.value as value FROM (SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan'] AND JSONExtractString(labels, 'status_code') IN ['STATUS_CODE_ERROR']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as A  GLOBAL INNER JOIN(SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as B  ON A.address = B.address AND A.ts = B.ts
2023-02-28T06:02:47.165Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:47.165Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts
2023-02-28T06:02:47.166Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:47.166Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.ts as ts, A.value / B.value as value FROM (SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as A  GLOBAL INNER JOIN(SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as B  ON A.ts = B.ts
2023-02-28T06:02:47.295Z	INFO	utils/time.go:12	GetMetricResult took 129.677912ms

2023-02-28T06:02:47.295Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 129.992533ms
2023-02-28T06:02:47.381Z	INFO	utils/time.go:12	GetMetricResult took 216.511378ms

2023-02-28T06:02:47.381Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 217.041976ms
2023-02-28T06:02:47.387Z	INFO	utils/time.go:12	GetMetricResult took 222.312969ms

2023-02-28T06:02:47.387Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 222.715429ms
2023-02-28T06:02:47.390Z	INFO	utils/time.go:12	GetMetricResult took 224.805653ms

2023-02-28T06:02:47.391Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 225.099565ms
2023-02-28T06:02:49.017Z	DEBUG	auth/auth.go:376	Login method called for user:

2023-02-28T06:02:49.017Z	INFO	app/server.go:236	/api/v1/login	timeTaken: 325.579µs
 value:
2023-02-28T06:02:49.058Z	INFO	app/server.go:236	/api/v1/user/{id}	timeTaken: 387.94µs
2023-02-28T06:02:49.179Z	INFO	app/server.go:236	/api/v1/featureFlags	timeTaken: 200.19µs
2023-02-28T06:02:49.179Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 195.92µs
2023-02-28T06:02:49.205Z	INFO	app/server.go:236	/api/v1/service/top_level_operations	timeTaken: 2.594889ms
2023-02-28T06:02:49.211Z	DEBUG	clickhouseReader/reader.go:1867
Copy code
SELECT
			quantile(0.5)(durationNano) as p50,
			quantile(0.95)(durationNano) as p95,
			quantile(0.99)(durationNano) as p99,
			COUNT(*) as numCalls,
			name
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND timestamp>= @start AND timestamp<= @end GROUP BY name ORDER BY p99 DESC LIMIT 10
2023-02-28T06:02:49.212Z	INFO	app/server.go:236	/api/v1/service/top_operations	timeTaken: 12.679458ms
2023-02-28T06:02:49.217Z	DEBUG	clickhouseReader/reader.go:869
		SELECT
			toStartOfInterval(timestamp, INTERVAL @interval minute) as time,
			quantile(0.99)(durationNano) as p99,
			quantile(0.95)(durationNano) as p95,
			quantile(0.50)(durationNano) as p50,
			count(*) as numCalls
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND name In [@names] AND timestamp>= @start AND timestamp<= @end GROUP BY time ORDER BY time DESC
2023-02-28T06:02:49.226Z	DEBUG	clickhouseReader/reader.go:897
		SELECT
			toStartOfInterval(timestamp, INTERVAL @interval minute) as time,
			count(*) as numErrors
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND name In [@names] AND timestamp>= @start AND timestamp<= @end AND statusCode=2 GROUP BY time ORDER BY time DESC
2023-02-28T06:02:49.226Z	INFO	app/server.go:236	/api/v1/service/overview	timeTaken: 28.123133ms
2023-02-28T06:02:49.271Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:49.271Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.ts as ts, A.value / B.value as value FROM (SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as A  GLOBAL INNER JOIN(SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as B  ON A.ts = B.ts
2023-02-28T06:02:49.272Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:49.272Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.address as address, A.ts as ts, A.value * 100 / B.value as value FROM (SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan'] AND JSONExtractString(labels, 'status_code') IN ['STATUS_CODE_ERROR']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as A  GLOBAL INNER JOIN(SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as B  ON A.address = B.address AND A.ts = B.ts
2023-02-28T06:02:49.273Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:49.273Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.address as address, A.ts as ts, A.value / B.value as value FROM (SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as A  GLOBAL INNER JOIN(SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as B  ON A.address = B.address AND A.ts = B.ts
2023-02-28T06:02:49.274Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:49.274Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts
2023-02-28T06:02:49.368Z	INFO	utils/time.go:12	GetMetricResult took 94.450179ms
Copy code
2023-02-28T06:02:49.369Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 94.759835ms
2023-02-28T06:02:49.477Z	INFO	utils/time.go:12	GetMetricResult took 205.540941ms

2023-02-28T06:02:49.477Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 205.949289ms
2023-02-28T06:02:49.489Z	INFO	utils/time.go:12	GetMetricResult took 216.242606ms

2023-02-28T06:02:49.489Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 216.649147ms
2023-02-28T06:02:49.490Z	INFO	utils/time.go:12	GetMetricResult took 218.363514ms

2023-02-28T06:02:49.490Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 218.789615ms
2023-02-28T06:02:49.773Z	INFO	app/server.go:236	/api/v1/configs	timeTaken: 591.196096ms
2023-02-28T06:02:50.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 13.714µs
2023-02-28T06:02:50.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 7.056µs
@Srikanth Chekuri Will it works or shall I share in file?
s
This looks good to me. Can you exec into clickhouse pod and share the output of this query?
Copy code
SELECT A.ts as ts, A.value / B.value as value FROM (SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as A  GLOBAL INNER JOIN(SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as B  ON A.ts = B.ts
s
Sure
@Srikanth Chekuri
Copy code
chi-signoz-release-clickhouse-cluster-0-0-0.chi-signoz-release-clickhouse-cluster-0-0.platform.svc.cluster.local :) SELECT A.ts as ts, A.value / B.value as value FROM (SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as A  GLOBAL INNER JOIN(SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as B  ON A.ts = B.ts

SELECT
    A.ts AS ts,
    A.value / B.value AS value
FROM
(
    SELECT
        ts,
        sum(value) AS value
    FROM
    (
        SELECT
            ts,
            runningDifference(value) / runningDifference(ts) AS value
        FROM
        (
            SELECT
                fingerprint,
                toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), toIntervalSecond(60)) AS ts,
                max(value) AS value
            FROM signoz_metrics.distributed_samples_v2
            GLOBAL INNER JOIN
            (
                SELECT fingerprint
                FROM signoz_metrics.distributed_time_series_v2
                WHERE (metric_name = 'signoz_external_call_latency_sum') AND (JSONExtractString(labels, 'service_name') IN ['oa-lending-loan'])
            ) AS filtered_time_series USING (fingerprint)
            WHERE (metric_name = 'signoz_external_call_latency_sum') AND (timestamp_ms >= 1677562369000) AND (timestamp_ms <= 1677564120000)
            GROUP BY
                fingerprint,
                ts
            ORDER BY
                fingerprint ASC,
                ts ASC
        )
        OFFSET 1
    )
    GROUP BY ts
    ORDER BY ts ASC
) AS A
GLOBAL INNER JOIN
(
    SELECT
        ts,
        sum(value) AS value
    FROM
    (
        SELECT
            ts,
            runningDifference(value) / runningDifference(ts) AS value
        FROM
        (
            SELECT
                fingerprint,
                toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), toIntervalSecond(60)) AS ts,
                max(value) AS value
            FROM signoz_metrics.distributed_samples_v2
            GLOBAL INNER JOIN
            (
                SELECT fingerprint
                FROM signoz_metrics.distributed_time_series_v2
                WHERE (metric_name = 'signoz_external_call_latency_count') AND (JSONExtractString(labels, 'service_name') IN ['oa-lending-loan'])
            ) AS filtered_time_series USING (fingerprint)
            WHERE (metric_name = 'signoz_external_call_latency_count') AND (timestamp_ms >= 1677562369000) AND (timestamp_ms <= 1677564120000)
            GROUP BY
                fingerprint,
                ts
            ORDER BY
                fingerprint ASC,
                ts ASC
        )
        OFFSET 1
    )
    GROUP BY ts
    ORDER BY ts ASC
) AS B ON A.ts = B.ts

Query id: 117b56be-2cc1-43cf-a5be-19fd8d3b4074

Ok.

0 rows in set. Elapsed: 0.098 sec. Processed 518.37 thousand rows, 120.84 MB (5.28 million rows/s., 1.23 GB/s.)
Above is query output
No data return
@Srikanth Chekuri Any idea why are we not getting database calls and external calls?
s
It’s not clear to me what’s happening. Shall we sync up at 2 pm?
s
yes
We can
s
Sure, I will share the meeting link.
s
👍
@Srikanth Chekuri Plz share meeting link
s
Sharing in a sec
s
joining
@Srikanth Chekuri Issue resolved
Thank you for the quick call and debugging the issue
v
@sudhanshu dev how did you resolved your issue?
s
It was related to the values.yaml. We created custom values.yaml file which overriding the otel collector metrics configuration
So we added the default configuration into custom values.yaml file
4 Views
Open in Slack
PreviousNext
Powered by