< Prashant Shahi> < nitya signoz> < Ankit Nayan> In services SigNoz Community #support

<@U02SS3ZAMKQ> <@U03BTNACHNX> <@U01HWQ1RTC2> In se...

sudhanshu dev

02/28/2023, 5:32 AM

@Prashant Shahi @nitya-signoz @Ankit Nayan In services, we are not getting all the metrics data like no data base calls, no external calls metrics etc. Below are the screen shots Could u plz help me to debug the issue?

sudhanshu dev

02/28/2023, 5:32 AM

Where should I debug?

nitya-signoz

02/28/2023, 5:36 AM

@Srikanth Chekuri

sudhanshu dev

02/28/2023, 5:38 AM

@Srikanth Chekuri Is this issue related to opentelemerty configuration ?

Srikanth Chekuri

02/28/2023, 5:46 AM

Yes, all these come from the otel-collector-metrics, is that pod running?

sudhanshu dev

02/28/2023, 5:46 AM

yes

sudhanshu dev

02/28/2023, 5:47 AM

where we deploy signoz?

sudhanshu dev

02/28/2023, 5:48 AM

Below is the list of pods

Srikanth Chekuri

02/28/2023, 5:49 AM

Can you share the logs of the query-service by loading the screen again?

sudhanshu dev

02/28/2023, 5:56 AM

sudhanshu dev

02/28/2023, 6:02 AM

logs generated for database calls

Copy code

2023-02-28T06:00:59.271Z	DEBUG	clickhouseReader/reader.go:1867
		SELECT
			quantile(0.5)(durationNano) as p50,
			quantile(0.95)(durationNano) as p95,
			quantile(0.99)(durationNano) as p99,
			COUNT(*) as numCalls,
			name
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND timestamp>= @start AND timestamp<= @end GROUP BY name ORDER BY p99 DESC LIMIT 10
2023-02-28T06:00:59.271Z	INFO	app/server.go:236	/api/v1/service/top_operations	timeTaken: 14.483965ms
2023-02-28T06:00:59.276Z	DEBUG	clickhouseReader/reader.go:869
		SELECT
			toStartOfInterval(timestamp, INTERVAL @interval minute) as time,
			quantile(0.99)(durationNano) as p99,
			quantile(0.95)(durationNano) as p95,
			quantile(0.50)(durationNano) as p50,
			count(*) as numCalls
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND name In [@names] AND timestamp>= @start AND timestamp<= @end GROUP BY time ORDER BY time DESC
2023-02-28T06:00:59.287Z	DEBUG	clickhouseReader/reader.go:897
		SELECT
			toStartOfInterval(timestamp, INTERVAL @interval minute) as time,
			count(*) as numErrors
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND name In [@names] AND timestamp>= @start AND timestamp<= @end AND statusCode=2 GROUP BY time ORDER BY time DESC
2023-02-28T06:00:59.287Z	INFO	app/server.go:236	/api/v1/service/overview	timeTaken: 31.303781ms
2023-02-28T06:00:59.340Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:00:59.340Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:00:59.340Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.ts as ts, A.value / B.value as value FROM (SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_db_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_db_latency_sum' AND timestamp_ms >= 1677562259000 AND timestamp_ms <= 1677564000000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as A  GLOBAL INNER JOIN(SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_db_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_db_latency_count' AND timestamp_ms >= 1677562259000 AND timestamp_ms <= 1677564000000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as B  ON A.ts = B.ts
2023-02-28T06:00:59.340Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT db_system,  ts, sum(value) as value FROM (SELECT db_system,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, db_system,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'db_system') as db_system, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_db_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_db_latency_count' AND timestamp_ms >= 1677562259000 AND timestamp_ms <= 1677564000000 GROUP BY fingerprint, db_system,ts ORDER BY fingerprint, db_system,  ts) OFFSET 1) GROUP BY db_system,ts ORDER BY db_system,  ts
2023-02-28T06:00:59.401Z	INFO	utils/time.go:12	GetMetricResult took 60.660535ms

2023-02-28T06:00:59.401Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 61.073809ms
2023-02-28T06:00:59.424Z	INFO	app/server.go:236	/api/v1/configs	timeTaken: 197.318715ms
2023-02-28T06:00:59.459Z	INFO	utils/time.go:12	GetMetricResult took 118.686484ms

2023-02-28T06:00:59.459Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 119.0515ms
2023-02-28T06:01:00.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 13.678µs
2023-02-28T06:01:00.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 9.285µs
2023-02-28T06:01:10.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 11.66µs
2023-02-28T06:01:10.982Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 5.334µs

sudhanshu dev

02/28/2023, 6:04 AM

logs for external calls

Copy code

2023-02-28T06:02:40.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 11.692µs
2023-02-28T06:02:40.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 9.867µs
2023-02-28T06:02:47.164Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:47.164Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:47.164Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.address as address, A.ts as ts, A.value / B.value as value FROM (SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as A  GLOBAL INNER JOIN(SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as B  ON A.address = B.address AND A.ts = B.ts
2023-02-28T06:02:47.164Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.address as address, A.ts as ts, A.value * 100 / B.value as value FROM (SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan'] AND JSONExtractString(labels, 'status_code') IN ['STATUS_CODE_ERROR']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as A  GLOBAL INNER JOIN(SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as B  ON A.address = B.address AND A.ts = B.ts
2023-02-28T06:02:47.165Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:47.165Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts
2023-02-28T06:02:47.166Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:47.166Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.ts as ts, A.value / B.value as value FROM (SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as A  GLOBAL INNER JOIN(SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562367000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as B  ON A.ts = B.ts
2023-02-28T06:02:47.295Z	INFO	utils/time.go:12	GetMetricResult took 129.677912ms

2023-02-28T06:02:47.295Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 129.992533ms
2023-02-28T06:02:47.381Z	INFO	utils/time.go:12	GetMetricResult took 216.511378ms

2023-02-28T06:02:47.381Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 217.041976ms
2023-02-28T06:02:47.387Z	INFO	utils/time.go:12	GetMetricResult took 222.312969ms

2023-02-28T06:02:47.387Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 222.715429ms
2023-02-28T06:02:47.390Z	INFO	utils/time.go:12	GetMetricResult took 224.805653ms

2023-02-28T06:02:47.391Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 225.099565ms
2023-02-28T06:02:49.017Z	DEBUG	auth/auth.go:376	Login method called for user:

2023-02-28T06:02:49.017Z	INFO	app/server.go:236	/api/v1/login	timeTaken: 325.579µs
 value:
2023-02-28T06:02:49.058Z	INFO	app/server.go:236	/api/v1/user/{id}	timeTaken: 387.94µs
2023-02-28T06:02:49.179Z	INFO	app/server.go:236	/api/v1/featureFlags	timeTaken: 200.19µs
2023-02-28T06:02:49.179Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 195.92µs
2023-02-28T06:02:49.205Z	INFO	app/server.go:236	/api/v1/service/top_level_operations	timeTaken: 2.594889ms
2023-02-28T06:02:49.211Z	DEBUG	clickhouseReader/reader.go:1867

sudhanshu dev

02/28/2023, 6:04 AM

Copy code

SELECT
			quantile(0.5)(durationNano) as p50,
			quantile(0.95)(durationNano) as p95,
			quantile(0.99)(durationNano) as p99,
			COUNT(*) as numCalls,
			name
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND timestamp>= @start AND timestamp<= @end GROUP BY name ORDER BY p99 DESC LIMIT 10
2023-02-28T06:02:49.212Z	INFO	app/server.go:236	/api/v1/service/top_operations	timeTaken: 12.679458ms
2023-02-28T06:02:49.217Z	DEBUG	clickhouseReader/reader.go:869
		SELECT
			toStartOfInterval(timestamp, INTERVAL @interval minute) as time,
			quantile(0.99)(durationNano) as p99,
			quantile(0.95)(durationNano) as p95,
			quantile(0.50)(durationNano) as p50,
			count(*) as numCalls
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND name In [@names] AND timestamp>= @start AND timestamp<= @end GROUP BY time ORDER BY time DESC
2023-02-28T06:02:49.226Z	DEBUG	clickhouseReader/reader.go:897
		SELECT
			toStartOfInterval(timestamp, INTERVAL @interval minute) as time,
			count(*) as numErrors
		FROM signoz_traces.distributed_signoz_index_v2
		WHERE serviceName = @serviceName AND name In [@names] AND timestamp>= @start AND timestamp<= @end AND statusCode=2 GROUP BY time ORDER BY time DESC
2023-02-28T06:02:49.226Z	INFO	app/server.go:236	/api/v1/service/overview	timeTaken: 28.123133ms
2023-02-28T06:02:49.271Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:49.271Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.ts as ts, A.value / B.value as value FROM (SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as A  GLOBAL INNER JOIN(SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as B  ON A.ts = B.ts
2023-02-28T06:02:49.272Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:49.272Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.address as address, A.ts as ts, A.value * 100 / B.value as value FROM (SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan'] AND JSONExtractString(labels, 'status_code') IN ['STATUS_CODE_ERROR']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as A  GLOBAL INNER JOIN(SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as B  ON A.address = B.address AND A.ts = B.ts
2023-02-28T06:02:49.273Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:49.273Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT A.address as address, A.ts as ts, A.value / B.value as value FROM (SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as A  GLOBAL INNER JOIN(SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts) as B  ON A.address = B.address AND A.ts = B.ts
2023-02-28T06:02:49.274Z	INFO	api/metrics.go:21	CustomMetricsFunction feature is not enabled in this plan
2023-02-28T06:02:49.274Z	INFO	clickhouseReader/reader.go:3084	Executing metric result query: SELECT address,  ts, sum(value) as value FROM (SELECT address,  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint, address,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  JSONExtractString(labels, 'address') as address, fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, address,ts ORDER BY fingerprint, address,  ts) OFFSET 1) GROUP BY address,ts ORDER BY address,  ts
2023-02-28T06:02:49.368Z	INFO	utils/time.go:12	GetMetricResult took 94.450179ms

sudhanshu dev

02/28/2023, 6:04 AM

Copy code

2023-02-28T06:02:49.369Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 94.759835ms
2023-02-28T06:02:49.477Z	INFO	utils/time.go:12	GetMetricResult took 205.540941ms

2023-02-28T06:02:49.477Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 205.949289ms
2023-02-28T06:02:49.489Z	INFO	utils/time.go:12	GetMetricResult took 216.242606ms

2023-02-28T06:02:49.489Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 216.649147ms
2023-02-28T06:02:49.490Z	INFO	utils/time.go:12	GetMetricResult took 218.363514ms

2023-02-28T06:02:49.490Z	INFO	app/server.go:236	/api/v2/metrics/query_range	timeTaken: 218.789615ms
2023-02-28T06:02:49.773Z	INFO	app/server.go:236	/api/v1/configs	timeTaken: 591.196096ms
2023-02-28T06:02:50.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 13.714µs
2023-02-28T06:02:50.981Z	INFO	app/server.go:236	/api/v1/version	timeTaken: 7.056µs

sudhanshu dev

02/28/2023, 6:05 AM

@Srikanth Chekuri Will it works or shall I share in file?

Srikanth Chekuri

02/28/2023, 6:16 AM

This looks good to me. Can you exec into clickhouse pod and share the output of this query?

Copy code

SELECT A.ts as ts, A.value / B.value as value FROM (SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as A  GLOBAL INNER JOIN(SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as B  ON A.ts = B.ts

sudhanshu dev

02/28/2023, 6:21 AM

Sure

sudhanshu dev

02/28/2023, 6:26 AM

@Srikanth Chekuri

Copy code

chi-signoz-release-clickhouse-cluster-0-0-0.chi-signoz-release-clickhouse-cluster-0-0.platform.svc.cluster.local :) SELECT A.ts as ts, A.value / B.value as value FROM (SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_sum' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_sum' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as A  GLOBAL INNER JOIN(SELECT  ts, sum(value) as value FROM (SELECT  ts, runningDifference(value)/runningDifference(ts) as value FROM(SELECT fingerprint,  toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), INTERVAL 60 SECOND) as ts, max(value) as value FROM signoz_metrics.distributed_samples_v2 GLOBAL INNER JOIN (SELECT  fingerprint FROM signoz_metrics.distributed_time_series_v2 WHERE metric_name = 'signoz_external_call_latency_count' AND JSONExtractString(labels, 'service_name') IN ['oa-lending-loan']) as filtered_time_series USING fingerprint WHERE metric_name = 'signoz_external_call_latency_count' AND timestamp_ms >= 1677562369000 AND timestamp_ms <= 1677564120000 GROUP BY fingerprint, ts ORDER BY fingerprint,  ts) OFFSET 1) GROUP BY ts ORDER BY  ts) as B  ON A.ts = B.ts

SELECT
    A.ts AS ts,
    A.value / B.value AS value
FROM
(
    SELECT
        ts,
        sum(value) AS value
    FROM
    (
        SELECT
            ts,
            runningDifference(value) / runningDifference(ts) AS value
        FROM
        (
            SELECT
                fingerprint,
                toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), toIntervalSecond(60)) AS ts,
                max(value) AS value
            FROM signoz_metrics.distributed_samples_v2
            GLOBAL INNER JOIN
            (
                SELECT fingerprint
                FROM signoz_metrics.distributed_time_series_v2
                WHERE (metric_name = 'signoz_external_call_latency_sum') AND (JSONExtractString(labels, 'service_name') IN ['oa-lending-loan'])
            ) AS filtered_time_series USING (fingerprint)
            WHERE (metric_name = 'signoz_external_call_latency_sum') AND (timestamp_ms >= 1677562369000) AND (timestamp_ms <= 1677564120000)
            GROUP BY
                fingerprint,
                ts
            ORDER BY
                fingerprint ASC,
                ts ASC
        )
        OFFSET 1
    )
    GROUP BY ts
    ORDER BY ts ASC
) AS A
GLOBAL INNER JOIN
(
    SELECT
        ts,
        sum(value) AS value
    FROM
    (
        SELECT
            ts,
            runningDifference(value) / runningDifference(ts) AS value
        FROM
        (
            SELECT
                fingerprint,
                toStartOfInterval(toDateTime(intDiv(timestamp_ms, 1000)), toIntervalSecond(60)) AS ts,
                max(value) AS value
            FROM signoz_metrics.distributed_samples_v2
            GLOBAL INNER JOIN
            (
                SELECT fingerprint
                FROM signoz_metrics.distributed_time_series_v2
                WHERE (metric_name = 'signoz_external_call_latency_count') AND (JSONExtractString(labels, 'service_name') IN ['oa-lending-loan'])
            ) AS filtered_time_series USING (fingerprint)
            WHERE (metric_name = 'signoz_external_call_latency_count') AND (timestamp_ms >= 1677562369000) AND (timestamp_ms <= 1677564120000)
            GROUP BY
                fingerprint,
                ts
            ORDER BY
                fingerprint ASC,
                ts ASC
        )
        OFFSET 1
    )
    GROUP BY ts
    ORDER BY ts ASC
) AS B ON A.ts = B.ts

Query id: 117b56be-2cc1-43cf-a5be-19fd8d3b4074

Ok.

0 rows in set. Elapsed: 0.098 sec. Processed 518.37 thousand rows, 120.84 MB (5.28 million rows/s., 1.23 GB/s.)

sudhanshu dev

02/28/2023, 6:26 AM

Above is query output

sudhanshu dev

02/28/2023, 6:27 AM

No data return

sudhanshu dev

02/28/2023, 7:10 AM

@Srikanth Chekuri Any idea why are we not getting database calls and external calls?

Srikanth Chekuri

02/28/2023, 7:38 AM

It’s not clear to me what’s happening. Shall we sync up at 2 pm?

sudhanshu dev

02/28/2023, 7:38 AM

yes

sudhanshu dev

02/28/2023, 7:38 AM

We can

Srikanth Chekuri

02/28/2023, 7:38 AM

Sure, I will share the meeting link.

sudhanshu dev

02/28/2023, 7:39 AM

👍

sudhanshu dev

02/28/2023, 8:29 AM

@Srikanth Chekuri Plz share meeting link

Srikanth Chekuri

02/28/2023, 8:30 AM

Sharing in a sec

Srikanth Chekuri

02/28/2023, 8:31 AM

https://us06web.zoom.us/j/84382846357?pwd=dlcwUHhuM09MamtIbTdyMjBxakdEZz09

sudhanshu dev

02/28/2023, 8:32 AM

joining

sudhanshu dev

02/28/2023, 10:35 AM

@Srikanth Chekuri Issue resolved

sudhanshu dev

02/28/2023, 10:35 AM

Thank you for the quick call and debugging the issue

Vinayak Singh

03/01/2023, 4:33 AM

@sudhanshu dev how did you resolved your issue?

sudhanshu dev

03/01/2023, 4:39 AM

It was related to the values.yaml. We created custom values.yaml file which overriding the otel collector metrics configuration

sudhanshu dev

03/01/2023, 4:39 AM

So we added the default configuration into custom values.yaml file

4 Views

Open in Slack

Previous Next