Slackbot
08/30/2022, 3:16 PMAnkit Nayan
Ankit Nayan
Prashant Shahi
08/30/2022, 6:13 PMPrashant Shahi
08/30/2022, 6:17 PM6060
of query-service container:
kubectl -n platform port-forward pod/my-release-signoz-query-service-0 6060:6060
In another terminal, run the following to obtain pprof data:
β’ CPU Profile
curl "<http://localhost:6060/debug/pprof/profile?seconds=30>" -o query-service.pprof -v
β’ Heap Profile
curl "<http://localhost:6060/debug/pprof/heap>" -o query-service-heap.pprof -v
Alexei Zenin
08/30/2022, 6:34 PMAlexei Zenin
09/20/2022, 4:01 PMAlexei Zenin
09/20/2022, 4:03 PMAnkit Nayan
select count() from signoz_metrics.time_series_v2;
select count() from signoz_metrics.samples_v2 where timestamp_ms > toUnixTimestamp(now() - INTERVAL 30 MINUTE)*1000;
Ankit Nayan
Alexei Zenin
09/20/2022, 7:29 PMSELECT count()
FROM signoz_metrics.time_series_v2
Query id: 195009d4-26ea-49e1-86fb-15dc7de0313a
ββcount()ββ
β 2449091 β
βββββββββββ
1 row in set. Elapsed: 0.002 sec.
SELECT count()
FROM signoz_metrics.samples_v2
WHERE timestamp_ms > (toUnixTimestamp(now() - toIntervalMinute(30)) * 1000)
Query id: 1941f1fd-56c8-4a6d-8f28-e6fc9bdab919
ββcount()ββ
β 2714 β
βββββββββββ
1 row in set. Elapsed: 0.005 sec. Processed 40.58 thousand rows, 324.64 KB (8.54 million rows/s., 68.31 MB/s.)
Alexei Zenin
09/20/2022, 7:37 PMAlexei Zenin
09/22/2022, 2:48 PMAnkit Nayan
Ankit Nayan
Our containers come up and down dozens of times per day, would that affect it (due to rescheduling of EC2 Spot instances)?yes .. heavily...
Ankit Nayan
Ankit Nayan
Alexei Zenin
09/22/2022, 2:53 PMAnkit Nayan
Ankit Nayan
Ankit Nayan
what's the retention for metrics?this can be seen in the UI => settings page
Ankit Nayan
Ankit Nayan
truncate table signoz_metrics.time_series_v2;
and, restart otel-collectorsAlexei Zenin
09/22/2022, 3:01 PMAlexei Zenin
09/22/2022, 3:01 PMAnkit Nayan
Ankit Nayan
Ankit Nayan
do you see charts whose data you did not collect say in the last 1hr?what's the oldest data you want to see in charts?
Alexei Zenin
09/22/2022, 3:03 PMAnkit Nayan
Alexei Zenin
09/22/2022, 3:04 PMAlexei Zenin
09/22/2022, 3:05 PMAnkit Nayan
Alexei Zenin
09/22/2022, 3:05 PMAnkit Nayan
Alexei Zenin
09/22/2022, 3:05 PMAnkit Nayan
Ankit Nayan
select metric_name, count() as count from signoz_metrics.time_series_v2 group by metric_name order by count desc limit 10;
Ankit Nayan
Ankit Nayan
select metric_name, count() as count from signoz_metrics.samples_v2 where timestamp_ms > toUnixTimestamp(now() - INTERVAL 120 MINUTE)*1000 group by metric_name order by count desc limit 10 ;
Alexei Zenin
09/22/2022, 5:23 PMSELECT
metric_name,
count() AS count
FROM signoz_metrics.time_series_v2
GROUP BY metric_name
ORDER BY count DESC
LIMIT 10
Query id: d4f1d9a8-067d-48e4-a75e-4fff3f3ef6e2
ββmetric_nameβββββββββββββββββββββββββββββββββββββ¬ββcountββ
β otelcol_processor_batch_batch_send_size_bucket β 765371 β
β otelcol_exporter_enqueue_failed_spans β 208634 β
β otelcol_exporter_enqueue_failed_metric_points β 208634 β
β otelcol_exporter_enqueue_failed_log_records β 208634 β
β otelcol_process_uptime β 107054 β
β otelcol_process_cpu_seconds β 107054 β
β otelcol_process_memory_rss β 107054 β
β otelcol_process_runtime_heap_alloc_bytes β 107054 β
β otelcol_process_runtime_total_alloc_bytes β 107054 β
β otelcol_exporter_queue_size β 107054 β
ββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββ
10 rows in set. Elapsed: 0.096 sec. Processed 2.50 million rows, 2.59 MB (25.92 million rows/s., 26.89 MB/s.)
Alexei Zenin
09/22/2022, 5:23 PMSELECT
metric_name,
count() AS count
FROM signoz_metrics.samples_v2
WHERE timestamp_ms > (toUnixTimestamp(now() - toIntervalMinute(120)) * 1000)
GROUP BY metric_name
ORDER BY count DESC
LIMIT 10
Query id: fef0992c-1d77-4a6b-9465-8e7aeea03f75
ββmetric_nameββββββββββββββββββββββββββββββββββββ¬βcountββ
β up β 66 β
β scrape_samples_scraped β 66 β
β scrape_series_added β 66 β
β scrape_duration_seconds β 66 β
β scrape_samples_post_metric_relabeling β 66 β
β otelcol_exporter_enqueue_failed_metric_points β 59 β
β otelcol_exporter_enqueue_failed_log_records β 59 β
β otelcol_exporter_enqueue_failed_spans β 59 β
β otelcol_process_memory_rss β 33 β
β otelcol_process_runtime_total_alloc_bytes β 33 β
βββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββ
10 rows in set. Elapsed: 0.022 sec. Processed 28.82 thousand rows, 261.97 KB (1.31 million rows/s., 11.92 MB/s.)
Alexei Zenin
09/22/2022, 5:24 PMAnkit Nayan
Ankit Nayan
truncate table signoz_metrics.time_series_v2;
and then restart otel-collectorsAlexei Zenin
09/22/2022, 5:32 PMAnkit Nayan
Ankit Nayan
Alexei Zenin
09/22/2022, 5:33 PMAnkit Nayan
Ankit Nayan
Alexei Zenin
09/22/2022, 5:34 PMAnkit Nayan
Ankit Nayan
select metric_name, count() as count from signoz_metrics.time_series_v2 where metric_name ilike '%signoz%' group by metric_name;
Ankit Nayan
select metric_name, count() as count from signoz_metrics.samples_v2 where timestamp_ms > toUnixTimestamp(now() - INTERVAL 1 DAY)*1000 and metric_name ilike '%signoz%' group by metric_name;
Alexei Zenin
09/22/2022, 8:34 PMSELECT
metric_name,
count() AS count
FROM signoz_metrics.time_series_v2
WHERE metric_name ILIKE '%signoz%'
GROUP BY metric_name
Query id: 14b718d9-4522-401b-9f15-aa362870b0be
ββmetric_nameβββββββββββββββββββββββββ¬ββcountββ
β signoz_latency_bucket β 105108 β
β signoz_latency_count β 5532 β
β signoz_external_call_latency_count β 862 β
β signoz_db_latency_sum β 211 β
β signoz_db_latency_count β 211 β
β signoz_calls_total β 5754 β
β signoz_latency_sum β 5532 β
β signoz_external_call_latency_sum β 862 β
ββββββββββββββββββββββββββββββββββββββ΄βββββββββ
8 rows in set. Elapsed: 0.266 sec. Processed 2.50 million rows, 2.59 MB (9.41 million rows/s., 9.76 MB/s.)
Alexei Zenin
09/22/2022, 8:35 PMSELECT
metric_name,
count() AS count
FROM signoz_metrics.samples_v2
WHERE (timestamp_ms > (toUnixTimestamp(now() - toIntervalDay(1)) * 1000)) AND (metric_name ILIKE '%signoz%')
GROUP BY metric_name
Query id: a5255015-e491-434c-80c7-cffd8c850e44
ββmetric_nameβββββββββββββββββββββββββ¬βcountββ
β signoz_latency_bucket β 16986 β
β signoz_latency_count β 894 β
β signoz_external_call_latency_count β 199 β
β signoz_db_latency_sum β 136 β
β signoz_db_latency_count β 136 β
β signoz_calls_total β 894 β
β signoz_latency_sum β 894 β
β signoz_external_call_latency_sum β 199 β
ββββββββββββββββββββββββββββββββββββββ΄ββββββββ
8 rows in set. Elapsed: 0.023 sec. Processed 90.94 thousand rows, 560.54 KB (3.93 million rows/s., 24.22 MB/s.)
Ankit Nayan
Ankit Nayan
Alexei Zenin
10/13/2022, 8:46 PMAlexei Zenin
10/13/2022, 8:46 PMAlexei Zenin
10/13/2022, 8:47 PMAnkit Nayan
v0.11.2
you won't need to truncate anymore. Use metrics freely nowAnkit Nayan
im assuming the data is independent of each one and will have no impact on span metrics scrapping if done in 1:1 fashion instead of 1:Nthat is true and should work if scrape configs are correct. But why would you want to do 1:1, won't it be too much overkill?
Alexei Zenin
10/14/2022, 4:44 AM