I ve migrated a fair chunk of services to SigNoz in our test

Question

I ve migrated a fair chunk of services to SigNoz in our test environment and have `929234` time series at the moment The query service needed to have its memory increased to 3GB for the container to not crash due to OOM Our PROD setup is much more extensive and has more traffic so will this need dozens of GB to operate the backend service Is there anything I am doing wrong I know there are optimizations coming but seems tough to handle this linearly increasing memory in terms of operations

Ankit Nayan · Answer

< Alexei Zenin> what memory usage do you see in steady state We know a reason that causes spike in memory usage during boot up which can be fixed in a few days for run in single instance

Ankit Nayan · Answer

< Prashant Shahi> can you share steps to get memory pprof dump this will help us understand more

Prashant Shahi · Answer

Sure here it is

Prashant Shahi · Answer

Port forward pprof port `6060` of query service container ```kubectl n platform port forward pod my release signoz query service 0 6060 6060``` In another terminal run the following to obtain pprof data CPU Profile ```curl o query service pprof v``` Heap Profile ```curl o query service heap pprof v```

Alexei Zenin · Answer

Thanks ill see if I can do this sometime soon will map these instructions to ECS

Alexei Zenin · Answer

Haven t had the time to do any of the profiling but hit memory limit again with 4000MB allocated to container Seems it was fine for a week or so hovering around 70 max usage but then eventually hit 98 and started crashing with OOM Now just cycles will need to bump to 5000MB

Alexei Zenin · Answer

Ingesting about 250K spans per hour 5 5 million spans per day = 6GB per day in Clickhouse

Ankit Nayan · Answer

< Alexei Zenin> what are the outputs of the below commands ```select count from signoz metrics time series v2 ``` ```select count from signoz metrics samples v2 where timestamp ms gt toUnixTimestamp now INTERVAL 30 MINUTE 1000 ```

Ankit Nayan · Answer

We probably got a lot of stale timeseries

Alexei Zenin · Answer

```SELECT count FROM signoz metrics time series v2 Query id 195009d4 26ea 49e1 86fb 15dc7de0313a count 2449091 1 row in set Elapsed 0 002 sec SELECT count FROM signoz metrics samples v2 WHERE timestamp ms gt toUnixTimestamp now toIntervalMinute 30 1000 Query id 1941f1fd 56c8 4a6d 8f28 e6fc9bdab919 count 2714 1 row in set Elapsed 0 005 sec Processed 40 58 thousand rows 324 64 KB 8 54 million rows s 68 31 MB s ```

Alexei Zenin · Answer

Out of curiosity when would a time series become stale Our containers come up and down dozens of times per day would that affect it due to rescheduling of EC2 Spot instances

Alexei Zenin · Answer

Does that confirm your suspicion ^

Ankit Nayan · Answer

oh you have 2 5M timeseries labels and you receive data from 90 timeseries per min 2714 datapoints in 30 mins

Ankit Nayan · Answer

gt Our containers come up and down dozens of times per day would that affect it due to rescheduling of EC2 Spot instances yes heavily

Ankit Nayan · Answer

do you have any timeseries whose frequency of data collection is gt 1min

Ankit Nayan · Answer

and what s the retention for metrics I am trying to think of a way out of this

Alexei Zenin · Answer

not sure is there a query i can run Currently it is until disk reaches some move factor then unloads to S3

Ankit Nayan · Answer

you configure scraping interval in your otel collectors

Ankit Nayan · Answer

not a way to get that by api IMO

Ankit Nayan · Answer

gt what s the retention for metrics this can be seen in the UI = gt settings page

Ankit Nayan · Answer

Let s just truncate the timeseries table and restart otel collectors this should free up memory and will start writing timeseries labels again

Ankit Nayan · Answer

```truncate table signoz metrics time series v2 ``` and restart otel collectors

Alexei Zenin · Answer

Right now unlimited retention haven t set it meant a query to see the timeseries gt 1min In terms of scraping most of the traces are being pushed to the collectors via an agent which then send to the SigNoz collector stack Which scraping are you referring to

Alexei Zenin · Answer

truncating the table would delete the data

Ankit Nayan · Answer

not the data just the labels do you see charts whose data you did not collect say in the last 1hr

Ankit Nayan · Answer

wait let me think of a smarter solution

Ankit Nayan · Answer

gt do you see charts whose data you did not collect say in the last 1hr what s the oldest data you want to see in charts

Alexei Zenin · Answer

all of the services we are running are still running atm from the services page view don t have any dashboards as of yet

Ankit Nayan · Answer

oh so no other metrics than what is there by default in signoz

Alexei Zenin · Answer

yeah we are only sending traces atm via OTEL java agent

Alexei Zenin · Answer

maybe those are the OTEL collector metrics which I think SigNoz collectors scrape

Ankit Nayan · Answer

we need to get deeper let me come back post dinner in an hr or so

Alexei Zenin · Answer

no problem appreciate it

Ankit Nayan · Answer

do you run otel collectors in spot instances

Alexei Zenin · Answer

yeah

Ankit Nayan · Answer

there it is then

Ankit Nayan · Answer

```select metric name count as count from signoz metrics time series v2 group by metric name order by count desc limit 10 ```

Ankit Nayan · Answer

what s the output of above

Ankit Nayan · Answer

and ```select metric name count as count from signoz metrics samples v2 where timestamp ms gt toUnixTimestamp now INTERVAL 120 MINUTE 1000 group by metric name order by count desc limit 10 ```

Alexei Zenin · Answer

```SELECT metric name count AS count FROM signoz metrics time series v2 GROUP BY metric name ORDER BY count DESC LIMIT 10 Query id d4f1d9a8 067d 48e4 a75e 4fff3f3ef6e2 metric name count otelcol processor batch batch send size bucket 765371 otelcol exporter enqueue failed spans 208634 otelcol exporter enqueue failed metric points 208634 otelcol exporter enqueue failed log records 208634 otelcol process uptime 107054 otelcol process cpu seconds 107054 otelcol process memory rss 107054 otelcol process runtime heap alloc bytes 107054 otelcol process runtime total alloc bytes 107054 otelcol exporter queue size 107054 10 rows in set Elapsed 0 096 sec Processed 2 50 million rows 2 59 MB 25 92 million rows s 26 89 MB s ```

Alexei Zenin · Answer

```SELECT metric name count AS count FROM signoz metrics samples v2 WHERE timestamp ms gt toUnixTimestamp now toIntervalMinute 120 1000 GROUP BY metric name ORDER BY count DESC LIMIT 10 Query id fef0992c 1d77 4a6b 9465 8e7aeea03f75 metric name count up 66 scrape samples scraped 66 scrape series added 66 scrape duration seconds 66 scrape samples post metric relabeling 66 otelcol exporter enqueue failed metric points 59 otelcol exporter enqueue failed log records 59 otelcol exporter enqueue failed spans 59 otelcol process memory rss 33 otelcol process runtime total alloc bytes 33 10 rows in set Elapsed 0 022 sec Processed 28 82 thousand rows 261 97 KB 1 31 million rows s 11 92 MB s ```

Alexei Zenin · Answer

I guess I can disable this Doesn t seem critical to have

Ankit Nayan · Answer

yes disable metrics collection for this

Ankit Nayan · Answer

apply ```truncate table signoz metrics time series v2 ``` and then restart otel collectors

Alexei Zenin · Answer

thanks ill give this a try

Ankit Nayan · Answer

curious I don t see signoz s APM metrics in the list

Ankit Nayan · Answer

how many services are you running

Alexei Zenin · Answer

75 services

Ankit Nayan · Answer

weird they should be coming up

Ankit Nayan · Answer

let me pull up a couple of more queries to understand deeper

Alexei Zenin · Answer

services are test environment basically no traffic

Ankit Nayan · Answer

ok even then no traffic in last 2 hrs must be then

Ankit Nayan · Answer

```select metric name count as count from signoz metrics time series v2 where metric name ilike signoz group by metric name ```

Ankit Nayan · Answer

now let s check how many APM metrics datapoints you received in a day ```select metric name count as count from signoz metrics samples v2 where timestamp ms gt toUnixTimestamp now INTERVAL 1 DAY 1000 and metric name ilike signoz group by metric name ```

Alexei Zenin · Answer

```SELECT metric name count AS count FROM signoz metrics time series v2 WHERE metric name ILIKE signoz GROUP BY metric name Query id 14b718d9 4522 401b 9f15 aa362870b0be metric name count signoz latency bucket 105108 signoz latency count 5532 signoz external call latency count 862 signoz db latency sum 211 signoz db latency count 211 signoz calls total 5754 signoz latency sum 5532 signoz external call latency sum 862 8 rows in set Elapsed 0 266 sec Processed 2 50 million rows 2 59 MB 9 41 million rows s 9 76 MB s ```

Alexei Zenin · Answer

```SELECT metric name count AS count FROM signoz metrics samples v2 WHERE timestamp ms gt toUnixTimestamp now toIntervalDay 1 1000 AND metric name ILIKE signoz GROUP BY metric name Query id a5255015 e491 434c 80c7 cffd8c850e44 metric name count signoz latency bucket 16986 signoz latency count 894 signoz external call latency count 199 signoz db latency sum 136 signoz db latency count 136 signoz calls total 894 signoz latency sum 894 signoz external call latency sum 199 8 rows in set Elapsed 0 023 sec Processed 90 94 thousand rows 560 54 KB 3 93 million rows s 24 22 MB s ```

Ankit Nayan · Answer

hmm very less datapoints not big load

Ankit Nayan · Answer

<https signoz community slack com archives C01HWQ1R0BC p1663867897689989 thread ts=1661872578 524639 amp cid=C01HWQ1R0BC> you can do this

Alexei Zenin · Answer

Looks like this worked I removed the scrapping of internal metrics and truncated the table memory usage is very low now

Alexei Zenin · Answer

Just to confirm the otel collector metrics container can be run as a sidecar for each otel collector container

Alexei Zenin · Answer

I have a Load balancer in front of my otel collectors and each one has a local sidecar otel collector metrics to scrap the prometheus endpoint im assuming the data is independent of each one and will have no impact on span metrics scrapping if done in 1 1 fashion instead of 1 N

Ankit Nayan · Answer

< Alexei Zenin> with release `v0 11 2` you won t need to truncate anymore Use metrics freely now

Ankit Nayan · Answer

gt im assuming the data is independent of each one and will have no impact on span metrics scrapping if done in 1 1 fashion instead of 1 N that is true and should work if scrape configs are correct But why would you want to do 1 1 won t it be too much overkill

Alexei Zenin · Answer

Ease of deployment no need to do any dynamic scrape config setup and needing to restart metrics collector i run them on Fargate spot so IPs change all the time