https://signoz.io logo
#support
Title
# support
d

Divyanshu Negi

05/10/2023, 3:19 AM
Hi Team, recently Signoz started to be much slower in our case to query the data for last 6 hours and sometimes the dashboard failing like with this error • is there any documentation to debug slow issues or optimise the signoz dashboard ?
we are in the latest version
0.18.3
s

Srikanth Chekuri

05/10/2023, 3:24 AM
What are the resources (CPU and Memory) given to ClickHouse?
d

Divyanshu Negi

05/10/2023, 3:29 AM
CleanShot 2023-05-10 at 11.29.32@2x.png
I have noticed when doing any search on signoz the clickhouse container CPU usage is going beyond 200%
using
docker stats | grep container_id
s

Srikanth Chekuri

05/10/2023, 4:05 AM
How many CPU cores are available?
a

Ankit Nayan

05/10/2023, 4:09 AM
@Srikanth Chekuri can you please get on a call with @Divyanshu Negi and check the query, the amount of data fetched, the processing speed of clickhouse, etc. We can discuss more based on the data. You if you have a set of commands, that can be run and will collect these data would also be good
d

Divyanshu Negi

05/10/2023, 4:09 AM
with
nproc
it shows 2, and no specific cpu core setting is done on the docker container, so it has access to the 2 CPU cores from host
a

Ankit Nayan

05/10/2023, 4:10 AM
2CPUs are less... how much data are you handling?
d

Divyanshu Negi

05/10/2023, 4:14 AM
In a 1 day this is traces count
s

Srikanth Chekuri

05/10/2023, 4:17 AM
2 CPU == 2 concurrent queries. What does the chart you mentioned above query?
d

Divyanshu Negi

05/10/2023, 4:18 AM
the chart above if you mean the clickhouse one, is the stats I got from
docker stats | grep container_id
and while filtering traces for 1 Day
s

Srikanth Chekuri

05/10/2023, 4:19 AM
No, I meant the original chat service chart
d

Divyanshu Negi

05/10/2023, 4:20 AM
ah ok, it has 4 panels 1- network latency 2 - GC metrics 3- error rates 4- memory usage
s

Srikanth Chekuri

05/10/2023, 4:24 AM
You gc queries the
stringTagMap
in traces which will get really slow and would lead to timeout for other panels since the gc query is not yet finished. Create a materialized column for it and check again.
d

Divyanshu Negi

05/10/2023, 4:28 AM
as a quick solution, increasing the CPU cores would be good enough ?
s

Srikanth Chekuri

05/10/2023, 4:38 AM
That might solve your issue temporarily, but I would recommend creating materialized column so you will not face the same issue.
d

Divyanshu Negi

05/10/2023, 4:42 AM
is there any doc ref I can look at on how to add this materialized column for our usecase ? we are sending the GC events via collector tracing service to signoz.
s

Srikanth Chekuri

05/10/2023, 5:03 AM
I don’t think we have docs for that yet. We will look into it since it might also effect the exporter. If you can exec into clickhouse -> clieckhouse-client and run the query and share the end of the output that would be helpful.
Copy code
SELECT toStartOfInterval(timestamp, INTERVAL 60 SECOND) AS ts, 
avg(durationNano) as value 
from signoz_traces.distributed_signoz_index_v2 where (timestamp >= (now() - toIntervalHour(6))) 
AND stringTagMap['gc_type']='GC' group by ts order by ts
a

Ankit Nayan

05/10/2023, 9:06 AM
@Vishal Sharma adding serviceName in the filters should speed up things, right?
@Vishal Sharma also, adding
mapContains(stringTagMap, 'gc_type')
will make the query faster?
v

Vishal Sharma

05/10/2023, 9:09 AM
adding serviceName in the filters should speed up things, right?
Yes, more filters means less DB rows scanned and it will be faster
also, adding
mapContains(stringTagMap, 'gc_type')
will make the query faster?
Yes if usecase is just to check if
gc_type
exists then this way it should be faster.
a

Ankit Nayan

05/10/2023, 9:36 AM
Yes if usecase is just to check if
gc_type
exists then this way it should be faster.
I meant adding
mapContains
along with
AND stringTagMap['gc_type']='GC'
v

Vishal Sharma

05/10/2023, 10:05 AM
I meant adding
mapContains
along with
AND stringTagMap['gc_type']='GC'
Tested this on a test environment. This doesn’t reduce number of rows scanned and no significant performance changes with
mapContains
filter