Hi Team, recently Signoz started to be much slower in our case to query the data for last 6 hours and sometimes the dashboard failing like with this error • is there any documentation to debug slow issues or optimise the signoz dashboard ?

What are the resources (CPU and Memory) given to ClickHouse?

CleanShot 2023-05-10 at 11.29.32@2x.png

How many CPU cores are available?

@Srikanth Chekuri can you please get on a call with @Divyanshu Negi and check the query, the amount of data fetched, the processing speed of clickhouse, etc. We can discuss more based on the data. You if you have a set of commands, that can be run and will collect these data would also be good

with

nproc

it shows 2, and no specific cpu core setting is done on the docker container, so it has access to the 2 CPU cores from host

2CPUs are less... how much data are you handling?

In a 1 day this is traces count

2 CPU == 2 concurrent queries. What does the chart you mentioned above query?

the chart above if you mean the clickhouse one, is the stats I got from

docker stats | grep container_id

and while filtering traces for 1 Day

No, I meant the original chat service chart

ah ok, it has 4 panels 1- network latency 2 - GC metrics 3- error rates 4- memory usage

You gc queries the

stringTagMap

in traces which will get really slow and would lead to timeout for other panels since the gc query is not yet finished. Create a materialized column for it and check again.

as a quick solution, increasing the CPU cores would be good enough ?

That might solve your issue temporarily, but I would recommend creating materialized column so you will not face the same issue.

is there any doc ref I can look at on how to add this materialized column for our usecase ? we are sending the GC events via collector tracing service to signoz.

I don’t think we have docs for that yet. We will look into it since it might also effect the exporter. If you can exec into clickhouse -> clieckhouse-client and run the query and share the end of the output that would be helpful.

SELECT toStartOfInterval(timestamp, INTERVAL 60 SECOND) AS ts, 
avg(durationNano) as value 
from signoz_traces.distributed_signoz_index_v2 where (timestamp >= (now() - toIntervalHour(6))) 
AND stringTagMap['gc_type']='GC' group by ts order by ts

@Vishal Sharma adding serviceName in the filters should speed up things, right?

adding serviceName in the filters should speed up things, right?

Yes, more filters means less DB rows scanned and it will be faster

Yes if usecase is just to check if

gc_type

exists then this way it should be faster.

I meant adding

mapContains

along with

AND stringTagMap['gc_type']='GC'

I meant adding

mapContains

along with

AND stringTagMap['gc_type']='GC'

Tested this on a test environment. This doesn’t reduce number of rows scanned and no significant performance changes with

mapContains

filter