Hi guys. Need help in clarifying storage concepts in Signoz. Issue: The size of data dir increases ...

Chitransh Popli

05/01/2024, 10:27 AM

Hi guys. Need help in clarifying storage concepts in Signoz. Issue: The size of data dir increases continuously. I am sending logs to Signoz and monitoring their size after processing. Initially, the data directory (

signoz/deploy/docker/clickhouse-setup/data

) size increased from 40MB to 155MB despite no new data being sent. After sending test logs, the size increased to 265MB. 15mins after all logs were received, it was 285MB. I have removed both hotrod containers as well as

tcplog/docker

receiver and Receiving logs in signoz via

httplogreceiver

. I am sure there is some concept aboutt data storage in Signoz thaat I am missing. Any help is appreciated.

Chitransh Popli

05/01/2024, 10:31 AM

Size log of the directory:

Copy code

2024-05-01 20:02:50 265M        ./docker/clickhouse-setup/data
2024-05-01 20:03:00 263M        ./docker/clickhouse-setup/data
2024-05-01 20:03:10 266M        ./docker/clickhouse-setup/data
2024-05-01 20:03:20 265M        ./docker/clickhouse-setup/data
2024-05-01 20:03:30 264M        ./docker/clickhouse-setup/data
2024-05-01 20:03:40 263M        ./docker/clickhouse-setup/data
2024-05-01 20:03:51 267M        ./docker/clickhouse-setup/data
2024-05-01 20:04:01 268M        ./docker/clickhouse-setup/data
2024-05-01 20:04:11 267M        ./docker/clickhouse-setup/data
2024-05-01 20:04:21 270M        ./docker/clickhouse-setup/data
2024-05-01 20:04:31 269M        ./docker/clickhouse-setup/data
2024-05-01 20:04:41 271M        ./docker/clickhouse-setup/data
2024-05-01 20:04:51 269M        ./docker/clickhouse-setup/data
2024-05-01 20:05:01 272M        ./docker/clickhouse-setup/data
2024-05-01 20:05:11 271M        ./docker/clickhouse-setup/data
2024-05-01 20:05:21 272M        ./docker/clickhouse-setup/data
2024-05-01 20:05:31 270M        ./docker/clickhouse-setup/data
2024-05-01 20:05:41 272M        ./docker/clickhouse-setup/data
2024-05-01 20:05:51 275M        ./docker/clickhouse-setup/data
2024-05-01 20:06:01 276M        ./docker/clickhouse-setup/data
2024-05-01 20:06:11 275M        ./docker/clickhouse-setup/data
2024-05-01 20:06:21 276M        ./docker/clickhouse-setup/data
2024-05-01 20:06:31 274M        ./docker/clickhouse-setup/data
2024-05-01 20:06:41 273M        ./docker/clickhouse-setup/data
2024-05-01 20:06:51 277M        ./docker/clickhouse-setup/data
2024-05-01 20:07:01 282M        ./docker/clickhouse-setup/data
2024-05-01 20:07:12 280M        ./docker/clickhouse-setup/data
2024-05-01 20:07:22 279M        ./docker/clickhouse-setup/data
2024-05-01 20:07:32 277M        ./docker/clickhouse-setup/data
2024-05-01 20:07:42 279M        ./docker/clickhouse-setup/data
2024-05-01 20:07:52 280M        ./docker/clickhouse-setup/data
2024-05-01 20:08:02 272M        ./docker/clickhouse-setup/data
2024-05-01 20:08:12 276M        ./docker/clickhouse-setup/data
2024-05-01 20:08:22 276M        ./docker/clickhouse-setup/data
2024-05-01 20:08:32 278M        ./docker/clickhouse-setup/data
2024-05-01 20:08:42 275M        ./docker/clickhouse-setup/data
2024-05-01 20:08:52 279M        ./docker/clickhouse-setup/data
2024-05-01 20:09:02 280M        ./docker/clickhouse-setup/data
2024-05-01 20:09:12 277M        ./docker/clickhouse-setup/data
2024-05-01 20:09:22 280M        ./docker/clickhouse-setup/data
2024-05-01 20:09:32 279M        ./docker/clickhouse-setup/data
2024-05-01 20:09:42 281M        ./docker/clickhouse-setup/data
2024-05-01 20:09:52 279M        ./docker/clickhouse-setup/data
2024-05-01 20:10:02 282M        ./docker/clickhouse-setup/data
2024-05-01 20:10:13 282M        ./docker/clickhouse-setup/data
2024-05-01 20:10:23 281M        ./docker/clickhouse-setup/data
2024-05-01 20:10:33 280M        ./docker/clickhouse-setup/data
2024-05-01 20:10:43 280M        ./docker/clickhouse-setup/data
2024-05-01 20:10:53 283M        ./docker/clickhouse-setup/data
2024-05-01 20:11:03 281M        ./docker/clickhouse-setup/data
2024-05-01 20:11:13 282M        ./docker/clickhouse-setup/data
2024-05-01 20:11:23 283M        ./docker/clickhouse-setup/data
2024-05-01 20:11:33 283M        ./docker/clickhouse-setup/data
2024-05-01 20:11:43 281M        ./docker/clickhouse-setup/data
2024-05-01 20:11:53 283M        ./docker/clickhouse-setup/data
2024-05-01 20:12:03 285M        ./docker/clickhouse-setup/data
2024-05-01 20:12:13 284M        ./docker/clickhouse-setup/data
2024-05-01 20:12:24 284M        ./docker/clickhouse-setup/data
2024-05-01 20:12:34 284M        ./docker/clickhouse-setup/data
2024-05-01 20:12:44 285M        ./docker/clickhouse-setup/data
2024-05-01 20:12:54 285M        ./docker/clickhouse-setup/data
2024-05-01 20:13:04 284M        ./docker/clickhouse-setup/data
2024-05-01 20:13:14 287M        ./docker/clickhouse-setup/data
2024-05-01 20:13:24 284M        ./docker/clickhouse-setup/data
2024-05-01 20:13:34 283M        ./docker/clickhouse-setup/data
2024-05-01 20:13:44 280M        ./docker/clickhouse-setup/data
2024-05-01 20:13:54 284M        ./docker/clickhouse-setup/data
2024-05-01 20:14:04 284M        ./docker/clickhouse-setup/data
2024-05-01 20:14:14 281M        ./docker/clickhouse-setup/data
2024-05-01 20:14:24 287M        ./docker/clickhouse-setup/data
2024-05-01 20:14:34 289M        ./docker/clickhouse-setup/data
2024-05-01 20:14:44 290M        ./docker/clickhouse-setup/data
2024-05-01 20:14:54 285M        ./docker/clickhouse-setup/data
2024-05-01 20:15:04 286M        ./docker/clickhouse-setup/data
2024-05-01 20:15:15 286M        ./docker/clickhouse-setup/data
2024-05-01 20:15:25 289M        ./docker/clickhouse-setup/data
2024-05-01 20:15:35 289M        ./docker/clickhouse-setup/data
2024-05-01 20:15:45 289M        ./docker/clickhouse-setup/data
2024-05-01 20:15:55 291M        ./docker/clickhouse-setup/data
2024-05-01 20:16:05 292M        ./docker/clickhouse-setup/data
2024-05-01 20:16:15 291M        ./docker/clickhouse-setup/data
2024-05-01 20:16:25 292M        ./docker/clickhouse-setup/data
2024-05-01 20:16:35 292M        ./docker/clickhouse-setup/data
2024-05-01 20:16:45 289M        ./docker/clickhouse-setup/data
2024-05-01 20:16:55 294M        ./docker/clickhouse-setup/data
2024-05-01 20:17:05 299M        ./docker/clickhouse-setup/data
2024-05-01 20:17:15 298M        ./docker/clickhouse-setup/data
2024-05-01 20:17:25 296M        ./docker/clickhouse-setup/data
2024-05-01 20:17:35 292M        ./docker/clickhouse-setup/data
2024-05-01 20:17:45 294M        ./docker/clickhouse-setup/data
2024-05-01 20:17:55 295M        ./docker/clickhouse-setup/data
2024-05-01 20:18:05 294M        ./docker/clickhouse-setup/data
2024-05-01 20:18:16 295M        ./docker/clickhouse-setup/data
2024-05-01 20:18:26 299M        ./docker/clickhouse-setup/data
2024-05-01 20:18:36 302M        ./docker/clickhouse-setup/data
2024-05-01 20:18:46 299M        ./docker/clickhouse-setup/data
2024-05-01 20:18:56 301M        ./docker/clickhouse-setup/data
2024-05-01 20:19:06 301M        ./docker/clickhouse-setup/data
2024-05-01 20:19:16 301M        ./docker/clickhouse-setup/data
2024-05-01 20:19:26 304M        ./docker/clickhouse-setup/data
2024-05-01 20:19:36 303M        ./docker/clickhouse-setup/data
2024-05-01 20:19:46 305M        ./docker/clickhouse-setup/data
2024-05-01 20:19:56 300M        ./docker/clickhouse-setup/data
2024-05-01 20:20:06 301M        ./docker/clickhouse-setup/data
2024-05-01 20:20:16 300M        ./docker/clickhouse-setup/data
2024-05-01 20:20:26 302M        ./docker/clickhouse-setup/data
2024-05-01 20:20:36 300M        ./docker/clickhouse-setup/data
2024-05-01 20:20:46 300M        ./docker/clickhouse-setup/data
2024-05-01 20:20:56 307M        ./docker/clickhouse-setup/data
2024-05-01 20:21:06 309M        ./docker/clickhouse-setup/data
2024-05-01 20:21:16 309M        ./docker/clickhouse-setup/data
2024-05-01 20:21:27 308M        ./docker/clickhouse-setup/data
2024-05-01 20:21:37 309M        ./docker/clickhouse-setup/data
2024-05-01 20:21:47 306M        ./docker/clickhouse-setup/data
2024-05-01 20:21:57 310M        ./docker/clickhouse-setup/data
2024-05-01 20:22:07 312M        ./docker/clickhouse-setup/data
2024-05-01 20:22:17 308M        ./docker/clickhouse-setup/data
2024-05-01 20:22:27 308M        ./docker/clickhouse-setup/data
2024-05-01 20:22:37 305M        ./docker/clickhouse-setup/data
2024-05-01 20:22:47 308M        ./docker/clickhouse-setup/data
2024-05-01 20:22:57 310M        ./docker/clickhouse-setup/data
2024-05-01 20:23:07 309M        ./docker/clickhouse-setup/data
2024-05-01 20:23:17 310M        ./docker/clickhouse-setup/data
2024-05-01 20:23:27 313M        ./docker/clickhouse-setup/data
2024-05-01 20:23:37 316M        ./docker/clickhouse-setup/data
2024-05-01 20:23:47 312M        ./docker/clickhouse-setup/data
2024-05-01 20:23:57 315M        ./docker/clickhouse-setup/data
2024-05-01 20:24:07 314M        ./docker/clickhouse-setup/data
2024-05-01 20:24:17 314M        ./docker/clickhouse-setup/data
2024-05-01 20:24:28 315M        ./docker/clickhouse-setup/data
2024-05-01 20:24:38 316M        ./docker/clickhouse-setup/data
2024-05-01 20:24:48 317M        ./docker/clickhouse-setup/data
2024-05-01 20:24:58 317M        ./docker/clickhouse-setup/data
2024-05-01 20:25:08 319M        ./docker/clickhouse-setup/data
2024-05-01 20:25:18 319M        ./docker/clickhouse-setup/data
2024-05-01 20:25:28 321M        ./docker/clickhouse-setup/data
2024-05-01 20:25:38 318M        ./docker/clickhouse-setup/data
2024-05-01 20:25:48 322M        ./docker/clickhouse-setup/data
^C%

Chitransh Popli

05/01/2024, 10:37 AM

One more thing to add is, the size stabilizes as soon as I do docker compose down:

Copy code

2024-05-01 20:33:48 349M        ./docker/clickhouse-setup/data
2024-05-01 20:33:58 350M        ./docker/clickhouse-setup/data
2024-05-01 20:34:08 350M        ./docker/clickhouse-setup/data
2024-05-01 20:34:18 350M        ./docker/clickhouse-setup/data
2024-05-01 20:34:28 350M        ./docker/clickhouse-setup/data
2024-05-01 20:34:38 350M        ./docker/clickhouse-setup/data
2024-05-01 20:34:48 350M        ./docker/clickhouse-setup/data
2024-05-01 20:34:58 350M        ./docker/clickhouse-setup/data
2024-05-01 20:35:08 350M        ./docker/clickhouse-setup/data
2024-05-01 20:35:19 350M        ./docker/clickhouse-setup/data

Stopped at 350M. Would have assumed the continuous increase was due to some cache/temp files if it fell.

Srikanth Chekuri

05/02/2024, 5:45 PM

There are some system tables that clickhouse uses. They also contribute the size. Please look at the output of this query. This indicates the compressed and uncompressed size of the data. If you keep sending data, the real usage would differes as the clickhouse optimizes data in background

Copy code

SELECT
	database,
	table,
	formatReadableSize(sum(data_compressed_bytes) AS size) AS compressed,
	formatReadableSize(sum(data_uncompressed_bytes) AS usize) AS uncompressed,
	round(usize / size, 2) AS compr_rate,
	sum(rows) AS rows,
	count() AS part_count
FROM system.parts
WHERE (active = 1) AND (database LIKE 'signoz_%')
GROUP BY
	database,
	table
ORDER BY size DESC;

Chitransh Popli

05/06/2024, 11:40 AM

Sorry I don't have a lot of experience with signoz, where do I need to run this query?

Srikanth Chekuri

05/06/2024, 3:28 PM

exec into clickhouse

Copy code

docker exec -it signoz-clickhouse

get into clickhouse console

Copy code

clickhouse-client

run the query

Chitransh Popli

05/06/2024, 6:33 PM

That was perfect. Truly appreciate the help.

3 Views

Open in Slack

Previous Next

SigNoz Community

SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.