High Memory usage for Zookeeper
I'm using a clickhouse cluster with three shards and one replica eac...
d
Daniel Hilgarth
10/22/2024, 8:26 PMHigh Memory usage for Zookeeper
I'm using a clickhouse cluster with three shards and one replica each, so three total instances of clickhouse. They are supported by three zookeeper instances. The zookeeper instances use about the same amount of memory as my clickhouse instances.
Zookeeper: 1,23GB, 1,19GB and 879 MB
Clickhouse: 1,24GB, 1,11GB and 2,03 GB
Is this expected?
This is a new install, I started from scratch 2 days ago with Signoz v0.56.0. I also threw away all previous data.
Clickhouse v24.1.2
Zookeeper Bitnami 3.7.1
Data in the system so far: see thread
Daniel Hilgarth
10/22/2024, 8:26 PM─database───────┬─table───────────────────────┬─compressed─┬─uncompressed─┬─compr_rate─┬─────rows─┬─part_count─┐
│ signoz_logs │ logs_v2 │ 3.48 GiB │ 49.55 GiB │ 14.22 │ 59052397 │ 26 │
│ signoz_traces │ durationSort │ 908.84 MiB │ 7.46 GiB │ 8.41 │ 9412798 │ 25 │
│ signoz_traces │ signoz_index_v2 │ 789.69 MiB │ 9.18 GiB │ 11.91 │ 9412798 │ 35 │
│ signoz_traces │ signoz_spans │ 556.53 MiB │ 17.55 GiB │ 32.3 │ 9412798 │ 25 │
│ signoz_traces │ span_attributes │ 224.53 MiB │ 1.52 GiB │ 6.91 │ 23056127 │ 8 │
│ system │ query_log │ 211.86 MiB │ 1.13 GiB │ 5.46 │ 1812799 │ 9 │
│ system │ trace_log │ 199.03 MiB │ 2.54 GiB │ 13.07 │ 8050078 │ 4 │
│ system │ part_log │ 162.62 MiB │ 1.17 GiB │ 7.39 │ 2179857 │ 10 │
│ system │ query_views_log │ 66.61 MiB │ 874.76 MiB │ 13.13 │ 766776 │ 9 │
│ system │ metric_log │ 54.52 MiB │ 213.03 MiB │ 3.91 │ 195833 │ 6 │
│ system │ asynchronous_metric_log │ 38.03 MiB │ 700.94 MiB │ 18.43 │ 45911975 │ 9 │
│ signoz_traces │ signoz_error_index_v2 │ 22.02 MiB │ 546.75 MiB │ 24.83 │ 463865 │ 12 │
│ signoz_traces │ dependency_graph_minutes_v2 │ 13.87 MiB │ 20.05 MiB │ 1.45 │ 17795 │ 12 │
│ signoz_traces │ dependency_graph_minutes │ 13.87 MiB │ 20.00 MiB │ 1.44 │ 17799 │ 12 │
│ signoz_logs │ tag_attributes │ 18.25 KiB │ 194.96 KiB │ 10.68 │ 485 │ 3 │
│ signoz_logs │ usage │ 8.17 KiB │ 11.73 KiB │ 1.44 │ 56 │ 3 │
│ signoz_traces │ usage │ 6.99 KiB │ 9.83 KiB │ 1.41 │ 48 │ 3 │
│ signoz_traces │ span_attributes_keys │ 3.80 KiB │ 5.60 KiB │ 1.47 │ 290 │ 5 │
│ signoz_traces │ usage_explorer │ 2.83 KiB │ 4.67 KiB │ 1.65 │ 246 │ 8 │
│ signoz_traces │ top_level_operations │ 1.54 KiB │ 2.77 KiB │ 1.8 │ 113 │ 1 │
│ signoz_logs │ logs_v2_resource │ 1.21 KiB │ 62.51 KiB │ 51.83 │ 227 │ 3 │
│ signoz_logs │ logs_resource_keys │ 1.18 KiB │ 1.84 KiB │ 1.56 │ 76 │ 4 │
│ signoz_traces │ schema_migrations │ 847.00 B │ 1.04 KiB │ 1.25 │ 62 │ 2 │
│ signoz_logs │ schema_migrations │ 512.00 B │ 518.00 B │ 1.01 │ 30 │ 2 │
│ signoz_logs │ logs_attribute_keys │ 325.00 B │ 248.00 B │ 0.76 │ 13 │ 3 │
│ signoz_metrics │ schema_migrations │ 266.00 B │ 272.00 B │ 1.02 │ 16 │ 1 │
└────────────────┴─────────────────────────────┴────────────┴──────────────┴────────────┴──────────┴────────────┘
s
Srikanth Chekuri
10/23/2024, 3:51 AMThe default zk installation from bitnami pre-allocates 1 GB of heap memory. You don't need that much heap memory most of the time. There is a
heapSized
Daniel Hilgarth
10/23/2024, 7:49 AMGreat, thanks. I just did that and it looks fine so far. What would be a good indicator that Zookeeper needs in fact more heap?
s
Srikanth Chekuri
10/28/2024, 6:08 PMThe jvm heap memory used metric emitted by ZK will be good thing to monitor. If the memory is filling too fast and too frequently causing gc, then it's better idea to provision more memory.
29 Views
SigNoz Community
SigNoz is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to DataDog, NewRelic, etc.
Powered by