https://signoz.io logo
Join Slack
Powered by
Hi Team, We are evaluating SigNoz for metrics/logs...
# support
a
Hi Team, We are evaluating SigNoz for metrics/logs for kubernetes and want to compress it under 2 GB Max Usage, While we have tuned timeouts and clickhouse settings to reach within 2.5 GB, The next step is to combine/transfer signoz-k8s-infra-otel-agent into signoz-otel-collector, Has anyone tried that or has any advise on this?
We are basically thinking to migrate all recievers/processors to otel-collector
s
You are not going to reduce the memory footprint by reducing the containers. The 
*agent*
exists to collect logs on nodes and send them to main signoz.
a
Yeah i mounted the hostPath of the only node of my cluster in otel-collector and managed to send logs, but you are right it ended up taking more memory than what both agent and collector were taking combined. I thought i would save on some sort of memory overhead associated with having a separate collector. Anyways, thanks for the reply @Srikanth Chekuri, Any advice on how to further decreasing the SigNoz stack memory usage for the above described use case would be greatly appreciated!
s
What volume of data ingestion do you expect for this instance?
a
@Srikanth Chekuri I could not find a way to measure my current data ingestion volume, please let me know if there is, else here are the clickhouse table stats for logs/metrics scraping_interval for metrics is set to 15m signoz_metrics.time_series_v2 (TTL: 12 hours)
Copy code
Total rows	22406877
Total bytes	599970852
Total bytes uncompressed	14380494605
signoz_logs.logs (TTL: 5 hours)
Copy code
Total rows	261990
Total bytes	11324792
Total bytes uncompressed	359655034
Here is the current memory distribution across the stack, The memory limits have been set according to the observed spikes
Copy code
NAME                                         CPU_LIMIT   MEMORY_LIMIT
chi-signoz-clickhouse-cluster-0-0-0          600m        1320Mi
signoz-alertmanager-0                        30m         48Mi
signoz-frontend-66dc8f86b7-7vhpz             100m        30Mi
signoz-k8s-infra-otel-agent-5lmtl            50m         280Mi
signoz-kube-state-metrics-5dbbd76fb8-pktxv   10m         25Mi
signoz-otel-collector-5f86845f94-llmjt       300m        220Mi
signoz-query-service-0                       75m         260Mi
signoz-schema-migrator-upgrade-d785d         <none>      <none>
NAME                                         CPU(cores)   MEMORY(bytes)   
chi-signoz-clickhouse-cluster-0-0-0          53m          981Mi           
signoz-alertmanager-0                        1m           34Mi            
signoz-frontend-66dc8f86b7-7vhpz             1m           15Mi            
signoz-k8s-infra-otel-agent-5lmtl            49m          193Mi           
signoz-kube-state-metrics-5dbbd76fb8-pktxv   1m           20Mi            
signoz-otel-collector-5f86845f94-llmjt       7m           144Mi           
signoz-query-service-0                       3m           121Mi           
Total Memory Limit(Mi): 2183
Total Memory Usage(Mi): 1508
Total CPU Limit(m): 1165
Total CPU Usage(m): 115
s
Are you still trying to squeeze more and reduce memory of this setup?
a
@Srikanth Chekuri Yes, Not by much though, We need to have the Total Memory Limit under 1900 Mi for SIgNoz to be able to safely fit in our existing k8s clusters without affecting the application. Already playing with the following ClickHouse parameters to reduce peak memory usage, Please let me know if you feel any parameter is different than expected or if there is scope for anything other than clickhouse to be optimized in terms of reducing the peak memory usage
Copy code
profiles: 
    default/max_insert_block_size: 388449
    default/max_bytes_before_external_group_by: 150000000
    default/max_bytes_before_external_sort: 150000000 
    default/max_block_size: 1024
    default/max_download_threads: 1
    default/input_format_parallel_parsing: 0                                            
    default/output_format_parallel_formatting: 0     

  # -- Default user profile configuration for Clickhouse. !!! Please DO NOT override this !!!
  defaultProfiles:
    default/allow_experimental_window_functions: "1"
    default/allow_nondeterministic_mutations: "1"
  defaultSettings:
    format_schema_path: /etc/clickhouse-server/config.d/
    user_scripts_path: /var/lib/clickhouse/user_scripts/
    user_defined_executable_functions_config: '/etc/clickhouse-server/functions/custom-functions.xml'
    max_server_memory_usage_to_ram_ratio: 0.9
    mark_cache_size: 0
    index_mark_cache_size: 0                                                                                                                                
    uncompressed_cache_size: 0
10 Views
Open in Slack
PreviousNext
Powered by