Hi team 👋, Self Hosted in Kubernets ( helm chart ) I have an issue with the default "Infra Monitoring" in SigNoz. My problem is: The memory and CPU utilization shown by SigNoz is higher (e.g. Memory ~85%) compared to what I get from Kubernetes directly using kubectl top nodes (Memory ~55%). I've already configured my OpenTelemetry Collector to exclude unnecessary metrics (cached, buffers, slab, allocatable) and cleared old metrics from ClickHouse. However, the built-in Infrastructure → Infra Monitoring doesn't match these metrics and I can't change there through the UI. Could you please help me understand: How can I modify or override the default used in the built-in Infra Monitoring ? Or how can I achieve that the built-in Infra Monitoring reflects exactly the same values as the ones I see with kubectl top nodes? Thanks in advance for your help! 🙏

i have the same problem here but is more with memory

Please review this issue. It is very critical for our work.

@Andrii Sereda i think signoz monitoring graph counts buffer/cache memory as "used" can you share output of free -h command in the server?

@Abhay Unfortunately you won't see anything new - the result is the same as "top" if as you say it's "buffer/cache memory" can it be excluded from the collection? If in fact it's a lie and I need to see the actual data..

Hi, two different things are being compared here. The k8s memory metrics are designed to better represent the actual memory pressure on nodes for scheduling and resource management purposes, while host OS metrics provide a more direct view of physical memory allocation. The host section of infra monitoring shows the memory usage from the OS perspective and indeed includes the cache memory as mentioned by @Abhay. They will never exactly match because k8s uses a different heuristic to show the usage that you get with the "kubectl top" command. The apples-to-apples comparison is to go to the nodes tab in the k8s infra monitoring, and see the used memory and allocatable memory from k8s.

@Andrii Sereda what you have shared is not same as top it is total memory requested by all the pod running on this node, not the actual usage. what signoz infra shows is actual usage, comparison of both will always be different. can compare output of below command with signoz data :

kubectl top node

I wouldn't recommend excluding certain states from host memory usage metric in an attempt to make it match with the top from k8s metrics server. One is from host OS and another is from k8s metrics server https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md#how-memory-usage-is-calculated

@Srikanth Chekuri I understand, but I need it because other data is of no use - in our case.

What other data are you referreing to? The memory usage differs with different states?

@Abhay That's the problem. I need the actual display as in "kubectl top node" - "Infra Monitoring" Because this is the only thing that matters to me, other information doesn't give me any benefit. That's why I'm asking if it's possible to disable other metrics in "Infra Monitoring" via the helm chart or somehow. So that it would be as convenient as possible for us.

I want these metrics to be transferred to Signoz without previous data.

Just to make sure I follow you correctly, you are saying you are looking for SigNoz to show top view as if were to show by lens or kubectl?

@Andrii Sereda try importing signoz dashboard for k8s node metrics : https://github.com/SigNoz/dashboards/blob/main/k8s-infra-metrics/kubernetes-node-metrics-detailed.json

I am going to summarise the points 1. Host metrics are different from k8s metrics and won't match exactly. It will do no good for you to exclude certain states from system_memory_usage 2. If you're looking for similar values as k8s top X, use the k8s section of infra monitoring. 3. The node metrics dashboard shared above is nothing different from the k8s node monitoring, they are the same 4. The data is collected at an interval of 30-60s using the kubelet receiver, which means data is a minute or so behind.

let me explain again. @Abhay @Srikanth Chekuri Please I have cluster metrics: "kubectl top node" (or Lens) They display actual data at the moment. signoz - at the moment displays other data in "Infra Monitoring" that do not match the actual data. I know that you can use other dashboards - and customize the display. The question is - how to make it so that in "Infra Monitoring" I need to see actual data as "kubectl top node" or "lens" and not a full display of those parameters that are not needed (only the actual data in "Infra Monitoring" is needed)! for ease of use and alert settings.

how to make it so that in "Infra Monitoring" I need to see actual data as "kubectl top node"

There are two sections, Hosts and K8s, use the K8s nodes part because they are the metrics reported kubelet https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/documentation.md

@Srikanth Chekuri Once again I show that the data is somehow different! Here are the screenshots - the same node - but the data is different. For some reason signoz displays the wrong data. I compared all the winds on the services Cloud Watch 9.8 Lens 9.8 kubectl top node 9.8 signoz 12.06

open the detailed view and show memory usage chart data.

@Srikanth Chekuri Only signoz "Allocatable Capacity" - shows it as already used memory by default - and all others show only "Usage" Is it possible to disable the display of "Allocatable Capacity" in "Infra Monitoring" so that the dashboards match and are correct

> Only signoz "Allocatable Capacity" - shows it as already used memory by default How did you arrive at this interpretation? It clearly says "allocatable". > Is it possible to disable the display of "Allocatable Capacity" in "Infra Monitoring" On the listing page, it is already shown

Okey. Then I don't understand what the factual difference is. @Srikanth Chekuri