Hi everyone! I need some help understand Metrics in Signoz, or OTEL to be more specific. We have EKS cluster which is being monitored by Signoz (logs/traces/metrics) and I try to show the memory utilization of out nodes on agraph. I've used k8s_node_memory_usage - but the results I get are way off the values I get when executing

kubectl top nodes

. Any idea why I get different results? Also, is there a document with better explenation regarding each metric? Even CPU utilization in Signoz seems to be far from the real utilization when watching with 'kubectl top nodes' or in EC2 dashboard.

What is the query? Or do you use some existing dashboard? We use these ourselves and they are correct https://github.com/SigNoz/dashboards/tree/main/k8s-infra-metrics.

this is just great.. Thank you for sharing this as I can see now what was wrong with my settings. while I was using

k8s_node_memory_usage

it looks like what I was looking for is

k8s_node_memory_working_set

also instead of

k8s_node_memory_available

i should have used

k8s_node_allocatable_memory

Is there any valid documentation to describe the meaning of each metric?

The metrics documentation is available here https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/documentation.md

Thanks for you help! I'll look into it

Which part is not clear?

what is usage an what is working set for example I would assume that usage is what i will get performing

kubectl top nodes

Usage is total memory used (this includes the cache such as page cache etc...); Some part of this memory can be evicted when the kernel gets a request from another process for memory. The working set is the amount of memory required for the process to run. working <= usage.

thanks for this detailed explanation