https://signoz.io logo
#general
Title
# general
b

Brian Richardson

10/03/2022, 12:08 PM
hello, my Signoz pods started getting evicted overnight due to low ephemeral storage. There's plenty of disk on the node, so I'm not sure what it's complaining about. I even put limits on the ephemeral storage that a pod could use, but something is still running the pods out of this storage type. How do I configure more or figure out what's eating it all up?
a

Ankit Nayan

10/03/2022, 12:09 PM
you should set retention time to your data https://signoz.io/docs/userguide/retention-period/
b

Brian Richardson

10/03/2022, 12:10 PM
it is a brand new cluster with no data...
I literally just left it running overnight with nothing feeding it, came back this morning and all the pods were evicted
a

Ankit Nayan

10/03/2022, 12:19 PM
oh..weird
can you paste the container logs/events about the error?
which ephemeral storage are you using.. might be something to do with that
b

Brian Richardson

10/03/2022, 12:38 PM
hmm I reset the cluster, going to try from scratch.. I've moved /var/lib/kubelet and /var/lib/containerd to the data volume, so let's see if that improves things
I haven't configured anything for ephemeral storage, just whatever the cluster does by default
well, it is alive again. Let's see if it lasts until tomorrow... I found this article to be helpful in regaining control of my k8s cluster:
4 Views