hello, my Signoz pods started getting evicted over...
# generalb
Brian Richardson
10/03/2022, 12:08 PMhello, my Signoz pods started getting evicted overnight due to low ephemeral storage. There's plenty of disk on the node, so I'm not sure what it's complaining about. I even put limits on the ephemeral storage that a pod could use, but something is still running the pods out of this storage type. How do I configure more or figure out what's eating it all up?
a
Ankit Nayan
10/03/2022, 12:09 PM
you should set retention time to your data https://signoz.io/docs/userguide/retention-period/
b
Brian Richardson
10/03/2022, 12:10 PMit is a brand new cluster with no data...
Brian Richardson
10/03/2022, 12:10 PMI literally just left it running overnight with nothing feeding it, came back this morning and all the pods were evicted
a
Ankit Nayan
10/03/2022, 12:19 PM
oh..weird
Ankit Nayan
10/03/2022, 12:19 PM
can you paste the container logs/events about the error?
Ankit Nayan
10/03/2022, 12:20 PM
which ephemeral storage are you using.. might be something to do with that
b
Brian Richardson
10/03/2022, 12:38 PMhmm I reset the cluster, going to try from scratch.. I've moved /var/lib/kubelet and /var/lib/containerd to the data volume, so let's see if that improves things
Brian Richardson
10/03/2022, 12:39 PMI haven't configured anything for ephemeral storage, just whatever the cluster does by default
Brian Richardson
10/03/2022, 1:14 PMwell, it is alive again. Let's see if it lasts until tomorrow... I found this article to be helpful in regaining control of my k8s cluster:
Brian Richardson
10/03/2022, 1:14 PM5 Views