ah, ok I've sort of proven that now. Redirecting the query service to use a hostPath temporarily for the signoz.db file, the query service comes up. Anyone have any idea why I cannot use an NFS provision for signoz.db file without getting: ? The issue is definitely NFS related
We've worked around it. There are issues relating to any sort of file-share mounting that relates to the uid/gid that each respective component is using. zookeeper pod is one example that prevented me from mounting a share on the K8s node and using hostpath provision for it.
However, all pods work fine on hostpath, as long as you don't try to mount it on a fileshare, as long as we modify the various place where the securityContext is specifying different uid's that don't resolve all of the permissions issues.
In the end, we worked around it by either altering the gid/uids, and not mounting. Just leaving on the hostpath. Create a cron job to back up every 15 minutes or so to the share
Strange thing about zookeeper was that even if you alter its securityContext to runas root.....it will still write some of its data to the volume as uid 1000