Kubernetes - pods share a specific directory on NFS with mountOptions

Question

I'm am trying to set up a kubernetes service where multiple pods share the same directory on an NFS volume. The NFS volume has a bunch of data pre-generated outside kubernetes in a specific directory for the pods to both read & write.

I can think of two ways to try this, neither of which are working properly. The way we are currently doing it is by using the NFS option for creating a volume directly in the pod like this:

kind: Deployment
spec:
  template:
    spec:
      containers:
         ...
      volumes:
        - name: embeddings
          nfs:
            path: /data/embeddings
            server: fs-blahblah.efs.us-west-2.amazonaws.com

This mostly works. All the pods see the same directory on NFS, which is pre-populated with data, and they can all access it. But AFAIK that there is no way to specify NFS mount options. The only other allowed option is readonly. ref and we need to tune some things in the NFS client like noac.

The other way I can think of is to use a PersistentVolume. If you create a PV using NFS you can specify the mount options:

kind: PersistentVolume
spec:
  mountOptions:
    - nfsvers=4.1
    - noac

But I don't know how to get a pod to access a specific directory within a PV. My understanding is this isn't possible - because pods can't mount PV's they can only mount PVC's, and AFAIK you can't pick a specific directory for a PVC, nor have multiple pods share a PVC which is what I specifically want. Plus the semantics for PVC just seem wrong - I don't want k8s to put storage limits on this drive, I want all the pods to use all the space on the disk they want/need. But maybe it can work anyway?

So, how can I get multiple pods to access the same specific directory on an NFS volume while specifying mountOptions?

Answer 1

NFS volume support the access mode ReadWriteMany. Meaning you can mount it multiple times.

Since a claim only tries to find a volume that matches its spec, you still mount the volume and not the claim. The claim is just a way to express how the volume, you want to mount, should look like.
Perhaps some cluster admin has already created volumes which you could claim, if they match your criteria. Or you let a storage class create the volume based on the claim.
In the below example, the spec explicitly asks for a volume with that name, so in that case no other volume could match, even if it had all the other criteria given.
You could further strengthen that bond by using a claimRef on the volume, which would prevent any other claim, than the referenced one, to claim the volume.

You are right, in that you have to go the volume + claim way, if you want to specify mount options for the NFS.

Note that the capacity isn't actually doing anything in this example, volumes for pre-existing NFS have the capacity of the actual NFS and not what you specify. You still need to provide some capacity in your manifest, as it's required by Kubernetes. It can be some arbitrary value, it doesn't matter.

Another, thing is the empty storage class. This is because you don't want to create the NFS via a storage class, but use a pre created NFS outside Kubernetes. If you don't give a storage class, the default class will be used, which is not what you want. Hence, the empty string is required.

Regarding the path, if you want to mount different path inside the NFS to different pods or the same NFS under different path to the same pod to different locations, you can use a subPath. Although it doesn't seem to be required from your question.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-nfs-volume
spec:
  persistentVolumeReclaimPolicy: Retain
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    path: /data/embeddings
    server: fs-blahblah.efs.us-west-2.amazonaws.com
  mountOptions:
    - vers=4
    - minorversion=1
    - noac
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-nfs-claim
spec:
  volumeName: my-nfs-volume
  storageClassName: ""
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mounter-fleet
spec:
  replicas: 5
  selector:
    matchLabels:
      app: mounter-fleet
  template:
    metadata:
      labels:
        app: mounter-fleet
    spec:
      containers:
        - name: mounter
          image: busybox
          command: ["sleep", "infinity"]
          volumeMounts:
            - name: nfs
              mountPath: /mnt/nfs
      volumes:
        - name: nfs
          persistentVolumeClaim:
            claimName: my-nfs-claim