10

My company bought a software we're trying to deploy on IBM cloud, using kubernetes and given private docker repository. Once deployed, there is always a Kubernetes error : "Back-off restarting failed container". So I read logs in order to understand why the container is restarting and here is the error :

Caused by: java.io.FileNotFoundException: /var/yseop-log/yseop-manager.log (Permission denied)

So I deduced that I just had to change permissions in the Kubernetes file. Since I'm using a deployment, I tried the following initContainer :

initContainers:
    - name: permission-fix
      image: busybox
      command: ['sh', '-c']
      args: ['chmod -R 777 /var']
      volumeMounts:
        - mountPath: /var/yseop-engine
          name: yseop-data
        - mountPath: /var/yseop-data/yseop-manager
          name: yseop-data
        - mountPath: /var/yseop-log
          name: yseop-data

This didn't worked because I'm not allowed to execute chmod on read-only folders as non root user.

So I tried remounting those volumes, but that also failed, because I'm not a root user.

I then found out about running as User and group. In order to find out which User and group I had to write in my security context, I read the dockerfile and here is the user and group :

 USER 1001:0

So I tought I could just write this in my deployment file :

  securityContext: 
      runAsUser: 1001  
      rusAsGroup: 0

Obvisouly, that didn't worked neither, because I'm not allowed to run as group 0

So I still don't know what to do in order to properly deploy this image. The image is working when doing a docker pull and exec on m computer, but it's not working on Kubernetes.

Here is my complete Volume file :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    ibm.io/auto-create-bucket: "true"
    ibm.io/auto-delete-bucket: "false"
    ibm.io/bucket: ""
    ibm.io/secret-name: "cos-write-access"
    ibm.io/endpoint: https://s3.eu-de.cloud-object-storage.appdomain.cloud
  name: yseop-pvc
  namespace: ns
  labels:
    app: yseop-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: ibmc
  volumeMode: Filesystem

And here is my full deployment file :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: yseop-manager
  namespace: ns
spec:
  selector:
    matchLabels:
      app: yseop-manager
  template:
    metadata:
      labels:
        app: yseop-manager
    spec:
      securityContext: 
          runAsUser: 1001  
          rusAsGroup: 0
      initContainers:
        - name: permission-fix
          image: busybox
          command: ['sh', '-c']
          args: ['chmod -R 777 /var']
          volumeMounts:
            - mountPath: /var/yseop-engine
              name: yseop-data
            - mountPath: /var/yseop-data/yseop-manager
              name: yseop-data
            - mountPath: /var/yseop-log
              name: yseop-data
      containers:
        - name: yseop-manager
          image:IMAGE
          imagePullPolicy: IfNotPresent
          env:
            - name: SECURITY_USERS_DEFAULT_ENABLED
              value: "true"
          ports:
            - containerPort: 8080
          volumeMounts:
            - mountPath: /var/yseop-engine
              name: yseop-data
            - mountPath: /var/yseop-data/yseop-manager
              name: yseop-data
            - mountPath: /var/yseop-log
              name: yseop-data
      imagePullSecrets:
        - name: regcred
      volumes:
        - name: yseop-data
          persistentVolumeClaim:
            claimName: yseop-pvc

Thanks for helping

MarcoHiro
  • 128
  • 1
  • 1
  • 11
  • could you modify your busybox not to exit to early, login to it and check permissions you have? Shall be ran as root. Do you have any issues for `yseop-engine` or `yseop-manager`? – Nick Aug 04 '20 at 20:19

2 Answers2

7

Can you please try including supplementary group ID in the security context like

SecurityContext:
   runAsUser: 1001  
   fsGroup: 2000

By Default runAsGroup is 0 which is root. Below link might give more insight about this. https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

Working Yaml Content

apiVersion: apps/v1
kind: Deployment
metadata:
  name: yseop-manager
  namespace: ns
spec:
  selector:
    matchLabels:
      app: yseop-manager
  template:
    metadata:
      labels:
        app: yseop-manager
    spec:
      securityContext: 
          fsGroup: 2000 
      initContainers:
        - name: permission-fix
          image: busybox
          command: ['sh', '-c']
          args: ['chown -R root:2000 /var']
          volumeMounts:
            - mountPath: /var/yseop-engine
              name: yseop-data
            - mountPath: /var/yseop-data/yseop-manager
              name: yseop-data
            - mountPath: /var/yseop-log
              name: yseop-data
      containers:
        - name: yseop-manager
          image:IMAGE
          imagePullPolicy: IfNotPresent
          securityContext:
             runAsUser: 1001
             runAsGroup: 2000
          env:
            - name: SECURITY_USERS_DEFAULT_ENABLED
              value: "true"
          ports:
            - containerPort: 8080
          volumeMounts:
            - mountPath: /var/yseop-engine
              name: yseop-data
            - mountPath: /var/yseop-data/yseop-manager
              name: yseop-data
            - mountPath: /var/yseop-log
              name: yseop-data
      imagePullSecrets:
        - name: regcred
      volumes:
        - name: yseop-data
          persistentVolumeClaim:
            claimName: yseop-pvc
Kiruba
  • 1,322
  • 8
  • 15
  • Thanks for this reply. I tried with fsGroup = 1000 but it does not work, Initcontainer permission fix is always back-off restarting failed container. I tried to remove said initcontainer, so the pod was executed but I still have the permission denied error. That being said, the reason behind this back-off restarting is that chmod command fails on /var folder. – MarcoHiro Aug 04 '20 at 13:02
  • Can you try to execute the pod and traverse to the path and see the permission for that folder. **kubectl exec -it yseop-manager -- sh; check ls /var and ls /var/yseop-log** just to with what permission actually the folder structure has got. – Kiruba Aug 04 '20 at 14:37
  • Permissions for /var :`total 25 drwxr-xr-x 1 root root 4096 Aug 5 07:28 . drwxr-xr-x 1 root root 4096 Aug 5 07:28 .. drwxr-xr-x 3 root root 4096 Aug 5 07:28 run drwxr-xr-x 3 root root 4096 Jul 27 19:19 spool drwxr-xr-x 2 root root 4096 Jul 27 19:19 www drwxr-xr-x 3 root root 4096 Aug 5 07:28 yseop-data drwxrwxr-x 1 root root 0 Jan 1 1970 yseop-engine drwxrwxr-x 1 root root 0 Jan 1 1970 yseop-log` And permissions for /var/yseop-log `total 5 drwxrwxr-x 1 root root 0 Jan 1 1970 . drwxr-xr-x 1 root root 4096 Aug 5 07:28 ..` – MarcoHiro Aug 05 '20 at 07:38
  • I have edit the above answer and included the yaml file content which might work for you. Please do let me know if it works fine. – Kiruba Aug 05 '20 at 08:59
  • I'm now getting the follwing error : Error: container has runAsNonRoot and image will run as root. I went into logs and permission-fix failed, here is the log : Error from server (BadRequest): container "permission-fix" in pod "yseop-manager-f5b7f854c-26vcg" is waiting to start: CreateContainerConfigError – MarcoHiro Aug 05 '20 at 09:13
  • Please make sure the UID 1001 is available and also please include **runAsNonRoot: true in spec.containers.securityContext**. Mostly this error will come only when UID is not present but we can try enforcing to run as not root user by setting runAsNonRoot: true value, – Kiruba Aug 05 '20 at 09:35
  • So I managed to fix that. I was not told that my company put restrictive PSP, thus there is no way I would be able to directly write in volumes. However, I could just mount emptydir as volumes and then create a PVC that would store those data. – MarcoHiro Aug 06 '20 at 08:14
2

I was not told by my company that we do have restrictives Pod Security Policies. Because of that, volumes are Read-only and there is no way I could have written anything in said volumes.

The solution is as follow :

      volumes:
    - name: yseop-data
      emptyDir: {}

Then, I have to specify a path in volumeMounts (Which was already done) and create a PVC, so my Data would be persistent.

MarcoHiro
  • 128
  • 1
  • 1
  • 11