11

I have read many links similar to my issue, but none of them were helping me to resolve the issue.

Similar Links:

  1. Failed to exec into the container due to permission issue after executing 'systemctl daemon-reload'
  2. OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
  3. CI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
  4. OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: unknown
  5. Fail to execute docker exec
  6. OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "open /proc/self/fd: no such file or directory": unknown

Problem Description:

I have created a new Kubernetes cluster using Kubespray. When I wanted to execute some commands in one of containers I faced to the following error:

Executed Command
kubectl exec -it -n rook-ceph rook-ceph-tools-68d847b88d-7kw2v -- sh
Error:

OCI runtime exec failed: exec failed: unable to start container process: open /dev/pts/1: operation not permitted: unknown command terminated with exit code 126

I have also logged in to the node, which runs the pod, and try executing the container using docker exec command, but the error was not changed.

Workarounds:

  • As I have found, the error code (126) implies that the permissions are insufficient, but I haven't faced this kind of error (like executing sh) in Docker or Kubernetes.

  • I have also checked whether SELinux is enabled or not (as it has been said in the 3rd link).

    apt install policycoreutils
sestatus
# Output
SELinux status:                 disabled

  • In the 5th link, it was said to check whether you have updated the kernel, and I didn't upgrade anything on the nodes.

    id; stat /dev/pts/0
# output
uid=0(root) gid=0(root) groups=0(root)
  File: /dev/pts/0
  Size: 0               Blocks: 0          IO Block: 1024   character special file
Device: 18h/24d Inode: 3           Links: 1     Device type: 88,0
Access: (0600/crw-------)  Uid: (    0/    root)   Gid: (    5/     tty)
Access: 2022-08-21 12:01:25.409456443 +0000
Modify: 2022-08-21 12:01:25.409456443 +0000
Change: 2022-08-21 11:54:47.474457646 +0000
 Birth: -

  • Also tried /bin/sh instead of sh or /bin/bash, but not worked and the same error occurred.

Can anyone help me to find the root cause of this problem and then solve it?

Mostafa Ghadimi
  • 5,883
  • 8
  • 64
  • 102
  • 1
    ```kubectl run debug --image=alpine -i --tty -- sh``` can you check if this command work – Adiii Aug 21 '22 at 12:42
  • @Adiii It works, but it doesn't work for executing the existing pods like `rook-ceph-tools-68d847b88d-7kw2v`, I've mentioned it here. – Mostafa Ghadimi Aug 21 '22 at 12:43
  • is there any volume mount? is the image support your platform? `k describe node node_name | grep "kubernetes.io/arch"` – Adiii Aug 21 '22 at 12:49
  • `docker inspect image my-image | grep Architecture` the node and image architecture should match, if not then you need to rebuild image – Adiii Aug 21 '22 at 12:50
  • `docker build --platform linux/x86-64 -t demo-image .` or `platform linux/amd64` – Adiii Aug 21 '22 at 12:51
  • Can you [edit] the question to include a [mcve]; what is the application code required to reproduce the issue? Do you think this is a duplicate of the [existing question you linked to](https://stackoverflow.com/questions/73379718/ci-runtime-exec-failed-exec-failed-unable-to-start-container-process-open-de)? Or do you think this is about the way you installed the cluster (in which case another site like [sf] may be more appropriate)? – David Maze Aug 21 '22 at 12:59
  • @Adiii Yes! they are the same and the architecture is `amd64`. – Mostafa Ghadimi Aug 21 '22 at 13:22
  • @DavidMaze No it is not depend on the way I have installed the cluster. I was just for describing what has happened to me. No, it's not the duplicate question, since non of them work properly and the problem is still unresolved. – Mostafa Ghadimi Aug 21 '22 at 13:27
  • Try `kubectl exec -i -n rook-ceph rook-ceph-tools-68d847b88d-7kw2v -- sh` without the -t. You might get a running shell without tty. Just test at empty prompt. E.g. `date` and `exit`. If it works, you have hit a bug in runc and have to update to v1.1.4 – D. Lohrsträter Oct 17 '22 at 13:40

1 Answers1

7

This issue may relate to docker, first drain your node.

kubectl drain <node-name>

Second, SSH to the node and restart docker service.

systemctl restart docker.service

At the end try to execute your command.

Mostafa Ghadimi
  • 5,883
  • 8
  • 64
  • 102
Xirehat
  • 1,155
  • 1
  • 8
  • 20
  • If you are using Docker Compose, bringing your containers down and back up again also solves the problem. My question is what might be causing Docker to do this? – Erich Sep 07 '22 at 14:35
  • @Erich I'm not sure this is exactly Docker related, but this solution fix his problem. – Xirehat Sep 18 '22 at 19:18
  • 2
    looks like this was a bug in `runc` 1.1.3 that has since been fixed. i upgraded containerd on my machine and haven't seen this issue since. https://github.com/opencontainers/runc/pull/3554 – Erich Sep 21 '22 at 12:41
  • @Erich Definitely not just docker. I'm getting this with Podman under runc. I'm on Almalinux 8 which has runc 1.1.3. Hopefully runc 1.1.4 shows up soon. I thought this was some bug in Cockpit's Podman plugin, but guess not.... – Ungeheuer Oct 19 '22 at 22:54