7

I'm trying to copy a local Python file to a running container on Kubernetes and it fails:

$ kubectl cp /path/to/file.py namespace/pod:/path/in/container/file.py
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
command terminated with exit code 2

I know the tar binary must be available in the container, and it is.

Does anyone know what's going on here and how I can solve this?

UPDATE:

After some more testing I can confirm that this only happens on nodes that run nvidia-docker rather than the normal docker. When piping things into kubectl exec on these nodes the stream is always empty.

So the following command yields an empty file in the pod running on a GPU-enabled node while the file is non-empty on other nodes without GPU support:

cat nonempty_file.txt | kubectl exec -i pod -- tee /home/jovyan/empty_file.txt

This has been tested using the exact same image/container on both nodes.

Simon G.
  • 441
  • 8
  • 10
  • can you try with this I solved mine error with go program https://stackoverflow.com/a/55901093/3053228 – prashant Oct 21 '19 at 04:41
  • 2
    The fact this is working on one node and not the other node sounds very very fishy. Can you double check that the container is indeed the same when you run copy on each and update your question showing this? – Dandy Oct 21 '19 at 05:03
  • Also, watch out for the version spread between your `kubectl` binary and the server, since [the version difference is actually constrained](https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl) – mdaniel Oct 21 '19 at 06:20
  • I actually updated `kubectl` while trying to fix this issue. The minor version of the client was 14 before and it gave me the same error. – Simon G. Oct 21 '19 at 06:23
  • could you try this one: `cat /path/to/file.py | kubectl exec -it pod -- tee path/in/container/file.py 2>/dev/null` – Mark Oct 21 '19 at 11:27
  • @Hanx that command fails with "Unable to use a TTY - input is not a terminal or the right kind of file" – Simon G. Oct 23 '19 at 23:48
  • 1. If you are using some official image please provide more infomration about the image. 2. You can use [ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#create-a-configmap) and apply your script into the POD. 3. You can use [volume](https://kubernetes.io/docs/concepts/storage/volumes/) and do the same. 4. You can add your file in docker file during build stage. – Mark Oct 24 '19 at 06:03
  • Turns out it actually works on one node and not the other - using the same image. The difference is that the node it isn't working on is running nvidia-docker, which I suspect is the culprit here. I've updated the original question accordingly. – Simon G. Nov 01 '19 at 01:11
  • I can't test it but could you please try as workaround and use configmap or volumes instead for this specific task as described above? – Mark Nov 04 '19 at 12:12
  • Is there any update on this? I can't think of a reason why kubectl cp would break for nvidia-docker containers. Thanks for the effort you've all put into debugging this so far. I'm on 1.14.9 client and EKS 1.14.9+. – Thomas Havlik Jan 16 '20 at 15:41
  • UPDATE: `kubectl cp` into pod does not seem to work at all with the `AL2_x86_64_GPU` AMI type. Works fine with `AL2_x86_64`. This issue is probably also related to https://github.com/NVIDIA/nvidia-docker/issues/1091 – Thomas Havlik Jan 16 '20 at 16:55
  • Issue persists, so I've reported it to AWS as well: https://github.com/awslabs/amazon-eks-ami/issues/453. Interestingly Azure doesn't have this problem. – Simon G. Apr 20 '20 at 15:04

2 Answers2

1

Problem is solved by updating EKS AMI version.

Please install new release: eks-ami-releases.

See: eks-ami-kubectl.

Malgorzata
  • 6,409
  • 1
  • 10
  • 27
-1

Please take a look for the docs:

  • Supported releases and component skew,

    Nodes may lag masters components by up to two minor versions but should be at a version no newer than the master; a client should be skewed no more than one minor version from the master, but may lead the master by up to one minor version. For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and should work with v1.2, v1.3, and v1.4 clients.

  • other github issues,

  • k8s docs,

kubectl is supported within one minor version (older or newer) of kube-apiserver.

  • v1.16.0 release

    kubectl cp no longer supports copying symbolic links from containers; to support this use case, see kubectl cp --help for examples using tar directly opy files and directories to and from containers.

Examples: # !!!Important Note!!! # Requires that the 'tar' binary is present in your container # image. If 'tar' is not present, 'kubectl cp' will fail.

Please try and install appropriate kubectl version.

Hope this help

Mark
  • 3,644
  • 6
  • 23
  • 1
    As noted in the comments to the original question this issue is not due to the version difference. I had the matching version of `kubectl` installed when I first experienced the issue and subsequently updated `kubectl` in an attempt to fix my problem. – Simon G. Oct 25 '19 at 07:23
  • Did you try this [solution](https://stackoverflow.com/questions/39886146/kubectl-attach-unable-to-use-a-tty-container-es-node-did-not-allocate-one) – Mark Oct 28 '19 at 08:53