Failed to pull image "velero/velero-plugin-for-gcp:v1.1.0" while installing Velero in GKE Cluster

Question

I'm trying to install and configure Velero for kubernetes backup. I have followed the link to configure it in my GKE cluster. The installation went fine, but velero is not working.

I am using google cloud shell for running all my commands (I have installed and configured velero client in my google cloud shell)

On further inspection on velero deployment and velero pods, I found out that it is not able to pull the image from the docker repository.

kubectl get pods -n velero
NAME                      READY   STATUS              RESTARTS   AGE
velero-5489b955f6-kqb7z   0/1     Init:ErrImagePull   0          20s

Error from velero pod (kubectl describe pod) (output redacted for readability - only relevant info shown below)

    Events:
  Type     Reason     Age               From                                                  Message
  ----     ------     ----              ----                                                  -------
  Normal   Scheduled  38s               default-scheduler                                     Successfully assigned velero/velero-5489b955f6-kqb7z to gke-gke-cluster1-default-pool-a354fba3-8674
  Warning  Failed     22s               kubelet, gke-gke-cluster1-default-pool-a354fba3-8674  Failed to pull image "velero/velero-plugin-for-gcp:v1.1.0": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning  Failed     22s               kubelet, gke-gke-cluster1-default-pool-a354fba3-8674  Error: ErrImagePull
  Normal   BackOff    21s               kubelet, gke-gke-cluster1-default-pool-a354fba3-8674  Back-off pulling image "velero/velero-plugin-for-gcp:v1.1.0"
  Warning  Failed     21s               kubelet, gke-gke-cluster1-default-pool-a354fba3-8674  Error: ImagePullBackOff
  Normal   Pulling    8s (x2 over 37s)  kubelet, gke-gke-cluster1-default-pool-a354fba3-8674  Pulling image "velero/velero-plugin-for-gcp:v1.1.0"

Command used to install velero: (some of the values are given as variables)

velero install \
     --provider gcp \
     --plugins velero/velero-plugin-for-gcp:v1.1.0 \
     --bucket $storagebucket \
     --secret-file ~/velero-backup-storage-sa-key.json

Velero Version

velero version
Client:
        Version: v1.4.2
        Git commit: 56a08a4d695d893f0863f697c2f926e27d70c0c5
<error getting server version: timed out waiting for server status request to be processed>

GKE version

v1.15.12-gke.2

That seems like a firewall blocking connection to docker registry? Is it happening only with this image from docker registry? — Arghya Sadhu, Aug 14 '20 at 13:53
@ArghyaSadhu it is happening only with this image. I am able to deploy other images without any issues.. Looking at the describe pod, I can see the below: Containers: velero: Container ID: Image: velero/velero:v1.4.2 — srsn, Aug 14 '20 at 15:00
@Sreesan Other public images from docker hub registry works? — Arghya Sadhu, Aug 14 '20 at 15:01
Image: velero/velero:v1.4.2 (Is this issue because of the image url?) velero has created custom resource definitions for deploying this — srsn, Aug 14 '20 at 15:02
@mario this is a private cluster but I can deploy other services without any issues (for eg: I have deployed nginx successfully) — srsn, Aug 14 '20 at 15:03
@ArghyaSadhu yes, they works without any issues...for example, I deployed nginx without any issue using the command "kubectl create deployment nginx --image=nginx" — srsn, Aug 14 '20 at 15:05

mario · Accepted Answer · 2020-08-14T17:35:37.360

Isn't this a Private Cluster ? – mario 31 mins ago

@mario this is a private cluster but I can deploy other services without any issues (for eg: I have deployed nginx successfully) – Sreesan 15 mins ago

Well, this is a know limitation of GKE Private Clusters. As you can read in the documentation:

Can't pull image from public Docker Hub

Symptoms

A Pod running in your cluster displays a warning in kubectl describe such as Failed to pull image: rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Potential causes

Nodes in a private cluster do not have outbound access to the public internet. They have limited access to Google APIs and services, including Container Registry.

Resolution

You cannot fetch images directly from Docker Hub. Instead, use images hosted on Container Registry. Note that while Container Registry's Docker Hub mirror is accessible from a private cluster, it should not be exclusively relied upon. The mirror is only a cache, so images are periodically removed, and a private cluster is not able to fall back to Docker Hub.

You can also compare it with this answer.

It can be easily verified on your own by making a simple experiment. Try to run two different nginx deployments. First based on image nginx (which equals to nginx:latest) and the second one based on nginx:1.14.2.

While the first scenario is perfectly feasible because the nginx:latest image can be pulled from Container Registry's Docker Hub mirror which is accessible from a private cluster, any attempt of pulling nginx:1.14.2 will fail which you'll see in Pod events. It happens because the kubelet is not able to find this version of the image in GCR and it tries to pull it from public docker registry (https://registry-1.docker.io/v2/), which in Private Clusters is not possible. "The mirror is only a cache, so images are periodically removed, and a private cluster is not able to fall back to Docker Hub." - as you can read in docs.

If you still have doubts, just ssh into your node and try to run following commands:

curl https://cloud.google.com/container-registry/

curl https://registry-1.docker.io/v2/

While the first one works perfectly, the second one will eventually fail:

curl: (7) Failed to connect to registry-1.docker.io port 443: Connection timed out

Reason ? - "Nodes in a private cluster do not have outbound access to the public internet."

Solution ?

You can search what is currently available in GCR here.

In many cases you should be able to get the required image if you don't specify it's exact version (by default latest tag is used). While it can help with nginx, unfortunatelly no version of velero/velero-plugin-for-gcp is currently available in Google Container Registry's Docker Hub mirror.

Granting private nodes outbound internet access by using Cloud NAT seems the only reasonable solution that can be applied in your case.

mario thanks for the detailed explanation for the issue and the possible solution..I solved the final part of the problem (ie, using velero images in private gke clusters) without using cloud NAT. Instead I used gcr.io for storing the images. I followed the steps in given in https://velero.io/docs/main/on-premises/ to save the velero images in my private gcr (not the gcr mirror) and then used that private images to install velero in my private gke cluster — srsn, Aug 15 '20 at 11:02
You're welcome! Glad I could help. Uploading an image to private gcr is also a great solution! — mario, Aug 15 '20 at 11:48

score 0 · Answer 2 · answered Aug 24 '22 at 13:04

0

I solved this problem by realizing that version of:

velero/velero-plugin-for-gcp

is not following the version of:

velero/velero

For example, now latest versions are: velero/velero:v1.9.1 and velero/velero-plugin-for-gcp:v1.5.0

answered Aug 24 '22 at 13:04

Ivan Aracki

4,861
11
59
73

Failed to pull image "velero/velero-plugin-for-gcp:v1.1.0" while installing Velero in GKE Cluster

2 Answers2

Can't pull image from public Docker Hub

Solution ?

Linked