How do I force Kubernetes to re-pull an image?

Question

I have the following replication controller in Kubernetes on GKE:

apiVersion: v1
kind: ReplicationController
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  replicas: 2
  selector:
    app: myapp
    deployment: initial
  template:
    metadata:
      labels:
        app: myapp
        deployment: initial
    spec:
      containers:
      - name: myapp
        image: myregistry.com/myapp:5c3dda6b
        ports:
        - containerPort: 80
      imagePullPolicy: Always
      imagePullSecrets:
        - name: myregistry.com-registry-key

Now, if I say

kubectl rolling-update myapp --image=us.gcr.io/project-107012/myapp:5c3dda6b

the rolling update is performed, but no re-pull. Why?

Answer 1

Kubernetes will pull upon Pod creation if either (see updating-images doc):

• Using images tagged :latest
• imagePullPolicy: Always is specified

This is great if you want to always pull. But what if you want to do it on demand: For example, if you want to use some-public-image:latest but only want to pull a newer version manually when you ask for it. You can currently:

• Set imagePullPolicy to IfNotPresent or Never and pre-pull: Pull manually images on each cluster node so the latest is cached, then do a kubectl rolling-update or similar to restart Pods (ugly easily broken hack!)
• Temporarily change imagePullPolicy, do a kubectl apply, restart the pod (e.g. kubectl rolling-update), revert imagePullPolicy, redo a kubectl apply (ugly!)
• Pull and push some-public-image:latest to your private repository and do a kubectl rolling-update (heavy!)

No good solution for on-demand pull. If that changes, please comment; I'll update this answer.

Answer 2

One has to group imagePullPolicy inside the container data instead of inside the spec data. However, I filed an issue about this because I find it odd. Besides, there is no error message.

So, this spec snippet works:

spec:
  containers:
  - name: myapp
    image: myregistry.com/myapp:5c3dda6b
    ports:
    - containerPort: 80
    imagePullPolicy: Always
  imagePullSecrets:
    - name: myregistry.com-registry-key

Answer 3

There is a comand to directly do that:

Create a new kubectl rollout restart command that does a rolling restart of a deployment.

The pull request got merged. It is part of the version 1.15 (changelog) or higher.

Answer 4

My hack during development is to change my Deployment manifest to add the latest tag and always pull like so

image: etoews/my-image:latest
imagePullPolicy: Always

Then I delete the pod manually

kubectl delete pod my-app-3498980157-2zxhd

Because it's a Deployment, Kubernetes will automatically recreate the pod and pull the latest image.

Answer 5

A popular workaround is to patch the deployment with a dummy annotation (or label):

kubectl patch deployment <name> -p \
  "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

Assuming your deployment meets these requirements, this will cause K8s to pull any new image and redeploy.

Answer 6

Now, the command kubectl rollout restart deploy YOUR-DEPLOYMENT combined with a imagePullPolicy: Always policy will allow you to restart all your pods with a latest version of your image.

Answer 7

1. Specify strategy as:

  strategy: 
    type: Recreate
    rollingUpdate: null

2. Make sure you have different annotation for each deployment. Helm does it like:

  template:
    metadata:
      labels:
        app.kubernetes.io/name: AppName
        app.kubernetes.io/instance: ReleaseName
      annotations:
        rollme: {{ randAlphaNum 5 | quote }}

3. Specify image pull policy - Always

      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: Always

Answer 8

# Linux

kubectl patch deployment <name> -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

# windows

kubectl patch deployment <name> -p (-join("{\""spec\"":{\""template\"":{\""metadata\"":{\""annotations\"":{\""date\"":\""" , $(Get-Date -Format o).replace(':','-').replace('+','_') , "\""}}}}}"))

Answer 9

This answer aims to force an image pull in a situation where your node has already downloaded an image with the same name, therefore even though you push a new image to container registry, when you spin up some pods, your pod says "image already present".

For a case in Azure Container Registry (probably AWS and GCP also provides this):

1. You can look to your Azure Container Registry and by checking the manifest creation date you can identify what image is the most recent one.

2. Then, copy its digest hash (which has a format of sha256:xxx...xxx).

3. You can scale down your current replica by running command below. Note that this will obviously stop your container and cause downtime.

kubectl scale --replicas=0 deployment <deployment-name> -n <namespace-name>

4. Then you can get the copy of the deployment.yaml by running:

kubectl get deployments.apps <deployment-name> -o yaml > deployment.yaml

5. Then change the line with image field from <image-name>:<tag> to <image-name>@sha256:xxx...xxx, save the file.

6. Now you can scale up your replicas again. New image will be pulled with its unique digest.

Note: It is assumed that, imagePullPolicy: Always field is present in the container.

Answer 10

Having gone through all the other answers and not being satisfied, I found much better solution here: https://cloud.google.com/kubernetes-engine/docs/how-to/updating-apps

It works without using latest tag or imagePullPolicy: Always. It also works if you push new image to the same tag by specifying image sha256 digest.

Steps:

1. get image SHA256 from docker hub (see image below)
2. find your deployment using kubectl get deployments
3. kubectl set image deployment/<your-deployment> <your_container_name>=<some/image>@sha256:<your sha>
4. kubectl scale deployment <your-deployment>--replicas=0
5. kubectl scale deployment <your-deployment>--replicas=original replicas count

Note: Rollout might also work instead of scale but in my case we don't have enough hardware resources to create another instance and k8s gets stuck.

Answer 11

Apparently now when you run a rolling-update with the --image argument the same as the existing container image, you must also specify an --image-pull-policy. The following command should force a pull of the image when it is the same as the container image:

kubectl rolling-update myapp --image=us.gcr.io/project-107012/myapp:5c3dda6b --image-pull-policy Always

Answer 12

The rolling update command, when given an image argument, assumes that the image is different than what currently exists in the replication controller.

Answer 13

You can define imagePullPolicy: Always in your deployment file.

Answer 14

I have used kubectl rollout restart for my springboot api and it works.

kubectl rollout restart -f pod-staging.yml --namespace test

Yaml for the Deployment:

apiVersion: "apps/v1"
kind: "Deployment"
metadata:
    name: "my-api"
    labels:
      app: "my-api"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "my-api"
  template:
    metadata:
      labels:
        app: "my-api"
    spec:
      containers:
        - name: my-api
          image: harbor.url.com/mycompany/my-api:staging
          ports:
            - containerPort: 8099
              protocol: TCP
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8099
            initialDelaySeconds: 90
            periodSeconds: 10
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8099
            initialDelaySeconds: 90
            periodSeconds: 5
          envFrom:
            - configMapRef:
                name: "my-api-configmap"
          env:
            - name: "TOKEN_VALUE"
              valueFrom:
                secretKeyRef:
                  name: "my-api-secret"
                  key: "TOKEN_VALUE"
          resources:
            requests:
              memory: "512Mi"
              cpu: "500m"
            limits:
              memory: "2048Mi"
              cpu: "1000m"
      imagePullSecrets:
        - name: "my-ci-user"

Answer 15

The Image pull policy will always actually help to pull the image every single time a new pod is created (this can be in any case like scaling the replicas, or pod dies and new pod is created)

But if you want to update the image of the current running pod, deployment is the best way. It leaves you flawless update without any problem (mainly when you have a persistent volume attached to the pod) :)

Answer 16

The below solved my problem:

kubectl rollout restart

Answer 17

if you want to perform a direct image update on a specific pod, you can use kubectl set image also.

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

Answer 18

either you have deleted all the pods manually to get it recreated with pulling the image again.

or

run this below command kubectl rollout restart deployment/deployment_name kubectl rollout restart deployment/nginx

this command should recreate all the pods.

for both scenarios imagepullPolicy should be set as Always.

Answer 19

A one-liner solution based on invalidating Deployment hash by adding some new unique data, here: a timestamp-based environment variable (just like adding a "volatile" ENV to bust docker cache during image builds):

kubectl set env deployment/nginx REDEPLOY_TIME="$(date)"

or when using oc Client Tools under OCP/OKD:

oc set env dc/nginx REDEPLOY_TIME="$(date)"

It will trigger an automatic rolling re-deployment/re-pull even in older installations of k8s (not just in v1.15 or above, where kubectl rollout restart is the correct solution as described in this answer). In fact I verified this workaround even in the archaic Openshift 3.11 based on k8s 1.11 from mid-2018!

Note we need the usual prerequisites of imagePullPolicy: Always and a "rolling" container image tag such as latest.

Note: kudos and the original idea (using a YAML Deployment manifest file and sed) go back to this comment in the rather long-running k8s issue devoted to this now thankfully gone opinionated choice made initially by k8s devs.