Kubernetes job update with helm

Question

I have a Kubernetes cluster and I'm deploying my app there with Helm. Everything works fine, but one aspect, the Job update. As I've read, the Jobs are immutable and that's why they can't be updated, but I don't get, why is helm not creating a new job as it does for the Pods?

In the end, I want to achieve that my app code is deployed as a job, that runs DB migrations. I tried to do it as a Pod, but for pods, the restart policy can be only "Always", "never" is not supported, even though the doc says otherwise. How can I achieve this, so the Migration can be updated with every deployment (new image tag) and it runs once and not restarts?

Answer 1

You can use helm hooks here. Official Link: https://helm.sh/docs/topics/charts_hooks/

Once job is completed with "helm install", helm hook should delete it. Once you perform "helm upgrade", a new job should be triggered. Application logic should handle install and upgrade scenarios.

Below are some concepts related to helm hooks.

Types of Helm Hooks

• pre-install : hooks run after templates are rendered and before any resources are created in a Kubernetes cluster
• post-install : hooks run after all Kubernetes resources have been loaded
• pre-delete : hooks run before any existing resources are deleted from Kubernetes
• post-delete : hooks run after all Kubernetes resources have been deleted
• pre-upgrade : hooks run after chart templates have been rendered and before any resources are loaded into Kubernetes
• post-upgrade : hooks run after all Kubernetes resources have been upgraded
• pre-rollback : hooks run after templates have been rendered and before any resources are rolled back
• post-rollback : hooks run after all resources have been modified
• test : hooks run when helm test subcommand is executed

NOTE: One resource can implement multiple hooks:

Eg: annotations: "helm.sh/hook": post-install,post-upgrade

How Helm Chart Hooks Are Executed

• When a Helm chart containing hooks is executed, components like pods or jobs pertaining to hooks are not directly applied in a Kubernetes environment. Instead when a hook is executed, a new pod is created corresponding to the hook. If successfully run, they will be in "Completed" state.
• Any resources created by a Helm hook are un-managed Kubernetes objects. In other words, uninstalling a Helm chart using "helm uninstall" will not remove the underlying resources created by hooks. A separate deletion policy needs to be defined in the form of annotation if those resources need to be deleted.
• Any hook resources that must never be deleted should be annotated with "helm.sh/resource-policy: keep".

Helm Hook Annotations

• "helm.sh/hook": post-install
• "helm.sh/hook-weight": "-5" ## NOTE: This MUST be string
• "helm.sh/hook-delete-policy": hook-succeeded
• "helm.sh/resource-policy": keep

Hook Deletion Policies

• “helm.sh/hook-delete-policy" annotation to be used.

Three different deletion policies are supported which will decide when to delete the resources:

• before-hook-creation : Delete the previous resource before a new hook is launched
• hook-succeeded : Delete the resource after the hook is successfully executed
• hook-failed : Delete the resource if the hook failed during execution

NOTE: If no hook deletion policy annotation is specified, the before-hook-creation behavior is applied by default.

Hook Weights

• "helm.sh/hook-weight" annotation to be used.
• Hook weights can be positive or negative numbers but must be represented as strings.
• When Helm starts the execution cycle of hooks of a particular Kind it will sort those hooks in ascending order.

Hook weights ensure below:

• execute in the right weight sequence
• block each other
• all block main K8s resource from starting

Complete Execution Flow Example

1. Step-1: Create post-install and post-install hook YAML files

---

pre-install.yaml

apiVersion: v1
 kind: Pod
 metadata:
   name: hook-preinstall
   annotations:
     "helm.sh/hook": "pre-install"          ## Without this line, this becomes a normal K8s resource.
 spec:
   containers:
   - name: hook1-container
     image: busybox
     imagePullPolicy: IfNotPresent
     command: ['sh', '-c', 'echo The pre-install hook Pod is running  - hook-preinstall && sleep 15']
   restartPolicy: Never
   terminationGracePeriodSeconds: 0

---

post-install.yaml

apiVersion: v1
 kind: Pod
 metadata:
   name: hook-postinstall
   annotations:
     "helm.sh/hook": "post-install"         ## Without this line, this becomes a normal K8s resource.
 spec:
   containers:
   - name: hook2-container
     image: busybox
     imagePullPolicy: IfNotPresent
     command: ['sh', '-c', 'echo post-install hook Pod is running - hook-postinstall && sleep 10']
   restartPolicy: Never
   terminationGracePeriodSeconds: 0

---

2. Step-2: Install Helm Chart (Assuming other K8s resources are defined under /templates/ directory)

---

3. Get Pods:

---

$ kubectl get pods
NAME                                 READY   STATUS      RESTARTS   AGE
demohook-testhook-5ff88bb44b-qc4n2   1/1     Running     0          5m45s
hook-postinstall                     0/1     Completed   0          5m45s
hook-preinstall                      0/1     Completed   0          6m2s
$

---

4. Describe Pods and notice Started & Finished time of the pods:

---

$ kubectl describe pod demohook-testhook-5ff88bb44b-qc4n2 | grep -E 'Anno|Started:|Finished:'
$ kubectl describe pod hook-postinstall | grep -E 'Anno|Started:|Finished:'              
$ kubectl describe pod hook-preinstall | grep -E 'Anno|Started:|Finished:'

---