Restart a Kubernetes Job or Pod with a different command

Question

I'm looking for a way to quickly run/restart a Job/Pod from the command line and override the command to be executed in the created container.

For context, I have a Kubernetes Job that gets executed as a part of our deploy process. Sometimes that Job crashes and I need to run certain commands inside the container the Job creates to debug and fix the problem (subsequent Jobs then succeed).

The way I have done this so far is:

• Copy the YAML of the Job, save into a file
• Clean up the YAML (delete Kubernetes-managed fields)
• Change the command: field to tail -f /dev/null (so that the container stays alive)
• kubectl apply -f job.yaml && kubectl get all && kubectl exec -ti pod/foobar bash
• Run commands inside the container
• kubectl delete job/foobar when I am done

This is very tedious. I am looking for a way to do something like the following

kubectl restart job/foobar --command "tail -f /dev/null"

# or even better
kubectl run job/foobar --exec --interactive bash

---

I cannot use the run command to create a Pod:

kubectl run --image xxx -ti

because the Job I am trying to restart has certain volumeMounts and other configuration I need to reuse. So I would need something like kubectl run --from-config job/foobar.

---

Is there a way to achieve this or am I stuck with juggling the YAML definition file?

---

Edit: the Job YAML looks approx. like this:

apiVersion: batch/v1
kind: Job
metadata:
    name: database-migrations
    labels:
        app: myapp
        service: myapp-database-migrations
spec:
    backoffLimit: 0
    template:
        metadata:
            labels:
                app: myapp
                service: myapp-database-migrations
        spec:
            restartPolicy: Never
            containers:
                - name: migrations
                  image: registry.example.com/myapp:977b44c9
                  command:
                      - "bash"
                      - "-c"
                      - |
                          set -e -E
                          echo "Running database migrations..."
                          do-migration-stuff-here
                          echo "Migrations finished at $(date)"
                  imagePullPolicy: Always
                  volumeMounts:
                      -   mountPath: /home/example/myapp/app/config/conf.yml
                          name: myapp-config-volume
                          subPath: conf.yml
                      -   mountPath: /home/example/myapp/.env
                          name: myapp-config-volume
                          subPath: .env
            volumes:
                - name: myapp-config-volume
                  configMap:
                      name: myapp
            imagePullSecrets:
                -   name: k8s-pull-project

Answer 1

The commands you suggested don't exist. Take a look at this reference where you can find all available commands.

Based on that documentation the task of the Job is to create one or more Pods and continue retrying execution them until the specified number of successfully terminated ones will be achieved. Then the Job tracks the successful completions. You cannot just update the Job because these fields are not updatable. To do what's you want you should delete current job and create one once again.

---

I recommend you to keep all your configurations in files. If you have a problem with configuring job commands, practice says that you should modify these settings in yaml and apply to the cluster - if your deployment crashes - by storing the configuration in files, you have a backup.

If you are interested how to improve this task, you can try those 2 examples describe below:

Firstly I've created several files:

example job (job.yaml):

apiVersion: batch/v1
kind: Job
metadata:
  name: test1
spec:
  template:
    spec:
      containers:
      - name: test1
        image: busybox
        command: ["/bin/sh", "-c", "sleep 300"]
        volumeMounts:
        - name: foo
          mountPath: "/script/foo"
      volumes:
      - name: foo
        configMap:
          name: my-conf
          defaultMode: 0755
      restartPolicy: OnFailure

patch-file.yaml:

spec:
  template:
    spec:
      containers:
      - name: test1
        image: busybox
        command: ["/bin/sh", "-c", "echo 'patching test' && sleep 500"]

and configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-conf
data:
  test: |
    #!/bin/sh
    echo "skrypt test"

---

1. If you want to automate this process you can use plugin

A plugin is a standalone executable file, whose name begins with kubectl-. To install a plugin, move its executable file to anywhere on your PATH.

There is no plugin installation or pre-loading required. Plugin executables receive the inherited environment from the kubectl binary. A plugin determines which command path it wishes to implement based on its name.

Here is the file that can replace your job

A plugin determines the command path that it will implement based on its filename.

kubectl-job:

#!/bin/bash
kubectl patch -f job.yaml -p "$(cat patch-job.yaml)" --dry-run=client -o yaml | kubectl replace --force -f - && kubectl wait --for=condition=ready pod -l job-name=test1 && kubectl exec -it $(kubectl get pod -l job-name=test1 --no-headers -o custom-columns=":metadata.name") -- /bin/sh

This command uses an additional file (patch-job.yaml, see this link) - within we can put our changes for job.

Then you should change the permissions of this file and move it:

sudo chmod +x .kubectl-job
sudo mv ./kubectl-job /usr/local/bin

It's all done. Right now you can use it.

$ kubectl job
job.batch "test1" deleted
job.batch/test1 replaced
pod/test1-bdxtm condition met
pod/test1-nh2pv condition met
/ #

As you can see Job has been replaced (deleted and created).

---

2. You can also use single-line command, here is the example:

kubectl get job test1 -o json | jq "del(.spec.selector)" | jq "del(.spec.template.metadata.labels)" | kubectl patch -f - --patch '{"spec":  {"template":  {"spec":  {"containers": [{"name": "test1", "image": "busybox", "command": ["/bin/sh", "-c",  "sleep 200"]}]}}}}' --dry-run=client -o yaml | kubectl replace --force -f -

With this command you can change your job entering parameters "by hand". Here is the output:

job.batch "test1" deleted
job.batch/test1 replaced

As you can see this solution works as well.