91

I tried to run a Helm upgrade before running helm repo update and now it seems to be permanently stuck in "STATUS: pending-upgrade" and won't let me try to run the upgrade again.

Trying to run: helm upgrade --namespace coder --install --force --atomic --wait --version 1.13.2 --values ./coder.yaml coder coder/coder

outputs: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

Octoxan
  • 1,949
  • 2
  • 17
  • 34

8 Answers8

112

This solution worked for me:

kubectl get secrets
kubectl delete secret sh.helm.release.v1.<RELEASE_NAME>.v<LATEST_REVISION>

Following the resolution described in this issue

tom1299
  • 75
  • 10
duyetpt
  • 1,471
  • 1
  • 10
  • 12
  • 6
    [This answer](https://stackoverflow.com/a/65135726/8093394) explains why, but this one solved my problem. I've canceled a pipeline run in the middle of an `upgrade` command and there was no Release registered in helm to do `rollback` or `uninstall`. Deleting the helm's release secret was the only way to move on. +1 – Diego Rafael Souza May 14 '21 at 23:48
  • This also worked for me. – jph Oct 27 '21 at 17:44
  • Great solution! – Mark Davydov Feb 13 '22 at 13:29
  • 1
    just to add - I had to add --namespace to both kubectl commands to get them to work for my deployment (otherwise it assumes default namespace) – Andy Lorenz Mar 23 '22 at 11:14
  • this worked for me also, in the secret labels you should see "status : pending" – mohamed wael thabet Apr 20 '22 at 11:00
  • 1
    This is probably the worst advice. You WILL most probably end up with dangling resources. A rollback is by far the safest way to restore to previous state. If it is the first helm install (i.e. no release versions), just force delete. – Francois Nel Sep 30 '22 at 07:33
  • i did not see any helm releases other than "deployed" status with helm ls -A but still received the "an operation is running" message. After deleting the latest helm secret i could upgrade again. So: worked for me. – Blafasel42 Oct 20 '22 at 09:39
  • The only thing that helped for me. Rollbacks didn't do the trick. – Pointi Jun 13 '23 at 13:30
102

TLDR: You need to rollback to another version first and then helm upgrade again:

helm rollback <release> <revision> --namespace <namespace>

This can happen for a few reasons, but it ultimately occurs when there's an interruption during the upgrade/install process. Commonly, you SIGKILL (Ctrl C) while the deployment is ongoing.

You'll notice that if you helm ls --namespace <namespace> while it's stuck in STATUS: pending-upgrade state, you'll see the following without any other information:

NAME    NAMESPACE   REVISION    UPDATED STATUS  CHART   APP VERSION

The best workaround currently is to rollback to another version, and then helm upgrade again:

helm rollback <release> <revision> --namespace <namespace>

revision is optional, but you should try to provide it.

more resources:

EDIT

If this is your first revision, use:

helm delete <release> -n <namespace>
bwl1289
  • 1,655
  • 1
  • 12
  • 10
  • 24
    But what do you do if this is the first Revision? There is nothing to roll back to. – Explicitsoul Jan 28 '21 at 15:16
  • 3
    i also had this issue but using flux, and this solution worked for that as well – aL3891 Nov 05 '21 at 09:15
  • 3
    to get the stuck app listet by helm use `helm ls --namespace -all` – defonion May 16 '22 at 09:57
  • 1
    @Explicitsoul just helm delete (if there is no previous revision) – Francois Nel Sep 30 '22 at 07:38
  • if you have many deployments, you can run the following command to rollback all the failing deployments `helm ls -a -n | grep -e failed -e pending-upgrade | awk '{print $1} {print $2} {print $3 - 1}' | xargs -n 3 /bin/sh -c 'helm rollback $0 $2 -n $1'` – DC- Jan 29 '23 at 17:52
  • I want to add: This also works if you have a pending-rollback situation. In my case, I mistakenly killed the rollback job, leading to a forever pending-rollback state. Manually rolling back worked and allowed me to continue with my next deployment. – techfly Mar 09 '23 at 10:49
31

In case is useful to someone, and in response to explicitsoul's comment, what fixed it to me was just:

helm delete <release> -n <namespace>

That removed the pending install (in my case, the first one so I hadn't a previous release to rollback to) and then I was able to run the install again.

What caused the stuck process in my case was a CTRL-C canceling the install command, so don't do that.

Lucas Veljacic
  • 437
  • 4
  • 5
26

Here is what worked for me

  1. helm list --all This will list all the releases with their status
NAME  NAMESPACE       REVISION        UPDATED                                 STATUS               CHART                   APP VERSION
rel1  default         1               2021-06-04 14:15:37.652066 +0530 IST    deployed             rel1-3.32.0             0.46.0     
rel2  default         29              2021-06-18 11:02:38.779801 +0530 IST    pending-upgrade      rel2-0.0.1                     
rel3  default         3               2021-06-17 11:27:14.608042 +0530 IST    deployed             rel3-0.0.1
  1. Notice that rel2 has status as pending-update . This happened because I did a Ctrl+C while upgrade was in progress
  2. All i had to do was rollback to previous revision in this case 28 helm rollback rel2 28 --namespace default
NAME   NAMESPACE       REVISION        UPDATED                                 STATUS          CHART          APP VERSION
rel1   default         1               2021-06-04 14:15:37.652066 +0530 IST    deployed        rel1-3.32.0    0.46.0     
rel2   default         30              2021-06-18 11:26:07.555547 +0530 IST    deployed        rel2-0.0.1                     
rel3   default         3               2021-06-17 11:27:14.608042 +0530 IST    deployed        rel3-0.0.1
Manoj I
  • 1,089
  • 1
  • 13
  • 18
6

I am facing the same. I used helm 3.4.1... It happens when the deployment is in pending and you use --atomic (which in helm3 implies also --wait).

I could not get upgrade working. Worst is that even helm -n code list did not show anything, so I could not do:

helm -n code code

As helm3 holds such info in secrets, just clean the respective secret(s) and do install (or upgrade --install , but without --atomic). In your case something like

helm delete --namespace code secret sh.helm.release.v1.code.v1

(where last v1 is the release number, so maybe list and delete all if you are ok with that).

and afterwars helm install.

NOTE: old objects (pods,etc) will be there, so the new install will try to merge things. It went OK for me, but note -> It's a HACK :)

More on: https://github.com/helm/helm/issues/5595

ReSearchIT Eng
  • 111
  • 1
  • 3
4

In order to rollback to previous version you can just pass the release name:

helm rollback <RELEASE_NAME>

Where RELEASE_NAME can be seen when you run helm list --all -> under the NAME column.

(*) Add --namespace <namespace> if terminal context is not set to namespace.

Rot-man
  • 18,045
  • 12
  • 118
  • 124
3

These are the steps that worked for me:

  1. See the status of your deployment (my was pending all the time)

    helm list --all

  2. Rollback to the previous version, for me was working already here, next step optional

    helm rollback <NAMESPACE_NAME> <Previous Version> --namespace <NAMESPACE_NAME>

  3. In case you want new/another deploy

    helm upgrade . . .

Boba Fett likes JS
  • 747
  • 3
  • 10
  • 21
1

kubectl get secrets kubectl delete secret sh.helm.release.v1.<RELEASE_NAME>.v<LATEST_REVISION>

By using the above command, it will remove the existing secrets in the middle of helm upgrade which will run and delete the stucked helm upgrade and it will generate the new one to proceed with the new upgrade.

Venkata Satya Karthik Varun Ku
  • 279
  • 1
  • 6