6

I made a Kafka and zookeeper as a statefulset and exposed Kafka to the outside of the cluster. However, whenever I try to delete the Kafka statefulset and re-create one, the data seemed to be gone? (when I tried to consume all the message using kafkacat, the old messages seemed to be gone) even if it is using the same PVC and PV. I am currently using EBS as my persistent volume.

Can someone explain to me what is happening to PV when I delete the statefulset? Please help me.

Rico
  • 58,485
  • 12
  • 111
  • 141
user11771691
  • 61
  • 1
  • 3

4 Answers4

2

I would probably look at how the persistent volume is created. If you run the command kubectl get pv you can see the Reclaim policy, if it is set to retain, then your volume will survive even when stateful set is deleted

srinivas kulkarni
  • 23
  • 5
  • the reclaim policy is set to retain. I guess whenever I tried to re-create or start again, I saw it connects to existing pvc and pv but it seemed like all the previous data wiped out. – user11771691 Jul 11 '19 at 19:29
  • That should not be the case! Persistent volumes should be able to hold data even after deletion. My guess is when you recreate a stateful set it could be creating a new PV? – srinivas kulkarni Jul 11 '19 at 19:32
  • No, i think it's still using the same pv. maybe re-created Kafka cluster is not able to find the existing data? – user11771691 Jul 11 '19 at 20:31
2

This is the expected behaviour , because the new statefulSet will create a new set of PVs and start over. ( if there is no other choice it can randomly land on old PVs as well , for example local volumes )

StatefulSet doesn't mean that kubernetes will remember what you were doing in some other old statefulset that u have deleted.

Statefulset means that if the pod is restarted or re-created for some reason, the same volume will be assigned to it. This doesn't mean that the volume will be assigned across the StatefulSets.

Ijaz Ahmad
  • 11,198
  • 9
  • 53
  • 73
2

I assume your scenario is that you have a statefulset which has got a persistentvolumeclaim definition in it - or it is just referencing an existing volume - and you try to delete it.

In this case the persistent volume will stay there. Also the pvc won't disappear;

This is so that you can, if you wanted to, remount the same volume to a different statefulset pod - or an update of the previous one thereof.

If you want to delete the persistent volume you should delete the pvc and the buond PVs will disappear.

iomv
  • 2,409
  • 18
  • 28
2

Kubernetes default prevent deleting PersistentVolumeClaims and bounded PersistentVolume objects when you scaling StatefulSet down or deleting them.

Retaining PersistentVolumeClaims is the default behavior, but you can configure the StatefulSet to delete them via the persistentVolumeClaimRetentionPolicy field.

This example shows part of StatefulSet manifest file, where retention policy causes deleting PersistentVolumeClaim when StatefulSet is scaled down, and retaining when StatefulSet is deleted.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: quiz
  spec:
    persistentVolumeClaimRetentionPolicy:
      whenScaled: Delete
      whenDeleted: Retain

Make sure you have properly configured StatefulSet manifest and Kafka cluster.

NOTE

If you want to delete a StatefulSet but keep the Pods and the PersistentVolumeClaims, you can use the --cascade=orphan option. In this case, the PersistentVolumeClaims will be preserved even if the retention policy is set to Delete.

Marko Lukša "Kubernetes in Action, Second Edition"

Mikolaj
  • 1,231
  • 1
  • 16
  • 34
  • It seems helm chart does not allow this property , installation fails stating reason "not valid spec", detail Error : INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(StatefulSet.metadata): unknown field "spec" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta – SKP Feb 02 '23 at 09:28