PVC size reconcile errors on 1.16 upgrade

MongoDB Percona Operator for MongoDB
1

Getting some reconciler errors after upgrade to 1.16.0 related to the size of my PVCs that have been increased since the creation of the cluster.

{
  "controller": "psmdb-controller",
  "object": {
    "name": "cluster-name",
    "namespace": "cluster-name"
  },
  "namespace": "cluster-name",
  "name": "cluster-name",
  "reconcileID": "9255cd50-fe90-4ee2-be5c-c5789d2b96e5",
  "error": "reconcile statefulsets: reconcile StatefulSet for rs0: reconcile PVCs for cluster-name-rs0: resize volumes if needed: requested storage (32Gi) is less than actual storage (256Gi)",
  "errorVerbose": "reconcile StatefulSet for rs0: reconcile PVCs for cluster-name-rs0: resize volumes if needed: requested storage (32Gi) is less than actual storage (256Gi)"
}

The problem is that the statefulset cannot be updated, either manually or through a change of the CR file, since they you cannot change those properties:

{
  "controller": "psmdb-controller",
  "object": {
    "name": "cluster-name",
    "namespace": "cluster-name"
  },
  "namespace": "cluster-name",
  "name": "cluster-name",
  "reconcileID": "2451bdaf-e606-4a90-bb68-2472292dd8f6",
  "error": "reconcile statefulsets: reconcile StatefulSet for rs0: update StatefulSet cluster-name-rs0: StatefulSet.apps \"cluster-name-rs0\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden",
  "errorVerbose": "reconcile StatefulSet for rs0: update StatefulSet cluster-name-rs0: StatefulSet.apps \"cluster-name-rs0\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden"
}
2

Ok, found a solution for this. You need to delete the statefulset without killing the associated pods to avoid downtime.

kubectl delete sts cluster-name-rs0 --cascade=orphan

And then re-apply the changes to the CR, which should recreate the statefulset with the proper values, all while keeping the pods alive in the meantime.