Facing issue with mongodb cluster in error state due to reconciler errors in operator pod

[Raj_Burnwal]

Team, I setup percona mongodb cluster using helm. So after using for some time the db, I expanded volume by just editing pvc for replicas( 0-2). Now cluster is in error state due to reconciler errors in operator pods and also description of psmdb cluster. Though pods are up and running and also mongosh working fine. I believe it will create issue in future if anything happens in order to recover pods or something. I saw this problem in first hand when I tried to enable PMM after updating secret data for PMM_SERVER_API_KEY in k8s secret of mongodb cluster. Not able to enable PMM for this cluster due to reconciler issues.

can you please suggest some solution?

You can check error logs.

2025-01-31T16:29:13.591Z	ERROR	Reconciler error	{"controller": "psmdb-controller", "object": {"name":"prodeu-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "prodeu-psmdb-db", "reconcileID": "05feb733-8767-4605-b015-3a04ee4f36ad", "error": "reconcile statefulsets: reconcile StatefulSet for rs0: update StatefulSet prodeu-psmdb-db-rs0: StatefulSet.apps \"prodeu-psmdb-db-rs0\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden", "errorVerbose": "reconcile StatefulSet for rs0: update StatefulSet prodeu-psmdb-db-rs0: StatefulSet.apps \"prodeu-psmdb-db-rs0\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:489

[Ege_Gunes]

Hi @Raj_Burnwal,

This is expected behavior since it’s forbidden to update volume claim templates in statefulsets.

You need to follow the official procedure of updating the volume size in cr.yaml and letting operator do the resize operation. Please check this documentation: Horizontal and vertical scaling - Percona Operator for MongoDB

To fix your current problem, you can delete the statefulset with --cascade=orphan:

kubectl delete statefulset prodeu-psmdb-db-rs0 --cascade=orphan

This command will delete the statefulset without deleting the rs0 pods and operator will recreate the statefulset. I recommend updating your cr.yaml with the new size of PVCs and applying it before doing the operation, so new statefulset will be created with updated template.

[Raj_Burnwal]

@Ege_Gunes

Horizontal and vertical scaling - Percona Operator for MongoDB

Thanks @Ege_Gunes It worked like a charm. Also I’ll follow the given link for storage expansion next time if required. I hope it works with Helm values as well?

Also do we support azure storage for storing backup using PMM UI? I can see only s3 and local path. I know its doable through k8s custom resources and values with backup.enabled: true.