Percona Operator for MongoDB not able to apply changes on running instance

MongoDB Percona Operator for MongoDB
1

Hi,

I have an instance of Mongodb cluster installed by Percona Operator on Openshift ( Percona Server MongoDB Operator v1.7.0).
Since a few days ago, the changes applied on the cluster instance resource definitions are not applied on the statefulsets (cfg or replicaset). The only logs I found are from the percona operator pod with this message :

{"level":"info","ts":1623838760.8197567,"logger":"cmd","msg":"Git commit: c12b69e2c41efc67336a890039394250420f60bb Git branch: release-1-7-0"}
{"level":"info","ts":1623838760.8198013,"logger":"cmd","msg":"Go Version: go1.14.15"}
{"level":"info","ts":1623838760.819812,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1623838760.8198216,"logger":"cmd","msg":"operator-sdk Version: v0.17.1"}
{"level":"info","ts":1623838760.8205972,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1623838765.5465813,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1623838765.5546856,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1623838770.2623434,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1623838770.2627563,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1623838770.26814,"logger":"controller_psmdb","msg":"server version","platform":"openshift","version":"undefined (v4.0+)"}
{"level":"info","ts":1623838770.2691066,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1623838770.2694118,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1623838770.2694328,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"psmdb-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1623838770.2698176,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbrestore-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1623838770.269959,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbbackup-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1623838770.3702178,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"psmdb-controller"}
{"level":"info","ts":1623838770.370256,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"psmdb-controller","worker count":1}
{"level":"info","ts":1623838770.3704336,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbrestore-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1623838770.3712387,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbbackup-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1623838770.470894,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"perconaservermongodbrestore-controller"}
{"level":"info","ts":1623838770.4709818,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"perconaservermongodbrestore-controller","worker count":1}
{"level":"info","ts":1623838770.4713585,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"perconaservermongodbbackup-controller"}
{"level":"info","ts":1623838770.571519,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"perconaservermongodbbackup-controller","worker count":1}
{"level":"error","ts":1623838770.851839,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go...
{"level":"error","ts":1623838771.927495,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go...
{"level":"error","ts":1623838772.9993627,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623838774.0769784,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623838775.1550205,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623838776.2235157,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623838777.2941067,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623838778.3859608,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623838779.448452,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go...
{"level":"error","ts":1623838780.809681,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go...
{"level":"error","ts":1623838783.4408083,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623838788.6679401,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623838798.9833922,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623838819.546329,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go...
{"level":"error","ts":1623838860.578178,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go...
{"level":"error","ts":1623838942.5682397,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623839106.505805,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go...
{"level":"error","ts":1623839434.2589436,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623839528.4559617,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...
{"level":"error","ts":1623840089.6871831,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"smc-os2-bigd/sccluster","error":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps \"sccluster-cfg\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/g...

Any clues on why ?
thanks
David

1 Like
2

Hello @David_Da_Costa ,

that is weird. Could you please provide the exact steps to reproduce? Or at least what and how are you trying to change.

Any permission changes recently?

1 Like
3

Hi @Sergey_Pronin ,

This happens when I try to change the mem and cpu limits of the replicaset.
But I don’t understand how to troubleshoot better this issue.

We didn’t change permissions or roles.

1 Like
4

@David_Da_Costa it sounds very strange - it should be fixed.
do you have reliable steps to reproduce?

1 Like
5

Hi @Mykola.
The steps to reproduce are :

  • I edit the percona CRD specs for cpu and memory limits for a replicaset.
    specs-changed

  • The statefulset is not updated.

  • The error message above are displayed in the percona operator pod

  • and the state of the percona crd state stays at “ClusterInitializing”

crd-state
crd-state2642×764 171 KB

And why is the percona instance state still initializing?
regards,

1 Like
6

So if you have a running mongodb cluster, how do you apply the new values ?

I always do like:

kubectl apply -f cr_with_diff_values.yaml

in conjunction with sharding=on - this will update w/o any interruption. this is done very nicely.

1 Like
7

The cluster is running since beginning of May and since mid-June, the percona operator don’t apply new values. It was working fine until “something” happened which the operator doesn’t like.

I’m able to update the CR with new values (with kubectl apply or by openshift console) but it’s not applied to the resources.

1 Like
8

“Sometimes” you have to restart the operator pod.

1 Like
9

I restarted the pod and still the same issue.

I still have the error log : “error”:“reconcile StatefulSet for cfg: update StatefulSet sccluster-cfg: StatefulSet.apps "sccluster-cfg" is invalid: spec: Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘template’, and ‘updateStrategy’ are forbidden”.

It seems there is a configuration mismatch regarding the cluster cfg StatefulSet. And I don’t know where to look in the operator pod.

1 Like
10

@David_Da_Costa is something else (except resources) was changed.
I cannot reproduce the issue when I just change resources

1 Like
11

There are few ways to apply changes to k8s primitives or mongodb components through the operator:

  1. recommended: change original cr.yaml and apply through kubectl apply -f cr.yaml. Operator detects the changes and applies them.
  2. patch Custom Resource object directly in kubernetes kubectl patch psmdb my-cluster or kubectl edit psmdb my-cluster. Just ensure you have all the values aligned in cr.yaml as well.

If kubernetes primitives are changed manually (like editing the statefulset), Operator is going to change them back right away.
Judging by the log - I think you changed something manually, @David_Da_Costa and now there is some desync.

1 Like
12

@David_Da_Costa I wouldn’t recommend this, but you can also try to remove sccluster-cfg statefulset with --cascade=orphan option, Operator will recreate it, but it can be dangerous. you potentially can lose data.

1 Like
13

It’s too dangerous because the mongoDB Cluster is a production environment.

1 Like
14

Is there any updates on this?
We have a same issue changing “resources.requests.storage” for rs.
Our CR has been applied by ArgoCD and psmdb-db based on helm chart.
So CR is in sync now with changes, but operator throws that error:
is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template'...

15

Problem solved using kubectl delete sts --cascade=orphan command. The deployment volume had been increased in a wrong way.

16

Hey @qonalex - please have a look at these blog posts:

They esplain the details behind storage modifications and reasoning behind it.

17

Thank you @Sergey_Pronin . This is exactly where I got the knowledge to fix my problem. :slight_smile: