# Upgrade from 1.6.0 to 1.7.0 fails **URL:** \ **Category:** Percona Operator for MongoDB\ **Tags:** mongodb\ **Created:** [January 24, 2022, 5:24pm UTC](https://forums.percona.com/t/upgrade-from-1-6-0-to-1-7-0-fails/14079 "2022-01-24T17:24:25Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![andreyk](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@andreyk](https://forums.percona.com/u/andreyk)\ **Post date:** [January 24, 2022, 5:24pm UTC](https://forums.percona.com/t/upgrade-from-1-6-0-to-1-7-0-fails/14079/1 "2022-01-24T17:24:25Z")
I’m upgrading an operator for mongodb server from 1.6.0 to 1.7.0. Once I deploy a new operator with 1.7.0 image, my psmdb goes into initializing state and stays in this state forever. While 1.7.0 operator produces the following logs: ```auto {"level":"error","ts":1643043912.3269353,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"psmdb-controller","request":"datalayer/mongodb","error":"reconcile StatefulSet for rs0: update StatefulSet mongodb-rs0: StatefulSet.apps \"mongodb-rs0\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden","errorVerbose":"reconcile StatefulSet for rs0: update StatefulSet mongodb-rs0: StatefulSet.apps \"mongodb-rs0\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:350\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} ``` If I roll back operator to 1.6.0 image, my psmdb goes into ready state and there are no errors in the log anymore. What am I missing? Thanks, Andrey ---
**Author:** ![Ege\_Gunes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ege_gunes/32/4595_2.png) [@Ege\_Gunes](https://forums.percona.com/u/Ege_Gunes)\ **Post date:** [January 25, 2022, 6:05am UTC](https://forums.percona.com/t/upgrade-from-1-6-0-to-1-7-0-fails/14079/2 "2022-01-25T06:05:04Z")
Hi @andreyk, Could you please share how you tried to upgrade? Have you used helm or installed/upgraded manually? ---
**Author:** ![andreyk](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@andreyk](https://forums.percona.com/u/andreyk)\ **Post date:** [January 25, 2022, 9:14am UTC](https://forums.percona.com/t/upgrade-from-1-6-0-to-1-7-0-fails/14079/3 "2022-01-25T09:14:26Z")
We use ArgoCD to deploy applications into our k8s cluster. Changes to templates: ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/8/832db8799cb7248806e1d0cbb0b87e66774ff0f2.png) Then I sync it into the cluster. ---
**Author:** ![andreyk](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@andreyk](https://forums.percona.com/u/andreyk)\ **Post date:** [January 27, 2022, 12:27pm UTC](https://forums.percona.com/t/upgrade-from-1-6-0-to-1-7-0-fails/14079/4 "2022-01-27T12:27:21Z")
I’ve created a new PSMDB with operator 1.7.0 and compared the generated StatefulSet with the one created by 1.6.0. And there is a difference in the label selectors (left side is 1.6.0, right side is 1.7.0): ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/f/f358cb236c004206aaf8c54402bc9fc4cd447cf1.png) So with 2 PSMDBs (1 created by 1.6.0, another by 1.7.0), when I deploy 1.6.0 operator, 1.6.0 PSMDB goes into “ready” state, while 1.7.0 PSMDB goes into “initializing” state. When I deploy 1.7.0 operator, 1.6.0 PSMDB goes into “initializing” state, while 1.7.0 PSMDB goes into “ready” state. Is there any way to align the stateful set without recreating it? ---
**Author:** ![Sergey\_Pronin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sergey_pronin/32/14887_2.png) [@Sergey\_Pronin](https://forums.percona.com/u/Sergey_Pronin)\ **Post date:** [January 27, 2022, 12:37pm UTC](https://forums.percona.com/t/upgrade-from-1-6-0-to-1-7-0-fails/14079/5 "2022-01-27T12:37:09Z")
@andreyk I’m not sure at what state is your cluster now and what is going on. Could you please share the pods, psmdb objects and stateful sets? ---
**Author:** ![andreyk](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@andreyk](https://forums.percona.com/u/andreyk)\ **Post date:** [January 27, 2022, 1:35pm UTC](https://forums.percona.com/t/upgrade-from-1-6-0-to-1-7-0-fails/14079/6 "2022-01-27T13:35:29Z")
@Sergey_Pronin Currently I have 1.7.0 operator deployed and two psmdb objects: ```auto NAME ENDPOINT STATUS AGE mongodb mongodb-rs0.datalayer.svc.cluster.local initializing 3d mongodb-test mongodb-test-rs0-test.datalayer.svc.cluster.local ready 96m ``` “mongodb” was created with the 1.6.0 operator and “mongodb-test” was created with 1.7.0 operator. If I deploy 1.6.0 operator then the states will switch. mongodb-test - initializing; mongodb - ready; Unfortunately I can’t attach files here, so I’ve uploaded all the files to the drive: [https://drive.google.com/drive/folders/1LGV6AFXJxKQa5aPRNGTNhJOpyLfjsB8I?usp=sharing](https://drive.google.com/drive/folders/1LGV6AFXJxKQa5aPRNGTNhJOpyLfjsB8I?usp=sharing)