Percona cluster not ready i.e error state after upgrade to mongodb 8

getting below error on mongodb operator pod but other pods are working fine like mongos, rs and config.

/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224

2025-03-04T17:27:52.797Z INFO Cluster state changed {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"dev01-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "dev01-psmdb-db", "reconcileID": "f6512ee5-1a46-4901-9ad3-afaedee0da7d", "previous": "error", "current": "ready"}

2025-03-04T17:27:52.811Z ERROR failed to update cluster status {"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"dev01-psmdb-db","namespace":"mongodb"}, "namespace": "mongodb", "name": "dev01-psmdb-db", "reconcileID": "f6512ee5-1a46-4901-9ad3-afaedee0da7d", "replset": "rs0", "error": "write status: PerconaServerMongoDB.psmdb.percona.com \"dev01-psmdb-db\" is invalid: [replsets.cfg.members: Invalid value: \"object\": replsets.cfg.members in body must be of type array: \"object\", replsets.rs0.members: Invalid value: \"object\": replsets.rs0.members in body must be of type array: \"object\"]", "errorVerbose": "PerconaServerMongoDB.psmdb.percona.com \"dev01-psmdb-db\" is invalid: [replsets.cfg.members: Invalid value: \"object\": replsets.cfg.members in body must be of type array: \"object\", replsets.rs0.members: Invalid value: \"object\": replsets.rs0.members in body must be of type array: \"object\"]\nwrite status\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).writeStatus\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/status.go:269\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).updateStatus\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/status.go:238\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:266\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:470\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:116\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:303\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.3/pkg/internal/controller/controller.go:224\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}

github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile.func1

used below value file for mongodb upgrade

pmm:
  enabled: true
replsets:
  rs0:
    configuration: "operationProfiling:\n  mode: slowOp  \n"
    name: rs0
    resources:
      limits:
        memory: 4G
    size: 3
    volumeSpec:
      pvc:
        resources:
          requests:
            storage: 50Gi
updateStrategy: RollingUpdate
upgradeOptions:
  apply: 8.0-recommended
  schedule: 10 * * * *

helm upgrade --install mongo-op percona/psmdb-operator --namespace mongodb

helm upgrade --install dev01 percona/psmdb-db -f mongo-values.yaml -n mongodb --version=1.19.1

what could be the issue here?

Hi,

Could you clarify whether you have upgraded CRDs and RBAC manually before upgrading operator using helm? Please check upgrade instruction for helm: Upgrade MongoDB and the Operator - Percona Operator for MongoDB
Also note, that the recommended way to upgrade database is using patch command Upgrade MongoDB and the Operator - Percona Operator for MongoDB .

2 Likes