Configuring sharding with mongo psmdb, switch sharding false to true

MongoDB Percona Operator for MongoDB
1

Description:

I’m trying to restore a backup of a non-sharded MongoDB cluster to a sharded one.

To do this, I first start the cluster with one replica set in a non-sharded state, restore the data, and everything is fine.

But when I switch to a sharded state and add the replica set rs1, the operator can’t start and configure the cluster.

Steps to Reproduce:

replsets:
  - name: rs0

#start cluster with
sharding:
enabled: false

apply cr.yaml

when change to

replsets:
  - name: rs0
...
  - name: rs1
...

sharding: 
  enabled: true

Снимок экрана в 2026-04-07 12-48-53
Снимок экрана в 2026-04-07 12-48-53752×469 43.5 KB

Logs:

psmdb

Message: Error: delete psmdb pods: is pod primary: failed to create standalone client: ping mongo: connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate │
│ using mechanism “SCRAM-SHA-1”: (AuthenticationFailed) Authentication failed.

Expected Result:

cluster with 2 shards

Actual Result:

The cluster won’t start
Partial replica sets won’t start
PSMDB is in error state

Additional Information:

I’d like a manual if I’m doing something wrong. Maybe I need to do this procedure through Unmanage?

2

│ 2026-04-07T10:17:39.754Z INFO Sharding is enabled, pausing the cluster {“controller”: “psmdb-controller”, “controllerGroup”: “psmdb.percona.com”, “controllerKind”: “PerconaServerMongoDB”, “PerconaServerMongoDB”: {“name”:“mong │
│ ocluster-psmdb”,“namespace”:“netsim-stage”}, “namespace”: “netsim-stage”, “name”: “mongocluster-psmdb”, “reconcileID”: “ed486af8-ce24-442a-a664-8472322a9ee7”} │
│ 2026-04-07T10:17:50.348Z ERROR failed to reconcile cluster {“controller”: “psmdb-controller”, “controllerGroup”: “psmdb.percona.com”, “controllerKind”: “PerconaServerMongoDB”, “PerconaServerMongoDB”: {“name”:“mongocluster-psm │
│ db”,“namespace”:“netsim-stage”}, “namespace”: “netsim-stage”, “name”: “mongocluster-psmdb”, “reconcileID”: “ed486af8-ce24-442a-a664-8472322a9ee7”, “replset”: “rs0”, “error”: “create system users: failed to get mongo client: ping mongo │
│ : connection() error occurred during connection handshake: auth error: unable to authenticate using mechanism “SCRAM-SHA-256”: (AuthenticationFailed) Authentication failed.”, “errorVerbose”: "connection() error occurred during conne │
│ ction handshake: auth error: unable to authenticate using mechanism “SCRAM-SHA-256”: (AuthenticationFailed) Authentication failed.

3

As a result, I manually changed statefull and launched shard replica sets, and manually added shards to the cluster. The data also seemed to resolve, but the users were never created. Psmdb also crashed with the same error message:

│ 2026-04-07T10:53:28.160Z ERROR failed to reconcile cluster {“controller”: “psmdb-controller”, “controllerGroup”: “psmdb.percona.com”, “controllerKind”: “PerconaServerMongoDB”, “PerconaServerMongoDB”: {“name”:“mongocluster-psm │
│ db”,“namespace”:“netsim-stage”}, “namespace”: “netsim-stage”, “name”: “mongocluster-psmdb”, “reconcileID”: “1e083c9c-6482-4595-8ef3-7b2c34212f75”, “replset”: “rs0”, “error”: “create system users: failed to get mongo client: ping mongo │
│ : connection() error occurred during connection handshake: auth error: unable to authenticate using mechanism “SCRAM-SHA-256”: (AuthenticationFailed) Authentication failed.”, “errorVerbose”: “connection() error occurred during conne │
│ ction handshake: auth error: unable to authenticate using mechanism “SCRAM-SHA-256”: (AuthenticationFailed) Authentication failed.\nping mongo\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo.Dial\n\t/go/src/githu │
│ b.com/percona/percona-server-mongodb-operator/pkg/psmdb/mongo/mongo.go:126\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb.MongoClient\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/client.go:62\ │
│ ngithub.com\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/connections.go:38\n │
│ github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).mongoClientWithRole\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermong │
│ odb/connections.go:60\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).createOrUpdateSystemUsers\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg │
│ /controller/perconaservermongodb/mgo.go:950\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb │
│ -operator/pkg/controller/perconaservermongodb/mgo.go:157\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona- │
│ server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:590\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.c │
│ om/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:448\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@ │
│ v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go: │
│ 479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/int │
│ ernal/controller.(*Controller[…]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\nfailed to get mongo clien │
│ t\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).createOrUpdateSystemUsers\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconas │
│ ervermongodb/mgo.go:952\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/contro │
│ ller/perconaservermongodb/mgo.go:157\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-opera │
│ tor/pkg/controller/perconaservermongodb/psmdb_controller.go:590\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-s │
│ erver-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:448\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal │
│ /controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/con │
│ troller-runtime/pkg/internal/controller.(*Controller[…]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*C │
│ ontroller[…]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\ncreate system users\ ngithub.com │
│ na-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:159\ngithub.com/ │
│ percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_con │
│ troller.go:590\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaserv │
│ ermongodb/psmdb_controller.go:448\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/contro │
│ ller-runtime/pkg/internal/controller.(*Controller[…]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Control │
│ ler[…]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).Start.func1.1\n\t/go/pkg/mod/sigs │
│ .k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693”}

2026-04-07T10:53:28.303Z ERROR Reconciler error {“controller”: “psmdb-controller”, “controllerGroup”: “psmdb.percona.com”, “controllerKind”: “PerconaServerMongoDB”, “PerconaServerMongoDB”: {“name”:“mongocluster-psmdb”,“namesp │
│ ace”:“netsim-stage”}, “namespace”: “netsim-stage”, “name”: “mongocluster-psmdb”, “reconcileID”: “1e083c9c-6482-4595-8ef3-7b2c34212f75”, “error”: “reconcile statefulsets: create system users: failed to get mongo client: ping mongo: con │
│ nection() error occurred during connection handshake: auth error: unable to authenticate using mechanism “SCRAM-SHA-256”: (AuthenticationFailed) Authentication failed.”, “errorVerbose”: “create system users: failed to get mongo clie │
│ nt: ping mongo: connection() error occurred during connection handshake: auth error: unable to authenticate using mechanism “SCRAM-SHA-256”: (AuthenticationFailed) Authentication failed.\nreconcile statefulsets\ngithub.com/percona/p │
│ ercona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:450\ns │
│ igs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:222\nsigs.k8s.io/controller-runtime/pkg/internal/controller │
│ .(*Controller[…]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:479\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).processNextWorkItem\n\t/go │
│ /pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/p │
│ kg/internal/controller/controller.go:313\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693”} │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler │
│ /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:495 │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem │
│ /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:438 │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 │
│ /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.1/pkg/internal/controller/controller.go:313

4

Hi, do you have a Secrets object with the same user passwords as in the original cluster??