Physical Restore from One Cluster to Another Fails with "not a member of the leader rs"

MongoDB Percona Operator for MongoDB
1

Hi Percona Team,

I am trying to restore a physical backup from one MongoDB cluster (source-cluster) to another cluster (target-cluster) in Kubernetes using the Percona Server for MongoDB Operator. Here is the setup and what I have tried:

Environment

  • Operator: Percona Server for MongoDB Operator

  • Source cluster: source-cluster (production-like)

    • Replica set of 3 nodes

    • Multiple databases and collections

    • Physical backup exists in S3

  • Target cluster: target-cluster (test)

    • Replica set of 3 nodes

    • Initially empty or with test collections

  • Both clusters share the same Secrets for users

  • No encryption key is used

  • S3 bucket is accessible by both clusters

Attempted Restore

CRD used:

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
  name: restore-test
spec:
  clusterName: target-cluster
  backupSource:
    type: physical
    destination: s3://my-backup-bucket/backup-name
    s3:
      region: ap-northeast-2
      endpointUrl: https://s3.ap-northeast-2.amazonaws.com

  • kubectl apply -f restore-crd.yaml executed

  • Percona operator shows events repeteadly:

    INFO    backup not found in PBM metadata. we need to resync...

  • INFO    waiting for resync to start (up to 30 seconds)

  • INFO    waiting for resync to finish (up to 2 hours)

  • INFO    resync finished

  • All 3 pods in target-cluster log repeatedly:

    not a member of the leader rs

  • Replica set status on target-cluster is normal:

    • PRIMARY: two-rs0-0

    • SECONDARY: two-rs0-1, two-rs0-2

    • All members healthy, optimes in sync

  • psmdb-restore CRD object’s status shows nothing (the “STATUS” section is blank)

Observations

  • Logical restore works as expected

  • Physical restore also works on same cluster

  • Physical restore never completes

  • S3 backup is confirmed present

  • Secrets are identical

  • No encryption key is set (physical backup unencrypted)

  • Logs indicate agent cannot recognize the leader in replica set

Question

Has anyone successfully performed a physical restore to a separate cluster using the Percona Operator without encryption keys? Are there additional requirements regarding replica set structure or pod names when restoring to a different cluster?

Any guidance on making the physical restore work across clusters would be appreciated.

Thanks in advance!

2

Hi @ciel,

I just tried to restore a physical backup to a new cluster with operator v1.20.1 and restore succeeded:

$ k get psmdb-restore restore1 -o yaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
  creationTimestamp: "2025-08-29T08:54:26Z"
  generation: 1
  name: restore1
  namespace: psmdb-11459
  resourceVersion: "1756457897207247017"
  uid: 4877c1df-442e-4cf9-8ed0-6c3366cdd346
spec:
  backupSource:
    destination: s3://operator-testing/2025-08-29T08:22:11Z
    s3:
      bucket: operator-testing
      credentialsSecret: minio-secret
      endpointUrl: http://minio-service:9000
      region: us-east-1
      serverSideEncryption: {}
    type: physical
  clusterName: cluster2
status:
  pbmName: "2025-08-29T08:56:34.034604124Z"
  state: ready

$ k get psmdb
NAME       ENDPOINT                                     STATUS   AGE
cluster1   cluster1-rs0.psmdb-11459.svc.cluster.local   ready    75m
cluster2   cluster2-rs0.psmdb-11459.svc.cluster.local   ready    44m

Before the restore I ensured that these secrets are identical:

$ copy-secret cluster1-secrets cluster2-secrets
$ copy-secret cluster1-mongodb-encryption-key cluster2-mongodb-encryption-key
$ copy-secret cluster1-mongodb-keyfile cluster2-mongodb-keyfile