Physical Restore from One Cluster to Another Fails with "not a member of the leader rs"

MongoDB Percona Operator for MongoDB
1

Hi Percona Team,

I am trying to restore a physical backup from one MongoDB cluster (source-cluster) to another cluster (target-cluster) in Kubernetes using the Percona Server for MongoDB Operator. Here is the setup and what I have tried:

Environment

  • Operator: Percona Server for MongoDB Operator

  • Source cluster: source-cluster (production-like)

    • Replica set of 3 nodes

    • Multiple databases and collections

    • Physical backup exists in S3

  • Target cluster: target-cluster (test)

    • Replica set of 3 nodes

    • Initially empty or with test collections

  • Both clusters share the same Secrets for users

  • No encryption key is used

  • S3 bucket is accessible by both clusters

Attempted Restore

CRD used:

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
  name: restore-test
spec:
  clusterName: target-cluster
  backupSource:
    type: physical
    destination: s3://my-backup-bucket/backup-name
    s3:
      region: ap-northeast-2
      endpointUrl: https://s3.ap-northeast-2.amazonaws.com

  • kubectl apply -f restore-crd.yaml executed

  • Percona operator shows events repeteadly:

    INFO    backup not found in PBM metadata. we need to resync...

  • INFO    waiting for resync to start (up to 30 seconds)

  • INFO    waiting for resync to finish (up to 2 hours)

  • INFO    resync finished

  • All 3 pods in target-cluster log repeatedly:

    not a member of the leader rs

  • Replica set status on target-cluster is normal:

    • PRIMARY: two-rs0-0

    • SECONDARY: two-rs0-1, two-rs0-2

    • All members healthy, optimes in sync

  • psmdb-restore CRD object’s status shows nothing (the “STATUS” section is blank)

Observations

  • Logical restore works as expected

  • Physical restore also works on same cluster

  • Physical restore never completes

  • S3 backup is confirmed present

  • Secrets are identical

  • No encryption key is set (physical backup unencrypted)

  • Logs indicate agent cannot recognize the leader in replica set

Question

Has anyone successfully performed a physical restore to a separate cluster using the Percona Operator without encryption keys? Are there additional requirements regarding replica set structure or pod names when restoring to a different cluster?

Any guidance on making the physical restore work across clusters would be appreciated.

Thanks in advance!