MongoDB Sharded Cluster Stuck Initializing Due to Denylisted Sync Source

MongoDB Percona Operator for MongoDB
2

This occurs as i was provisioning a fresh cluster and doing a mongosync from a different cluster to this new one.

The shard’s secondary node fell so far behind during the bulk data sync that the primary’s oplog entries rolled off before the secondary could apply them, forcing it into a full initial sync (RECOVERING) instead of simple catch-up. Because it couldn’t become a healthy SECONDARY, the Percona operator left the CR “Progressing” and Argo CD never marked it Healthy.

The solution is to to do a full initial scan by deleting the underlining data. As this is a secondary replica its relatively safe to do. Although mongodb does not have an inbuilt way to do this.

llm reccomended to delete the PVC → PV → Pod of the shard, i also had to do it in this order to let the operation hang otherwise pod is just restarted and rebinds to the same pvc.

Once a new pv/pvc/pod is created it starts an initial sync and should eventually catch up to the primary of that shard. I was able to monitor the progress using kubectl -n mongodb describe perconaservermongodb psmdb-default-sharded and prometheus metricis of the persistent volume.

    rs2:
      added_as_shard:  true
      Initialized:     true
      Members:
        psmdb-default-sharded-rs2-0:
          Name:       psmdb-default-sharded-rs2-0.psmdb-default-sharded-rs2.mongodb.svc.cluster.local:27017
          State:      1
          State Str:  PRIMARY
        psmdb-default-sharded-rs2-1:
          Name:       psmdb-default-sharded-rs2-1.psmdb-default-sharded-rs2.mongodb.svc.cluster.local:27017
          State:      2
          State Str:  SECONDARY
        psmdb-default-sharded-rs2-2:
          Name:       psmdb-default-sharded-rs2-2.psmdb-default-sharded-rs2.mongodb.svc.cluster.local:27017
          State:      2
          State Str:  SECONDARY

I could not find any documentation on percona around this issue, so i don’t know if this is the most suitable solution but it is one that works and will get the state of the current cluster back to ready/healthy.

To prevent this from happening in the future - There are a few solutions i can come across to prevent this from happening in the first place.

  1. Increase the size of your replicas during the mongosync ( seems to work - currently trying this).
  • Update: problem still occurs despite using a larger instance
  1. Set size rs2.size = 1 during the mongosync and increase to 3 once its done.
  2. Increase the size of oplog.