Efficiently Adding a Secondary Node with the 5TB data to a Percona MongoDB Replica Set Without Full Resync

Currently we are having a percona mongodb replicaset setup with one primary and two secondary nodes and its having around 5TB data in the cluster. We are planning to add additional secondary node to this replica set. As we are running this cluster in the AWS Ec2, we have took the snapshot of the volume from existing node and tried to attach it to the new secondary node so that it doesn’t have to sync entire 5TB data on this. But when we do so, the new secondary node is trying to resync the data from the existing nodes by deleting all the data in the data folder. Resyncing entire 5TB is causing slowness in the primary node or something crashing the primary and it also takes more than 10 days to complete. Is there any proper way to accomplish this without having to resync from scratch? We are looking for a solution to perform something like copy data to new server without having to wait for 2 weeks. Please advise the best recommended approach for this in percona mongodb.

1 Like

Hello @nagaa .

You created this question in Percona Operator for MongoDB section. Please confirm if you use the Operator. The answer is going to be different depending on the technology you are using.

hi @Sergey_Pronin
We are not using the Percona Operator for MongoDB. Instead, we are running Percona MongoDB’s Docker image as primary and secondary containers in a replica set on EC2 machines.

Hello @nagaa

Are you following proper steps like below?

  1. Take volume snapshot where MongoDB data files are stored from existing node (secondary)
  2. Attach volume to new node.
  3. Make sure you have same replica set name in the MongoDB configuration file as your existing members.
  4. Start the MongoDB service and add the new node to the replica set from current Primary.

Ideally above steps will add new node in the replica set. If you’re still encountering the issue, please share output of “rs.conf()” of replica set along with MongoDB configuration file of new node to help further. If possible try to share relevant MongoDB log file.

1 Like