Hi all,
While testing replicaset PITR restores of multi-terrabyte mongodb clusters using PBM, I observed that PBM replays the oplog serially. PITR restore for a 4 hour window of oplog size of about 46GB took 2+ days to complete. I patched PBM to apply the oplog in parallel using multiple threads. It did improve but not ideal as it took 24hrs to complete and cluster was mostly idle during the run. This performance profile matches what was reported in PITR restore performance as there was no resource contraints on the cluster and it was mostly idle.
Hardware specs was about 64k iops and io2 storage for the nodes.
I have a PR in draft state currently that adds a --native-pitr option for PBM PITR restores built on PBM 2.16.0 and PSMDB v7. Basically PBM appends the PITR window from the backup storage (S3 in my case) to the restored node’s oplog and mongod then replays it with its own multi-threaded applier during startup recovery. With this approach the whole restore (which includes the 4hr PTR window) on a 12TB cluster completed in 4hrs while on an 8TB cluster, 4Hr oplog window with 5GB oplog the restore completed in about 30 minutes. This excludes time for creating/initializing the volumes tho as that is a separate process. Data validation was done successfully as there was no missing data and accuracy was verified.
Using this approach means there is no need for a separate pbm oplog-replay run after restoring the base snapshot as the PITR restore can now be completed in a single run when you run pbm restore-finish command.
It currently supports replicasets only, I believe the same approach could be extended to sharded clusters.
I would appreciate if this could be reviewed when i open the PR and any feedback on the approach would be welcome.