I’m looking for a way to backup large mongodb instances around 3.5 TB.
Came across PITR feature in PBM. Went through the docs and understood how it works mostly.
It was mentioned that “once PITR restore is done, the operation changes the timeline of oplog slices. Therefore, all oplog slices made after the restore time stamp and before the last backup become invalid”. Can someone please provide more information on this. I couldn’t find it on official docs.
1 Like
Hi Siva.
“once PITR restore is done, the operation changes the timeline of oplog slices. Therefore, all oplog slices made after the restore time stamp and before the last backup become invalid”.
It’s meant to be pointing out that once you do a restore you won’t be able to do another PITR restore until after another full snapshot has been done. Eg. let’s say the backups are at ~1am each day. Let’s say you have a production disaster at 14:00 on Wed, and do a restore (This might be either the full snapshot at ~1am, or some point in time after). You wont be able to run another PITR restore today because the oplog history starts diverging more or less immediately after the first restore.
backup large mongodb instances around 5 TB.
Is that per node / replicaset? If so I would not recommend a ‘logical’ methods such as using mongodump or Percona Backup for MongoDB. Each TB takes enough time to copy to a backup server as you would already know and ‘logical’ methods usually experience a disk read amplification costs that makes it even slower. Percona Backup for MongoDB wasn’t intended for data sets of this size per server. These mongodb backup tools read the documents in each collection by _id order, but documents usually aren’t persisted serially by that order in the on-disk pages/blocks. Random access, including coming back to each page many times, happens instead of efficient, serial disk file copying. In one real-world case I saw every filesystem disk block was being accessed several dozen times each by the time the backup was complete. Instead of reading 10TB from disk it was reading 120TB i.i.rc.
So if you find you have > 1 TB of data per node I would recommend using physical backup methods - Percona Server for MongoDB’s createBackup command, or a filesystem snapshot, or a hidden secondary that you stop and make the file copies of whilst they are ‘cold’. This does mean going without point in time recovery unless you choose to script a restore process using oplog yourself.
On the other hand if you have a large cluster where there is not too much data per shard, and especially that most data is in memory, Percona Backup for MongoDB is a very suitable. It’s parallelized processing per shard comes into good effect in this case.
2 Likes
I’m sorry I mentioned 5TB. I meant 3.5TB for whole instance. And yeah, that cleared most of my queries. Thanks Akira.
2 Likes