Backing up a small PSMDB 8.0 replica set in Docker without PBM and NFS: recommended approaches?

MongoDB Percona Server for MongoDB
1

Hello,

I’m looking for pointers and opinions on the simplest reliable way to back up a MongoDB replica set without using PBM and a shared NFS storage.

Current setup

  • Percona Server for MongoDB 8.0.19-7, running in Docker containers (percona/percona-server-mongodb:8.0.19-7)
  • 4-node replica set: Primary, 2 Secondaries, 1 Hidden member
  • Two independent replica sets on the same hosts, both small: a logical backup is about 200 MB for the larger one and under 10 MB for the smaller one; a physical backup is about 2 GB
  • Today: PBM 2.13.0 with filesystem storage on an NFS share, PITR enabled, daily logical and weekly physical backups
  • The NFS server runs on the hidden node itself, so backups live on a member of the cluster they protect
  • All VMs are covered by a daily VM-level backup (snapshot based). We do not control the exact time each VM is backed up, only the global job.

Why we are considering a change

PBM works, but the NFS part added complexity and caused real issues:

  • containers starting before the NFS mount and silently writing to the local directory
  • orphaned backup files
  • boot ordering and mount options to get right on every node

Management would like to reduce the number of moving parts. An RPO of 24 hours may be acceptable (12 hours would be better…still to be confirmed), so PITR could be a nice-to-have rather than a requirement.

Options we are evaluating

  1. mongodump --oplog --gzip --archive from the hidden member, scheduled with cron, written to a dedicated local disk on that node, with simple file-based retention. The daily VM backup then carries the dump files off the node.
  2. createBackup (Percona hot backup) on the hidden member, to a local directory or to an S3-compatible storage if one is available.
  3. VM-level snapshot only, relying on the fact that data files and journal are on the same volume, so the snapshot should be crash-consistent and recoverable by WiredTiger.
  4. Keep PBM but switch storage to S3, if an S3-compatible object storage is available internally, to remove NFS and keep PITR.

Questions

  1. For a dataset this small, is mongodump --oplog from a hidden secondary a sound choice for a daily backup? Any pitfalls to be aware of (consistency guarantees, users and roles, index rebuild time on restore, --oplogReplay behavior)?
  2. With createBackup, what is the recommended procedure to restore a whole replica set from a single member’s backup (forced reconfig to a single-member replica set, then initial sync of the others)? Are there any Docker-specific considerations for backupDir?
  3. Is a crash-consistent VM snapshot of the hidden member considered a valid backup in practice? Any recommendation on using fsyncLock() on the hidden member if the backup software can run pre/post scripts?
  4. Without PBM, is there a reasonably simple way to get some point-in-time capability, for example periodically dumping local.oplog.rs incrementally, or is that more trouble than it’s worth?
  5. In your experience, which option gives the best balance between simplicity and restore reliability for a small replica set like this one?

Any links to official documentation, blog posts or real-world experiences would be very welcome.

Thank you!

Francesco

2

Hi @MoosEsooM,
In short - if you have, or can get, any S3-compatible service - this will be the best solution for you. If you can’t, frequent mongodump --oplog from the hidden member with VM snapshots as a second layer is a considerable setup for 200 MB of data.

To answer your questions directly:

  1. Using mongodump --oplog

A 200 MB dump should take seconds (1minute max?), and a logical restore of that size is fast, portable across versions, and easy to test. However I’d watch out for:

  • Renames, drops or index builds that happen while the dump runs can make the dump fail or leave oplog entries that won’t replay cleanly. Shouldn’t usually happen in that short dump time… but worth to watch.

  1. createBackup is a good tool, but at this size, it adds restore complexity compared with a dump and gives you little in return. Details: Hot Backup - Percona Server for MongoDB 8.3

  2. VM snapshot of the hidden member
    It’s valid in principle. MongoDB’s own docs accept filesystem snapshots as a backup method when journaling is on and data and journal are on the same volume, because WiredTiger recovers from a crash-consistent image. In practice, check two things:

  • The snapshot is atomic across the disk(s) holding the Docker volume. Watch for LVM spanning disks and for volumes excluded from the backup job.
  • The dbPath really lives on the snapshotted disk.

Also, fsyncLock() in pre/post scripts operates on a hidden member and provides a clean checkpoint, but it isn’t strictly necessary with the journal. The risk is that the member stays locked, replication stalls, and eventually it falls off the oplog, requiring a resync. If you do use it:

  • Lock only job for the snapshot creation,.
  • Make the unlock unconditional (trap or finally).
  • Monitor that member’s lag.

Personally, I’d skip the lock and rely on crash consistency, because the dumps are your primary backup anyway.

Restoring is the same as with hot backups.

  1. PITR without PBM is possible, but you’d be rebuilding PBM’s PITR yourself. I’d call it more trouble than it’s worth. If you want finer RPO without PBM, dump more often; hourly is trivial at 200 MB. If you need true PITR, that’s the argument for PBM on S3.

  2. I’d definitely recommend going PBM + S3 (just use minio as a storage type if you don’t use genuine AWS S3 but S3-compatible). See MinIO and S3-compatible storage - Percona Backup for MongoDB