How to properly use oplogOnly with physical backups?

MongoDB Percona Backup for MongoDB
1

Hi!

If oplogOnly is enabled, and the Percona Mongodb server and the physical backup method are used, do I need to make any backups and set up a backup schedule?

From the documentation we can come to the conclusion that no. That oplog slices will somehow appear in the pbm list and that’s it. If restoration is necessary, it will be possible to use them somehow, but it is not clear how. Is everything right?

Please describe in more detail the process of using this oplogOnly parameter.

2

Hi @ecoatoms,

Oplog slicing runs periodically by PBM itself (internally). The period controlled by pitr.oplogSpanMin parameter. For example, if it is 1, the first and following oplog ranges are sliced every minute.

By pbm status, you can check if oplog slicing is enabled:

[...]

PITR incremental backup:
========================
Status [ON]

[...]

As for v2.2.1, it requires a Full Logical Backup to start slicing. Otherwise, you should see an error like:

[...]

PITR incremental backup:
========================
Status [ON]
! ERROR while running PITR backup: 2024-10-11T01:28:40.000+0300 E [rs0/rs0:27017] [pitr] init: catchup: no backup found. full backup is required to start PITR

[...]

With the pitr.oplogOnly, PBM ignores missed valid base snapshot and does slicing anyway.

For example, I enabled oplogOnly and, after some time, ran Full Logical Backup:

[...]


PITR incremental backup:
========================
Status [ON]

Currently running:
==================
(none)

Backups:
========
FS  /var/pbm
  Snapshots:
    2024-10-20T15:35:54Z 16.96KB <logical> [restore_to_time: 2024-10-20T15:35:59Z]
  PITR chunks [59.78KB]:
    2024-10-20T15:36:00Z - 2024-10-20T15:38:11Z
    2024-10-20T15:34:41Z - 2024-10-20T15:35:59Z (no base snapshot)

The 2024-10-20T15:34:41Z - 2024-10-20T15:35:59Z (no base snapshot) is an oplog range without a valid base snapshot.
The 2024-10-20T15:36:00Z - 2024-10-20T15:38:11Z is an oplog range that is based on a valid snapshot:
2024-10-20T15:35:54Z 16.96KB <logical> [restore_to_time: 2024-10-20T15:35:59Z]

2024-10-20T15:35:54Z is the backup name (base-snapshot value).
[restore_to_time: 2024-10-20T15:35:59Z] is the actual restore time of the backup (UTC time in ISO-8601). As you can see, it is 1 second before the first oplog range (backup-based).

To restore to a time, you need to specify the --base-snapshot and --time parameters. For example, let’s imagine we have:

  • physical backup 2024-11-20T16:35:51Z 1GB <physical> [restore_to_time: 2024-11-20T16:40:25Z]
  • oplog range: 2024-10-20T15:36:00Z - 2024-11-21T13:01:11Z

To restore to 2024-11-20T17:00:00Z, you should run:
pbm restore --base-snapshot="2024-11-20T16:35:51Z" --time="2024-11-20T17:00:00".
2024-11-20T16:35:51Z - backup name.
2024-11-20T17:00:00 - time in UTC (but without UTC designator (“Z”))

*NOTE: do not forget to shut down all mongos before running the command.

Point-in-Time Restore first restores a base-snapshot (as for v2.2.1, if you do not provide the option, PBM will use the most recent Full Logical Backup). Since v2.2.0, you can use a Full Physical or Incremental backup for the value.

We are going to extend the valid base snapshot to Full Physical and Incremental backups ([PBM-1162] Make physical/incremental backup as valid base-snapshot). Don’t miss it :slight_smile:

1 Like
3

Thanks!

It is not clear why the oplogOnly parameter is needed if it cannot be used without a base snapshot.

I had pitr.enabled = true, I created a logical backup on the 18th, but oplog only started recording when I added oplogOnly, that is on the 19th, why?

Без имени
Без имени1322×117 12.9 KB

4

proper order is:

  • enable PITR (oplogOnly=false)
  • make full logical backup

then it will start slicing.

If you do backup first and then enable PITR, PBM will try to fetch a missed oplog range between the last successful backup’s restore time and the time of enabling it (the last backup should be the full logical). however, if oplogOnly=true, PBM does not catch up oplog range from the last backup but just starts for the time of enabling. (keep in mind, MongoDB oplog uses capped collection and some load could impact oplog range availability. better to do backup after PITR on)

in your case, between full logical backup and the time of enabling PITR there are 3 backups made.
the last backup is non full logical backup.

the oplogOnly is useful if you plan to never use the full logical backup. you start the oplog slicing (background process) and do physical/incremental backups periodically. if you need to restore to a time, you will need to specify explicitly the base snapshot. (as I mentioned, this will be automated later)

but, it is impossible to start slicing based on Snapshot-based physical backups because PBM has limited control on user’s snapshots. (no workaround for oplogOnly)

1 Like
5

It was included from the very beginning, that’s the point.

That is, I can use all other types of backups (logical, physical and incremental base) or only logical? If only logical, then why?

6

Sorry, it was turned off. I remembered that to use a config file you need to run something like pbm config --file pbm_config.yaml, and despite the fact that it was included in the configuration file, in fact it was not.

7

It was included from the very beginning, that’s the point.

I lost context. What was included and where?

Sorry, it was turned off.

What was off and where?

That is, I can use all other types of backups (logical, physical and incremental base) or only logical?

For what do you want to use it?
Just to restore specific database snapshot? - obviously yes.
As a base snapshot in PITR - already answered (and it is documented) - “Since v2.2.0, you can use a Full Physical or Incremental backup for the value.”
To start oplog slicing - answered here.

If only logical, then why?

Answers to this kind of questions can be found in PBM Docs or Percona Blog. If there is no information, it is implementation details and can be changed (answers could not be the same between releases). But you can also look at source code.

I highly recommend reading the Percona Backup for MongoDB Documentation. It contains all the information.

PBM is an easy-to-use “toolset” when you have prepared backup and recovery strategies/scenarios. In this case, you just verify how much it works well when “replaying” the steps (the docs have answers for how).
Otherwise, read all the PBM docs/changelog or Percona Blog (“countless” articles). I believe we have something interesting there for inspiration.

I hope it will be helpful for you :slight_smile: