We are using percona MongoDB version 4.2.8 as a replica set across three machines and percona backup version 1.2.1 on centOS7. Currently our pbm-agents do a full backup every two hours (though we have now reduced this to once a day). Recently when the pbm-agents on the secondary machines are executing a backup, they spike to a memory usage of upwards of 24% and we think this has lead to our mongod instances being shut down by the kernel as it has run out of memory. Is there any way to configure the memory that the pbm-agent uses when it executes a backup?
Currently, there is no way to adjust memory consumption during backup.
Can you tell in absolute numbers how much memory does agent consume during the backup (how much Mb)? And what are the size of your DB and average document size?
It’s ok for an agent to increase memory consumption during the backup since it needs buffers to read and compress data. And it might be around 400-500 Mb.
Btw, since v1.3.0 PBM supports Point-In-Time recovery (https://www.percona.com/doc/percona-backup-mongodb/point-in-time-recovery.html) So you can leave your daily (nightly) full backups and the rest of the time cover by PITR which is by far not as heavy operation as a full backup.