Can't cancel pbm backup

MongoDB Percona Backup for MongoDB
1

I recently had a out of disk error in one of my MongoDB Instances. This happened while a backup was running. The backup never completed, but if I try to cancel it nothing happens.

pbm cancel-backup
Backup cancellation has started

It says Backup cancellation has started, but even after quite some time it is still marked as “running” when I execute pbm status.

There is nothing in the logs beside:

2021-04-28T07:39:47.000+0000 I got command cancelBackup <ts: 1619595586>

I tried deleting it but that doesn’t work as it’s still running:

pbm delete-backup 2021-04-27T00:19:49Z
Are you sure you want delete backup(s)? [y/N] y
Waiting for delete to be done ..
Error: deleting: unable to delete backup in running state

I tried disabling PITR as that helped with another problem before but to no avail.

The problem is that while it’s still running I can’t run another backup. So my PBM is basically unusable.

I guess I might be able to just remove it from the database directly, but I am hesistant to mess with the PBM documents without going through the command line as I have no idea if it might lead to unintended side effect.

MongoDB Version: 4.4.0
PBM Version: 1.4.1

1 Like
2

Hi @jaschaio

I suppose because out of disk pbm-agents weren’t able to receive the cancellation command or properly handle it.
Anyway, try to restart pbm-agents. And run a new backup in 30 sec after that. That should work.
After restarting agents its locks became stale in 15-20 sec. The next pbm operation will recognise it and remove such locks and marks the stale operation as failed. And system should be available again.