# Percona Operator cannot release backup lock – all subsequent backups stuck in “waiting”

**URL:** <https://forums.percona.com/t/percona-operator-cannot-release-backup-lock-all-subsequent-backups-stuck-in-waiting/39585>\
**Category:** Percona Operator for MongoDB\
**Tags:** percona, mongodb\
**Created:** [October 20, 2025, 2:12pm UTC](https://forums.percona.com/t/percona-operator-cannot-release-backup-lock-all-subsequent-backups-stuck-in-waiting/39585 "2025-10-20T14:12:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![bofh](https://avatars.discourse-cdn.com/v4/letter/b/f6c823/32.png) [@bofh](https://forums.percona.com/u/bofh)\
**Post date:** [October 20, 2025, 2:12pm UTC](https://forums.percona.com/t/percona-operator-cannot-release-backup-lock-all-subsequent-backups-stuck-in-waiting/39585/1 "2025-10-20T14:12:05Z")

</div>

## Description:

The Percona Server for MongoDB Operator becomes unable to perform new backups due to the message

> “Another backup is holding the lock.”

There appears to be no way to clear this backup lock, even after deleting related Kubernetes resources and restarting the operator.

## Steps to Reproduce:

- 
  1. **Create manual backup**

- 

```auto
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  name: manual-BACKUPVERSION
spec:
  psmdbCluster: x-y-mongodb-cluster
  storageName: x-y-backup

```

- 
  1. **Restore manual backup**

- 

```auto
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
  name: manual-restore
spec:
  clusterName: x-y-mongodb-cluster
  backupName: manual-2025-10-16-10-45-44

```

- 
  1. \*\*Attemp to run additional backups \*\*  
Manual backup or backup by cron job

## Version:

- **Percona Operator for MongoDB:** 1.20.1
- **Platform:** Red Hat OpenShift on AWS (ROSA)
- **OpenShift Version:** 4.19.14

## Logs:

```auto
INFO Acquiring the backup lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"manual-2025-10-20-15-11-13","namespace":"x-y"}, "namespace": "x-y", "name": "manual-2025-10-20-15-11-13", "reconcileID": "xxx"}

INFO Another backup is holding the lock {"controller": "psmdbbackup-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBBackup", "PerconaServerMongoDBBackup": {"name":"manual-2025-10-20-15-11-13","namespace":"x-y"}, "namespace": "x-y", "name": "manual-2025-10-20-15-11-13", "reconcileID": "xxx", "holder": "manual-2025-10-16-10-45-44-776ab071-f45d-4d11-a45c-8f04d0e2f20b"}

```

## Expected Result:

- New backups should start and complete successfully after the previous backup/restore has finished.

## Actual Result:

- All subsequent backups enter the `waiting` or `error` state, and the operator logs show:

## Additional Information:

- Deleted the `perconaservermongodbbackups.psmdb.percona.com` backup CR

- Deleted the `perconaservermongodbrestores.psmdb.percona.com` restore CR

- Deleted all related PVCs

- Deleted the MongoDB StatefulSet

- Restarted the `percona-server-mongodb-operator` deployment and pod

- Verified no entries exist in:

Despite these steps, new backups continue to report “Another backup is holding the lock”.

---

<div class="post-metadata">

**Author:** ![Sami\_Ahlroos](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sami_ahlroos/32/20214_2.png) [@Sami\_Ahlroos](https://forums.percona.com/u/Sami_Ahlroos)\
**Post date:** [October 21, 2025, 11:59am UTC](https://forums.percona.com/t/percona-operator-cannot-release-backup-lock-all-subsequent-backups-stuck-in-waiting/39585/2 "2025-10-21T11:59:00Z")

</div>

Hi,

That lock is a Kubernetes [Lease](https://kubernetes.io/docs/concepts/architecture/leases/). You should see `psmdb-clusterName-backup-lock` if you run `kubectl get lease`. Could you try deleting it, and see if that fixes the issue?  
I haven’t been able to reproduce this problem on my side, unfortunately.

---

<div class="post-metadata">

**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\
**Post date:** [October 21, 2025, 3:23pm UTC](https://forums.percona.com/t/percona-operator-cannot-release-backup-lock-all-subsequent-backups-stuck-in-waiting/39585/3 "2025-10-21T15:23:33Z")

</div>

Hi @bofh, could you please check your operator’s log and “grep” the ‘delete lease’ error?

---

<div class="post-metadata">

**Author:** ![bofh](https://avatars.discourse-cdn.com/v4/letter/b/f6c823/32.png) [@bofh](https://forums.percona.com/u/bofh)\
**Post date:** [October 21, 2025, 4:10pm UTC](https://forums.percona.com/t/percona-operator-cannot-release-backup-lock-all-subsequent-backups-stuck-in-waiting/39585/4 "2025-10-21T16:10:10Z")

</div>

Thanks @Sami_Ahlroos

I found the kubernetes lease with `kubectl get lease` and was able to delete it with `kubectl delete lease`

Queed backup-jobs were able to finish successfully and new jobs are running fine..

@Slava_Sarzhan  
Have been able to reproduce the behavior again with manual backup / restore procedure.

Here are the logs with the “release lease” error:

`2025-10-21T15:58:21.220Z ERROR failed to release the lock {“controller”: “psmdbbackup-controller”, “controllerGroup”: “``psmdb.percona.com``”, “controllerKind”: “PerconaServerMongoDBBackup”, “PerconaServerMongoDBBackup”: {“name”:“manual-2025-10-21-17-58-10”,“namespace”:“x-y”}, “namespace”: “x-y”, “name”: “manual-2025-10-21-17-58-10”, “reconcileID”: “049d05ee-08d6-4b34-a9be-d743bf66cac6”, “error”: “get lease: ``Lease.coordination.k8s.io`` "psmdb–backup-lock" not found”, “errorVerbose”: “``Lease.coordination.k8s.io`` "psmdb–backup-lock" not found\nget lease\``ngithub.com/percona/percona-server-mongodb-operator/pkg/k8s.ReleaseLease\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/k8s/lease.go:52\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:159\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:221\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[…]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700”``}`

**Afterwards deleted the lease manually:**  
`kubectl delete lease psmdb-x-y-mongodb-cluster-backup-lock`

`2025-10-21T16:03:58.338Z INFO Releasing backup lock {“controller”: “psmdbbackup-controller”, “controllerGroup”: “``psmdb.percona.com``”, “controllerKind”: “PerconaServerMongoDBBackup”, “PerconaServerMongoDBBackup”: {“name”:“cron-x-y-mongod-20251021160000-qtl7k”,“namespace”:“x-y”}, “namespace”: “x-y”, “name”: “cron-x-y-mongod-20251021160000-qtl7k”, “reconcileID”: “9672d4fa-621f-4c0c-bd25-797b6f2c4fad”, “lease”: “psmdb-x-y-mongodb-cluster-backup-lock”}`

---

<div class="post-metadata">

**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\
**Post date:** [October 21, 2025, 5:49pm UTC](https://forums.percona.com/t/percona-operator-cannot-release-backup-lock-all-subsequent-backups-stuck-in-waiting/39585/5 "2025-10-21T17:49:58Z")

</div>

@bofh, pelase check your RBAC and make sure that you have:

```auto
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete

```

I can’t reproduce it using GKE. We will test it on Red Hat OpenShift on AWS (ROSA) in the next few days. We will update you.

P.S. Do you have a cluster-wide deployment or a namespace-scoped one?

---

<div class="post-metadata">

**Author:** ![bofh](https://avatars.discourse-cdn.com/v4/letter/b/f6c823/32.png) [@bofh](https://forums.percona.com/u/bofh)\
**Post date:** [October 22, 2025, 11:24am UTC](https://forums.percona.com/t/percona-operator-cannot-release-backup-lock-all-subsequent-backups-stuck-in-waiting/39585/6 "2025-10-22T11:24:52Z")

</div>

On ROSA, there is no cluster-wide deployment available; therefore, we use a namespace-scoped deployment.

Regarding the RBAC permissions: should these rights be granted to the operator account or to the service account used by the MongoDB service?

Currently, the operator account has these permissions, while the MongoDB service account (default) does not.

```auto
percona-server-mongodb-operator

get: yes
list: yes
watch: yes
create: yes
update: yes
patch: yes
delete: yes

```

```auto
default

get: no
list: no
watch: no
create: no
update: no
patch: no
delete: no

```

---

<div class="post-metadata">

**Author:** ![Julio\_Pasinatto](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/julio_pasinatto/32/17643_2.png) [@Julio\_Pasinatto](https://forums.percona.com/u/Julio_Pasinatto)\
**Post date:** [October 27, 2025, 12:33pm UTC](https://forums.percona.com/t/percona-operator-cannot-release-backup-lock-all-subsequent-backups-stuck-in-waiting/39585/7 "2025-10-27T12:33:16Z")

</div>

Hi @bofh

I wasn’t able to reproduce this issue on ROSA with OpenShift 4.19.14.  
Could you share a bit more about how you deployed the operator? For example, did you install it from the repo, using Helm charts, or through OperatorHub (Community or Certified bundle)?

Just a heads-up — PSMDB Operator **v1.20.1** hasn’t been tested with OpenShift **4.19.x**. It’s only been verified on **4.14–4.18** , so while that might not be the exact cause, it could lead to some unexpected issues.

> **[Percona Operator for MongoDB - Percona Operator for MongoDB 1.20.1 (2025-06-04)](https://docs.percona.com/percona-operator-for-mongodb/RN/Kubernetes-Operator-for-PSMONGODB-RN1.20.1.html#supported-platforms)**
>
> Get started with the Operator