# PSMDB Operator flags scheduled backup CR as 'error' despite PBM success, breaking automated retention

**URL:** <https://forums.percona.com/t/psmdb-operator-flags-scheduled-backup-cr-as-error-despite-pbm-success-breaking-automated-retention/41015>\
**Category:** MongoDB\
**Created:** [July 1, 2026, 9:38am UTC](https://forums.percona.com/t/psmdb-operator-flags-scheduled-backup-cr-as-error-despite-pbm-success-breaking-automated-retention/41015 "2026-07-01T09:38:04Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![karam\_alhalabi](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/karam_alhalabi/32/22724_2.png) [@karam\_alhalabi](https://forums.percona.com/u/karam_alhalabi)\
**Post date:** [July 1, 2026, 9:38am UTC](https://forums.percona.com/t/psmdb-operator-flags-scheduled-backup-cr-as-error-despite-pbm-success-breaking-automated-retention/41015/1 "2026-07-01T09:38:05Z")

</div>

Hello,

I am running Percona Server for MongoDB on Kubernetes via the PSMDB Operator (v1.21.0) with PITR enabled. I am experiencing an issue where my scheduled cron backups are successfully executed by `pbm-agent` and uploaded to my DigitalOcean Spaces S3 bucket, but the `PerconaServerMongoDBBackup` Custom Resource (CR) transitions to an `error` state.

Because the Kubernetes CR stays in an `error` state, the operator’s configured backup retention policy (`type: count`, `count: 3`, `deleteFromStorage: true`) completely skips processing them, leading to unbounded storage growth in our object store.

#### **Environment Details:**

- **Operator Version:** psmdb-operator-1.21.3

- **PBM Version:** 2.11.0

- **Storage Provider:** DigitalOcean Spaces (S3 compatible)

- **Features Enabled:** Scheduled logical backups, PITR incremental streaming

#### **Observed Behavior & Diagnostic Data:**

When describing the failed `psmdb-backup` CR, the status shows it timed out waiting to progress past the initialization frame:

```auto
Status:
  Destination: s3://test-test2008/2026-07-01T04:00:00Z
  Error: backup did not progress from 'starting' state within the allowed timeout
  Last Transition: 2026-07-01T04:00:00Z
  Pbm Name: 2026-07-01T04:00:00Z
  State: error

```

However, checking the operator logs around that time frame shows that the backup leases/locks are processed and successfully released:

```auto
2026-06-30T10:53:36.311Z INFO releasing backup lock {"controller": "psmdbbackup-controller", "lease": "psmdb-mongodb-prod-1-psmdb-backup-lock", "holder": "cron-mongodb-prod-1-p-20260630020000-rbsfk-..."}
2026-06-30T10:54:00.290Z INFO PBM updating latest restorable time {"controller": "psmdb-controller", "backup": "cron-mongodb-prod-1-p-20260503020000-2fkch", "latestRestorableTime": "2026-06-30 10:53:56 +0000 UTC"}

```

Furthermore, checking the true underlying status directly via the `pbm` CLI inside the `backup-agent` container confirms that the snapshots **are perfectly valid and marked as a success** by the backup engine:

```auto
$ kubectl exec -it mongodb-prod-1-psmdb-rs0-0 -c backup-agent -- pbm status

PITR incremental backup:
========================
Status [ON]

Backups:
========
S3 sgp1 https://sgp1.digitaloceanspaces.com/test-test2008
  Snapshots:
    2026-06-30T02:00:00Z 8.44GB <logical> success [restore_to_time: 2026-06-30T02:21:50]
    2026-06-29T02:00:00Z 8.07GB <logical> success [restore_to_time: 2026-06-29T02:20:10]
    2026-06-28T02:00:00Z 8.22GB <logical> success [restore_to_time: 2026-06-28T02:21:01]

```

#### **The Problem:**

There appears to be a severe race condition or rigid timing mismatch between the Operator’s status reconcile loop and PBM’s actual agent initialization. Because a heavy cluster workload can delay the state transition from `starting` to `running` for over 60 seconds, the operator permanently flags the resource as an `error`.

Because it is flagged as an `error`, the operator’s retention manager refuses to evaluate it, causing old successful snapshots to stack up in S3 storage indefinitely.

Has anyone found a clean way to raise the internal threshold for the `starting` state timeout via the operator deployment configurations, or is there a patch planned to allow the retention engine to clean up backups that PBM natively marks as `success` regardless of the CR status?

---

<div class="post-metadata">

**Author:** ![Mayank\_Shah](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mayank_shah/32/15501_2.png) [@Mayank\_Shah](https://forums.percona.com/u/Mayank_Shah)\
**Post date:** [July 1, 2026, 11:35am UTC](https://forums.percona.com/t/psmdb-operator-flags-scheduled-backup-cr-as-error-despite-pbm-success-breaking-automated-retention/41015/2 "2026-07-01T11:35:34Z")

</div>

Hi @karam_alhalabi !

Starting from v1.22.0, we allow configuring a `startingDeadlineSeconds` on the backup spec. The default is 120s, but you can raise the value based on your requirements.

Let me know if this fixes it for you!