Hello,
I am running Percona Server for MongoDB on Kubernetes via the PSMDB Operator (v1.21.0) with PITR enabled. I am experiencing an issue where my scheduled cron backups are successfully executed by pbm-agent and uploaded to my DigitalOcean Spaces S3 bucket, but the PerconaServerMongoDBBackup Custom Resource (CR) transitions to an error state.
Because the Kubernetes CR stays in an error state, the operator’s configured backup retention policy (type: count, count: 3, deleteFromStorage: true) completely skips processing them, leading to unbounded storage growth in our object store.
Environment Details:
-
Operator Version: psmdb-operator-1.21.3
-
PBM Version: 2.11.0
-
Storage Provider: DigitalOcean Spaces (S3 compatible)
-
Features Enabled: Scheduled logical backups, PITR incremental streaming
Observed Behavior & Diagnostic Data:
When describing the failed psmdb-backup CR, the status shows it timed out waiting to progress past the initialization frame:
Status:
Destination: s3://test-test2008/2026-07-01T04:00:00Z
Error: backup did not progress from 'starting' state within the allowed timeout
Last Transition: 2026-07-01T04:00:00Z
Pbm Name: 2026-07-01T04:00:00Z
State: error
However, checking the operator logs around that time frame shows that the backup leases/locks are processed and successfully released:
2026-06-30T10:53:36.311Z INFO releasing backup lock {"controller": "psmdbbackup-controller", "lease": "psmdb-mongodb-prod-1-psmdb-backup-lock", "holder": "cron-mongodb-prod-1-p-20260630020000-rbsfk-..."}
2026-06-30T10:54:00.290Z INFO PBM updating latest restorable time {"controller": "psmdb-controller", "backup": "cron-mongodb-prod-1-p-20260503020000-2fkch", "latestRestorableTime": "2026-06-30 10:53:56 +0000 UTC"}
Furthermore, checking the true underlying status directly via the pbm CLI inside the backup-agent container confirms that the snapshots are perfectly valid and marked as a success by the backup engine:
$ kubectl exec -it mongodb-prod-1-psmdb-rs0-0 -c backup-agent -- pbm status
PITR incremental backup:
========================
Status [ON]
Backups:
========
S3 sgp1 https://sgp1.digitaloceanspaces.com/test-test2008
Snapshots:
2026-06-30T02:00:00Z 8.44GB <logical> success [restore_to_time: 2026-06-30T02:21:50]
2026-06-29T02:00:00Z 8.07GB <logical> success [restore_to_time: 2026-06-29T02:20:10]
2026-06-28T02:00:00Z 8.22GB <logical> success [restore_to_time: 2026-06-28T02:21:01]
The Problem:
There appears to be a severe race condition or rigid timing mismatch between the Operator’s status reconcile loop and PBM’s actual agent initialization. Because a heavy cluster workload can delay the state transition from starting to running for over 60 seconds, the operator permanently flags the resource as an error.
Because it is flagged as an error, the operator’s retention manager refuses to evaluate it, causing old successful snapshots to stack up in S3 storage indefinitely.
Has anyone found a clean way to raise the internal threshold for the starting state timeout via the operator deployment configurations, or is there a patch planned to allow the retention engine to clean up backups that PBM natively marks as success regardless of the CR status?