# Full and incremental backups conflicting and stopping all backups

**URL:** <https://forums.percona.com/t/full-and-incremental-backups-conflicting-and-stopping-all-backups/33157>\
**Category:** Percona Operator for PostgreSQL\
**Created:** [September 16, 2024, 12:45am UTC](https://forums.percona.com/t/full-and-incremental-backups-conflicting-and-stopping-all-backups/33157 "2024-09-16T00:45:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![seacom](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@seacom](https://forums.percona.com/u/seacom)\
**Post date:** [September 16, 2024, 12:45am UTC](https://forums.percona.com/t/full-and-incremental-backups-conflicting-and-stopping-all-backups/33157/1 "2024-09-16T00:45:48Z")

</div>

## Description:

I have incremental backups scheduled at 5 minute intervals and a nightly full backup. For the past several months this is worked great, the pgbackrest lock file has always conflicted causing a failed job somewhere but the operator would run the backup again and the issue would resolve it self.

However the addition of the latestrestorabletime update doesn’t play well with these as the conflicted backups produce a perconapgbackup that can’t verified/reconciled whatever.

I have no idea what happens in the background but this eventually leads to no backup jobs being created.

The operator doesn’t log anything to indicate there is a fault and backup jobs stop being created eventually, prior to that the backup jobs just log  
“HINT: is another pgBackRest process running?”. And the operator just loops over “Triggering PGBackup reconcile” and "Latest commit timestamp " with “latestRestorableTime”: “”.

There is no actual errors though.

It follows a pattern of exponential failures until backups stop altogether.

## Steps to Reproduce:

Schedule full and log backups to s3 to run at the same time for several days or until backup job pods stop being created (perconapgbackup objects will still be created without a status or s3 path).

## Version:

2.4.1

## Additional Information:

I’m pretty sure several of the other issues regarding backups on this version are just symptoms of this, i experienced a large azure and s3 bill as a result of the operator constantly trying to reconcile non existing backups.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [September 16, 2024, 2:08pm UTC](https://forums.percona.com/t/full-and-incremental-backups-conflicting-and-stopping-all-backups/33157/2 "2024-09-16T14:08:11Z")

</div>

> [@seacom](#):
>
> I have incremental backups scheduled at 5 minute intervals

Wow. That’s a bit crazy. For the sake of fixing your issue, have you tried changing this to something like 30m? If that fixes the issue, then the original problem was that you are taking incrementals too frequently and causing an internal race condition.

---

<div class="post-metadata">

**Author:** ![seacom](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@seacom](https://forums.percona.com/u/seacom)\
**Post date:** [September 16, 2024, 10:01pm UTC](https://forums.percona.com/t/full-and-incremental-backups-conflicting-and-stopping-all-backups/33157/3 "2024-09-16T22:01:13Z")

</div>

> [@matthewb](#):
>
> Wow. That’s a bit crazy.

Honestly I don’t think it is. 5 minutes is standard for most databases. Not to mention the zalando operator with WALG never had an issue keeping up.

> [@matthewb](#):
>
> For the sake of fixing your issue, have you tried changing this to something like 30m?

I have actually. It did not. It just delayed the time it took to get there as its an exponential problem. Because the issue is the incremental and full running at the same time and competing for the pgackrest lock file, when 1 fails and can’t get the lock the operator restarts the job until it does, however the backups that failed generate perconapgbackup objects which the operator gets stuck in a loop trying to reconcile because the perconapgbackup objects have no status of them.

This wasn’t an issue until latestrestorabletime was added.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [September 17, 2024, 4:35am UTC](https://forums.percona.com/t/full-and-incremental-backups-conflicting-and-stopping-all-backups/33157/4 "2024-09-17T04:35:36Z")

</div>

> [@seacom](#):
>
> Honestly I don’t think it is. 5 minutes is standard for most databases.

As someone who has worked on literally thousands of MySQL servers, across all industries (Medical, PCI, Fintech, gaming…) I can legitimately say, no, 5 min interval backups is absolutely not the standard. That would cause a crazy amount of overhead.

The most frequent interval backups I’ve seen were every 8hrs. _Standard_, is more like daily fulls with 5m rsync of the binary logs (or standing up a binlog server to receive immediate transaction copies). Multi-TB typically do a full once a week, then daily intervals, plus the binlog sync. MySQL binlogs are similar to PGSQL’s WAL archiving. I’ll ask one of my PGSQL consultants to chime in on this best practice.

Regarding your current issue, I’ll ping our operators tech lead and see if he has any insights.

---

<div class="post-metadata">

**Author:** ![Slava\_Sarzhan](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/slava_sarzhan/32/3196_2.png) [@Slava\_Sarzhan](https://forums.percona.com/u/Slava_Sarzhan)\
**Post date:** [September 17, 2024, 6:57am UTC](https://forums.percona.com/t/full-and-incremental-backups-conflicting-and-stopping-all-backups/33157/5 "2024-09-17T06:57:28Z")

</div>

Hi @seacom, We will try to reproduce this issue and the same time starting from v2.5.0 it will be possible to disable “latestRestorableTime” tracking via [percona-postgresql-operator/deploy/cr.yaml at main · percona/percona-postgresql-operator · GitHub](https://github.com/percona/percona-postgresql-operator/blob/main/deploy/cr.yaml#L307) option. I hope that we will have a release next week.

---

<div class="post-metadata">

**Author:** ![seacom](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@seacom](https://forums.percona.com/u/seacom)\
**Post date:** [September 17, 2024, 11:28am UTC](https://forums.percona.com/t/full-and-incremental-backups-conflicting-and-stopping-all-backups/33157/6 "2024-09-17T11:28:27Z")

</div>

> [@Slava\_Sarzhan](#):
>
> I hope that we will have a release next week.

Awesome thanks.

Perhaps I am confusing continuous archiving with incremental backups.

---

<div class="post-metadata">

**Author:** ![Emmanuel\_Tom\_Jose](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/emmanuel_tom_jose/32/23312_2.png) [@Emmanuel\_Tom\_Jose](https://forums.percona.com/u/Emmanuel_Tom_Jose)\
**Post date:** [February 26, 2026, 6:07am UTC](https://forums.percona.com/t/full-and-incremental-backups-conflicting-and-stopping-all-backups/33157/7 "2026-02-26T06:07:28Z")

</div>

We hit the same issue — full and incremental schedules competing for the pgBackRest lock, failed PerconaPGBackup objects piling up, and eventually all scheduled backups stopping. This persists beyond  
v2.4.x — similar behavior has been reported on [Can't start backup. Previous backup is still in progress](https://forums.percona.com/t/cant-start-backup-previous-backup-is-still-in-progress/37913) , where a stuck PerconaPGBackup in “Starting” state blocks all  
future backups.

Our workaround was to bypass the operator’s native scheduling entirely and use external Kubernetes CronJobs that kubectl exec into the repo-host pod. The key difference: the script checks if a backup  
is already running before attempting one, so it never creates a conflicting backup that would generate a stuck PerconaPGBackup object.

Here’s the core logic (same for both full and incremental, just change --type=full to --type=incr):

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: pgbackrest-full-weekly
namespace: postgres-operator
spec:
schedule: “30 2 * * 0”
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
ttlSecondsAfterFinished: 86400
template:
spec:
serviceAccountName: pgbackrest-cronjob-sa
containers:
- name: backup-trigger
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- |
set -e
# Find the repo-host pod
REPO_HOST=$(kubectl get pod -n postgres-operator 
-l postgres-operator.crunchydata.com/data=pgbackrest 
–field-selector=status.phase=Running 
-o jsonpath=‘{.items[0].metadata.name}’ 2>/dev/null)
if [-z “$REPO_HOST”]; then
echo “SKIP: No running repo-host pod found”
exit 0
fi

# Check stanza is initialized
STANZA_INFO=$(kubectl exec -n postgres-operator "$REPO_HOST" \
-c pgbackrest -- pgbackrest info --stanza=db \
--output=json 2>/dev/null) || {
echo "SKIP: stanza not ready yet"
exit 0
}

# KEY: Check if another backup holds the lock
if echo "$STANZA_INFO" | grep -q '"held":true'; then

echo "SKIP: Another backup is already running"
exit 0
fi

# Safe to run
kubectl exec -n postgres-operator "$REPO_HOST" \ -c pgbackrest -- pgbackrest backup – stanza=db --type=full --log-level-console=info 

restartPolicy: OnFailure

```

You’ll need a ServiceAccount + Role + RoleBinding with pods: [get, list] and pods/exec: [create] in the namespace.

Why this works where native scheduling doesn’t:

- concurrencyPolicy: Forbid — Kubernetes prevents overlapping CronJob runs
- ttlSecondsAfterFinished: 86400 — completed Jobs cleaned up after 24h, no backlog
- successfulJobsHistoryLimit/failedJobsHistoryLimit: 3 — caps Job object history
- The “held”:true lock check — gracefully skips instead of failing and creating stuck PerconaPGBackup objects
- Bypasses operator state tracking entirely — no [pgv2.percona.com/backup-in-progress](https://forums.percona.com/t/cant-start-backup-previous-backup-is-still-in-progress/37913) annotation issues

Remove the schedules block from your PerconaPGCluster CR when switching to this approach. We’ve been running this in production (weekly full + 4-hourly incremental to S3) for months with zero backlog  
issues.