Description:
Kubernetes.
We have Percona MongoDB Operator deployed at version 1.21.1.
We will now deploy a MongoDB database using the Helm Chart from GitHub [ percona-helm-charts/charts/psmdb-db at main · percona/percona-helm-charts · GitHub ].
Steps to Reproduce:
Once deployed and the data imported, we can see that the resync storage action is continuously running. This causes backups to stop after a certain amount of time, leaving the executions in a waiting state or simply without a status after several pending status.
finalizers:
- percona.com/delete-psmdb-pods-in-order
- percona.com/delete-psmdb-pvc
nameOverride: ""
fullnameOverride: ""
crVersion: 1.21.1
pause: false
unmanaged: false
unsafeFlags:
tls: false
replsetSize: false
mongosSize: false
terminationGracePeriod: false
backupIfUnhealthy: false
enableVolumeExpansion: true
multiCluster:
enabled: false
updateStrategy: SmartUpdate
upgradeOptions:
versionServiceEndpoint: https://check.percona.com
apply: disabled
schedule: "0 2 * * *"
setFCV: false
image:
repository: percona/percona-server-mongodb
tag: 6.0.25-20
imagePullPolicy: Always
secrets: {}
pmm:
enabled: false
image:
repository: percona/pmm-client
tag: 3.4.1
serverHost: monitoring-service
replsets:
rs0:
name: rs0
size: 3
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
podDisruptionBudget:
maxUnavailable: 2
expose:
enabled: false
type: ClusterIP
resources:
requests:
cpu: "500m"
memory: "1Gi"
volumeSpec:
pvc:
resources:
requests:
storage: 5Gi
hidden:
enabled: false
size: 2
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
podDisruptionBudget:
maxUnavailable: 1
resources:
limits:
cpu: "600m"
memory: "1Gi"
requests:
cpu: "300m"
memory: "1Gi"
volumeSpec:
pvc:
resources:
requests:
storage: 3Gi
nonvoting:
enabled: false
size: 3
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
podDisruptionBudget:
maxUnavailable: 1
resources:
limits:
cpu: "600m"
memory: "1Gi"
requests:
cpu: "300m"
memory: "1Gi"
volumeSpec:
pvc:
resources:
requests:
storage: 3Gi
arbiter:
enabled: false
size: 1
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
resources:
limits:
cpu: "600m"
memory: "1Gi"
requests:
cpu: "300m"
memory: "1Gi"
sharding:
enabled: false
balancer:
enabled: false
configrs:
size: 3
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
podDisruptionBudget:
maxUnavailable: 1
expose:
enabled: false
type: ClusterIP
resources:
limits:
cpu: "600m"
memory: "1Gi"
requests:
cpu: "300m"
memory: "1Gi"
volumeSpec:
pvc:
resources:
requests:
storage: 3Gi
mongos:
size: 3
affinity:
antiAffinityTopologyKey: "kubernetes.io/hostname"
podDisruptionBudget:
maxUnavailable: 1
resources:
limits:
cpu: "600m"
memory: "1Gi"
requests:
cpu: "300m"
memory: "1Gi"
expose:
enabled: false
type: ClusterIP
users:
- name: sample
db: sample
roles:
- name: dbOwner
db: sample
backup:
enabled: true
image:
repository: percona/percona-backup-mongodb
tag: 2.11.0
resources:
requests:
cpu: "500m"
memory: "1Gi"
storages:
s3:
main: true
type: s3
s3:
bucket: backups
prefix: "mongodb/sample"
credentialsSecret: minio-credentials
endpointUrl: http://minio.minio.svc.cluster.local:9000
insecureSkipTLSVerify: true
retryer:
numMaxRetries: 3
minRetryDelay: 100s
maxRetryDelay: 5m
pitr:
enabled: false
oplogOnly: false
tasks:
- name: daily-backup
enabled: true
schedule: "*/15 * * * *"
retention:
count: 7
type: count
deleteFromStorage: true
storageName: s3
type: physical
logcollector:
enabled: false
image:
repository: percona/fluentbit
tag: 4.0.1
# configuration: |
# [SERVICE]
# Flush 1
# Log_Level error
# Daemon off
resources:
requests:
memory: 100M
cpu: 200m
It doesn’t matter if the backup is scheduled every 15 minutes or every 24 hours.
Version:
Versión de mongodb: 6.0.25-20
Version de mongo backup: 2.11
Logs:
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:38:52.000+0000 I starting PITR routine
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:38:52.000+0000 I node: rs0/sample-db-psmdb-db-rs0-2.sample-db-psmdb-db-rs0.mongodb-sample.svc.cluster.local:27017
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:38:52.000+0000 I conn level ReadConcern: majority; WriteConcern: majority
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:38:52.000+0000 I listening for the commands
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 I got command resync <ts: 1762868346>, opid: 69133c7a012d514f2a9937fd
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 I got epoch {1762868336 1}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 D [resync] epoch set to {1762868338 21}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:25.000+0000 I got command resync <ts: 1762868356>, opid: 69133c84012d514f2a993809
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:25.000+0000 I got epoch {1762868348 17}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:25.000+0000 D [resync] get lock: duplicate operation: 69133c84012d514f2a993809 [Resync storage]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:25.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 I got command resync <ts: 1762868368>, opid: 69133c90012d514f2a993815
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 I got epoch {1762868359 2}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 D [resync] epoch set to {1762868360 19}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:47.000+0000 I got command resync <ts: 1762868379>, opid: 69133c9b012d514f2a993821
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:47.000+0000 I got epoch {1762868369 2}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:47.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:47.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:48.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:48.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:48.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:48.000+0000 D [resync] epoch set to {1762868371 22}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:48.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:59.000+0000 I got command resync <ts: 1762868390>, opid: 69133ca6012d514f2a99382d
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:59.000+0000 I got epoch {1762868382 17}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:59.000+0000 D [resync] get lock: duplicate operation: 69133ca6012d514f2a99382d [Resync storage]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:59.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:10.000+0000 I got command resync <ts: 1762868402>, opid: 69133cb2012d514f2a993839
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:10.000+0000 I got epoch {1762868391 1}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:10.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:10.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:11.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:11.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:11.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:11.000+0000 D [resync] epoch set to {1762868394 21}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:11.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 I got command resync <ts: 1762868413>, opid: 69133cbd012d514f2a993845
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 I got epoch {1762868404 2}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 D [resync] epoch set to {1762868405 21}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:33.000+0000 I got command resync <ts: 1762868425>, opid: 69133cc9012d514f2a993851
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:33.000+0000 I got epoch {1762868416 18}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] get lock: duplicate operation: 69133cc9012d514f2a993851 [Resync storage]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 I got command resync <ts: 1762868430>, opid: 69133cce012d514f2a993857
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 I got epoch {1762868421 1}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:31.000+0000 I got command resync <ts: 1762868430>, opid: 69133cce012d514f2a993857
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:31.000+0000 I got epoch {1762868421 1}
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:31.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 D [resync] epoch set to {1762868422 17}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:23.000+0000 I got command resync <ts: 1762868425>, opid: 69133cc9012d514f2a993851
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:23.000+0000 I got epoch {1762868422 17}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:23.000+0000 D [resync] get lock: duplicate operation: 69133cc9012d514f2a993851 [Resync storage]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:23.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 I got command resync <ts: 1762868436>, opid: 69133cd4012d514f2a99385d
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 I got epoch {1762868426 1}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:28.000+0000 I got command resync <ts: 1762868436>, opid: 69133cd4012d514f2a99385d
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:28.000+0000 I got epoch {1762868426 1}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:28.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:37.000+0000 I got command resync <ts: 1762868436>, opid: 69133cd4012d514f2a99385d
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:37.000+0000 I got epoch {1762868426 1}
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:37.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 D [resync] epoch set to {1762868428 19}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:29.000+0000 I got command resync <ts: 1762868430>, opid: 69133cce012d514f2a993857
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:29.000+0000 I got epoch {1762868428 19}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:29.000+0000 D [resync] get lock: duplicate operation: 69133cce012d514f2a993857 [Resync storage]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:29.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 I got command resync <ts: 1762868442>, opid: 69133cda012d514f2a993863
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 I got epoch {1762868431 1}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:42.000+0000 I got command resync <ts: 1762868442>, opid: 69133cda012d514f2a993863
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:42.000+0000 I got epoch {1762868431 1}
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:42.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] epoch set to {1762868433 17}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:34.000+0000 I got command resync <ts: 1762868436>, opid: 69133cd4012d514f2a99385d
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:34.000+0000 I got epoch {1762868433 17}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:34.000+0000 D [resync] get lock: duplicate operation: 69133cd4012d514f2a99385d [Resync storage]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:34.000+0000 D [resync] lock not acquired