[pitr] oplog slicer is paused for lock [Snapshot backup, opid: 692902205e216c21fa537eaf]
The oplog slicer remains paused indefinitely, preventing the backup from completing and affecting point-in-time recovery capabilities.
Steps to Reproduce:
-
Deploy Percona MongoDB cluster in Kubernetes
-
Create a backup schedule/trigger manual backup
-
Monitor backup agent logs
-
Observe the “oplog slicer is paused for lock” message
i run operator 1.21.1
The oplog slicer is paused for lock is not an error, and the cause of this is that PBM stops saving PITR chunks when a logical backup is running. For example, if you have 1TB of data, it’s expected that the backup will take a long time to complete.
Isn’t it ever continuing in your case?
Have you ever had a successful backup?
I have some backup ready and some backup fails due the same reason oplog slicer is paused for lock
after see the logs on another node of backup-agent
mark RS as error dump: dump namespaces: pay_687dsfsd851393.filemanager_logs: close: upload: "pay_687dsfsd851393.filemanager_logs": save during split-merge mw: upload to S3: operation error S3: PutObject, exceeded maximum number of attempts, 6, https response error StatusCode: 504, RequestID: N/A, HostID: N/A, api error GatewayTimeout: The server did not respond in time.:
can you help?
as per the log above you are getting timeouts talking to the storage. The problem is either the network or the remote storage then.
Manual backups execute successfully, however, the automated cron-based backups are failing with an error.
this my config
—————————-
backup:
enabled: true
image: perconalab/percona-server-mongodb-operator:main-backup
storages:
NBG1:
type: s3
s3:
bucket: test-bk-test
region: NBG1
credentialsSecret: mongodb-prod-1-backup-s3
endpointUrl: https://nbdc.your-objectstorage.com
retryer:
numMaxRetries: 5
minRetryDelay: 30ms
maxRetryDelay: 10m
prefix: ""
uploadPartSize: 20971520
maxUploadParts: 10000
pitr:
enabled: true
oplogOnly: false
oplogSpanMin: 10
compressionType: gzip
compressionLevel: 6
tasks:
# =============
# DAILY BACKUP at 12PM UTC 0
# =============
- name: mongodb-prod-1-backup-daily
enabled: true
retention:
count: 3 # keep 1 backup each day, 3 days max
type: count
deleteFromStorage: true
type: logical # best for storage, slower
schedule: 0 2 \* \* \* # daily at 2 AM UTC 0
storageName: NBG1
# =============
# WEEKLY BACKUP at SATURDAY 12AM UTC 0
# =============
- name: mongodb-prod-1-backup-weekly
enabled: true
retention:
count: 2 # keep 1 backup each week, 2 weeks max
type: count
deleteFromStorage: true
type: logical # best for storage, slower
schedule: 0 0 \* \* 5 # weekly at 12AM UTC 0 friday
storageName: NBG1
# =============
# MONTHLY BACKUP at SATURDAY 12AM UTC 0
# =============
- name: mongodb-prod-1-backup-MONTHLY
enabled: true
retention:
count: 3 # keep 1 backup each month, 3 months max
type: count
deleteFromStorage: true
type: logical # best for storage, slower
schedule: 0 4 1 \* \* # 1st day of every month at 4 AM
storageName: NBG1