[pitr] oplog slicer is paused for lock

[pitr] oplog slicer is paused for lock [Snapshot backup, opid: 692902205e216c21fa537eaf]
The oplog slicer remains paused indefinitely, preventing the backup from completing and affecting point-in-time recovery capabilities.

Steps to Reproduce:

  1. Deploy Percona MongoDB cluster in Kubernetes

  2. Create a backup schedule/trigger manual backup

  3. Monitor backup agent logs

  4. Observe the “oplog slicer is paused for lock” message

i run operator 1.21.1

The oplog slicer is paused for lock is not an error, and the cause of this is that PBM stops saving PITR chunks when a logical backup is running. For example, if you have 1TB of data, it’s expected that the backup will take a long time to complete.
Isn’t it ever continuing in your case?
Have you ever had a successful backup?

I have some backup ready and some backup fails due the same reason oplog slicer is paused for lock

after see the logs on another node of backup-agent

mark RS as error dump: dump namespaces: pay_687dsfsd851393.filemanager_logs: close: upload: "pay_687dsfsd851393.filemanager_logs": save during split-merge mw: upload to S3: operation error S3: PutObject, exceeded maximum number of attempts, 6, https response error StatusCode: 504, RequestID: N/A, HostID: N/A, api error GatewayTimeout: The server did not respond in time.:

can you help?

as per the log above you are getting timeouts talking to the storage. The problem is either the network or the remote storage then.

Manual backups execute successfully, however, the automated cron-based backups are failing with an error.
this my config

—————————-

backup:
enabled: true
image: perconalab/percona-server-mongodb-operator:main-backup

storages:
  NBG1:
    type: s3
    s3:
      bucket: test-bk-test
      region: NBG1
      credentialsSecret: mongodb-prod-1-backup-s3
      endpointUrl: https://nbdc.your-objectstorage.com  
      retryer:
        numMaxRetries: 5
        minRetryDelay: 30ms
        maxRetryDelay: 10m
      prefix: ""
      uploadPartSize: 20971520
      maxUploadParts: 10000

pitr:
  enabled: true
  oplogOnly: false

oplogSpanMin: 10

  compressionType: gzip
  compressionLevel: 6

tasks:
  # =============  
  #  DAILY BACKUP at 12PM UTC 0
  # =============
 - name: mongodb-prod-1-backup-daily 
   enabled: true  
   retention:  
      count: 3 # keep 1 backup each day, 3 days max  
      type: count  
      deleteFromStorage: true  
   type: logical # best for storage, slower    
   schedule: 0 2 \* \* \* # daily at 2 AM UTC 0  
   storageName: NBG1
  # =============  
  #  WEEKLY BACKUP at SATURDAY 12AM UTC 0
  # =============
 - name: mongodb-prod-1-backup-weekly 
   enabled: true  
   retention:  
      count: 2 # keep 1 backup each week, 2 weeks max   
      type: count  
      deleteFromStorage: true  
   type: logical # best for storage, slower    
   schedule: 0 0 \* \* 5 # weekly at 12AM UTC 0 friday    
   storageName: NBG1
  # =============  
  #  MONTHLY BACKUP at SATURDAY 12AM UTC 0
  # ============= 
 - name: mongodb-prod-1-backup-MONTHLY 
   enabled: true  
   retention:  
      count:  3 # keep 1 backup each month, 3 months max   
      type: count  
      deleteFromStorage: true  
   type: logical # best for storage, slower    
   schedule: 0 4 1 \* \* # 1st day of every month at 4 AM   
   storageName: NBG1