PBM Agent always running resync storage

MongoDB Percona Operator for MongoDB
1

Description:

Kubernetes.

We have Percona MongoDB Operator deployed at version 1.21.1.

We will now deploy a MongoDB database using the Helm Chart from GitHub [ percona-helm-charts/charts/psmdb-db at main · percona/percona-helm-charts · GitHub ].

Steps to Reproduce:

Once deployed and the data imported, we can see that the resync storage action is continuously running. This causes backups to stop after a certain amount of time, leaving the executions in a waiting state or simply without a status after several pending status.

finalizers:
  - percona.com/delete-psmdb-pods-in-order
  - percona.com/delete-psmdb-pvc

nameOverride: ""
fullnameOverride: ""


crVersion: 1.21.1
pause: false
unmanaged: false
unsafeFlags:
  tls: false
  replsetSize: false
  mongosSize: false
  terminationGracePeriod: false
  backupIfUnhealthy: false

enableVolumeExpansion: true
multiCluster:
  enabled: false
updateStrategy: SmartUpdate
upgradeOptions:
  versionServiceEndpoint: https://check.percona.com
  apply: disabled
  schedule: "0 2 * * *"
  setFCV: false

image:
  repository: percona/percona-server-mongodb
  tag: 6.0.25-20

imagePullPolicy: Always
secrets: {}
pmm:
  enabled: false
  image:
    repository: percona/pmm-client
    tag: 3.4.1
  serverHost: monitoring-service

replsets:
  rs0:
    name: rs0
    size: 3
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    podDisruptionBudget:
      maxUnavailable: 2
    expose:
      enabled: false
      type: ClusterIP
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
    volumeSpec:
      pvc:
        resources:
          requests:
            storage: 5Gi

    hidden:
      enabled: false
      size: 2
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: "600m"
          memory: "1Gi"
        requests:
          cpu: "300m"
          memory: "1Gi"
      volumeSpec:
        pvc:
          resources:
            requests:
              storage: 3Gi

    nonvoting:
      enabled: false
      size: 3
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: "600m"
          memory: "1Gi"
        requests:
          cpu: "300m"
          memory: "1Gi"
      volumeSpec:
        pvc:
          resources:
            requests:
              storage: 3Gi
    arbiter:
      enabled: false
      size: 1
      affinity:
        antiAffinityTopologyKey: "kubernetes.io/hostname"
      resources:
        limits:
          cpu: "600m"
          memory: "1Gi"
        requests:
          cpu: "300m"
          memory: "1Gi"

sharding:
  enabled: false
  balancer:
    enabled: false

  configrs:
    size: 3
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    podDisruptionBudget:
      maxUnavailable: 1
    expose:
      enabled: false
      type: ClusterIP
    resources:
      limits:
        cpu: "600m"
        memory: "1Gi"
      requests:
        cpu: "300m"
        memory: "1Gi"
    volumeSpec:
      pvc:
        resources:
          requests:
            storage: 3Gi

  mongos:
    size: 3
    affinity:
      antiAffinityTopologyKey: "kubernetes.io/hostname"
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      limits:
        cpu: "600m"
        memory: "1Gi"
      requests:
        cpu: "300m"
        memory: "1Gi"
    expose:
      enabled: false
      type: ClusterIP

users:
- name: sample
  db: sample
  roles:
    - name: dbOwner
      db: sample


backup:
  enabled: true
  image:
    repository: percona/percona-backup-mongodb
    tag: 2.11.0
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
  storages:
    s3:
      main: true
      type: s3
      s3:
        bucket: backups
        prefix: "mongodb/sample"
        credentialsSecret: minio-credentials
        endpointUrl: http://minio.minio.svc.cluster.local:9000
        insecureSkipTLSVerify: true
        retryer:
          numMaxRetries: 3
          minRetryDelay: 100s
          maxRetryDelay: 5m
      
  pitr:
    enabled: false
    oplogOnly: false

  tasks:
  - name: daily-backup
    enabled: true
    schedule: "*/15 * * * *"
    retention:
     count: 7
     type: count
     deleteFromStorage: true
    storageName: s3
    type: physical


logcollector:
  enabled: false
  image:
    repository: percona/fluentbit
    tag: 4.0.1
#  configuration: |
#    [SERVICE]
#        Flush        1
#        Log_Level    error
#        Daemon       off
  resources:
    requests:
      memory: 100M
      cpu: 200m

It doesn’t matter if the backup is scheduled every 15 minutes or every 24 hours.

Version:

Versión de mongodb: 6.0.25-20
Version de mongo backup: 2.11

Logs:

[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:38:52.000+0000 I starting PITR routine
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:38:52.000+0000 I node: rs0/sample-db-psmdb-db-rs0-2.sample-db-psmdb-db-rs0.mongodb-sample.svc.cluster.local:27017
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:38:52.000+0000 I conn level ReadConcern: majority; WriteConcern: majority
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:38:52.000+0000 I listening for the commands
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 I got command resync <ts: 1762868346>, opid: 69133c7a012d514f2a9937fd
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 I got epoch {1762868336 1}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 D [resync] epoch set to {1762868338 21}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:14.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:25.000+0000 I got command resync <ts: 1762868356>, opid: 69133c84012d514f2a993809
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:25.000+0000 I got epoch {1762868348 17}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:25.000+0000 D [resync] get lock: duplicate operation: 69133c84012d514f2a993809 [Resync storage]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:25.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 I got command resync <ts: 1762868368>, opid: 69133c90012d514f2a993815
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 I got epoch {1762868359 2}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 D [resync] epoch set to {1762868360 19}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:36.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:47.000+0000 I got command resync <ts: 1762868379>, opid: 69133c9b012d514f2a993821
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:47.000+0000 I got epoch {1762868369 2}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:47.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:47.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:48.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:48.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:48.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:48.000+0000 D [resync] epoch set to {1762868371 22}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:48.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:59.000+0000 I got command resync <ts: 1762868390>, opid: 69133ca6012d514f2a99382d
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:59.000+0000 I got epoch {1762868382 17}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:59.000+0000 D [resync] get lock: duplicate operation: 69133ca6012d514f2a99382d [Resync storage]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:39:59.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:10.000+0000 I got command resync <ts: 1762868402>, opid: 69133cb2012d514f2a993839
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:10.000+0000 I got epoch {1762868391 1}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:10.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:10.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:11.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:11.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:11.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:11.000+0000 D [resync] epoch set to {1762868394 21}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:11.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 I got command resync <ts: 1762868413>, opid: 69133cbd012d514f2a993845
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 I got epoch {1762868404 2}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 D [resync] epoch set to {1762868405 21}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:21.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:33.000+0000 I got command resync <ts: 1762868425>, opid: 69133cc9012d514f2a993851
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:33.000+0000 I got epoch {1762868416 18}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] get lock: duplicate operation: 69133cc9012d514f2a993851 [Resync storage]
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 I got command resync <ts: 1762868430>, opid: 69133cce012d514f2a993857
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 I got epoch {1762868421 1}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:31.000+0000 I got command resync <ts: 1762868430>, opid: 69133cce012d514f2a993857
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:31.000+0000 I got epoch {1762868421 1}
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:31.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 D [resync] epoch set to {1762868422 17}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:22.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:23.000+0000 I got command resync <ts: 1762868425>, opid: 69133cc9012d514f2a993851
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:23.000+0000 I got epoch {1762868422 17}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:23.000+0000 D [resync] get lock: duplicate operation: 69133cc9012d514f2a993851 [Resync storage]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:23.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 I got command resync <ts: 1762868436>, opid: 69133cd4012d514f2a99385d
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 I got epoch {1762868426 1}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:28.000+0000 I got command resync <ts: 1762868436>, opid: 69133cd4012d514f2a99385d
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:28.000+0000 I got epoch {1762868426 1}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:28.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:37.000+0000 I got command resync <ts: 1762868436>, opid: 69133cd4012d514f2a99385d
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:37.000+0000 I got epoch {1762868426 1}
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:37.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 D [resync] epoch set to {1762868428 19}
[pod/sample-db-psmdb-db-rs0-2/backup-agent] 2025-11-11T13:40:44.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:29.000+0000 I got command resync <ts: 1762868430>, opid: 69133cce012d514f2a993857
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:29.000+0000 I got epoch {1762868428 19}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:29.000+0000 D [resync] get lock: duplicate operation: 69133cce012d514f2a993857 [Resync storage]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:29.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 I got command resync <ts: 1762868442>, opid: 69133cda012d514f2a993863
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 I got epoch {1762868431 1}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 I [resync] started
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:42.000+0000 I got command resync <ts: 1762868442>, opid: 69133cda012d514f2a993863
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:42.000+0000 I got epoch {1762868431 1}
[pod/sample-db-psmdb-db-rs0-0/backup-agent] 2025-11-11T13:40:42.000+0000 D [resync] lock not acquired
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] uploading ".pbm.init" [size hint: 6 (6.00B); part size: 52428800 (50.00MB)]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] got backups list: 1
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] bcp: 2025-11-11T13:31:24Z
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] got physical restores list: 0
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 D [resync] epoch set to {1762868433 17}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:33.000+0000 I [resync] succeed
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:34.000+0000 I got command resync <ts: 1762868436>, opid: 69133cd4012d514f2a99385d
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:34.000+0000 I got epoch {1762868433 17}
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:34.000+0000 D [resync] get lock: duplicate operation: 69133cd4012d514f2a99385d [Resync storage]
[pod/sample-db-psmdb-db-rs0-1/backup-agent] 2025-11-11T13:40:34.000+0000 D [resync] lock not acquired

2

Thanks for posting. I’ll submit a Jira ticket so our team can evaluate that:

3

Hi,

I am trying to reproduce it but without luck yet. Could you clarify whether you have deployed 1.21.1 operator or this was update from prev versions and how big data you have imported and how?

Thanks.

1 Like
5 
k describe po percona-server-mongodb-operator-67676cb868-f6wx4 | grep -i image:
    Image:        percona/percona-server-mongodb-operator:1.21.1

We are testing with the following information:

https://media.mongodb.org/zips.json

Initially, we were sending backups to a QNAP, but after observing this behavior, we tested it with Minio.

8

I have the same issue, this is partially preventing backups, because they wait for all the resync jobs to finish, looking at the operator logs, I see a lot of

PBM	configuration changed or resync is needed
PBM	Setting config
PBM	main storage changed. starting resync
9

@schlaubi, do you have more than one storage?

10

@arodgon ,

Could you pls also share how you are loading data.

11 
mongoimport -U username --db sample --file file.json
1 Like
12

Hi @arodgon ,

Seems I’ve managed to reproduce the issue. The reason is that the region is not set in storage config. Custom Resource options - Percona Operator for MongoDB

  backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.11.0
    storages:
      minio:
        main: true
        s3:
          bucket: operator-testing
          prefix: "mongodb/sample"
          credentialsSecret: minio-secret
          endpointUrl: http://minio-service:9000/
          insecureSkipTLSVerify: true
          region: us-east-1
          retryer:
            numMaxRetries: 3
            minRetryDelay: 100s
            maxRetryDelay: 5m
        type: s3

Could you try setting it and let us know if this helps?

@schlaubi , could you check whether you have region set in config and let us know? If you have it in place - please share your storage/cr config.

Thanks.

13

Yep that has fixed it THX!

It’s a bit confusing since my s3 provider does not use the region parameter, it should might mention this somewhere in the docs, that you can use us-east-1 in case your provider does not use regions

2 Likes
14

Hi @schlaubi ,

Thanks for update and confirming. We will consider the best way of improving current behavior and/or docs in scope of Jira .

15

Good afternoon,

There is no issues with Minio and the region now.

Our next step is to send backups to a QNAP using QuObjects, just as if we were using a regular S3 bucket as a minio, for example. Since QuObjects doesn’t accept regions, but Psmdb is a required parameter, we send the region “us-east-1” as a dummy value, but the agent fails with the error:

Cluster:
========
rs0:
  - sample-db-psmdb-db-rs0-0.sample-db-psmdb-db-rs0.sample.svc.cluster.local:27017 [S]: pbm-agent [v2.11.0] FAILED status:
      > ERROR with lost agent, last heartbeat: 1764078793
  - sample-db-psmdb-db-rs0-1.sample-db-psmdb-db-rs0.sample.svc.cluster.local:27017 [P]: pbm-agent [v2.11.0] FAILED status:
      > ERROR with lost agent, last heartbeat: 1764078592
  - sample-db-psmdb-db-rs0-2.sample-db-psmdb-db-rs0.sample.svc.cluster.local:27017 [S]: pbm-agent [v2.11.0] FAILED status:
      > ERROR with lost agent, last heartbeat: 1764078790

Configuration:

  storages:
    qnap:
      main: true
      type: s3
      s3:
        bucket: bucket
        prefix: "mongodb/sample"
        region: us-east-1
        credentialsSecret: secret
        endpointUrl: http://qnap:80
        insecureSkipTLSVerify: true
16

@arodgon, we haven’t tested QuObjects as a backup storage, but if it’s S3-compatible, it should work. Could you collect and check the agent logs from all pods? We might be able to find some useful information about this error.