Issues with configuring PITR Backup against S3 storage - no configuration applied "null"

MongoDB Percona Server for MongoDB
,
1

Hello,
I’m using Percona Operator with Percona Server MongoDB on Rancher running the EKS cluster.
I am trying to configure a backup “storages” section to S3 of PITR but for unknown reasons, the configuration is not applied and remains “null” every time without any error logs on operator.
I’m using the 1.18 version config with current backup section:

backup:
  enabled: true
  image:
    repository: percona/percona-backup-mongodb
    tag: 2.7.0-multi
  pitr:
    enabled: false
    oplogOnly: false
  storages: null
  tasks: null
crVersion: 1.18.0
enableVolumeExpansion: false

I have tried different configurations but none of them work.

example:

backup:
  storages:
    s3-storage:
      type: s3
      s3:
        bucket: s3://mongodb-pitr-backup
        region: us-east-1
        prefix: mongodb-backups/
        credentialsSecret: s3-backup-secret

Am I missing something?
Thanx

2

Hi @Yaroslav_Gorodetzky, you can find a working example in our tests percona-server-mongodb-operator/e2e-tests/pitr-sharded/conf/eks-some-name-rs0.yml at main · percona/percona-server-mongodb-operator · GitHub or in official documentation Configure storage for backups - Percona Operator for MongoDB . Try to test it. Also, if it does not work, please provide the operator log as well.

3

Hi Slava_Sarzhan and thank you for your quick response. I went through all the links you posted here and none of the configurations can get it to work. I recreated the new MongoDB from scratch with official documentation settings and provide it with you:

annotations: {}
backup:
  enabled: true
  image:
    repository: percona/percona-backup-mongodb
    tag: 2.7.0-multi
  pitr:
    enabled: false
    oplogOnly: false
  storages:
    s3-us-west:
      type: s3
      s3:
        bucket: S3-BACKUP-BUCKET-Test
        region: us-west-2
        credentialsSecret: tets-name-backup-s3
  tasks: null
crVersion: 1.18.0
enableVolumeExpansion: false
finalizers:
  - percona.com/delete-psmdb-pods-in-order
fullnameOverride: ''
image:
  repository: percona/percona-server-mongodb
  tag: 7.0.14-8-multi
imagePullPolicy: Always
multiCluster:
  enabled: false
nameOverride: ''
pause: false
pmm:
  enabled: false
  image:
    repository: percona/pmm-client
    tag: 2.43.2
  serverHost: monitoring-service
replsets:
  rs0:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    arbiter:
      affinity:
        antiAffinityTopologyKey: kubernetes.io/hostname
      enabled: false
      size: 1
    expose:
      enabled: false
      type: ClusterIP
    name: rs0
    nonvoting:
      affinity:
        antiAffinityTopologyKey: kubernetes.io/hostname
      enabled: false
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: 300m
          memory: 0.5G
        requests:
          cpu: 300m
          memory: 0.5G
      size: 3
      volumeSpec:
        pvc:
          resources:
            requests:
              storage: 3Gi
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      limits:
        cpu: 300m
        memory: 0.5G
      requests:
        cpu: 300m
        memory: 0.5G
    size: 3
    volumeSpec:
      pvc:
        resources:
          requests:
            storage: 3Gi
secrets: {}
sharding:
  balancer:
    enabled: true
  configrs:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    expose:
      enabled: false
      type: ClusterIP
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      limits:
        cpu: 300m
        memory: 0.5G
      requests:
        cpu: 300m
        memory: 0.5G
    size: 3
    volumeSpec:
      pvc:
        resources:
          requests:
            storage: 3Gi
  enabled: true
  mongos:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    expose:
      enabled: false
      type: ClusterIP
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      limits:
        cpu: 300m
        memory: 0.5G
      requests:
        cpu: 300m
        memory: 0.5G
    size: 3
unmanaged: false
unsafeFlags:
  backupIfUnhealthy: false
  mongosSize: false
  replsetSize: false
  terminationGracePeriod: false
  tls: false
updateStrategy: SmartUpdate
upgradeOptions:
  apply: disabled
  schedule: 0 2 * * *
  setFCV: false
  versionServiceEndpoint: https://check.percona.com

Logs from MongoDB operator:

|2024-12-30T13:59:36.580Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: a3b8eeec-6b94-4a70-a72e-7a9d0af8ed19, replset: rs0, size: 3, pods: 2}|
|---|---|---|---|
|2024-12-30T13:59:42.411Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: ee73ae7e-e4f8-41cb-8574-2750bc5253ef, replset: rs0, size: 3, pods: 2}|
|2024-12-30T13:59:43.234Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: a5153f7d-cb85-4ad3-8778-13417bd35655, replset: rs0, size: 3, pods: 2}|
|2024-12-30T13:59:48.132Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 5eac591d-2290-4316-8b88-21666784e659, replset: rs0, size: 3, pods: 2}|
|2024-12-30T13:59:53.862Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 77a4bc88-a053-4977-99dc-9163c99f0321, replset: rs0, size: 3, pods: 2}|
|2024-12-30T13:59:54.667Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 41a3189f-8be6-4c75-a0f0-9e751852d5fe, replset: rs0, size: 3, pods: 2}|
|2024-12-30T13:59:59.592Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 29cdb715-eed6-4e5d-b74f-0499eac7f474, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:05.414Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: d8f039e0-c205-4c1e-81a6-c98570811af2, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:06.886Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: df43c862-9b7e-41d3-9c7b-226c1c0a9cae, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:11.664Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 0d721f0f-4a14-4df1-abf2-86a6885278c5, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:17.456Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: b5047e50-fd71-4fcb-a082-9b62648437eb, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:23.187Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 3209e364-10d8-454a-9353-0e3497d77a02, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:28.885Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 9f1e990e-8e87-40d0-bfb5-ec92563534e7, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:34.810Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: e13753a5-92e2-4300-a71e-e8ec767f8a12, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:40.822Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 32f75b28-32df-49ff-a237-707c2b8fd108, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:46.637Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 792a2717-2c8b-4322-ac17-6cdf94ebbb7d, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:52.393Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 78634e16-16ea-4927-b5d6-d31be109f971, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:00:58.176Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: ee6d974b-c3cc-4426-a09b-108f5bf20f6a, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:01:03.995Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 0b95ee52-594b-403d-a0ec-701d64bbf483, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:01:09.764Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: ee597a72-695e-40ef-b02d-07ac5506a7eb, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:01:15.472Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 260c83a5-6374-4243-b126-fca998a86c7d, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:01:21.348Z|INFO|Waiting for the pods|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: f723a0f3-0dda-4eb0-9e95-e5ef55fdbac1, replset: rs0, size: 3, pods: 2}|
|2024-12-30T14:01:28.575Z|INFO|Tags changed|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 3ba8b5e7-7f6e-4db4-9d9e-10d3fd590162, host: psmdb-db-rs0-0.psmdb-db-rs0.mongodb-test.svc.cluster.local:27017, old: {nodeName:ip-172-21-65-176.ec2.internal,podName:psmdb-db-rs0-0,serviceName:psmdb-db}, new: {nodeName:ip-172-21-67-21.ec2.internal,podName:psmdb-db-rs0-0,serviceName:psmdb-db}}|
|2024-12-30T14:01:28.575Z|INFO|Fixing member configurations|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 3ba8b5e7-7f6e-4db4-9d9e-10d3fd590162, replset: rs0}|
|2024-12-30T14:01:33.170Z|INFO|Adding new member to replset|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 56681867-01e8-462f-a850-b65a742cc21a, _id: 2, host: psmdb-db-rs0-2.psmdb-db-rs0.mongodb-test.svc.cluster.local:27017}|
|2024-12-30T14:01:33.170Z|INFO|Adding new nodes|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 56681867-01e8-462f-a850-b65a742cc21a, replset: rs0}|
|2024-12-30T14:01:33.181Z|INFO|Configuring member votes and priorities|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 56681867-01e8-462f-a850-b65a742cc21a, replset: rs0}|
|2024-12-30T14:02:00.704Z|INFO|Cluster state changed|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 149b013b-3de4-4eff-b47a-b80e907bd9a2, previous: initializing, current: ready}|
|2024-12-30T14:02:01.284Z|INFO|update Mongo version to 7.0.14-8 (fetched from db)|{controller: psmdb-controller, object: {name:psmdb-db,namespace:mongodb-test}, namespace: mongodb-test, name: psmdb-db, reconcileID: 6bdff036-e5ba-40fd-b0fa-6e167e766bca}|

Post-deployment configuration:

annotations: {}
backup:
  enabled: true
  image:
    repository: percona/percona-backup-mongodb
    tag: 2.7.0-multi
  pitr:
    enabled: false
    oplogOnly: false
  storages: null
  tasks: null
crVersion: 1.18.0
enableVolumeExpansion: false
finalizers:
  - percona.com/delete-psmdb-pods-in-order
fullnameOverride: ''
image:
  repository: percona/percona-server-mongodb
  tag: 7.0.14-8-multi
imagePullPolicy: Always
multiCluster:
  enabled: false
nameOverride: ''
pause: false
pmm:
  enabled: false
  image:
    repository: percona/pmm-client
    tag: 2.43.2
  serverHost: monitoring-service
replsets:
  rs0:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    arbiter:
      affinity:
        antiAffinityTopologyKey: kubernetes.io/hostname
      enabled: false
      size: 1
    expose:
      enabled: false
      type: ClusterIP
    name: rs0
    nonvoting:
      affinity:
        antiAffinityTopologyKey: kubernetes.io/hostname
      enabled: false
      podDisruptionBudget:
        maxUnavailable: 1
      resources:
        limits:
          cpu: 300m
          memory: 0.5G
        requests:
          cpu: 300m
          memory: 0.5G
      size: 3
      volumeSpec:
        pvc:
          resources:
            requests:
              storage: 3Gi
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      limits:
        cpu: 300m
        memory: 0.5G
      requests:
        cpu: 300m
        memory: 0.5G
    size: 3
    volumeSpec:
      pvc:
        resources:
          requests:
            storage: 3Gi
secrets: {}
sharding:
  balancer:
    enabled: true
  configrs:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    expose:
      enabled: false
      type: ClusterIP
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      limits:
        cpu: 300m
        memory: 0.5G
      requests:
        cpu: 300m
        memory: 0.5G
    size: 3
    volumeSpec:
      pvc:
        resources:
          requests:
            storage: 3Gi
  enabled: true
  mongos:
    affinity:
      antiAffinityTopologyKey: kubernetes.io/hostname
    expose:
      enabled: false
      type: ClusterIP
    podDisruptionBudget:
      maxUnavailable: 1
    resources:
      limits:
        cpu: 300m
        memory: 0.5G
      requests:
        cpu: 300m
        memory: 0.5G
    size: 3
unmanaged: false
unsafeFlags:
  backupIfUnhealthy: false
  mongosSize: false
  replsetSize: false
  terminationGracePeriod: false
  tls: false
updateStrategy: SmartUpdate
upgradeOptions:
  apply: disabled
  schedule: 0 2 * * *
  setFCV: false
  versionServiceEndpoint: https://check.percona.com
global:
  cattle:
    systemProjectId: p-q4t78
4

I can’t reproduce it using EKS :frowning: but I do not have Rancher.