Does percona mongodb-operator minimal-cluster supports backup?

Raji · April 4, 2024, 7:29am

Description:

Does percona mongodb-operator minimal-cluster supports backup? When I use minimal-cluster and try to implement backup, I get error as " Error: no available agent(s) on replsets: rs0, cfg"

Steps to Reproduce:

Have used cw-bundle.yaml (cluster-wide)
Have used cr-minimal.yaml. The cluster is up and running with cfg, rs0 and mongos
The backup secrets are added.
Have applied backup.yaml

Version:

1.15.0

Logs:

kubectl describe psmdb-backup backup4 -n psmdb
Name:         backup4
Namespace:    psmdb
Labels:       <none>
Annotations:  <none>
API Version:  psmdb.percona.com/v1
Kind:         PerconaServerMongoDBBackup
Metadata:
  Creation Timestamp:  2024-04-04T05:45:54Z
  Finalizers:
    delete-backup
  Generation:        1
  Resource Version:  1912835
  UID:               0f6673c7-5b9c-4b60-abd9-444a3f5985cd
Spec:
  Cluster Name:  minimal-cluster
  Storage Name:  azure-blob
  Type:          logical
Status:
  Azure:
    Container:           mongos
    Credentials Secret:  my-cluster-azure-secret1
    Prefix:              psmdb
  Destination:           azure://mongos/psmdb/2024-04-04T05:46:16Z
  Error:                 no available agent(s) on replsets: rs0, cfg
  Last Transition:       2024-04-04T05:46:16Z
  Pbm Name:              2024-04-04T05:46:16Z
  Replset Names:
    cfg
    rs0
  Start:         2024-04-04T05:46:16Z
  State:         error
  Storage Name:  azure-blob
  Type:          logical
Events:          <none>

Expected Result:

Backup should be successful

Actual Result:

Error: no available agent(s) on replsets: rs0, cfg

Additional Information:

Able to perform backup with cr.yaml

kubectl get psmdb-backup -n psmdb
NAME      CLUSTER           STORAGE      DESTINATION                                 TYPE      STATUS   COMPLETED   AGE
backup4   minimal-cluster   azure-blob   azure://mongos/psmdb/2024-04-04T05:46:16Z   logical   error                49s

Tried to increase the cfg,rs0,mongos size as well to 2, but still same error

kubectl get pods -n psmdb | grep minimal
minimal-cluster-cfg-0      2/2     Running   0          2m3s
minimal-cluster-cfg-1      2/2     Running   0          2m32s
minimal-cluster-mongos-0   1/1     Running   0          6m46s
minimal-cluster-rs0-0      2/2     Running   0          2m
minimal-cluster-rs0-1      2/2     Running   0          2m31s

Raji · April 4, 2024, 7:35am

kubectl exec -it minimal-cluster-cfg-0 -c backup-agent -n psmdb -- bash
bash-4.4$ 
bash-4.4$ 
bash-4.4$ pbm logs
2024-04-04T07:28:16Z I [cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] starting PITR routine
2024-04-04T07:28:16Z I [cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] node: cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017
2024-04-04T07:28:16Z I [cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] listening for the commands
2024-04-04T07:28:53Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] pbm-agent:
Version:   2.3.0
Platform:  linux/amd64
GitCommit: 3b1c2e263901cf041c6b83547f6f28ac2879911f
GitBranch: release-2.3.0
BuildTime: 2023-09-20_14:42_UTC
GoVersion: go1.19
2024-04-04T07:28:53Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] starting PITR routine
2024-04-04T07:28:53Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] node: rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017
2024-04-04T07:28:53Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] listening for the commands
2024-04-04T07:29:02Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] pbm-agent:
Version:   2.3.0
Platform:  linux/amd64
GitCommit: 3b1c2e263901cf041c6b83547f6f28ac2879911f
GitBranch: release-2.3.0
BuildTime: 2023-09-20_14:42_UTC
GoVersion: go1.19
2024-04-04T07:29:02Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] starting PITR routine
2024-04-04T07:29:02Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] node: rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017
2024-04-04T07:29:02Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] listening for the commands
2024-04-04T07:29:57Z I [cfg/minimal-cluster-cfg-1.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] got command backup [name: 2024-04-04T07:29:57Z, compression: gzip (level: default)] <ts: 1712215797>
2024-04-04T07:29:57Z I [cfg/minimal-cluster-cfg-1.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] got epoch {1712215785 2}
2024-04-04T07:29:57Z I [cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] got command backup [name: 2024-04-04T07:29:57Z, compression: gzip (level: default)] <ts: 1712215797>
2024-04-04T07:29:57Z I [cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] got epoch {1712215785 2}
2024-04-04T07:29:57Z I [cfg/minimal-cluster-cfg-1.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] [backup/2024-04-04T07:29:57Z] mark backup as error `no available agent(s) on replsets: rs0, cfg`: <nil>
2024-04-04T07:29:57Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] got command backup [name: 2024-04-04T07:29:57Z, compression: gzip (level: default)] <ts: 1712215797>
2024-04-04T07:29:57Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] got epoch {1712215785 2}
2024-04-04T07:29:58Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] got command backup [name: 2024-04-04T07:29:57Z, compression: gzip (level: default)] <ts: 1712215797>
2024-04-04T07:29:58Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] got epoch {1712215785 2}
bash-4.4$ 
bash-4.4$ 
bash-4.4$ 
bash-4.4$ 
bash-4.4$ pbm status
^C
bash-4.4$

Raji · April 4, 2024, 7:37am

 kubectl exec -it minimal-cluster-rs0-0 -c backup-agent -n psmdb -- bash
bash-4.4$ 
bash-4.4$ 
bash-4.4$ pbm status


^C
bash-4.4$ pbm logs
2024-04-04T07:28:16Z I [cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] starting PITR routine
2024-04-04T07:28:16Z I [cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] node: cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017
2024-04-04T07:28:16Z I [cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] listening for the commands
2024-04-04T07:28:53Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] pbm-agent:
Version:   2.3.0
Platform:  linux/amd64
GitCommit: 3b1c2e263901cf041c6b83547f6f28ac2879911f
GitBranch: release-2.3.0
BuildTime: 2023-09-20_14:42_UTC
GoVersion: go1.19
2024-04-04T07:28:53Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] starting PITR routine
2024-04-04T07:28:53Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] node: rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017
2024-04-04T07:28:53Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] listening for the commands
2024-04-04T07:29:02Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] pbm-agent:
Version:   2.3.0
Platform:  linux/amd64
GitCommit: 3b1c2e263901cf041c6b83547f6f28ac2879911f
GitBranch: release-2.3.0
BuildTime: 2023-09-20_14:42_UTC
GoVersion: go1.19
2024-04-04T07:29:02Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] starting PITR routine
2024-04-04T07:29:02Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] node: rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017
2024-04-04T07:29:02Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] listening for the commands
2024-04-04T07:29:57Z I [cfg/minimal-cluster-cfg-1.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] got command backup [name: 2024-04-04T07:29:57Z, compression: gzip (level: default)] <ts: 1712215797>
2024-04-04T07:29:57Z I [cfg/minimal-cluster-cfg-1.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] got epoch {1712215785 2}
2024-04-04T07:29:57Z I [cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] got command backup [name: 2024-04-04T07:29:57Z, compression: gzip (level: default)] <ts: 1712215797>
2024-04-04T07:29:57Z I [cfg/minimal-cluster-cfg-0.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] got epoch {1712215785 2}
2024-04-04T07:29:57Z I [cfg/minimal-cluster-cfg-1.minimal-cluster-cfg.psmdb.svc.cluster.local:27017] [backup/2024-04-04T07:29:57Z] mark backup as error `no available agent(s) on replsets: rs0, cfg`: <nil>
2024-04-04T07:29:57Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] got command backup [name: 2024-04-04T07:29:57Z, compression: gzip (level: default)] <ts: 1712215797>
2024-04-04T07:29:57Z I [rs0/minimal-cluster-rs0-1.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] got epoch {1712215785 2}
2024-04-04T07:29:58Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] got command backup [name: 2024-04-04T07:29:57Z, compression: gzip (level: default)] <ts: 1712215797>
2024-04-04T07:29:58Z I [rs0/minimal-cluster-rs0-0.minimal-cluster-rs0.psmdb.svc.cluster.local:27017] got epoch {1712215785 2}
bash-4.4$ 
bash-4.4$ 
bash-4.4$ 
bash-4.4$ 
bash-4.4$ pbm backup
Error: backup pre-check: no available agent(s) on replsets: rs0, cfg
bash-4.4$ 
bash-4.4$ 
bash-4.4$ pbm backup --type logical
Error: backup pre-check: no available agent(s) on replsets: rs0, cfg
bash-4.4$ 
bash-4.4$ 
bash-4.4$

Below is the cr-minimal.yaml which I have used, added the backup config in the bottom

cat cr-minimal.yaml 
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
  name: minimal-cluster
spec:
  crVersion: 1.15.0
  image: percona/percona-server-mongodb:6.0.9-7
  allowUnsafeConfigurations: true
  upgradeOptions:
    apply: disabled
    schedule: "0 2 * * *"
  secrets:
    users: minimal-cluster
  replsets:
  - name: rs0
    size: 2
    volumeSpec:
      persistentVolumeClaim:
        resources:
          requests:
            storage: 3Gi

  sharding:
    enabled: true

    configsvrReplSet:
      size: 2
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 3Gi

    mongos:
      size: 2  
  backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.3.0
    serviceAccountName: percona-server-mongodb-operator
#    annotations:
#      iam.amazonaws.com/role: role-arn
#    resources:
#      limits:
#        cpu: "300m"
#        memory: "0.5G"
#      requests:
#        cpu: "300m"
#        memory: "0.5G"
    storages:
      azure-blob:
        type: azure
        azure:
          container: mongos
          prefix: psmdb
          credentialsSecret: my-cluster-azure-secret1
    pitr:
      enabled: false
      oplogOnly: false
#      oplogSpanMin: 10
      compressionType: gzip
      compressionLevel: 6
#    tasks:
#      - name: daily-s3-us-west
#        enabled: true
#        schedule: "0 0 * * *"
#        keep: 3
#        storageName: s3-us-west
#        compressionType: gzip
#        compressionLevel: 6
#      - name: weekly-s3-us-west
#        enabled: false
#        schedule: "0 0 * * 0"
#        keep: 5
#        storageName: s3-us-west
#        compressionType: gzip
#        compressionLevel: 6
#      - name: weekly-s3-us-west-physical
#        enabled: false
#        schedule: "0 5 * * 0"
#        keep: 5
#        type: physical
#        storageName: s3-us-west
#        compressionType: gzip
#        compressionLevel: 6

Sergey_Pronin · April 8, 2024, 9:11am

Hello @Raji ,

the backup will work even with 1 node.
You are on a right path and just must ensure that spec.backup section is there and enabled: is set to true.

This is what I added into my cr-minimal.yaml manifest:

  backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.3.0
    serviceAccountName: percona-server-mongodb-operator
    storages:
      sp-test:
        type: s3
        s3:
          bucket: MYBUCKET
          credentialsSecret: sp-secret
          region: us-central1
          prefix: ""
          endpointUrl: https://storage.googleapis.com

My backup manifest looks like this:

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBBackup
metadata:
  finalizers:
  - delete-backup
  name: backup1
spec:
  clusterName: minimal-cluster
  storageName: sp-test

I ran kubectl apply -f backup.yaml and it worked as expected:

NAME      CLUSTER           STORAGE   DESTINATION                           TYPE      STATUS   COMPLETED   AGE
backup1   minimal-cluster   sp-test   s3://MYBUCKET/2024-04-08T09:07:02Z   logical   ready    4m16s       4m49s

Please let me know if you still face any problems.

Topic		Replies	Views
Error in backups after upgrading mongodb version MongoDB percona , mongodb , closed-no-reply	0	163	May 28, 2024
Percona mongodb operator backup error Percona Operator for MongoDB	1	409	February 21, 2024
Failed to perform backup to S3 on Percona Operator for MongoDB Percona Operator for MongoDB percona , mongodb	5	1357	May 14, 2024
All Backups Error Percona Operator for MongoDB	4	1178	March 21, 2022
Percona backup for mongodb on Azure Kubernetes and store on azure File share blob storage Percona Backup for MongoDB	1	841	March 27, 2023