I have deployed percona mongodb server on my eks cluster using its operator, recently I upgraed the cr version to 1.16.0 and the mongodb version to 7.0.8-5, I have configured scheduled backups in my manifest but still when the back is triggered it gives error which is →
no available agent(s) on replsets: rs0, cfg
Also when checked the operator logs I found this →
2024-05-28T21:13:03.657Z INFO Creating or updating backup job {"controller": "psmdb-controller", "object": {"name":"perconamongodb-psmdb","namespace":"perconamongodb"}, "namespace": "perconamongodb", "name": "perconamongodb-psmdb", "reconcileID": "d535b9fa-c8f3-46b3-8e4c-3d2bd88bf020", "name": "mongobackup_night", "namespace": "perconamongodb", "schedule": "0 22 * * *"}
2024-05-28T21:13:06.084Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"perconamongodb-psmdb","namespace":"perconamongodb"}, "namespace": "perconamongodb", "name": "perconamongodb-psmdb", "reconcileID": "d535b9fa-c8f3-46b3-8e4c-3d2bd88bf020", "error": "create pbm object: create PBM connection to perconamongodb-psmdb-rs0-0.perconamongodb-psmdb-rs0.perconamongodb.svc.cluster.local:27017,perconamongodb-psmdb-rs0-1.perconamongodb-psmdb-rs0.perconamongodb.svc.cluster.local:27017,perconamongodb-psmdb-rs0-2.perconamongodb-psmdb-rs0.perconamongodb.svc.cluster.local:27017: create mongo connection: ping: connection() error occurred during connection handshake: dial tcp: lookup perconamongodb-psmdb-rs0-0.perconamongodb-psmdb-rs0.perconamongodb.svc.cluster.local on 172.20.0.10:53: no such host", "errorVerbose": "connection() error occurred during connection handshake: dial tcp: lookup perconamongodb-psmdb-rs0-0.perconamongodb-psmdb-rs0.perconamongodb.svc.cluster.local on 172.20.0.10:53: no such host\nping\ncreate mongo connection\ncreate PBM connection to perconamongodb-psmdb-rs0-0.perconamongodb-psmdb-rs0.perconamongodb.svc.cluster.local:27017,perconamongodb-psmdb-rs0-1.perconamongodb-psmdb-rs0.perconamongodb.svc.cluster.local:27017,perconamongodb-psmdb-rs0-2.perconamongodb-psmdb-rs0.perconamongodb.svc.cluster.local:27017\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.NewPBM\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:173\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).updatePITR\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/backup.go:376\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:434\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\ncreate pbm object\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).updatePITR\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/backup.go:378\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:434\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.1/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
2024-05-28T21:22:01.557Z INFO Creating or updating backup job {"controller": "psmdb-controller", "object": {"name":"perconamongodb-psmdb","namespace":"perconamongodb"}, "namespace": "perconamongodb", "name": "perconamongodb-psmdb", "reconcileID": "895834c1-0945-4849-95af-9a8ed552968c", "name": "mongobackup_night", "namespace": "perconamongodb", "schedule": "0 0 * * *"}
2024-05-28T21:22:57.667Z INFO Starting backup {"controller": "psmdbbackup-controller", "object": {"name":"ondemand-local-backup","namespace":"perconamongodb"}, "namespace": "perconamongodb", "name": "ondemand-local-backup", "reconcileID": "558d1cbf-361a-4d7a-9ac8-b5dbfa7c1376", "backup": "ondemand-local-backup", "storage": "backupstorage"}
2024-05-28T21:22:57.667Z INFO Setting PBM config {"controller": "psmdbbackup-controller", "object": {"name":"ondemand-local-backup","namespace":"perconamongodb"}, "namespace": "perconamongodb", "name": "ondemand-local-backup", "reconcileID": "558d1cbf-361a-4d7a-9ac8-b5dbfa7c1376", "backup": "perconamongodb-psmdb"}
2024-05-28T21:23:08.672Z INFO Sending backup command {"controller": "psmdbbackup-controller", "object": {"name":"ondemand-local-backup","namespace":"perconamongodb"}, "namespace": "perconamongodb", "name": "ondemand-local-backup", "reconcileID": "558d1cbf-361a-4d7a-9ac8-b5dbfa7c1376", "backupCmd": "backup [name: 2024-05-28T21:23:08Z, compression: gzip (level: default)] <ts: 0>"}
and here is my configuration for backup section →
backup:
enabled: true
image:
repository: percona/percona-backup-mongodb
tag: 2.4.1
serviceAccountName: percona-server-mongodb-operator
annotations:
iam.amazonaws.com/role: arn:aws:iam::645193536862:role/prod-percona-s3-role
# podSecurityContext: {}
# containerSecurityContext: {}
# resources:
# limits:
# cpu: "300m"
# memory: "0.5G"
# requests:
# cpu: "300m"
# memory: "0.5G"
storages:
backupstorage:
type: s3
s3:
bucket: "ds-mongodb-backup"
credentialsSecret: percona-backup-secret
# serverSideEncryption:
# kmsKeyID: 1234abcd-12ab-34cd-56ef-1234567890ab
# sseAlgorithm: aws:kms
# sseCustomerAlgorithm: AES256
# sseCustomerKey: Y3VzdG9tZXIta2V5
# retryer:
# numMaxRetries: 3
# minRetryDelay: 30ms
# maxRetryDelay: 5m
region: us-east-1
prefix: "perconabackup"
# uploadPartSize: 10485760
# maxUploadParts: 10000
storageClass: STANDARD
insecureSkipTLSVerify: true
# minio:
# type: s3
# s3:
# bucket: MINIO-BACKUP-BUCKET-NAME-HERE
# region: us-east-1
# credentialsSecret: my-cluster-name-backup-minio
# endpointUrl: http://minio.psmdb.svc.cluster.local:9000/minio/
# prefix: ""
# azure-blob:
# type: azure
# azure:
# container: CONTAINER-NAME
# prefix: PREFIX-NAME
# endpointUrl: https://accountName.blob.core.windows.net
# credentialsSecret: SECRET-NAME
pitr:
enabled: false
oplogOnly: false
# oplogSpanMin: 10
# compressionType: gzip
# compressionLevel: 6
# configuration:
# backupOptions:
# priority:
# "localhost:28019": 2.5
# "localhost:27018": 2.5
# timeouts:
# startingStatus: 33
# oplogSpanMin: 10
# restoreOptions:
# batchSize: 500
# numInsertionWorkers: 10
# numDownloadWorkers: 4
# maxDownloadBufferMb: 0
# downloadChunkMb: 32
# mongodLocation: /usr/bin/mongo
# mongodLocationMap:
# "node01:2017": /usr/bin/mongo
# "node03:27017": /usr/bin/mongo
tasks:
- name: mongobackup_night
enabled: true
schedule: "0 0 * * *"
keep: 5
storageName: backupstorage
compressionType: gzip
type: logical
- name: mongobackup_day
enabled: true
schedule: "0 12 * * *"
keep: 5
storageName: backupstorage
compressionType: gzip
type: logical
Earlier version of it was working perfectly fine with same configs, the only change while updating I did was this here →
unsafeFlags:
tls: false
replsetSize: false
mongosSize: false
terminationGracePeriod: false
backupIfUnhealthy: true
I set the parameter backupIfUnhealthy as true.
Does this has any effect over it or am I doing something wrong.