Controller": "psmdbbackup-controller", "object": {"name" 5d", "error": "reconcile backup: cluster not found", "errorVerbose": "cluster not found\ngithub.com/percona/percona-server

Hi Team
We are using below link and created percona mongdb cluster, its running fine and able to create DB’s.
But when we are trying to backup in multi -namespace env we are facing cluster not found issue.

nstall Percona Operator for MongoDB in multi-namespace (cluster-wide) mode.

Here are the logs
will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For mor
2024-03-28T05:50:30.613Z ERROR Reconciler error {“controller”: “psmdbbackup-controller”, “object”: {“name”
7f", “error”: “reconcile backup: set backup config with storage s3-us-east: get PBM config: get s3 credentials secret: Sec
ret "nbi-mongo-backup-s3-secret" not found”, “errorVerbose”: “Secret "nbi-mongo-backup-s3-secret" not found\nget s3 cr
5d”, “backup”: “backup1”, “error”: “cluster not found”, “errorVerbose”: “cluster not found\ngithub.com/percona/percona-ser
2024-03-28T09:57:42.330Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The resul
t will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For mor
2024-03-28T09:57:42.330Z ERROR Reconciler error {“controller”: “psmdbbackup-controller”, “object”: {“name”
5d”, “error”: “reconcile backup: cluster not found”, “errorVerbose”: “cluster not found\ngithub.com/percona/percona-server
120f-8763-4562-bf28-21c6203ae356”, “error”: "Get "https://check.percona.com/versions/v1/psmdb-operator/1.15.0/disabled?ba

Hi Team,

We are using below link and created percona mongdb cluster, its running fine and able to create DB’s.

But when we are trying to implement backup in multi -namespace env we are facing “cluster not found” or “secret not found” or “agent not found” issue.

We are observing the below errors on operator pods:

2024-03-28T05:44:55.819Z	ERROR	failed to make backup	{"controller": "psmdbbackup-controller", "object": {"name":"backup1","namespace":"onap"}, "namespace": "onap", "name": "backup1", "reconcileID": "d7c65d35-6389-4dd9-9bfe-3be09205259f", "backup": "backup1", "error": "create pbm object: create PBM connection to : create mongo connection: create mongo client: error parsing uri: must have at least 1 host", "errorVerbose": "error parsing uri: must have at least 1 host\ncreate mongo client\ngithub.com/percona/percona-backup-mongodb/pbm.connect\n\t/go/pkg/mod/github.com/percona/percona-backup-mongodb@v1.8.1-0.20230920143330-3b1c2e263901/pbm/pbm.go:448\ngithub.com/percona/percona-backup-mongodb/pbm.New\n\t/go/pkg/mod/github.com/percona/percona-backup-mongodb@v1.8.1-0.20230920143330-3b1c2e263901/pbm/pbm.go:264\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.NewPBM\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:154\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).newBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/backup.go:33\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:176\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\ncreate mongo connection\ngithub.com/percona/percona-backup-mongodb/pbm.New\n\t/go/pkg/mod/github.com/percona/percona-backup-mongodb@v1.8.1-0.20230920143330-3b1c2e263901/pbm/pbm.go:266\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.NewPBM\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:154\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).newBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/backup.go:33\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:176\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\ncreate PBM connection to \ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.NewPBM\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:156\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).newBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/backup.go:33\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:176\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\ncreate pbm object\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).newBackup\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/backup.go:35\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:176\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"}
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile.func1
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:129
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile
	/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:178
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler

describe Backup log:

kubectl describe psmdb-backup backup1
Warning: Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-bas
ed tokens.
Name:         backup1
Namespace:    onap
Labels:       argocd.argoproj.io/instance=onap-nbi-onn
Annotations:  argocd.argoproj.io/tracking-id: onap-nbi-onn:psmdb.percona.com/PerconaServerMongoDBBackup:onap/backup1
API Version:  psmdb.percona.com/v1
Kind:         PerconaServerMongoDBBackup
Metadata:
  Creation Timestamp:  2024-03-28T09:57:42Z
  Finalizers:
    delete-backup
  Generation:        1
  Resource Version:  69577654
  UID:               c6b2621f-b95e-4556-8959-d2a9035b2ac2
Spec:
  Cluster Name:  nbi-mongo
  Storage Name:  s3-us-east
  Type:          logical
Status:
  Error:  cluster not found
  State:  error
Events:   <none>

Kindly check and share us an update on the same.