No available agent(s) on replsets pbm

MongoDB Percona Backup for MongoDB
, ,
1

Currently i have issue with my pbm agents on mongodb they tell me no available agent on replsets when i do any backup (physical - incremental). however my pbm status shows the pbm agent was OK on the all node also my mongodb replsets currently on healthy state.

Here the pbm status output (none of them confidential because it’s lab env)
3 VM running instance of mongodb community 4.4.25 with pbm agent 2.3.0 installed
1 VM as remote storage s3 minio

[root@al-mongobackup ~]# pbm status
Cluster:
========
rs0:
  - rs0/10.63.23.55:27017 [P]: pbm-agent v2.3.0 OK
  - rs0/al-mongorestore:27017 [S]: pbm-agent v2.3.0 OK
  - rs0/al-mongorestore2:27017 [S]: pbm-agent v2.3.0 OK


PITR incremental backup:
========================
Status [OFF]

Currently running:
==================
(none)

Backups:
========
S3 us-east-1 s3://http://10.63.23.117:9000/mongodb-backup-55 
  (none)

Error output
[root@al-mongobackup ~]# pbm backup --type incremental --base
Error: backup pre-check: no available agent(s) on replsets: rs0

pbm logs output
[root@al-mongobackup ~]# pbm logs
2023-10-16T09:32:53Z I [rs0/10.63.23.55:27017] starting PITR routine
2023-10-16T09:32:53Z I [rs0/10.63.23.55:27017] node: rs0/10.63.23.55:27017
2023-10-16T09:32:53Z I [rs0/10.63.23.55:27017] listening for the commands
2023-10-16T09:34:26Z I [rs0/al-mongorestore:27017] got command resync <ts: 1697448866>
2023-10-16T09:34:26Z I [rs0/al-mongorestore:27017] got epoch {1697448866 9}
2023-10-16T09:34:26Z I [rs0/10.63.23.55:27017] got command resync <ts: 1697448866>
2023-10-16T09:34:26Z I [rs0/10.63.23.55:27017] got epoch {1697448864 1}
2023-10-16T09:34:26Z I [rs0/10.63.23.55:27017] [resync] started
2023-10-16T09:34:26Z I [rs0/10.63.23.55:27017] [resync] succeed
2023-10-16T09:34:27Z I [rs0/al-mongorestore2:27017] got command resync <ts: 1697448866>
2023-10-16T09:34:27Z I [rs0/al-mongorestore2:27017] got epoch {1697448866 9}
2023-10-16T09:39:36Z I [rs0/10.63.23.55:27017] pbm-agent:
Version:   2.3.0
Platform:  linux/amd64
GitCommit: 3b1c2e263901cf041c6b83547f6f28ac2879911f
GitBranch: release-2.3.0
BuildTime: 2023-09-20_14:42_UTC
GoVersion: go1.19
2023-10-16T09:39:36Z I [rs0/10.63.23.55:27017] starting PITR routine
2023-10-16T09:39:36Z I [rs0/10.63.23.55:27017] node: rs0/10.63.23.55:27017
2023-10-16T09:39:36Z I [rs0/10.63.23.55:27017] listening for the commands
2023-10-16T12:08:36Z E [rs0/10.63.23.55:27017] [agentCheckup] check storage connection: storage check failed with: get S3 object header: RequestError: send request failed
caused by: Head "http://10.63.23.117:9000/mongodb-backup-55/.pbm.init": dial tcp 10.63.23.117:9000: connect: connection refused
2023-10-16T12:08:41Z E [rs0/10.63.23.55:27017] [agentCheckup] check storage connection: storage check failed with: get S3 object header: RequestError: send request failed
caused by: Head "http://10.63.23.117:9000/mongodb-backup-55/.pbm.init": dial tcp 10.63.23.117:9000: connect: connection refused
2023-10-16T12:08:46Z E [rs0/10.63.23.55:27017] [agentCheckup] check storage connection: storage check failed with: get S3 object header: RequestError: send request failed
caused by: Head "http://10.63.23.117:9000/mongodb-backup-55/.pbm.init": dial tcp 10.63.23.117:9000: connect: connection refused
2023-10-16T12:08:51Z E [rs0/10.63.23.55:27017] [agentCheckup] check storage connection: storage check failed with: get S3 object header: RequestError: send request failed
caused by: Head "http://10.63.23.117:9000/mongodb-backup-55/.pbm.init": dial tcp 10.63.23.117:9000: connect: connection refused
2023-10-16T12:08:56Z E [rs0/10.63.23.55:27017] [agentCheckup] check storage connection: storage check failed with: get S3 object header: RequestError: send request failed
caused by: Head "http://10.63.23.117:9000/mongodb-backup-55/.pbm.init": dial tcp 10.63.23.117:9000: connect: connection refused
[root@al-mongobackup ~]#

hopefully there was an solution.
thank you!

2

Hi @Alif_Irawan ,

Physical / Incremental backup works with PSMDB only.

The error says that there is no available agent that can do physical/incremental backup (because all agents are connected to CE). You need at least one PSMDB instance in each replset to be able to do physical backup.

PBM Docs: Physical backups and restores

3

Also, take a look:

4

HI, thank you. i just barely know that we need single instance of psmdb on the replicaset.
that solve my problem.

5

@Dmytro_Zghoba I have the same problem and I want to clarify some things:
I have a testing sharded cluster with mongodb, not the Percona Server, just regular mongodb v4.4.22

[root@elk-mongo-1 ~]# pbm status
Cluster:
========
cnfgsrvr0:
  - cnfgsrvr0/elk-mongo-1.amb.corp:27019 [P]: pbm-agent v2.3.0 OK
  - cnfgsrvr0/elk-mongo-2.amb.corp:27019 [S]: pbm-agent v2.3.0 OK
  - cnfgsrvr0/elk-mongo-3.amb.corp:27019 [S]: pbm-agent v2.3.0 OK
sh2:
  - sh2/elk-mongo-3.amb.corp:27018 [P]: pbm-agent v2.3.0 OK
sh0:
  - sh0/elk-mongo-1.amb.corp:27018 [P]: pbm-agent v2.3.0 OK
sh1:
  - sh1/elk-mongo-2.amb.corp:27018 [P]: pbm-agent v2.3.0 OK


PITR incremental backup:
========================
Status [OFF]

Currently running:
==================
(none)

Backups:
========
FS  /mnt/backup/mongo
  (none)

(we need this specific version to integrate into the customer’s mongoDB later, because they need specifically incremental backups).
And I have no Percona Server at all, and when I do incremental backup try I get this:

[root@elk-mongo-1 ~]# pbm backup --type incremental --base
Error: backup pre-check: no available agent(s) on replsets: cnfgsrvr0, sh0, sh1, sh2

So what specific steps do I need to do in order to make incremental backups work?

6

I installed additional PSMDB server for every replicasets, At least it started working.

7

The PSMDB instance is added as a member in replica set. My replica set demands authentication with keyfile. I set up and configure using Docker-compose.

version: '3.7'
services:
  psmdb:
    image: percona/percona-server-mongodb:4.4
    container_name: rep_psmdb
    # restart: always
    ports:
      - "27018:27017"
    volumes:
      - ./rep_psmdb_data:/data/db
      - ./mongodb-keyfile:/data/keyfile/mongodb-keyfile
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: Password
    command:
      - mongod
      - --replSet 
      - aReplSet
      - --keyFile
      - /data/keyfile/mongodb-keyfile

I encountered error:

"msg":"Read security file failed","attr":{"error":{"code":30,"codeName":"InvalidPath","errmsg":"error opening file: /data/keyfile/mongodb-keyfile: bad file"}}

Can you give me a reminder, I’m appreciate that.