Pbm-agent encounters an error in my self hosted replica set with 3 nodes

MongoDB Percona Backup for MongoDB
1

The erros is:
pbm-agent[3751]: 2024-11-13T11:29:23.000+0800 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result

The replica set rs1 is manually built on binary tarballs:

percona-mongodb-mongosh-2.3.0-x86_64.tar.gz
percona-backup-mongodb-2.7.0-x86_64.tar.gz
percona-server-mongodb-7.0.14-8-x86_64.bookworm.tar.gz

mongodb-rs1-node1 mongod.conf:

storage:
  dbPath: /var/lib/mongod
  directoryPerDB: true
net:
  bindIp: localhost,mongodb-rs1-node1
  port: 27017
systemLog:
  logAppend: true
  destination: file
  path: "/var/log/mongod/mongod.log"
replication:
  replSetName: "rs1"
processManagement:
  fork: true
  pidFilePath: /var/run/mongod.pid
setParameter:
  enableLocalhostAuthBypass: true
...

mongodb-rs1-node2 mongod.conf:

storage:
  dbPath: /var/lib/mongod
  directoryPerDB: true
net:
  bindIp: localhost,mongodb-rs1-node2
  port: 27017
systemLog:
  logAppend: true
  destination: file
  path: "/var/log/mongod/mongod.log"
replication:
  replSetName: "rs1"
processManagement:
  fork: true
  pidFilePath: /var/run/mongod.pid
setParameter:
  enableLocalhostAuthBypass: true
...

mongodb-rs1-node3 mongod.conf:

storage:
  dbPath: /var/lib/mongod
  directoryPerDB: true
net:
  bindIp: localhost,mongodb-rs1-node3
  port: 27017
systemLog:
  logAppend: true
  destination: file
  path: "/var/log/mongod/mongod.log"
replication:
  replSetName: "rs1"
processManagement:
  fork: true
  pidFilePath: /var/run/mongod.pid
setParameter:
  enableLocalhostAuthBypass: true
...

The start command is: /opt/mongodb/bin/mongod -f /etc/mongod.conf

Mongosh login tip : mongosh --host mongodb-rs1-node1 --port 27017


Current Mongosh Log ID:	67341fa2098ed10c1ca5b90d
Connecting to:		mongodb://mongodb-rs1-node1:27017/?directConnection=true&appName=mongosh+2.3.0
Using MongoDB:		7.0.14-8
Using Mongosh:		2.3.0
mongosh 2.3.3 is available for download: https://www.mongodb.com/try/download/shell

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

------
   The server generated these startup warnings when booting
   2024-11-12T19:03:51.682+08:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
   2024-11-12T19:04:00.310+08:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted. You can use percona-server-mongodb-enable-auth.sh to fix it
------

rs1 [direct: primary] test>

==================pbm-agent===================
The following statements run in primary:

db.getSiblingDB("admin").createRole({ "role": "pbmAnyAction",
      "privileges": [
         { "resource": { "anyResource": true },
           "actions": [ "anyAction" ]
         }
      ],
      "roles": []
   });

db.getSiblingDB("admin").createUser({user: "pbm",
       "pwd": "pbmpwd",
       "roles" : [
          { "db" : "admin", "role" : "readWrite", "collection": "" },
          { "db" : "admin", "role" : "backup" },
          { "db" : "admin", "role" : "clusterMonitor" },
          { "db" : "admin", "role" : "restore" },
          { "db" : "admin", "role" : "pbmAnyAction" }
       ]
    });

run pbm-agent:

root@RD-TEST-TWJ-7:~# export PBM_MONGODB_URI="mongodb://pbm:pbmpwd@localhost:27017/?authSource=admin"
root@RD-TEST-TWJ-7:~# pbm-agent

                    %                        _____
                   %%%                      |  __ \
                 ###%%%%%%%%%%%%*           | |__) |__ _ __ ___ ___  _ __   __ _
                ###  ##%%      %%%%         |  ___/ _ \ '__/ __/ _ \| '_ \ / _` |
              ####     ##%       %%%%       | |  |  __/ | | (_| (_) | | | | (_| |
             ###        ####      %%%       |_|   \___|_|  \___\___/|_| |_|\__,_|
           ,((###         ###     %%%         _____                       _
          (((( (###        ####  %%%%        / ____|                     | |
         (((     ((#         ######         | (___   __ _ _   _  __ _  __| |
       ((((       (((#        ####           \___ \ / _` | | | |/ _` |/ _` |
      /((          ,(((        *###          ____) | (_| | |_| | (_| | (_| |
    ////             (((         ####       |_____/ \__, |\__,_|\__,_|\__,_|
   ///                ((((        ####                 | |
 /////////////(((((((((((((((((########                |_|   Join @ squad.percona.com/mongodb

** Join Percona Squad! **
Participate in monthly SWAG raffles, get early access to new product features,
invite-only ”ask me anything” sessions with database performance experts.

Interested? Fill in the form at squad.percona.com/mongodb

2024-11-13T12:02:52.000+0800 I starting PITR routine
2024-11-13T12:02:52.000+0800 I pbm-agent:
Version:   2.7.0
Platform:  linux/amd64
GitCommit: 957ac501f939b1b784fcdcf61dcc5b9bf171e0b9
GitBranch: release-2.7.0
BuildTime: 2024-10-02_12:55_UTC
GoVersion: go1.22.8
2024-11-13T12:02:52.000+0800 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
2024-11-13T12:02:52.000+0800 I node: rs1/mongodb-rs1-node1:27017
2024-11-13T12:02:52.000+0800 I conn level ReadConcern: majority; WriteConcern: majority
2024-11-13T12:02:52.000+0800 I listening for the commands
2024-11-13T12:02:57.000+0800 E [agentCheckup] check storage connection: unable to get storage: get config: get: mongo: no documents in result
^C2024-11-13T12:03:00.000+0800 D [agentCheckup] deleting agent status
2024-11-13T12:03:00.000+0800 E listening commands: context canceled
2024-11-13T12:03:01.000+0800 I change stream was closed
2024/11/13 12:03:01 Exit: <nil>
2

rs.status()

rs1 [direct: primary] test> rs.status()
{
  set: 'rs1',
  date: ISODate('2024-11-13T04:05:24.290Z'),
  myState: 1,
  term: Long('5'),
  syncSourceHost: '',
  syncSourceId: -1,
  heartbeatIntervalMillis: Long('2000'),
  majorityVoteCount: 2,
  writeMajorityCount: 2,
  votingMembersCount: 3,
  writableVotingMembersCount: 3,
  optimes: {
    lastCommittedOpTime: { ts: Timestamp({ t: 1731470721, i: 6 }), t: Long('5') },
    lastCommittedWallTime: ISODate('2024-11-13T04:05:21.974Z'),
    readConcernMajorityOpTime: { ts: Timestamp({ t: 1731470721, i: 6 }), t: Long('5') },
    appliedOpTime: { ts: Timestamp({ t: 1731470721, i: 6 }), t: Long('5') },
    durableOpTime: { ts: Timestamp({ t: 1731470721, i: 6 }), t: Long('5') },
    lastAppliedWallTime: ISODate('2024-11-13T04:05:21.974Z'),
    lastDurableWallTime: ISODate('2024-11-13T04:05:21.974Z')
  },
  lastStableRecoveryTimestamp: Timestamp({ t: 1731470717, i: 2 }),
  electionCandidateMetrics: {
    lastElectionReason: 'electionTimeout',
    lastElectionDate: ISODate('2024-11-13T03:57:10.829Z'),
    electionTerm: Long('5'),
    lastCommittedOpTimeAtElection: { ts: Timestamp({ t: 0, i: 0 }), t: Long('-1') },
    lastSeenOpTimeAtElection: { ts: Timestamp({ t: 1731470193, i: 4 }), t: Long('3') },
    numVotesNeeded: 2,
    priorityAtElection: 1,
    electionTimeoutMillis: Long('10000'),
    numCatchUpOps: Long('0'),
    newTermStartDate: ISODate('2024-11-13T03:57:10.899Z'),
    wMajorityWriteAvailabilityDate: ISODate('2024-11-13T03:57:11.113Z')
  },
  members: [
    {
      _id: 0,
      name: 'mongodb-rs1-node1:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      uptime: 508,
      optime: { ts: Timestamp({ t: 1731470721, i: 6 }), t: Long('5') },
      optimeDate: ISODate('2024-11-13T04:05:21.000Z'),
      lastAppliedWallTime: ISODate('2024-11-13T04:05:21.974Z'),
      lastDurableWallTime: ISODate('2024-11-13T04:05:21.974Z'),
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      electionTime: Timestamp({ t: 1731470230, i: 1 }),
      electionDate: ISODate('2024-11-13T03:57:10.000Z'),
      configVersion: 1,
      configTerm: 5,
      self: true,
      lastHeartbeatMessage: ''
    },
    {
      _id: 1,
      name: 'mongodb-rs1-node2:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 503,
      optime: { ts: Timestamp({ t: 1731470721, i: 6 }), t: Long('5') },
      optimeDurable: { ts: Timestamp({ t: 1731470721, i: 6 }), t: Long('5') },
      optimeDate: ISODate('2024-11-13T04:05:21.000Z'),
      optimeDurableDate: ISODate('2024-11-13T04:05:21.000Z'),
      lastAppliedWallTime: ISODate('2024-11-13T04:05:21.974Z'),
      lastDurableWallTime: ISODate('2024-11-13T04:05:21.974Z'),
      lastHeartbeat: ISODate('2024-11-13T04:05:23.251Z'),
      lastHeartbeatRecv: ISODate('2024-11-13T04:05:23.455Z'),
      pingMs: Long('0'),
      lastHeartbeatMessage: '',
      syncSourceHost: 'mongodb-rs1-node1:27017',
      syncSourceId: 0,
      infoMessage: '',
      configVersion: 1,
      configTerm: 5
    },
    {
      _id: 2,
      name: 'mongodb-rs1-node3:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      uptime: 503,
      optime: { ts: Timestamp({ t: 1731470721, i: 6 }), t: Long('5') },
      optimeDurable: { ts: Timestamp({ t: 1731470721, i: 6 }), t: Long('5') },
      optimeDate: ISODate('2024-11-13T04:05:21.000Z'),
      optimeDurableDate: ISODate('2024-11-13T04:05:21.000Z'),
      lastAppliedWallTime: ISODate('2024-11-13T04:05:21.974Z'),
      lastDurableWallTime: ISODate('2024-11-13T04:05:21.974Z'),
      lastHeartbeat: ISODate('2024-11-13T04:05:23.248Z'),
      lastHeartbeatRecv: ISODate('2024-11-13T04:05:23.250Z'),
      pingMs: Long('0'),
      lastHeartbeatMessage: '',
      syncSourceHost: 'mongodb-rs1-node2:27017',
      syncSourceId: 1,
      infoMessage: '',
      configVersion: 1,
      configTerm: 5
    }
  ],
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1731470721, i: 6 }),
    signature: {
      hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
      keyId: Long('0')
    }
  },
  operationTime: Timestamp({ t: 1731470721, i: 6 })
}
3 
export PBM_MONGODB_URI="mongodb://pbm:pbmpwd@mongodb-rs1-node1:27017/?authSource=admin&replSetName=rs1"
pbm status

Error: get status of cluster: get cluster status: get config: get: mongo: no documents in result

4

Hello, have you configured the agent to use the remote storage as per Configure remote backup storage - Percona Backup for MongoDB ?
please show the pbm_config.yaml along with the output of pbm status command.

5

Thank for your reply. After i have configured the remote backup storage, it works. For that time when i was creating a new topic, i just want to check the pbm-agent started successfully and may the error be a warning which constantly remind me that i have’nt set the backup storage and give me some information about that, like the mongosh login warning.