PBM Agent failed status

MongoDB Percona Backup for MongoDB
1

Hi Team,

PBM agents are in failed status, even if i start it.. after sometime it went to failed state.
I don’t find any error messages in the pbm logs.
how to trouble shoot it.

[mongod@tora-pl211 ~]$ pbm status
Cluster:
========
shard3ReplSet:
  - shard3ReplSet/pmgo-pl206.int.compumark.com:27018 [P]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092393
  - shard3ReplSet/pmgo-pl205.int.compumark.com:27018 [S]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092394
  - shard3ReplSet/pmgo-pl201.int.compumark.com:27028 [S]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092394
shard2ReplSet:
  - shard2ReplSet/pmgo-pl204.int.compumark.com:27018 [P]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092393
  - shard2ReplSet/pmgo-pl202.int.compumark.com:27018 [S]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092394
  - shard2ReplSet/pmgo-pl205.int.compumark.com:27028 [S]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092394
configReplSet:
  - configReplSet/pmgo-pl204.int.compumark.com:27019 [S]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092394
  - configReplSet/pmgo-pl202.int.compumark.com:27019 [P]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092391
  - configReplSet/pmgo-pl206.int.compumark.com:27029 [S]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092393
shard1ReplSet:
  - shard1ReplSet/pmgo-pl203.int.compumark.com:27018 [P]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092393
  - shard1ReplSet/pmgo-pl201.int.compumark.com:27018 [S]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092394
  - shard1ReplSet/pmgo-pl202.int.compumark.com:27028 [S]: pbm-agent  FAILED status:
      > ERROR with ERROR: lost agent, last heartbeat: 1753092394


[mongod@tora-pl211 ~]$ cat /etc/pbm_config.yaml
pitr:
  enabled: false
  oplogSpanMin: 0
  compression: s2
storage:
  type: s3
  s3:
    region: us-east-1
    bucket: cm-mongo-******
    prefix: percona/backup/
    credentials:
      access-key-id: ***
      secret-access-key: *********
    maxUploadParts: 10000
    storageClass: STANDARD
    insecureSkipTLSVerify: false
    retryer:
      numMaxRetries: 10
      minRetryDelay: 60
      maxRetryDelay: 60
2

Hi, pbm logs are stored in journal usually. Can you check

journactl -u pbm-agent -f

also check that you have configured properly the pbm-agent URI so that each agents points to their local host

3 
-- Logs begin at Tue 2025-07-15 03:23:38 CDT. --
Jul 15 08:23:47 tora-pl211 systemd[1]: Started pbm-agent.
Jul 15 08:24:18 tora-pl211 pbm-agent[4856]: 2025/07/15 08:24:18 Exit: connect to PBM: create mongo connection: ping: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: localhost:27017, Type: Unknown, Last error: dial tcp [::1]:27017: connect: connection refused }, ] }
Jul 15 08:24:18 tora-pl211 systemd[1]: pbm-agent.service: Main process exited, code=exited, status=1/FAILURE
Jul 15 08:24:18 tora-pl211 systemd[1]: pbm-agent.service: Failed with result 'exit-code'.

This is the pbm-agent URI used.

export PBM_MONGODB_URI="mongodb://pbmuser:pbmuser@pmgo-pl206.int.compumark.com:27029,pmgo-pl202.int.compumark.com:27019,pmgo-pl204.int.compumark.com:27019/?authSource=admin&replicaSet=configReplSet"```
4

The configuration of pbm-agent is not correct. It seems your replset runs on port 27029 but agent is pointed to 27017.

5

Not sure what’s wrong. the same setup works earlier for me.

Please assist how to fix it.

[mongod@tora-pl211 ~]$ echo $PBM_MONGODB_URI
mongodb://pbmuser:pbmuser@pmgo-pl206.int.compumark.com:27029,pmgo-pl202.int.compumark.com:27019,pmgo-pl204.int.compumark.com:27019/?authSource=admin&replicaSet=configReplSet

6

the variable PBM_MONGODB_URI only affects the current shell session. What I mean is to check the configuration of the pbm-agent that is installed on each node. In RHEL this is typically stored in /etc/sysconfig/pbm-agent.
See this section if you have doubts: Configure authentication in MongoDB - Percona Backup for MongoDB