Error: connect to mongodb: get config server connection URI: mongo: no documents in result

Vitaly_Kargin · January 12, 2024, 3:49am

Description:

Hello, i’m using PBM fo Mongodb community 7.0 Sharded Cluster
3 mongos
3 config servers - replicaset
3 shard, each is replicaset (rs01 rs02 rs03)

I installed Percona Backup For MongoDB 2.3.1 on:

mongo cfg servers
mongo shard servers

cat /etc/percona/pbm_config.yml 
pitr:
  enabled: true
  oplogOnly: false
  oplogSpanMin: 10
  compression: s2
  compressionLevel: 2

storage:
  type: s3
  s3:
    region: eu
    bucket: superset-mongo-shard
    prefix: percona-backups
    endpointUrl: ******
    credentials:
      access-key-id: ****
      secret-access-key: ****

Problem:

pbm agent starts only on config servers.
but on shards systemd log - ‘Failed to start Percona Backup Agent For MongoDB.’

If I run pbm status, I get error “Error: connect to mongodb: get config server connection URI: mongo: no documents in result”

What should I do?

P.S. The same config works on other MongoDB Replicaset (NON-Sharded)

Mukesh_Kumar · January 12, 2024, 10:00am

Hi @Vitaly_Kargin
I hope you’re following the doc to setup the pbm in the sharded cluster.

Also make sure to set the correct pbm-uri and pbm-agent for config and shards. Sharing an example of PBM_URI and pbm-agent along with it path.
Config server pbm uri:
export PBM_MONGODB_URI="mongodb://pbm_user:pbm_cred@config-server-hostname:port/?authSource=admin&replSetName=cfg_replSetName"

Shard server pbm uri:

export PBM_MONGODB_URI="mongodb://pbm_user:pbm_cred@shard-server-hostname:port/?authSource=admin&replSetName=shard_replSetName"

pbm-agent on shards:
sudo cat /etc/sysconfig/pbm-agent (for redhat and derivatives) or /etc/default/pbm-agent (for debian/Ubuntu )
PBM_MONGODB_URI="mongodb://pbm_user:pbm_cred@config-server-hostname:port/?authSource=admin"

pbm-agent for config:
PBM_MONGODB_URI=mongodb://pbm_user:pbm_cred@shard-server-hostname:port/?authSource=admin

Please adjust your user/cred/replSetName, shard/config_host:port accordingly.

Apart from this please share the logs of pbm-agent from below, if still you caught an error while configuring the pbm for sharded cluster:

 journalctl -u pbm-agent.service

Thanks,
Mukesh

Vitaly_Kargin · January 12, 2024, 11:29am

I dont understand.
You wrote:

pbm-agent on shards:
sudo cat /etc/sysconfig/pbm-agent (for redhat and derivatives) or /etc/default/pbm-agent (for debian/Ubuntu )
PBM_MONGODB_URI="mongodb://pbm_user:pbm_cred@config-server-hostname:port/?authSource=admin"

pbm-agent for config:
PBM_MONGODB_URI=mongodb://pbm_user:pbm_cred@shard-server-hostname:port/?authSource=admin

But how can I indicate shard-server-hostname in pbm-agent for config??? I have 10 shards (10 replicaset shards). Which shard should specify in mongo-config server?
And why should i specify config-server-hostname on pbm-agent on shards?

Look:
my hosts:
mongocfg-1 (replSetName: cfg)
mongocfg-2 (replSetName: cfg)
mongocfg-3 (replSetName: cfg)

mongoshard-1-1 (replSetName: rs01)
mongoshard-1-2 (replSetName: rs01)
mongoshard-1-3 (replSetName: rs01)

mongoshard-2-1 (replSetName: rs02)
mongoshard-2-2 (replSetName: rs02)
mongoshard-2-3 (replSetName: rs02)

root@mongocfg-1:~# cat /etc/percona/pbm-agent-env
PBM_MONGODB_URI=“mongodb://perconaBackup:perconaBackup@127.0.0.1:27017/?authSource=admin”

root@mongoshard-1-1:~# cat /etc/percona/pbm-agent-env
PBM_MONGODB_URI=“mongodb://perconaBackup:perconaBackup@127.0.0.1:27017/?authSource=admin”

and other the same uri

p.s. It still doesn’t work

Vitaly_Kargin · January 12, 2024, 11:42am

And I have to say, that I find in docu nothing about connection string for sharded cluster

Mukesh_Kumar · January 12, 2024, 11:49am

Hi @Vitaly_Kargin
Yes, correct PBM_MONGODB_URI in pbm-agent for config/shard should be like below:

pbm-agent on config:
sudo cat /etc/sysconfig/pbm-agent (for redhat and derivatives) or /etc/default/pbm-agent (for debian/Ubuntu )
PBM_MONGODB_URI="mongodb://pbm_user:pbm_cred@config-server-hostname:port/?authSource=admin"

pbm-agent for shard:
PBM_MONGODB_URI=mongodb://pbm_user:pbm_cred@shard-server-hostname:port/?authSource=admin

PBM_MONGODB_URI of pbm-agent for both shards and configs looks fine.
Please share the pbm-agent logs so that we can troubleshoot it further.

Thanks,
Mukesh

Vitaly_Kargin · January 12, 2024, 1:48pm

All logs from root@mongoshard-1-1:~#

cat /var/log/pbm-agent.log
Error: connect to mongodb: get config server connection URI: mongo: no documents in result
Error: connect to mongodb: get config server connection URI: mongo: no documents in result
Error: connect to mongodb: get config server connection URI: mongo: no documents in result
Error: connect to mongodb: get config server connection URI: mongo: no documents in result
Error: connect to mongodb: get config server connection URI: mongo: no documents in result
Error: connect to mongodb: get config server connection URI: mongo: no documents in result

journalctl -u pbm-agent.service
-- Journal begins at Sun 2023-12-24 23:22:05 UTC, ends at Fri 2024-01-12 13:45:12 UTC. --
Jan 08 21:42:17 mongoshard-1-1 systemd[1]: Starting Percona Backup Agent For MongoDB...
Jan 08 21:42:17 mongoshard-1-1 systemd[440246]: pbm-agent.service: Failed to determine user credentials: No such process
Jan 08 21:42:17 mongoshard-1-1 systemd[440246]: pbm-agent.service: Failed at step USER spawning /usr/bin/pbm: No such process
Jan 08 21:42:17 mongoshard-1-1 systemd[1]: pbm-agent.service: Control process exited, code=exited, status=217/USER
Jan 08 21:42:17 mongoshard-1-1 systemd[1]: pbm-agent.service: Failed with result 'exit-code'.
Jan 08 21:42:17 mongoshard-1-1 systemd[1]: Failed to start Percona Backup Agent For MongoDB.
Jan 08 21:42:22 mongoshard-1-1 systemd[1]: pbm-agent.service: Scheduled restart job, restart counter is at 1.
Jan 08 21:42:22 mongoshard-1-1 systemd[1]: Stopped Percona Backup Agent For MongoDB.
Jan 08 21:42:22 mongoshard-1-1 systemd[1]: Starting Percona Backup Agent For MongoDB...
Jan 08 21:42:22 mongoshard-1-1 systemd[440302]: pbm-agent.service: Failed to determine user credentials: No such process
Jan 08 21:42:22 mongoshard-1-1 systemd[440302]: pbm-agent.service: Failed at step USER spawning /usr/bin/pbm: No such process
Jan 08 21:42:22 mongoshard-1-1 systemd[1]: pbm-agent.service: Control process exited, code=exited, status=217/USER
Jan 08 21:42:22 mongoshard-1-1 systemd[1]: pbm-agent.service: Failed with result 'exit-code'.
Jan 08 21:42:22 mongoshard-1-1 systemd[1]: Failed to start Percona Backup Agent For MongoDB.
Jan 08 21:42:27 mongoshard-1-1 systemd[1]: pbm-agent.service: Scheduled restart job, restart counter is at 2.
Jan 08 21:42:27 mongoshard-1-1 systemd[1]: Stopped Percona Backup Agent For MongoDB.
Jan 08 21:42:27 mongoshard-1-1 systemd[1]: Starting Percona Backup Agent For MongoDB...
Jan 08 21:42:27 mongoshard-1-1 systemd[440303]: pbm-agent.service: Failed to determine user credentials: No such process
Jan 08 21:42:27 mongoshard-1-1 systemd[440303]: pbm-agent.service: Failed at step USER spawning /usr/bin/pbm: No such process
Jan 08 21:42:27 mongoshard-1-1 systemd[1]: pbm-agent.service: Control process exited, code=exited, status=217/USER
Jan 08 21:42:27 mongoshard-1-1 systemd[1]: pbm-agent.service: Failed with result 'exit-code'.
Jan 08 21:42:27 mongoshard-1-1 systemd[1]: Failed to start Percona Backup Agent For MongoDB.
Jan 08 21:42:33 mongoshard-1-1 systemd[1]: pbm-agent.service: Scheduled restart job, restart counter is at 3.
Jan 08 21:42:33 mongoshard-1-1 systemd[1]: Stopped Percona Backup Agent For MongoDB.
Jan 08 21:42:33 mongoshard-1-1 systemd[1]: Starting Percona Backup Agent For MongoDB...

systemctl cat pbm-agent.service 
# /lib/systemd/system/pbm-agent.service
[Unit]
Description=Percona Backup Agent For MongoDB
Documentation=https://docs.percona.com/
After=network.target

[Service]
RestartSec=5s
Restart=on-failure
EnvironmentFile=-/etc/percona/pbm-agent-env
Type=simple
User=mongodb
Group=mongodb
PermissionsStartOnly=true
ExecStartPre=/usr/bin/pbm config --file /etc/percona/pbm_config.yml
ExecStart=/usr/bin/pbm-agent

StandardOutput=append:/var/log/pbm-agent.log
StandardError=append:/var/log/pbm-agent.log

[Install]
WantedBy=multi-user.target

cat /etc/percona/pbm_config.yml
pitr:
  enabled: true
  oplogOnly: false
  oplogSpanMin: 10
  compression: s2
  compressionLevel: 2

storage:
  type: s3
  s3:
    region: eu
    bucket: superset-mongo-shard
    prefix: percona-backups
    endpointUrl: *****
    credentials:
      access-key-id: *****
      secret-access-key: *****

cat /etc/percona/pbm-agent-env
PBM_MONGODB_URI="mongodb://perconaBackup:perconaBackup@127.0.0.1:27017/?authSource=admin"

pbm version
Version:   2.3.1
Platform:  linux/amd64
GitCommit: 8c4265cfb2d9a7581b782a829246d8fcb6c7d655
GitBranch: release-2.3.1
BuildTime: 2023-11-29_13:34_UTC
GoVersion: go1.19

pbm logs
Error: connect to mongodb: get config server connection URI: mongo: no documents in result

Vitaly_Kargin · January 13, 2024, 4:59am

I solved the problem
It was network problem between shard and cfg.
Thank you for participating.

Mukesh_Kumar · January 16, 2024, 12:53pm

Glad to know. thanks.

Topic		Replies	Views
Percona mongodb sharding not conected	9	1246	September 6, 2023
Having trouble with pbm-agent and PBM_MONGODB_URI settings Percona Backup for MongoDB	4	697	September 14, 2022
Pbm-agent in shard servers not working Percona Backup for MongoDB	2	779	August 23, 2022
Pbm backup error in a replicaset non-sharded scenario Percona Backup for MongoDB	3	891	August 11, 2021
PBM Backup Issues on Config Server Nodes vs. Shard Nodes in Kubernetes Percona Backup for MongoDB percona	1	164	March 1, 2024

Error: connect to mongodb: get config server connection URI: mongo: no documents in result

Description:

Problem:

What should I do?

Related Topics