MongoDB backup failed

Percona Monitoring and Management (PMM) PMM 2.x
1

I have tried to configure PMM to manage MongoDB backups, unfortunately backup is not created.
I have followed the documentation to add my MongoDB to PMM server and it looks good, I got information form the server.

When tried to create a manual backup it drops an error without any logs on PMM server side. I’m have checked pmm-agent on the DB server and I saw the following error is /var/log/messages

10 14:27:24 vmccpipmongo1td pmm-agent[2128860]: pbm config error: {"Error":"connect to mongodb: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: 10.140.225.153:27017, Type: Unknown, Last error: connection(10.140.225.153:27017[-64]) socket was unexpectedly closed: EOF }, ] }"}

My server is configured for secure connection, but when I added it to PMM-server the TLS verification was ignored
pmm-admin add mongodb --tls --tls-skip-verify --host=10.140.225.153 --port=27017 --username=pmm_user --password=pmm_password --authentication-database=admin --service-name=uatPrimary --cluster=uat

If this is a connection issue how pmm-agent can send other data to pmm-server ? It is weird to me.

Any suggestion how to fix it ?

2

The PBM as not configured originally, but it is working now.
Unfortunately from PMM I still got the error:

Jan 12 15:41:55 vmccpipmongo1td pmm-agent[2552500]: pbm config error: {"Error":"connect to mongodb: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: 10.140.225.153:27017, Type: Unknown, Last error: connection(10.140.225.153:27017[-64]) socket was unexpectedly closed: EOF }, ] }"}

How can I re-configure pmm-agent to start to use proper pbm configuration?

3

How can I re-configure pmm-agent to start to use proper pbm configuration?

Here is the documentation with the command for adding a MongoDB instance with the required prerequisites,

4

@lalit.choudhary

  1. Backup is enabled

    PMM-settings
    PMM-settings1057×384 23.1 KB

  2. I’m using local storage of the mongodb node, with pbm cli I able too create backup, so this should be OK.

  3. PMM-client is installed on mongodb node, pmm-agent up and running.

root@vmccpipmongo1td:/home/admin $ pmm-admin status
Agent ID : /agent_id/16304382-ba6a-486b-a354-4fc128cdc9c8
Node ID  : /node_id/595d815d-9257-42fd-8106-277dc0e3f23d
Node name: vmccpipmongo1td

PMM Server:
        URL    : https://10.140.225.154:8443/
        Version: 2.37.0

PMM Client:
        Connected        : true
        Time drift       : 95.176µs
        Latency          : 312.545µs
        Connection uptime: 100
        pmm-admin version: 2.37.0
        pmm-agent version: 2.37.0
Agents:
        /agent_id/2f33a325-7cfd-4ec8-9c26-0a3499e33401 vmagent Running 42000
        /agent_id/36174260-32f5-49eb-9f3f-93041c545d00 mongodb_exporter Running 42001
        /agent_id/41d72d4f-18cf-463d-864a-b32cb0e3b532 mongodb_profiler_agent Running 0
        /agent_id/771a7389-a593-437d-b5f0-8d60e950fde8 node_exporter Running 42002

  1. PBM is intalled on mongodb node, it is up and running. I able to do backup from cli, so the configuraiton should be OK.

  2. pmm is run with root user, and root is able to use mongod binary from anywhere

$ echo $PATH
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin

  1. Mongodb node added as cluster

    Capture1
    Capture1881×202 11.5 KB

  2. mongodb node run as rs

rs0 [direct: primary] test> rs.status
[Function: status] AsyncFunction {
  apiVersions: [ 0, 0 ],
  returnsPromise: true,
  serverVersions: [ '0.0.0', '999.999.999' ],
  topologies: [ 'ReplSet', 'Sharded', 'LoadBalanced', 'Standalone' ],
  returnType: { type: 'unknown', attributes: {} },
  deprecated: false,
  platforms: [ 'Compass', 'Browser', 'CLI' ],
  isDirectShellCommand: false,
  acceptsRawInput: false,
  shellCommandCompleter: undefined,
  help: [Function (anonymous)] Help
}
  1. pmm has pmm user and pbm has pbm user in mongodb, and both have all the rights which was described in the related documentation
 {
    _id: 'admin.pbmuser',
    userId: new UUID("830caf50-effe-410d-a13f-04d084782d3f"),
    user: 'pbmuser',
    db: 'admin',
    roles: [
      { role: 'restore', db: 'admin' },
      { role: 'readWrite', db: 'admin' },
      { role: 'clusterMonitor', db: 'admin' },
      { role: 'pbmAnyAction', db: 'admin' },
      { role: 'backup', db: 'admin' }
    ],
    mechanisms: [ 'SCRAM-SHA-1', 'SCRAM-SHA-256' ]
  },
  {
    _id: 'admin.pmm',
    userId: new UUID("c50c830c-9464-4542-9c5c-f780e8d1f0b2"),
    user: 'pmm',
    db: 'admin',
    roles: [
      { role: 'readWrite', db: 'admin' },
      { role: 'explainRole', db: 'admin' },
      { role: 'restore', db: 'admin' },
      { role: 'read', db: 'local' },
      { role: 'backup', db: 'admin' },
      { role: 'pbmAnyAction', db: 'admin' },
      { role: 'clusterMonitor', db: 'admin' }
    ],
    mechanisms: [ 'SCRAM-SHA-1', 'SCRAM-SHA-256' ]
  }
  1. I dont see any issue here.

So to summarize the installation I dont see any issue here. What I don’t understand is that:
If the pbm cli backup works properly why pmm compains to pbm config erro ???

10 14:27:24 vmccpipmongo1td pmm-agent[2128860]: pbm config error: {"Error":"connect to mongodb: create mongo connection: mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: 10.140.225.153:27017, Type: Unknown, Last error: connection(10.140.225.153:27017[-64]) socket was unexpectedly closed: EOF }, ] }"}

From where pmm should get the pba configuration? How to double check this settings ?

5

@lalit.choudhary

Any suggestion what else can be checked ? I have reviewed everything 2x times, but no progress.

6

Interesting pursuance,

I have done exactly same configuration in production environment as it in UAT (where I have the issue) to have at least the pbm cli backup solution, and there the PMM controlled backup works as expected. The only different compared to UAT environment is that the UAT is a single server mongoDB node running as replica set.

1 Like