PBM Access on tls encrypted cluster

MongoDB Percona Backup for MongoDB
1

I have a three nodes replicaSet configured with tls encryption
I configured my PBM_MONGODB_URI as follow:

PBM_MONGODB_URI="mongodb://pbmuser:PASSWORD@mongodb1:27017,mongodb2:27017,mongodb3:27017/?replicaSet=rs0&tls=true&tlsCertificateKeyFile=/certificate/client.pem&tlsCAFile=/certificate/mongoca.pem&tlsAllowInvalidCertificates=true&tlsAllowInvalidHostnames=true"

As soon as I try to configure the PBM I receive an error

connection rejected"},"remote":"10.0.0.4:52344","connectionId":217}}
{"t":{"$date":"2021-02-17T13:13:49.393+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn217","msg":"Connection ended","attr":{"remote":"10.0.0.4:52344","connectionId":217,"connectionCount":12}}
{"t":{"$date":"2021-02-17T13:13:49.893+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.0.4:52350","connectionId":218,"connectionCount":13}}
{"t":{"$date":"2021-02-17T13:13:49.901+00:00"},"s":"E",  "c":"NETWORK",  "id":23255,   "ctx":"conn218","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2021-02-17T13:13:49.901+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn218","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.0.0.4:52350","connectionId":218}}

If I try to connect to the DB with the client the following uri is working:

mongo "mongodb://mongodb1:27017,mongodb2:27017,mongodb3:27017/?replicaSet=rs0" --tls --tlsCAFile /certificate/mongoca.pem --tlsCertificateKeyFile /certificate/client.pem --username "pbmuser" -p

and

mongo "mongodb://mongodb1:27017,mongodb2:27017,mongodb3:27017/?replicaSet=rs0&tls=true&tlsCertificateKeyFile=/certificate/client.pem&tlsCAFile=/certificate/mongoca.pem&tlsAllowInvalidCertificates=true&tlsAllowInvalidHostnames=true" -u pbmuser -p

has the same issue as the pbm command.

Any Idea?

2 Likes
2

Hello Joel.

In the drivers released to support v4.2 MongoDB server the TLS options started to include filepath type arguments such “tlsCertificateKeyFile”. Well, in some drivers - I know the golang driver version in PBM does at least.

The surprise though is the mongo shell, even v4.2 and v4.4, does not accept them (eg. “tlsCertificateKeyFile”, “tlsCAFile”) as connection string URI arguments. In small print, in the documentation, it currently has this:

This connection string option [= several “tls*” options] is not available for the mongo shell. Use the command-line option instead.

Why, I don’t know. I guess that C++ client code the mongo shell is using, which was basically the C++ Mongo driver code once upon a time, has since v4.2 become out of sync with driver specs in general.

3 Likes
3

Hello Akira,

Also I definitely have the following results with the mongo cli client

[root@mongodb1 ~]# mongo "mongodb://mongodb1:27017,mongodb2:27017,mongodb3:27017/?replicaSet=visibility0" --tls --tlsCAFile /root/certificate/mongoca.pem --tlsCertificateKeyFile /root/certificate/client.pem --username "pbmuser" -p
Percona Server for MongoDB shell version v4.4.3-5
Enter password:
connecting to: mongodb://mongodb1.az.dsv.com:27017,mongodb2.az.dsv.com:27017,mongodb3.az.dsv.com:27017/?compressors=disabled&gssapiServiceName=mongodb&replicaSet=visibility0
Implicit session: session { "id" : UUID("12f0691e-02ab-4479-8508-7acd14fe0bc3") }

Percona Server for MongoDB server version: v4.4.3-5

The server generated these startup warnings when booting:

2021-02-17T11:01:00.293+00:00: While invalid X509 certificates may be used to connect to this server, they will not be considered permissible for authentication
2021-02-17T11:01:00.293+00:00: This server will not perform X.509 hostname validation. This may allow your server to make or accept connections to untrusted parties
---
visibility0:PRIMARY>

This one is working, and the next one is principally the same, but don’t work

[root@mongodb1 ~]# mongo "mongodb://mongodb1.az.dsv.com:27017,mongodb2.az.dsv.com:27017,mongodb3.az.dsv.com:27017/?replicaSet=visibility0&tls=true&tlsCAFile=/root/certificate/mongoca.pem&tlsCertificateKeyFile=/root/certificate/client.pem" --username "pbmuser" -p
Percona Server for MongoDB shell version v4.4.3-5
Enter password:
connecting to: mongodb://mongodb1.az.dsv.com:27017,mongodb2.az.dsv.com:27017,mongodb3.az.dsv.com:27017/?compressors=disabled&gssapiServiceName=mongodb&replicaSet=visibility0&tls=true&tlsCAFile=%2Froot%2Fcertificate%2Fmongoca.pem&tlsCertificateKeyFile=%2Froot%2Fcertificate%2Fclient.pem
{"t":{"$date":"2021-02-18T09:26:53.141Z"},"s":"E",  "c":"NETWORK",  "id":23256,   "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: self signed certificate in certificate chain"}}
{"t":{"$date":"2021-02-18T09:26:53.147Z"},"s":"E",  "c":"NETWORK",  "id":23256,   "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: self signed certificate in certificate chain"}}

The error in the logs is no SSL certificate file

{"t":{"$date":"2021-02-18T09:32:17.871+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.0.0.4:33846","connectionId":339,"connectionCount":13}}
{"t":{"$date":"2021-02-18T09:32:17.881+00:00"},"s":"E",  "c":"NETWORK",  "id":23255,   "ctx":"conn339","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2021-02-18T09:32:17.881+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn339","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.0.0.4:33846","connectionId":339}}

And as the second format is the one that is needed by the PBM in the PBM_MONGODB_URI the pbm-agent does not start at all or at least no backup can be done!

1 Like
4

Hi Joel.

I was mixed up a bit reading the above.

Yes, “no SSL certificate provided by peer” on the server showed that a client connected without providing TLS certificate credentials.

But the client in this case was just the mongo shell again. Not pbm-agent.

Even though the classic mongo shell can accept a URI string it is known not to support the “tlsCertificateKeyFile” and “tlsCAFile” options. (For future-proofing the value of this blog post: What the “mongosh” NodeJS-based shell released in 2020 does is unknown to me at the moment.)

So although in principle the two mongo shell commands below should be the same they are not.

mongo "mongodb://mongodb1:27017,mongodb2:27017,mongodb3:27017/?replicaSet=visibility0" --tls --tlsCAFile /root/certificate/mongoca.pem --tlsCertificateKeyFile /root/certificate/client.pem --username "pbmuser" -p SECRET
mongo "mongodb://pbmuser:SECRET@mongodb1:27017,mongodb2:27017,mongodb3:27017/?replicaSet=visibility0&tls=true&tlsCAFile=/root/certificate/mongoca.pem&tlsCertificateKeyFile=/root/certificate/client.pem"

But we’re not debugging mongo shell, we’re debugging the pbm-agent (or pbm CLI).

The original problem is still unresolved. That is this URI below looks good to me:

PBM_MONGODB_URI="mongodb://pbmuser:PASSWORD@mongodb1:27017,mongodb2:27017,mongodb3:27017/?replicaSet=rs0&tls=true&tlsCertificateKeyFile=/certificate/client.pem&tlsCAFile=/certificate/mongoca.pem&tlsAllowInvalidCertificates=true&tlsAllowInvalidHostnames=true"

I expect it to work so long as:

  • The mongod nodes were also configured to accept ‘invalid certificates’ and ‘invalid hostnames’ (Not that this is production-acceptable, but we’re just testing.)
  • The forward slash (“/”) characters in the TLS file paths aren’t a problem 
 but I don’t think they are. If you are worried you can replace them with “%2F” (the URI-encoding of “/”)

Could you please try connecting with the pbm CLI, say to run “pbm list” as a dummy command. (I suggest the pbm CLI just for speed. It uses the connection code but it is less time to use it than configure and start the pbm-agent.)

1 Like
5

Hello Akira,

thanks for your reply!

Also a part of it is now working, if I start the pbm-agent manually it’s working

nohup pbm-agent --mongodb-uri "mongodb://pbmuser: MyPassWord @mongodb1:27017/?replicaSet=visibility0&tls=true&tlsCertificateKeyFile=%2Froot%2Fcertificate%2Fclient.pem&tlsCAFile=%2Froot%2Fcertificate%2Fmongoca.pem&tlsAllowInvalidCertificates=true&tlsAllowInvalidHostnames=true" >> /var/log/pbm-agent.log 2>&1 &

and so on, on the other server

If I put the same information in the file /etc/sysconfig/pbm-agent
The pbm-agent don’t start at all with only this error (Active: inactive (dead))

[root@mongodb1 ~]# systemctl start pbm-agent
[root@mongodb1 ~]# systemctl status pbm-agent
● pbm-agent.service - pbm-agent
   Loaded: loaded (/usr/lib/systemd/system/pbm-agent.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2021-03-02 10:10:05 UTC; 3s ago
  Process: 3199034 ExecStart=/usr/bin/pbm-agent (code=exited, status=0/SUCCESS)
Main PID: 3199034 (code=exited, status=0/SUCCESS)

Mar 02 10:10:05 mongodb1.az.dsv.com systemd[1]: Started pbm-agent.

[root@mongodb1 ~]# journalctl -xe -u pbm-agent
Mar 02 10:10:05 mongodb1 systemd[1]: Started pbm-agent.
-- Subject: Unit pbm-agent.service has finished start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit pbm-agent.service has finished starting up.
--
-- The start-up result is done. 
[root@mongodb1 ~]#
[root@mongodb1 ~]# ps ax | grep pbm
3198458 pts/0    R+     0:00 grep --color=auto pbm

No pbm-agent started

I tried any syntax in the pbm-agent file, no one is working as soon as the encryption is enabled.

1 Like
6

Hi Joel.

It is strange to me that “journalctl -u pbm-agent” would only show one line (I’m ignoring the other lines added by the -x option). Is it the case that the “systemctl status pbm-agent” and “journalctl -u pbm-agent” commands were run within 30 secs? By default the mongodb driver will try to connect for 30 secs before returning an error.

The message above is just an observation about the log info or lack of it. Let’s go on with the connection error

I think your observation that PBM_MONGODB_URI, absorbed through the /etc/sysconfig/pbm-agent source shell file when pbm-agent is (re)started by systemctl, is not working but using --mongodb-uri commandline argument does is going to be the key troubleshooting discovery. I started a reproduction today but haven’t been able to complete it before other meetings are beginning. Sorry I’ll have to come back next week.

1 Like