PMM Server/Client Docker SSL Issue

Percona Monitoring and Management (PMM) PMM 2.x
1

Following using the docs: Secure - Percona Monitoring and Management

Using docker images:
percona/pmm-server:2
percona/pmm-client:2

I’m running pmm-server on vm0 and I’m able to bring up the UI and see that the connection is secure.
Docker command:
docker run --detach --restart always
–publish 443:443
-v ${pmm_server_volume_name}:${pmm_server_volume_path}
-v ${MONGO_OPT_PATH}/certificates/pmm:/srv/nginx
–name pmm-server
${pmm_server_image}

Certificates mounted to /srv/nginx:
[root@2103881373c7 nginx]# ls -lrt
total 28
…
-rw-r-----. 1 root root 1679 Feb 15 20:39 certificate.key
-rw-r-----. 1 root root 2908 Feb 15 20:39 certificate.crt
-rw-r-----. 1 root root 3042 Feb 15 20:39 ca-certs.pem
-rw-r-----. 1 root root 245 Feb 15 20:39 dhparam.pem

On vm2, I’m running pmm-client and it is unclear from the docs on how I need to mount certificates. I’ve mounted the same KeyFactor certificate to /srv/nginx and I mounted a self-signed certificate used for TLS to percona mongodb in /certs.
docker run
–name pmm-client
-d
–rm
-e PMM_AGENT_SERVER_ADDRESS=${pmm_server_address}
-e PMM_AGENT_SERVER_USERNAME=${pmm_server_username}
-e PMM_AGENT_SERVER_PASSWORD=${pmm_server_password}
-e PMM_AGENT_SETUP=1
-e PMM_AGENT_CONFIG_FILE=config/pmm-agent.yaml
-v /opt/app/mongo/local/bin/certificates/pmm-client:/certs
-v /opt/app/mongo/local/bin/certificates/pmm:/srv/nginx
-v ${pmm_client_volume_name}:${pmm_client_volume_path}
${pmm_client_image}

pmm-client container exits almost immediately (6-7 seconds) with this in docker logs:
Checking local pmm-agent status…
pmm-agent is not running.
Registering pmm-agent on PMM Server…
Failed to register pmm-agent on PMM Server: Post “https://<dns_name>:443/v1/management/Node/Register”: x509: certificate signed by unknown authority.

When I run a curl command, I’m seeing “Peer’s Certificate issuer not recognized”:
curl -v https://admin:@<dns_name>

  • About to connect() to <dns_name> port 443 (#0)
  • Trying 10.0.1.8…
  • Connected to <dns_name> (10.0.1.8) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • Server certificate:
  •   subject: CN=<redacted>

  •   start date: Feb 15 00:00:00 2023 GMT

  •   expire date: Feb 13 23:59:59 2024 GMT

  •   common name: <redacted>

  •   issuer: CN=DigiCert TLS RSA SHA256 2020 CA1,O=DigiCert Inc,C=US
  • NSS error -8179 (SEC_ERROR_UNKNOWN_ISSUER)
  • Peer’s Certificate issuer is not recognized.
  • Closing connection 0
    curl: (60) Peer’s Certificate issuer is not recognized.

echo “Getting KeyFactor intermediary chain and certificate authority”
openssl pkcs12 -in ${MONGO_OPT_PATH}/certificates/pmm/${cert_name}.pfx -password pass: -passout pass:${cert_password} -clcerts -nokeys | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’ > ${MONGO_OPT_PATH}/certificates/pmm/ca-certs.pem
openssl pkcs12 -in ${MONGO_OPT_PATH}/certificates/pmm/${cert_name}.pfx -password pass: -passout pass:${cert_password} -cacerts -nokeys -chain | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’ > ${MONGO_OPT_PATH}/certificates/pmm/ca-certs.pem

Am I generating the ca-certs.pem file correctly or any ideas of what the issue is?

3

Hi jbl1,

You could set PMM_AGENT_SERVER_INSECURE_TLS=1 to skip tls cert verfication until we have a proper answer.

4

Hi Denys, I have successfully tested pmm-server/pmm-client with the non-TLS flag before. I am specifically working on setting up a secure connection from pmm-client to pmm-server and pmm-client to percona mongodb.

5

theoretically you need to put it (serts) somewhere inside of system directory where all other certs are, so the lib would find it in a standard location.

So far I don’t have correct answer, where they should be and how.

6

Yeah that’s what I was thinking. The documentation provides a step-by-step solution to adding certs to /srv/nginx for pmm-server but not for pmm-client. Please let me know when your team has tested a successful solution or when you have an answer.

7

Hi Denys, any solution to this yet?

8

Hi @jbl1,

I tried, and it worked for me to mount cert on this location for the pmm-client container:
-v ~/tmp/certs-pmm/certificate.crt:/etc/pki/tls/certs/ca-bundle.crt

Go looks for certificates in these locations:

Also, please make sure you have proper CN/SAN in you certs so connection wouldn’t be refused because of incorrect domain name:
requested domain name does not match the server's certificate.

Please check Automate the SSL Certificate Lifecycle of your Percona Monitoring and Management Server - Percona Database Performance Blog as well.

Thanks,
Denys

9

Hi Denys, thanks for the update. I tried your solution by mounting certificate.crt. Good news is that the x509: certificate signed by unknown authority error doesn’t show in the logs. On the other hand, I’m not sure why pmm-client didn’t start up. I included the docker logs but I’m not sure how else to debug this.

-v /opt/app/mongo/local/bin/certificates/pmm/certificate.crt:/etc/pki/tls/certs/ca-bundle.crt

INFO[2023-02-24T14:59:42.459+00:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml.  component=local-server
INFO[2023-02-24T14:59:42.461+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=local-server
INFO[2023-02-24T14:59:42.463+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=local-server
INFO[2023-02-24T14:59:42.463+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=local-server
INFO[2023-02-24T14:59:42.463+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=local-server
INFO[2023-02-24T14:59:42.463+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=local-server
INFO[2023-02-24T14:59:42.463+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=local-server
INFO[2023-02-24T14:59:42.463+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=local-server
INFO[2023-02-24T14:59:42.463+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=local-server
INFO[2023-02-24T14:59:42.463+00:00] Updating PMM Server address from "<dns_name>" to "<dns_name>:443".  component=local-server
INFO[2023-02-24T14:59:42.465+00:00] Stopped.                                      component=local-server/JSON
INFO[2023-02-24T14:59:42.465+00:00] Done.                                         component=local-server
INFO[2023-02-24T14:59:42.465+00:00] Process: context canceled.                    agentID=/agent_id/6facb169-6027-4ef9-839c-135064a40d1f component=agent-process type=vm_agent
INFO[2023-02-24T14:59:42.465+00:00] Sending status: STOPPING (port 42001).        agentID=/agent_id/6facb169-6027-4ef9-839c-135064a40d1f component=agent-process type=vm_agent
INFO[2023-02-24T14:59:42.466+00:00] Starting 0, restarting 0, and stopping 2 agent processes.  component=supervisor
INFO[2023-02-24T14:59:42.466+00:00] Supervisor Changes() channel drained.         component=client
INFO[2023-02-24T14:59:42.466+00:00] Supervisor QANRequests() channel drained.     component=client
INFO[2023-02-24T14:59:42.466+00:00] Connection closed.                            component=client
INFO[2023-02-24T14:59:42.481+00:00] Jobs runner Messages() channel drained.       component=client
INFO[2023-02-24T14:59:42.481+00:00] Actions runner Results() channel drained.     component=client
INFO[2023-02-24T14:59:42.481+00:00] Process: context canceled.                    agentID=/agent_id/22fb7102-e716-458f-801a-ca5b0fcfad34 component=agent-process type=node_exporter
INFO[2023-02-24T14:59:42.481+00:00] Sending status: STOPPING (port 42000).        agentID=/agent_id/22fb7102-e716-458f-801a-ca5b0fcfad34 component=agent-process type=node_exporter
INFO[2023-02-24T14:59:42.481+00:00] Done.                                         component=client
10

@jbl1 that needs investigation, probably some other bug with SSL client-server connection. Could you please open a bug in jira.percona.com ?

11

Hi @Denys_Kondratenko, I was on vacation. I have opened a ticket here: [PMM-11767] PMM Server/Client Docker SSL - Percona JIRA.

12

Hi @Denys_Kondratenko, any idea when this ticket will at least be assigned? Created it 10 days ago.

13

Hi @jbl1,

I’ve tried to run PMM server + client with self-signed certificates, scenario seems to work fine. Please find steps on how to set this up below:

  1. Generate certificates (I’ve used GitHub - OpenVPN/easy-rsa: easy-rsa - Simple shell based CA utility for convenience):

$ ./easyrsa init-pki
$ ./easyrsa --req-cn=Percona --batch build-ca nopass
$ ./easyrsa --req-ou=server --subject-alt-name=DNS:pmm-server,IP:172.18.0.1 --batch build-server-full pmm-server nopass
$ cp pki/ca.crt /tmp/certs/ca-certs.pem
$ cp pki/private/pmm-server.key /tmp/certs/certificate.key
$ cp pki/issued/pmm-server.crt /tmp/certs/certificate.crt
$ openssl dhparam -out /tmp/certs/dhparam.pem 2048
$ ls -lah /tmp/certs/
-r–r–r–. 1 root root 1,2K Apr 6 08:46 ca-certs.pem
-r–r–r–. 1 root root 4,6K Apr 6 08:47 certificate.crt
-r–r–r–. 1 root root 1,7K Apr 6 08:47 certificate.key
-r–r–r–. 1 root root 424 Apr 6 08:47 dhparam.pem

  1. Run PMM server and client:

$ docker run --name pmm-server -d --publish 443:443 -v /tmp/certs:/srv/nginx percona/pmm-server:2
$ docker run --name pmm-client -d -e PMM_AGENT_SERVER_ADDRESS=172.18.0.1 -e PMM_AGENT_SERVER_USERNAME=*** -e PMM_AGENT_SERVER_PASSWORD=*** -e PMM_AGENT_SETUP=1 -e PMM_AGENT_CONFIG_FILE=config/pmm-agent.yaml -v /tmp/certs:/etc/pki/tls/certs percona/pmm-client:2
===================================================
INFO[2023-04-06T05:58:05.595+00:00] Updating PMM Server address from “172.18.0.1” to “172.18.0.1:443”. component=main
INFO[2023-04-06T05:58:05.600+00:00] Connected to 172.18.0.1:443. component=client

Regarding your particular setup it’s still unclear why pmm-agent exits, therefore please try either to use the steps above and/or enable debug for pmm-agent (PMM_AGENT_DEBUG, for more details pmm-agent - PMM Client agent - Percona Monitoring and Management) to check if any additional errors appear in logs

1 Like