Installed Percona-pmm and its failing to add RDS instances with a certificate error

Percona Monitoring and Management (PMM) PMM 2.x
1

Hey All, trying to troubleshoot this and running into some weird issues.

  1. I built a new docker image pulling from percona/pmm-server:2.22.0
  2. Deployed this and ran the container interactively, while I can get it to run and I can add an RDS instance manually i.e. I provide it the username/pass and node parameters, I cannot get discovery to work at all, nor can I get system metrics to show up. I get the query tool and some rds metrics to show, but node metrics are empty even though I have enabled enhanced monitoring or both nodes.

The error I get while trying to discover is:
RequestError: send request failed caused by: Post “https://rds.us-west-1.amazonaws.com/”: x509: certificate has expired or is not yet valid: current time 2021-10-14T23:47:41Z is after 2020-11-04T21:29:21Z

I put in an IAM user which should work according to the docs but its not per the expired cert there.

Also I scanned most of the certs I could find in the container and NONE of them are even close to expiring. I also downloaded the latest rds-root.pem (from here https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem ) and still continue to get the error.

I also notice curl doesn’t work as well - not sure if its related or just a side issue.

pmm-admin status
Agent ID: pmm-server
Node ID : pmm-server

PMM Server:
	URL    : https://127.0.0.1:443/
	Version: 2.22.0

PMM Client:
	Connected        : true
	Time drift       : 94.549µs
	Latency          : 304.912µs
	pmm-admin version: 2.22.0
	pmm-agent version: 2.22.0
Agents:
	/agent_id/06760184-e481-4ce3-8b26-151db06528fa mysql_perfschema_agent Running
	/agent_id/17407fd8-a007-489a-8c4b-0a1cf389edb7 mysqld_exporter Running
	/agent_id/22ec0f9b-7e83-4f15-8463-b0398b0e5159 mysqld_exporter Running
	/agent_id/44b5d6b6-f555-42d2-8820-dccbcb1c7b95 node_exporter Running
	/agent_id/7deac61a-edfa-4fa3-a595-81a04b54d8df postgresql_pgstatements_agent Running
	/agent_id/bb7e49c3-6b19-47ce-855a-21d8d54acdd3 postgres_exporter Running
	/agent_id/e831def4-2617-442c-9924-71f3faa3353c mysql_perfschema_agent Running

supervisorctl status
alertmanager                     RUNNING   pid 14564, uptime 0:52:21
clickhouse                       RUNNING   pid 14558, uptime 0:52:21
cron                             RUNNING   pid 14561, uptime 0:52:21
dashboard-upgrade                EXITED    Oct 14 10:15 PM
dbaas-controller                 STOPPED   Not started
grafana                          RUNNING   pid 14559, uptime 0:52:21
nginx                            RUNNING   pid 14560, uptime 0:52:21
pmm-agent                        RUNNING   pid 746, uptime 0:16:03
pmm-managed                      RUNNING   pid 14570, uptime 0:52:21
pmm-update-perform               STOPPED   Not started
postgresql                       RUNNING   pid 14557, uptime 0:52:21
prometheus                       STOPPED   Not started
qan-api2                         RUNNING   pid 14705, uptime 0:52:20
victoriametrics                  RUNNING   pid 14562, uptime 0:52:21
vmalert                          RUNNING   pid 14563, uptime 0:52:21

Everything is running just seems to think it has the wrong rds ca root cert, and for the life of me I cannot find a place to specify it or update it. Any ideas? Thanks

From pmm-managed.log:

caused by: Post "https://rds.us-west-1.amazonaws.com/": x509: certificate has expired or is not yet valid: current time 2021-10-14T21:00:31Z is after 2020-11-04T21:29:21Z```
2

Hi @sgales,

could you please check yum info ca-certificates and maybe update reinstall it?

BTW are there any certificates in /srv/nginx ?

Thanks,
Denys

1 Like
3

Thanks for the response! Here’s the data you requested.

# yum info ca-certificates
....
Installed Packages
Name        : ca-certificates
Arch        : noarch
Version     : 2021.2.50
Release     : 72.el7_9
Size        : 901 k
Repo        : installed
From repo   : updates
Summary     : The Mozilla CA root certificate bundle
URL         : http://www.mozilla.org/
License     : Public Domain
Description : This package contains the set of CA certificates chosen by the
            : Mozilla Foundation for use with the Internet PKI.

# ls -las /srv/nginx/
total 24
0 drwxr-xr-x 2 root root  115 Sep 21 10:27 .
0 drwxr-xr-x 1 root root  138 Sep 21 10:28 ..
8 -rw-r--r-- 1 root root 6016 Sep 21 10:27 ca-certs.pem
4 -rw-r--r-- 1 root root  137 Sep 21 10:27 certificate.conf
4 -rw-r--r-- 1 root root  977 Sep 21 10:27 certificate.crt
4 -rw-r--r-- 1 root root 1704 Sep 21 10:27 certificate.key
4 -rw-r--r-- 1 root root  424 Sep 21 10:27 dhparam.pem

I tried a yum reinstall ca-certificates but still getting tls cert issues:

# curl -vvvX POST "https://localhost/v1/management/RDS/Discover" -H "accept: application/json" -H "authorization: Basic <auth token from swagger>" -H "Content-Type: application/json" -d "{ \"aws_access_key\": \"<AWS_ACCESS>\", \"aws_secret_key\": \"<AWS_SECRET_KEY>\"}"
* About to connect() to localhost port 443 (#0)
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Server certificate:
* 	subject: O=Main Org.
* 	start date: Sep 21 10:27:31 2021 GMT
* 	expire date: Sep 21 10:27:31 2022 GMT
* 	common name: (nil)
* 	issuer: O=Main Org.
* NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
* Peer's certificate issuer has been marked as not trusted by the user.
* Closing connection 0
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
1 Like