Good evening all and Merry Christmas. For those of you who celebrate, I hope everyone has a wonderful holiday with your family. Now on to my dilemma.
I have a Proxysql cluster that is serving for the Galera cluster and I would like to monitor the stats using the PMM server I have running for monitoring the Galera environment. As a test, I manually installed the pmm-client on each docker container, then configured it with the following command:
pmm-admin config --server pmmserver:443 --server-user pmm --server-password pmm --server-insecure-ssl --force --bind-address
I checked to ensure that they are active on each container:
SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD
proxysql:metrics proxysql02 10.0.7.232–>10.0.7.160:42004 DOWN YES YES
I checked to ensure that they are able to communicate:
root@proxysql02:/# pmm-admin check-network
PMM Network Status
Server Address | pmmserver:443
Client Address | 10.0.7.232 (10.0.7.160)
-
System Time
NTP Server (0.pool.ntp.org) | 2018-12-23 23:11:46 +0000 UTC
PMM Server | 2018-12-23 23:11:46 +0000 GMT
PMM Client | 2018-12-23 23:11:46 +0000 UTC
PMM Server Time Drift | OK
PMM Client Time Drift | OK
PMM Client to PMM Server Time Drift | OK -
Connection: Client → Server
SERVER SERVICE STATUS
Consul API OK
Prometheus API OK
Query Analytics API OK
Connection duration | 1.186102ms
Request duration | -665.377µs
Full round trip | 520.725µs
I also checked the local log in order to ensure that they are gathering statistics:
root@proxysql02:/# cat /var/log/pmm-proxysql-metrics-42004.log
time=“2018-12-23T13:28:40Z” level=info msg=“Starting proxysql_exporter for stats:stats@tcp(localhost:6032)/” source=“proxysql_exporter.go:65”
time=“2018-12-23T13:28:41Z” level=info msg=“HTTP Basic authentication is enabled.” source=“basic_auth.go:105”
time=“2018-12-23T13:28:41Z” level=info msg=“Starting HTTPS server for [URL]https://10.0.7.160:42004/metrics[/URL] …” source=“server.go:106”
2018/12/23 23:11:46 http: TLS handshake error from 10.0.7.160:38430: tls: first record does not look like a TLS handshake
2018/12/23 23:11:46 http: TLS handshake error from 10.0.7.160:38432: tls: first record does not look like a TLS handshake
So, client services are able to communicate with the server and I am running the clients for the Galera cluster using the same configuration. So why aren’t the stats being sent to the pmm server and therefore viewable in grafana?
Edit:
I figured this out. It was a ip routing issue.