We have two Percona cluster node hosts (in the same cluster - which also features a garbd node) where the PMM client is not producing the right output at its metrics endpoints. Consulting the relevant pmm*metrics.log files reveals:
<this_host_ip>: tls: first record does not look like a TLS handshake.
On these hosts issue the command,
$ curl <this_host_ip>:42000
And you get an empty response. netstat -l reveals that the port is listening but it’s not returning anything.
There are other PMM client hosts which do not suffer from the same issue. On these you can
$ curl <this_host_ip>:42000
and get html relating to the linux:metrics endpoint.
The PMM server isn’t set up to use SSL/TLS and other hosts (including some different clusters’ nodes) do successfully set up an endpoint and communicate with the server without there being any setup relating to TLS. I don’t really know what to do next. What could possibly be different about the failing hosts? The problem appears to be with the client itself not producing the endpoint correctly rather than communication between hosts.
Also pmm-admin check-network reports the Client ← Server communication as ‘DOWN’ but Client → Server as ‘OK’.
Regards,
jholloway