PMM Client endpoints return an empty response. Log files show tls handshake error

[jholloway]

We have two Percona cluster node hosts (in the same cluster - which also features a garbd node) where the PMM client is not producing the right output at its metrics endpoints. Consulting the relevant pmm*metrics.log files reveals:

<this_host_ip>: tls: first record does not look like a TLS handshake.

On these hosts issue the command,

$ curl <this_host_ip>:42000

And you get an empty response. netstat -l reveals that the port is listening but it’s not returning anything.

There are other PMM client hosts which do not suffer from the same issue. On these you can

$ curl <this_host_ip>:42000

and get html relating to the linux:metrics endpoint.

The PMM server isn’t set up to use SSL/TLS and other hosts (including some different clusters’ nodes) do successfully set up an endpoint and communicate with the server without there being any setup relating to TLS. I don’t really know what to do next. What could possibly be different about the failing hosts? The problem appears to be with the client itself not producing the endpoint correctly rather than communication between hosts.

Also pmm-admin check-network reports the Client <-- Server communication as ‘DOWN’ but Client --> Server as ‘OK’.

Regards,

jholloway

[weber]

The command to check should be:
curl --insecure https://<this_host_ip>:42000

If you created services using 1.0.7 client and server is also the same version.

TLS handshake is not a problem and can be ignored.
The above curl command should work fine from the inside pmm-server contaner:
docker exec -ti pmm-server curl --insecure https://<this_host_ip>:42000

Otherwise, this is a firewall problem.

[jholloway]

Thanks for your swift response. In this case the issue was that the client I’d installed (1.07) was not the same version as the server and other clients (1.05). I updated these latter things and it worked perfectly.

[weber]

Yes, with 1.0.7 release, we require to match client and server version.
Most likely, we will add a check to check-network output to verify versions, so they are the same to prevent potential issues.