Have a setup of PMM server (v1.7.0) installed as docker and monitoring linux and mysql metrics of 3 PXC nodes using PMM client of same version. This setup was working fine without any issue till last Thusday. Since then, I am observing a flapping in the state of linux metrics from 2 nodes. PMM list shows the service as up always, but when we query pmm check-network, we can see the Client <-- Server status for linux-metrics as down and after couple of seconds or minutes then it is back as OK for couple of minutes before flapping again. During the same time when check on ‘prometheus/targets’, the linux metrics is down due to error “context deadline exceeded”.
The issue is with only 2 nodes out of 3. All the nodes and PMM server are in same N/W and DC. It will be great if I get some guidance on how to troubleshoot the issue.