PMM Linux targets flapping state

Percona Monitoring and Management (PMM) PMM 1.x
1

Hello,

Have a setup of PMM server (v1.7.0) installed as docker and monitoring linux and mysql metrics of 3 PXC nodes using PMM client of same version. This setup was working fine without any issue till last Thusday. Since then, I am observing a flapping in the state of linux metrics from 2 nodes. PMM list shows the service as up always, but when we query pmm check-network, we can see the Client ← Server status for linux-metrics as down and after couple of seconds or minutes then it is back as OK for couple of minutes before flapping again. During the same time when check on ‘prometheus/targets’, the linux metrics is down due to error “context deadline exceeded”.

The issue is with only 2 nodes out of 3. All the nodes and PMM server are in same N/W and DC. It will be great if I get some guidance on how to troubleshoot the issue.

With Regards
Raghupradeep

2

The issue is resolved. There was a NFS mount on the server and it was in hung state and it was causing the issue. When I removed the NFS mount, things started to work fine.

3

Hello raghupradeep first of all, sorry this one slipped the net but also I wanted to say thank you for updating the post since it could well help someone in the future. Appreciated!