seeing gaps during data collection in PMM dashboard

Percona Monitoring and Management (PMM) PMM 1.x
1

Hello,
We’ve installed PMM and it looks great! But the thing is that we see some gaps in between the charts as showing in the attachment.
From client side pmm-mysql-queries.log, we see these warnings:

2017/10/11 22:37:40.590505 WARNING data-sender Waiting for API to ack qan_1507761420012724193: read tcp x.x.x.x:50024->x.x.x.x:80: i/o timeout
2017/10/11 22:37:49.293295 WARNING data-sender Waiting for API to ack qan_1507761420012724193: read tcp x.x.x.x:50026->x.x.x.x:80: i/o timeout
2017/10/11 22:47:07.464413 WARNING data-sender Waiting for API to ack qan_1507761960013420750: read tcp x.x.x.x:50046->x.x.x.x:80: i/o timeout
2017/10/11 22:54:28.530116 WARNING data-sender Waiting for API to ack qan_1507762440012218450: read tcp x.x.x.x:50062->x.x.x.x:80: i/o timeout
2017/10/11 22:54:37.078760 WARNING data-sender Waiting for API to ack qan_1507762440012218450: read tcp x.x.x.x:50064->x.x.x.x:80: i/o timeout

What could cause this? Is there any parameter we should tune?

Thank you

photoid=49833
photoid=498331633×672 101 KB

2

Please share the output of pmm-admin check-network executed from the client side showing gaps in PMM Server.

Can you also share the full command you’ve used to create the pmm-server container?

3

Hi, sorry, I was out of town.

Here is the command output:

pmm-admin check-network
PMM Network Status

Server | x.x.x.x
Client | x.x.x.x

  • Client → Server

SERVER SERVICE CONNECTIVITY

Consul API

4

We see a similar problem. Increasing CPUs assigned to the VM has reduced but not eliminated the gaps. When I run the network-check, the status for client → servers is always OK but the status for server → client is sometimes OK and sometimes DOWN. Network roundtrip is ~1ms so I doubt there is any actual network issue. Is the DOWN status a sign that the server is too busy to connect to the client (essentially a timeout on server side rather than on client)?