seeing gaps during data collection in PMM dashboard

Hello,
We’ve installed PMM and it looks great! But the thing is that we see some gaps in between the charts as showing in the attachment.
From client side pmm-mysql-queries.log, we see these warnings:

2017/10/11 22:37:40.590505 WARNING data-sender Waiting for API to ack qan_1507761420012724193: read tcp x.x.x.x:50024->x.x.x.x:80: i/o timeout
2017/10/11 22:37:49.293295 WARNING data-sender Waiting for API to ack qan_1507761420012724193: read tcp x.x.x.x:50026->x.x.x.x:80: i/o timeout
2017/10/11 22:47:07.464413 WARNING data-sender Waiting for API to ack qan_1507761960013420750: read tcp x.x.x.x:50046->x.x.x.x:80: i/o timeout
2017/10/11 22:54:28.530116 WARNING data-sender Waiting for API to ack qan_1507762440012218450: read tcp x.x.x.x:50062->x.x.x.x:80: i/o timeout
2017/10/11 22:54:37.078760 WARNING data-sender Waiting for API to ack qan_1507762440012218450: read tcp x.x.x.x:50064->x.x.x.x:80: i/o timeout

What could cause this? Is there any parameter we should tune?

Thank you

Please share the output of pmm-admin check-network executed from the client side showing gaps in PMM Server.

Can you also share the full command you’ve used to create the pmm-server container?

Hi, sorry, I was out of town.

Here is the command output:

pmm-admin check-network
PMM Network Status

Server | x.x.x.x
Client | x.x.x.x

  • Client --> Server

SERVER SERVICE CONNECTIVITY


Consul API

We see a similar problem. Increasing CPUs assigned to the VM has reduced but not eliminated the gaps. When I run the network-check, the status for client -> servers is always OK but the status for server -> client is sometimes OK and sometimes DOWN. Network roundtrip is ~1ms so I doubt there is any actual network issue. Is the DOWN status a sign that the server is too busy to connect to the client (essentially a timeout on server side rather than on client)?