Hey everyone! I’m very impressed with the progress on PMM so far. However, I am running into an issue that just started recently around the time of upgrading to 1.10. Query analytics has no data. Looking at the logs, I see:
WARNING data-sender Waiting for API to ack qan_XXXXXXXXXXXXXX: read tcp XXX.XXX.XXX.XXX:37794->XXX.XXX.XXX.XXX:443: i/o timeout
Every single QAN times out…
But, pmm-admin check-network shows everything is ok from a connectivity standpoint. Linux and MySQL metrics are working correctly, and the server doesn’t appear to be under any kind of strain based on the Prometheus dashboard. I have upgraded to 1.11.0 and the problem remains (I was hoping it was a bug). Any ideas on how to troubleshoot slow response from the QAN API?