We have a docker-based PMM server which was originally set up by someone from Percona. It had been running 1.7.0. I followed the instructions for upgrading a PMM server using docker (https://www.percona.com/doc/percona-monitoring-and-management/deploy/server/docker.upgrading.html#update-server-docker).
With the 1.7.0 install, the PMM server had been scraping our MySQL server linux targets every second. I could see this by visiting https:///prometheus/targets. However now with 1.10, the PMM server is only scraping every 5 minutes. It seems to scrape all of the linux targets at the same time, but it waits 5 minutes between scrapes. I’ve checked the config (https:///prometheus/config) and everything looks the same.
For doing MySQL scraping on the same servers, it is even more erratic. mysql-hr is set to 1s, mysql-mr 5s, and mysql-lr 1m. But it take minutes to scrape -hr and -mr. I’ve seen some -mr waiting up to an hour to be scraped.
It is also supposed to be scraping Grafana every second, but that is currently sitting at 49 minutes since the last scrape.
The machine is not overloaded. It is sitting at ~10% cpu and 0.5% memory used right now.
Any help in figuring this out would be appreciated!