I setup a PMM server and install pmm-clients on my MySQL servers so that I can also collect OS metrics. All pmm-clients work properly except one client, named C, which has some missing data intermittently on the MySQL Replication page. On the machine which pmm-client is not working properly, it has three MySQL targets reported on the PMM server, which are metrics-lr, metrics-mr and metrics-hr. The /prometheus/target page complains a target out of three on host C is down (metrics-mr) and is in out of order sample status. However, the other two MySQL targets [LEFT][COLOR=#252C2F] on host C is reported up.
I also visit /prometheus and draw the graph of the target which is not working properly. There is a interval without data. Maybe they are dropped because Prometheus thinks they are invalid but I don’t know why and don’t know how can I solve it. Any ideas?
Hi greenhand
Is there heavy load on the MySQL server? Often this can lead to timeouts when scraping, and because Prometheus scrapes so frequently, it will give up and discard late samples.
Also do you have NTP to ensure time is precise between PMM Server and host C?
Is there anything in /var/log/pmm-metrics* ?
Hi greenhand , I had exactly the same problem today. a small “ntpdate” on the pmm-server and sql server fixed the problem.
Hi greenhand ,
I had exactly the same problem today. a small “ntpdate” on the pmm-server and sql server fixed the problem.