Not the answer you need?
Register and ask your own question!

Prometheus of PMM complains out of order sample

I setup a PMM server and install pmm-clients on my MySQL servers so that I can also collect OS metrics. All pmm-clients work properly except one client, named C, which has some missing data intermittently on the MySQL Replication page. On the machine which pmm-client is not working properly, it has three MySQL targets reported on the PMM server, which are metrics-lr, metrics-mr and metrics-hr. The /prometheus/target page complains a target out of three on host C is down (metrics-mr) and is in out of order sample status. However, the other two MySQL targets
on host C is reported up.
I also visit /prometheus and draw the graph of the target which is not working properly. There is a interval without data. Maybe they are dropped because Prometheus thinks they are invalid but I don't know why and don't know how can I solve it. Any ideas?

Comments

  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi greenhand

    Is there heavy load on the MySQL server? Often this can lead to timeouts when scraping, and because Prometheus scrapes so frequently, it will give up and discard late samples.

    Also do you have NTP to ensure time is precise between PMM Server and host C?

    Is there anything in /var/log/pmm-metrics* ?
  • cece46cece46 Entrant Current User Role Beginner
    Hi greenhand , I had exactly the same problem today. a small "ntpdate" on the pmm-server and sql server fixed the problem.
  • cece46cece46 Entrant Current User Role Beginner
    Hi greenhand ,
    I had exactly the same problem today. a small "ntpdate" on the pmm-server and sql server fixed the problem.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.