Not the answer you need?
Register and ask your own question!

Prometheus Log Warning.

marciomarcio ContributorCurrent User Role Contributor
Hi!

I'm using PMM 1.10 in Docker, and I see this warning in prometheus.log

time="2018-05-15T16:50:45Z" level=warning msg="Scrape health sample discarded" error="sample timestamp out of order" sample=up{instance="node1.test", job="mysql"} => 1 @[1526403040.94] source="scrape.go:587"
time="2018-05-15T16:50:45Z" level=warning msg="Scrape duration sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="node1.test", job="mysql"} => 4.780033556 @[1526403040.94] source="scrape.go:590"
time="2018-05-15T16:50:45Z" level=warning msg="Scrape sample count sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="node1.test", job="mysql"} => 4.780033556 @[1526403040.94] source="scrape.go:593"
time="2018-05-15T16:50:45Z" level=warning msg="Scrape sample count post-relabeling sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="node1.test", job="mysql"} => 4.780033556 @[1526403040.94] source="scrape.go:596"
time="2018-05-15T16:51:03Z" level=warning msg="Error on ingesting out-of-order samples" numDropped=2 source="scrape.go:534"
time="2018-05-15T16:51:03Z" level=warning msg="Scrape health sample discarded" error="sample timestamp out of order" sample=up{instance="node3.test", job="mysql"} => 1 @[1526403060.109] source="scrape.go:587"
time="2018-05-15T16:51:03Z" level=warning msg="Scrape duration sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="node3.test", job="mysql"} => 3.732717454 @[1526403060.109] source="scrape.go:590"
time="2018-05-15T16:51:03Z" level=warning msg="Scrape sample count sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="node3.test", job="mysql"} => 3.732717454 @[1526403060.109] source="scrape.go:593"
time="2018-05-15T16:51:03Z" level=warning msg="Scrape sample count post-relabeling sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="node3.test", job="mysql"} => 3.732717454 @[1526403060.109] source="scrape.go:596"

All my clients have this configuration (from pmm-admin check-network):

* System Time
NTP Server (0.pool.ntp.org) | 2018-05-15 13:49:57 -0300 -03
PMM Server | 2018-05-15 16:49:57 +0000 GMT
PMM Client | 2018-05-15 13:49:55 -0300 -03
PMM Server Time Drift | OK
PMM Client Time Drift | OK
PMM Client to PMM Server Time Drift | OK

Does this warnings have any relation about the timezone between clients and pmm-server ?

BR

Comments

  • akajlaakajla Entrant Current User Role Novice
    Hi, I am facing same issue on VirtualBox PMM instance, why no one is responding in this forum ?
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Patron
    Hi

    It's best that you open a new post, since if a post has not been answered in > 12 months it's not likely to be answered now, realistically. If there's no response it is likely that the question is answered elsewhere, or possibly not enough information is provided, or possibly the problem is too 'local' to the question asker making a response difficult to achieve without having sight of the environment.

    This post refers to 1.10 version of PMM. If you are still on that version, then you likely need to upgrade.

    To get the best chance of a response – though please understand that this is not a support queue, it is an open source community forum that helps users of our free, open source software – please follow the guidelines for asking questions that you can find here: https://www.percona.com/forums/quest...o-report-a-bug

    Thanks for your interest in our software!
  • akajlaakajla Entrant Current User Role Novice
    Hi Lorraine,

    Thanks for your reply and explanation. Sure I'll open new thread in case required in future.

    Any way this issue has bee resolved, I think it was due to timestamp conflict and it took some time to match the metrics timestamps with the current time.

    Regards,
    aKajla
  • lorraine.pocklingtonlorraine.pocklington Percona Community Manager Legacy User Role Patron
    That's good news that you fixed it.
    And also THANK YOU for saying what the issue was for you as it could help others in the future. So that's greatly appreciated!
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.