Prometheus Log Warning.

Percona Monitoring and Management (PMM) PMM 1.x
1

Hi!

I’m using PMM 1.10 in Docker, and I see this warning in prometheus.log

time=“2018-05-15T16:50:45Z” level=warning msg=“Scrape health sample discarded” error=“sample timestamp out of order” sample=up{instance=“node1.test”, job=“mysql”} => 1 @[1526403040.94] source=“scrape.go:587”
time=“2018-05-15T16:50:45Z” level=warning msg=“Scrape duration sample discarded” error=“sample timestamp out of order” sample=scrape_duration_seconds{instance=“node1.test”, job=“mysql”} => 4.780033556 @[1526403040.94] source=“scrape.go:590”
time=“2018-05-15T16:50:45Z” level=warning msg=“Scrape sample count sample discarded” error=“sample timestamp out of order” sample=scrape_duration_seconds{instance=“node1.test”, job=“mysql”} => 4.780033556 @[1526403040.94] source=“scrape.go:593”
time=“2018-05-15T16:50:45Z” level=warning msg=“Scrape sample count post-relabeling sample discarded” error=“sample timestamp out of order” sample=scrape_duration_seconds{instance=“node1.test”, job=“mysql”} => 4.780033556 @[1526403040.94] source=“scrape.go:596”
time=“2018-05-15T16:51:03Z” level=warning msg=“Error on ingesting out-of-order samples” numDropped=2 source=“scrape.go:534”
time=“2018-05-15T16:51:03Z” level=warning msg=“Scrape health sample discarded” error=“sample timestamp out of order” sample=up{instance=“node3.test”, job=“mysql”} => 1 @[1526403060.109] source=“scrape.go:587”
time=“2018-05-15T16:51:03Z” level=warning msg=“Scrape duration sample discarded” error=“sample timestamp out of order” sample=scrape_duration_seconds{instance=“node3.test”, job=“mysql”} => 3.732717454 @[1526403060.109] source=“scrape.go:590”
time=“2018-05-15T16:51:03Z” level=warning msg=“Scrape sample count sample discarded” error=“sample timestamp out of order” sample=scrape_duration_seconds{instance=“node3.test”, job=“mysql”} => 3.732717454 @[1526403060.109] source=“scrape.go:593”
time=“2018-05-15T16:51:03Z” level=warning msg=“Scrape sample count post-relabeling sample discarded” error=“sample timestamp out of order” sample=scrape_duration_seconds{instance=“node3.test”, job=“mysql”} => 3.732717454 @[1526403060.109] source=“scrape.go:596”

All my clients have this configuration (from pmm-admin check-network):

  • System Time
    NTP Server (0.pool.ntp.org) | 2018-05-15 13:49:57 -0300 -03
    PMM Server | 2018-05-15 16:49:57 +0000 GMT
    PMM Client | 2018-05-15 13:49:55 -0300 -03
    PMM Server Time Drift | OK
    PMM Client Time Drift | OK
    PMM Client to PMM Server Time Drift | OK

Does this warnings have any relation about the timezone between clients and pmm-server ?

BR

2

Hi, I am facing same issue on VirtualBox PMM instance, why no one is responding in this forum ?

3

Hi

It’s best that you open a new post, since if a post has not been answered in > 12 months it’s not likely to be answered now, realistically. If there’s no response it is likely that the question is answered elsewhere, or possibly not enough information is provided, or possibly the problem is too ‘local’ to the question asker making a response difficult to achieve without having sight of the environment.

This post refers to 1.10 version of PMM. If you are still on that version, then you likely need to upgrade.

To get the best chance of a response – though please understand that this is not a support queue, it is an open source community forum that helps users of our free, open source software – please follow the guidelines for asking questions that you can find here: [URL=“https://www.percona.com/forums/questions-discussions/percona-monitoring-and-management/50690-pmm-troubleshooting-and-how-to-report-a-bug”]https://www.percona.com/forums/quest...o-report-a-bug[/URL]

Thanks for your interest in our software!

4

Hi Lorraine,

Thanks for your reply and explanation. Sure I’ll open new thread in case required in future.

Any way this issue has bee resolved, I think it was due to timestamp conflict and it took some time to match the metrics timestamps with the current time.

Regards,
aKajla

5

That’s good news that you fixed it.
And also THANK YOU for saying what the issue was for you as it could help others in the future. So that’s greatly appreciated!