# Prometheus Log Warning.

**URL:** <https://forums.percona.com/t/prometheus-log-warning/6356>\
**Category:** PMM 1.x\
**Created:** [May 15, 2018, 11:28am UTC](https://forums.percona.com/t/prometheus-log-warning/6356 "2018-05-15T11:28:42Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![marcio](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@marcio](https://forums.percona.com/u/marcio)\
**Post date:** [May 15, 2018, 11:28am UTC](https://forums.percona.com/t/prometheus-log-warning/6356/1 "2018-05-15T11:28:42Z")

</div>

Hi!

I’m using PMM 1.10 in Docker, and I see this warning in prometheus.log

time=“2018-05-15T16:50:45Z” level=warning msg=“Scrape health sample discarded” error=“sample timestamp out of order” sample=up{instance=“node1.test”, job=“mysql”} =\> 1 @[1526403040.94] source=“scrape.go:587”  
time=“2018-05-15T16:50:45Z” level=warning msg=“Scrape duration sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“node1.test”, job=“mysql”} =\> 4.780033556 @[1526403040.94] source=“scrape.go:590”  
time=“2018-05-15T16:50:45Z” level=warning msg=“Scrape sample count sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“node1.test”, job=“mysql”} =\> 4.780033556 @[1526403040.94] source=“scrape.go:593”  
time=“2018-05-15T16:50:45Z” level=warning msg=“Scrape sample count post-relabeling sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“node1.test”, job=“mysql”} =\> 4.780033556 @[1526403040.94] source=“scrape.go:596”  
time=“2018-05-15T16:51:03Z” level=warning msg=“Error on ingesting out-of-order samples” numDropped=2 source=“scrape.go:534”  
time=“2018-05-15T16:51:03Z” level=warning msg=“Scrape health sample discarded” error=“sample timestamp out of order” sample=up{instance=“node3.test”, job=“mysql”} =\> 1 @[1526403060.109] source=“scrape.go:587”  
time=“2018-05-15T16:51:03Z” level=warning msg=“Scrape duration sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“node3.test”, job=“mysql”} =\> 3.732717454 @[1526403060.109] source=“scrape.go:590”  
time=“2018-05-15T16:51:03Z” level=warning msg=“Scrape sample count sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“node3.test”, job=“mysql”} =\> 3.732717454 @[1526403060.109] source=“scrape.go:593”  
time=“2018-05-15T16:51:03Z” level=warning msg=“Scrape sample count post-relabeling sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“node3.test”, job=“mysql”} =\> 3.732717454 @[1526403060.109] source=“scrape.go:596”

All my clients have this configuration (from pmm-admin check-network):

- System Time  
NTP Server ([0.pool.ntp.org](http://0.pool.ntp.org)) | 2018-05-15 13:49:57 -0300 -03  
PMM Server | 2018-05-15 16:49:57 +0000 GMT  
PMM Client | 2018-05-15 13:49:55 -0300 -03  
PMM Server Time Drift | OK  
PMM Client Time Drift | OK  
PMM Client to PMM Server Time Drift | OK

Does this warnings have any relation about the timezone between clients and pmm-server ?

BR

---

<div class="post-metadata">

**Author:** ![akajla](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@akajla](https://forums.percona.com/u/akajla)\
**Post date:** [March 7, 2019, 7:13am UTC](https://forums.percona.com/t/prometheus-log-warning/6356/2 "2019-03-07T07:13:38Z")

</div>

Hi, I am facing same issue on VirtualBox PMM instance, why no one is responding in this forum ?

---

<div class="post-metadata">

**Author:** ![lorraine.pocklington](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/lorraine.pocklington/32/37_2.png) [@lorraine.pocklington](https://forums.percona.com/u/lorraine.pocklington)\
**Post date:** [March 7, 2019, 7:47am UTC](https://forums.percona.com/t/prometheus-log-warning/6356/3 "2019-03-07T07:47:02Z")

</div>

Hi

It’s best that you open a new post, since if a post has not been answered in \> 12 months it’s not likely to be answered now, realistically. If there’s no response it is likely that the question is answered elsewhere, or possibly not enough information is provided, or possibly the problem is too ‘local’ to the question asker making a response difficult to achieve without having sight of the environment.

This post refers to 1.10 version of PMM. If you are still on that version, then you likely need to upgrade.

To get the best chance of a response – though please understand that this is not a support queue, it is an open source community forum that helps users of our free, open source software – please follow the guidelines for asking questions that you can find here: [URL=“[https://www.percona.com/forums/questions-discussions/percona-monitoring-and-management/50690-pmm-troubleshooting-and-how-to-report-a-bug](https://www.percona.com/forums/questions-discussions/percona-monitoring-and-management/50690-pmm-troubleshooting-and-how-to-report-a-bug)”][https://www.percona.com/forums/quest...o-report-a-bug[/URL]](https://www.percona.com/forums/quest...o-report-a-bug%5B/URL%5D)

Thanks for your interest in our software!

---

<div class="post-metadata">

**Author:** ![akajla](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@akajla](https://forums.percona.com/u/akajla)\
**Post date:** [March 7, 2019, 10:48pm UTC](https://forums.percona.com/t/prometheus-log-warning/6356/4 "2019-03-07T22:48:09Z")

</div>

Hi L[orraine](https://www.percona.com/forums/member/41223-lorraine-pocklington),

Thanks for your reply and explanation. Sure I’ll open new thread in case required in future.

Any way this issue has bee resolved, I think it was due to timestamp conflict and it took some time to match the metrics timestamps with the current time.

Regards,  
aKajla

---

<div class="post-metadata">

**Author:** ![lorraine.pocklington](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/lorraine.pocklington/32/37_2.png) [@lorraine.pocklington](https://forums.percona.com/u/lorraine.pocklington)\
**Post date:** [March 15, 2019, 5:17am UTC](https://forums.percona.com/t/prometheus-log-warning/6356/5 "2019-03-15T05:17:06Z")

</div>

That’s good news that you fixed it.  
And also THANK YOU for saying what the issue was for you as it could help others in the future. So that’s greatly appreciated!
