# prometheus high cpu **URL:** \ **Category:** PMM 1.x\ **Created:** [June 8, 2017, 4:30am UTC](https://forums.percona.com/t/prometheus-high-cpu/5666 "2017-06-08T04:30:53Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![liuqian](https://avatars.discourse-cdn.com/v4/letter/l/3da27b/32.png) [@liuqian](https://forums.percona.com/u/liuqian)\ **Post date:** [June 8, 2017, 4:30am UTC](https://forums.percona.com/t/prometheus-high-cpu/5666/1 "2017-06-08T04:30:53Z")
The following host load conditions ![5J~]`@627170GAUL@KZHGBM.png|690x383](upload://1uxBEM23ATwcB1hDlEXC0jx1E9l.png) ![J2]LQ%]T7D42Z5`P(FH2[31.png|472x500](https://forums.percona.com/uploads/short-url/aiL0dvL0E52SGsny1uR082kbHUj.png) ---
**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\ **Post date:** [June 8, 2017, 7:37am UTC](https://forums.percona.com/t/prometheus-high-cpu/5666/2 "2017-06-08T07:37:32Z")
what problems do you have (except high cpu numbers)? do you see any issues in prometheus log? you can open it with the following command help ```auto docker exec -it pmm-server less /var/log/prometheus.log ``` ---
**Author:** ![cloud-admin](https://avatars.discourse-cdn.com/v4/letter/c/b38774/32.png) [@cloud-admin](https://forums.percona.com/u/cloud-admin)\ **Post date:** [July 20, 2017, 2:04pm UTC](https://forums.percona.com/t/prometheus-high-cpu/5666/3 "2017-07-20T14:04:33Z")
We are seeing the same issue since moving to 1.2.0. We removed pmm-data volumes and started fresh two days ago (7-18 roughly 12pm) . Prometheous CPU usage jumped, disk io and load climbing steadily since install time. prometheus.log is filled with the following: time=“2017-07-20T18:40:10Z” level=warning msg=“Scrape duration sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“db2”, job=“mysql”} =\> 0.882654579 @[1500576009.988] source=“scrape.go:590” time=“2017-07-20T18:40:10Z” level=warning msg=“Scrape sample count sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“db2”, job=“mysql”} =\> 0.882654579 @[1500576009.988] source=“scrape.go:593” time=“2017-07-20T18:40:10Z” level=warning msg=“Scrape sample count post-relabeling sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“db2”, job=“mysql”} =\> 0.882654579 @[1500576009.988] source=“scrape.go:596” time=“2017-07-20T18:40:12Z” level=warning msg=“Storage has entered rushed mode.” chunksToPersist=1032 memoryChunks=37175 source=“storage.go:1867” urgencyScore=0.803 time=“2017-07-20T18:40:12Z” level=info msg=“Completed initial partial maintenance sweep through 763 in-memory fingerprints in 25.691535331s.” source=“storage.go:1398” time=“2017-07-20T18:40:14Z” level=info msg=“Storage has left rushed mode.” chunksToPersist=1002 memoryChunks=37242 source=“storage.go:1857” urgencyScore=0.569 Time is synced between hosts and within docker, except docker is on UTC. nms1:~ : date Thu Jul 20 12:32:37 PDT 2017 nms1:~ : ssh db1 date Thu Jul 20 12:32:37 PDT 2017 nms1:~ : ssh db2 date Thu Jul 20 12:32:37 PDT 2017 nms1:~ : sudo docker exec -it pmm-server date Thu Jul 20 19:32:38 UTC 2017 ![photoid=49107](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/b/bc1c2847ad21d829e1def5b9e4140454ea1a9ac4.jpeg) ![photoid=49108](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/e/e3fe600ca7d6aef84ad95a6e4754dd45c7a60f7e.png) ---
**Author:** ![cloud-admin](https://avatars.discourse-cdn.com/v4/letter/c/b38774/32.png) [@cloud-admin](https://forums.percona.com/u/cloud-admin)\ **Post date:** [July 21, 2017, 10:45am UTC](https://forums.percona.com/t/prometheus-high-cpu/5666/4 "2017-07-21T10:45:50Z")
Issue is resolved. Removed all client services (pmm-admin rm --all), removed pmm-server and pmm-data, then added pmm-server with -e METRICS\_RESOLUTION=5s -e METRICS\_MEMORY=786432 options, then added back all clients same as before. Adding those two options seems to have done the trick. Load and disk io on Prometheus server is fine and steady, and no more storage rushed mode. Still seeing “sample timestamp out of order” and “sample discarded” messages though. ![photoid=49148](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/1/10d7e8268fb1cf776a1538ad49e0e3c4c3b98966.jpeg) ---
**Author:** ![Mykola](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/mykola/32/30_2.png) [@Mykola](https://forums.percona.com/u/Mykola)\ **Post date:** [July 24, 2017, 2:25am UTC](https://forums.percona.com/t/prometheus-high-cpu/5666/5 "2017-07-24T02:25:59Z")
[cloud-admin@sbwell.com](https://percona.vanillacommunities.com/profile/x/x/36135), thank you for your feedback. we are going to increase METRICS\_MEMORY default value soon. ---
**Author:** ![bckim](https://avatars.discourse-cdn.com/v4/letter/b/67e7ee/32.png) [@bckim](https://forums.percona.com/u/bckim)\ **Post date:** [December 12, 2017, 2:33am UTC](https://forums.percona.com/t/prometheus-high-cpu/5666/6 "2017-12-12T02:33:45Z")
Is there any update about “sample timestamp out of order” log? thanks ---
**Author:** ![Registering\_Sucks](https://avatars.discourse-cdn.com/v4/letter/r/b782af/32.png) [@Registering\_Sucks](https://forums.percona.com/u/Registering_Sucks)\ **Post date:** [April 19, 2018, 10:16am UTC](https://forums.percona.com/t/prometheus-high-cpu/5666/7 "2018-04-19T10:16:18Z")
We are seeing excessive CPU use from prometheus and there are gaps in the data. Where is the log hidden in the current version (1.9.1)? There is no prometheus.log in /var/log or anywhere else on the filesystem.