# some errors are ouccured in prometheus log **URL:** \ **Category:** PMM 1.x\ **Created:** [December 12, 2017, 3:36am UTC](https://forums.percona.com/t/some-errors-are-ouccured-in-prometheus-log/6076 "2017-12-12T03:36:05Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![bckim](https://avatars.discourse-cdn.com/v4/letter/b/67e7ee/32.png) [@bckim](https://forums.percona.com/u/bckim)\ **Post date:** [December 12, 2017, 3:36am UTC](https://forums.percona.com/t/some-errors-are-ouccured-in-prometheus-log/6076/1 "2017-12-12T03:36:05Z")
hi, all I am using prometheus, but there are some error ouccured in prometheus log. – error log time=“2017-12-12T08:19:16Z” level=warning msg=“Scrape health sample discarded” error=“sample timestamp out of order” sample=up{instance=“malldb25”, job=“mysql”} =\> 1 @[1234] source=“scrape.go:587” time=“2017-12-12T08:19:16Z” level=warning msg=“Scrape duration sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“malldb25”, job=“mysql”} =\> 4.728769441 @[1234] source=“scrape.go:590” time=“2017-12-12T08:19:16Z” level=warning msg=“Scrape sample count sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“malldb25”, job=“mysql”} =\> 4.728769441 @[1234] source=“scrape.go:593” time=“2017-12-12T08:19:16Z” level=warning msg=“Scrape sample count post-relabeling sample discarded” error=“sample timestamp out of order” sample=scrape\_duration\_seconds{instance=“malldb25”, job=“mysql”} =\> 4.728769441 @[1234] source=“scrape.go:596” too many logs are storing now. please let me know why these errors are ouccured. – docker run command docker run -d -p 80:80 -e METRICS\_MEMORY=48318382080 -e METRICS\_RETENTION=2160h -e QUERIES\_RETENTION=2160h -e METRICS\_RESOLUTION=5s –volumes-from pmm-data –name pmm-server –restart always percona/pmm-server:latest – prometheus.yml # Managed by pmm-managed. DO NOT EDIT. * * * global: scrape\_interval: 5s scrape\_timeout: 5s evaluation\_interval: 5s scrape\_configs: - job\_name: prometheus scrape\_interval: 5s scrape\_timeout: 5s metrics\_path: /prometheus/metrics scheme: http static\_configs: - targets: - localhost:9090 labels: instance: prometheus - job\_name: linux scrape\_interval: 5s scrape\_timeout: 5s metrics\_path: /metrics scheme: http static\_configs: - targets: - localhost:9100 labels: instance: pmm-server consul\_sd\_configs: - server: localhost:8500 datacenter: dc1 tag\_separator: ‘,’ scheme: http services: - linux:metrics basic\_auth: username: pmm password: pmm tls\_config: insecure\_skip\_verify: true relabel\_configs: - source\_labels: [\__meta\_consul\_tags] separator: ; regex: .\*,alias_([-\w:.]+),.\* target\_label: instance replacement: $1 action: replace - source\_labels: [\_\_meta\_consul\_tags] separator: ; regex: ._,scheme\_https,._ target\_label: **scheme** replacement: https action: replace - job\_name: proxysql scrape\_interval: 5s scrape\_timeout: 5s metrics\_path: /metrics scheme: http consul\_sd\_configs: - server: localhost:8500 datacenter: dc1 tag\_separator: ‘,’ scheme: http services: - proxysql:metrics basic\_auth: username: pmm password: pmm tls\_config: insecure\_skip\_verify: true relabel\_configs: - source\_labels: [\__meta\_consul\_tags] separator: ; regex: .\*,alias_([-\w:.]+),.\* target\_label: instance replacement: $1 action: replace - source\_labels: [\_\_meta\_consul\_tags] separator: ; regex: ._,scheme\_https,._ target\_label: **scheme** replacement: https action: replace - job\_name: mongodb scrape\_interval: 5s scrape\_timeout: 5s metrics\_path: /metrics scheme: http consul\_sd\_configs: - server: localhost:8500 datacenter: dc1 tag\_separator: ‘,’ scheme: http services: - mongodb:metrics basic\_auth: username: pmm password: pmm tls\_config: insecure\_skip\_verify: true relabel\_configs: - source\_labels: [\__meta\_consul\_tags] separator: ; regex: .\*,alias_([-\w:.]+),.\* target\_label: instance replacement: $1 action: replace - source\_labels: [\_\_meta\_consul\_tags] separator: ; regex: ._,scheme\_https,._ target\_label: **scheme** replacement: https action: replace - source\_labels: [\__meta\_consul\_tags] separator: ; regex: .\*,cluster_([-\w:.]+),.\* target\_label: cluster replacement: $1 action: replace - job\_name: mysql-hr scrape\_interval: 5s scrape\_timeout: 5s metrics\_path: /metrics-hr scheme: http consul\_sd\_configs: - server: localhost:8500 datacenter: dc1 tag\_separator: ‘,’ scheme: http services: - mysql:metrics basic\_auth: username: pmm password: pmm tls\_config: insecure\_skip\_verify: true relabel\_configs: - separator: ; regex: (.\*) target\_label: job replacement: mysql action: replace - source\_labels: [\__meta\_consul\_tags] separator: ; regex: .\*,alias_([-\w:.]+),.\* target\_label: instance replacement: $1 action: replace - source\_labels: [\_\_meta\_consul\_tags] separator: ; regex: ._,scheme\_https,._ target\_label: **scheme** replacement: https action: replace - job\_name: mysql-mr scrape\_interval: 5s scrape\_timeout: 5s metrics\_path: /metrics-mr scheme: http consul\_sd\_configs: - server: localhost:8500 datacenter: dc1 tag\_separator: ‘,’ scheme: http services: - mysql:metrics basic\_auth: username: pmm password: pmm tls\_config: insecure\_skip\_verify: true relabel\_configs: - separator: ; regex: (.\*) target\_label: job replacement: mysql action: replace - source\_labels: [\__meta\_consul\_tags] separator: ; regex: .\*,alias_([-\w:.]+),.\* target\_label: instance replacement: $1 action: replace - source\_labels: [\_\_meta\_consul\_tags] separator: ; regex: ._,scheme\_https,._ target\_label: **scheme** replacement: https action: replace - job\_name: mysql-lr scrape\_interval: 1m scrape\_timeout: 5s metrics\_path: /metrics-lr scheme: http consul\_sd\_configs: - server: localhost:8500 datacenter: dc1 tag\_separator: ‘,’ scheme: http services: - mysql:metrics basic\_auth: username: pmm password: pmm tls\_config: insecure\_skip\_verify: true relabel\_configs: - separator: ; regex: (.\*) target\_label: job replacement: mysql action: replace - source\_labels: [\__meta\_consul\_tags] separator: ; regex: .\*,alias_([-\w:.]+),.\* target\_label: instance replacement: $1 action: replace - source\_labels: [\_\_meta\_consul\_tags] separator: ; regex: ._,scheme\_https,._ target\_label: **scheme** replacement: https action: replace - job\_name: rds-mysql-hr scrape\_interval: 1s scrape\_timeout: 1s metrics\_path: /metrics-hr relabel\_configs: - target\_label: job replacement: mysql - job\_name: rds-mysql-mr scrape\_interval: 5s scrape\_timeout: 1s metrics\_path: /metrics-mr relabel\_configs: - target\_label: job replacement: mysql - job\_name: rds-mysql-lr scrape\_interval: 1m scrape\_timeout: 5s metrics\_path: /metrics-lr relabel\_configs: - target\_label: job replacement: mysql – top result top - 18:05:17 up 22 days, 1:04, 1 user, load average: 0.68, 1.16, 1.25 Tasks: 444 total, 1 running, 443 sleeping, 0 stopped, 0 zombie %Cpu(s): 6.9 us, 0.2 sy, 0.0 ni, 92.9 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 65692272 total, 33868000 free, 2573836 used, 29250436 buff/cache KiB Swap: 8191996 total, 8164396 free, 27600 used. 62573256 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 29289 1000 20 0 5564340 1.605g 12892 S 220.2 2.6 94:18.48 prometheus 29520 1000 20 0 4151424 43404 6504 S 2.3 0.1 0:30.39 percona-qan-api 29299 root 20 0 2427432 30316 5048 S 1.3 0.0 0:20.82 node\_exporter 5813 root 20 0 148464 2452 1464 R 0.7 0.0 0:00.03 top 29283 polkitd 20 0 10.344g 115144 6864 S 0.7 0.2 0:17.54 mysqld thanks ---
**Author:** ![Peter](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/peter/32/2_2.png) [@Peter](https://forums.percona.com/u/Peter)\ **Post date:** [December 24, 2017, 8:41pm UTC](https://forums.percona.com/t/some-errors-are-ouccured-in-prometheus-log/6076/2 "2017-12-24T20:41:05Z")
Unless excessive you should not be concerned. “Prometheus” Dashboard has a graph to show you how many samples are being discarded every second