extremely high cpu cost

Percona Monitoring and Management (PMM) PMM 1.x
#1

hi,when install and running for some days, prometheus cost CPU extremely high,it always uses nearly 100%CPU,
and when show cpu information of database server,it’s very slow…

prometheus log was fulled with:
level=warning msg=“Error on ingesting samples with different value but same timestamp” numDropped=1 source=“scrape.go:537”
level=warning msg=“Scrape duration sample discarded” error=“sample with repeated timestamp but different value” sample=scrape_duration_seconds{instance="", job=“mysql”} => 0.546582099 @[1503888448.092] source=“scrape.go:590”

i’m using pmm 1.2.0 now,please kindly help me to solve this problem,
thanks in advance.

1902×723 69.3 KB

1679×317 17 KB

#2

i also uploaded screenshot of iotop

1684×333 7.54 KB

#3

Hi romber

Your server seems adequately provisioned for hardware (16 cores, 32GB RAM), but a couple observations: [LIST=1]
[]Do you have all servers in sync with time? Check with pmm-admin check-network to verify that your clocks are synchronised
[]Did you do the docker-based installation? You have quite a bit of unused RAM that could be allocated to Prometheus - see https://www.percona.com/doc/percona-monitoring-and-management/faq.html#how-to-control-memory-consumption-for-pmm
[/LIST] If you continue to have issues with high CPU, please send us the output of our pmm-info.sh debugging tool, which you can read about here:

https://www.percona.com/blog/2016/11/01/percona-monitoring-and-management-pmm-information-script/

The essential steps are:

wget https://raw.githubusercontent.com/Percona-QA/percona-qa/master/pmm-info.sh
vi pmm-info.sh
sh ./pmm-info.sh
#4

hi Michael Coburn ,
thanks for you answer.

  1. Do you have all servers in sync with time? Check with pmm-admin check-network to verify that your clocks are synchronised
    yes,i’ll upload the output of pmm-admin check-network command.i run the command on one of mysql server.

  2. Did you do the docker-based installation?
    yes,i use docker-based installation,and i already set 8GB RAM to pmm
    as this server is also used for recovery test,although this happened rarely,i have to left some RAM to recovery mysql database.
    is 8GB of RAM not enough for pmm-server? i’m monitoring about 20~30 mysql servers and the number is still growing…
    how much RAM does it need ?
    i also find out that I/O is high used by prometheus,is it normal?

  3. about pmm-info.sh
    i’ll upload the output of pmm-info.sh script.i run it on pmm-server

#5

is anyone still keeping an eye on it?

#6

Hi romber , my apologies for the delay.

I’d like to ask that you upgrade to latest PMM Server (1.2.2), because in 1.2.1 we introduced a memory fix that may be related to your high CPU utilisation:
https://www.percona.com/doc/percona-monitoring-and-management/release-notes/1.2.1.html [INDENT]PMM-1232: Update the default value of the METRICS_MEMORY configuration setting

Previous versions of PMM Server used a different value for the METRICS_MEMORY configuration setting which allowed Prometheus to use up to 768MB of memory.

PMM Server 1.2.0 used the storage.local.target-heap-size setting, its default value being 256MB. Unintentionally, this value reduced the amount of memory that Prometheus could use. As a result, the performance of Prometheus was affected.

To improve the performance of Prometheus, the default setting of storage.local.target-heap-size has been set to 768 MB.[/INDENT]
Can you let us know if this diminishes the CPU utlisation? We await your feedback!

#7

hi Michael Coburn , thanks for your reply.
i’ve upgraded to 1.2.2 for both of pmm server and client,and increased the METRICS_MEMORY to 16GB, decreased the METRICS_RETENTION to 360 hours.
but high cpu cost on pmm server still exists…
here is what i run:
docker run -d
-p 80:80
–volumes-from pmm-data
–name pmm-server
–restart always
-e SERVER_USER=pmm
-e SERVER_PASSWORD=xxx
-e METRICS_MEMORY=17179869184
-e METRICS_RETENTION=360h
percona/pmm-server:1.2.2

640×506 6.64 KB

1456×809 33.5 KB

845×656 18.9 KB