Not the answer you need?
Register and ask your own question!

extremely high cpu cost

romberromber ContributorCurrent User Role Supporter
hi,when install and running for some days, prometheus cost CPU extremely high,it always uses nearly 100%CPU,
and when show cpu information of database server,it's very slow..

prometheus log was fulled with:
level=warning msg="Error on ingesting samples with different value but same timestamp" numDropped=1 source="scrape.go:537"
level=warning msg="Scrape duration sample discarded" error="sample with repeated timestamp but different value" sample=scrape_duration_seconds{instance="<database server ip>", job="mysql"} => 0.546582099 @[1503888448.092] source="scrape.go:590"

i'm using pmm 1.2.0 now,please kindly help me to solve this problem,
thanks in advance.

Comments

  • romberromber Contributor Current User Role Supporter
    i also uploaded screenshot of iotop
  • romberromber Contributor Current User Role Supporter
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi romber

    Your server seems adequately provisioned for hardware (16 cores, 32GB RAM), but a couple observations:
    1. Do you have all servers in sync with time? Check with pmm-admin check-network to verify that your clocks are synchronised
    2. Did you do the docker-based installation? You have quite a bit of unused RAM that could be allocated to Prometheus - see https://www.percona.com/doc/percona-monitoring-and-management/faq.html#how-to-control-memory-consumption-for-pmm
    If you continue to have issues with high CPU, please send us the output of our pmm-info.sh debugging tool, which you can read about here:

    https://www.percona.com/blog/2016/11/01/percona-monitoring-and-management-pmm-information-script/

    The essential steps are:
    wget https://raw.githubusercontent.com/Percona-QA/percona-qa/master/pmm-info.sh
    vi pmm-info.sh
    sh ./pmm-info.sh
    
  • romberromber Contributor Current User Role Supporter
    hi Michael Coburn ,
    thanks for you answer.
    1. Do you have all servers in sync with time? Check with pmm-admin check-network to verify that your clocks are synchronised
    yes,i'll upload the output of pmm-admin check-network command.i run the command on one of mysql server.
    2. Did you do the docker-based installation?
    yes,i use docker-based installation,and i already set 8GB RAM to pmm
    as this server is also used for recovery test,although this happened rarely,i have to left some RAM to recovery mysql database.
    is 8GB of RAM not enough for pmm-server? i'm monitoring about 20~30 mysql servers and the number is still growing...
    how much RAM does it need ?
    i also find out that I/O is high used by prometheus,is it normal?

    3. about pmm-info.sh
    i'll upload the output of pmm-info.sh script.i run it on pmm-server
  • romberromber Contributor Current User Role Supporter
    is anyone still keeping an eye on it?
  • Michael CoburnMichael Coburn Principal Architect, Percona Percona Staff Role
    Hi romber , my apologies for the delay.

    I'd like to ask that you upgrade to latest PMM Server (1.2.2), because in 1.2.1 we introduced a memory fix that may be related to your high CPU utilisation:
    https://www.percona.com/doc/percona-monitoring-and-management/release-notes/1.2.1.html
    PMM-1232: Update the default value of the METRICS_MEMORY configuration setting

    Previous versions of PMM Server used a different value for the METRICS_MEMORY configuration setting which allowed Prometheus to use up to 768MB of memory.

    PMM Server 1.2.0 used the storage.local.target-heap-size setting, its default value being 256MB. Unintentionally, this value reduced the amount of memory that Prometheus could use. As a result, the performance of Prometheus was affected.

    To improve the performance of Prometheus, the default setting of storage.local.target-heap-size has been set to 768 MB.
    Can you let us know if this diminishes the CPU utlisation? We await your feedback!
  • romberromber Contributor Current User Role Supporter
    hi Michael Coburn , thanks for your reply.
    i've upgraded to 1.2.2 for both of pmm server and client,and increased the METRICS_MEMORY to 16GB, decreased the METRICS_RETENTION to 360 hours.
    but high cpu cost on pmm server still exists...
    here is what i run:
    docker run -d \
    -p 80:80 \
    --volumes-from pmm-data \
    --name pmm-server \
    --restart always \
    -e SERVER_USER=pmm \
    -e SERVER_PASSWORD=xxx \
    -e METRICS_MEMORY=17179869184 \
    -e METRICS_RETENTION=360h \
    percona/pmm-server:1.2.2
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.