Not the answer you need?
Register and ask your own question!

pmm client side mysqld_exporter consuming a lot of memory

Hi,
I finally got pmm to work after opening up port 42001 42002 on my client (where the mysql db is located)

While looking at the new metrics coming in, i was surprised to see memory available down to 4%. (usually around 40-50% free)

Running top on the database server, i see this:
3066 root      20   0 2825676   2.5g   1240 S   1.3 68.0   4:41.44 mysqld_exporter

thats 68% memory held by mysqld_exporter and it stays there.

Tasks:  84 total,   1 running,  83 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.9 us,  5.5 sy,  0.0 ni, 65.5 id, 26.7 wa,  0.0 hi,  0.2 si,  0.2 st
KiB Mem :  3880184 total,   111948 free,  3665160 used,   103076 buff/cache
KiB Swap:  4194300 total,  3543804 free,   650496 used.    40916 avail Mem

Noted that the server is on the weak side of 4GB memory. but the server is a new startup and rarely goes beyond 50% of memory usage yet.

Any advice on this? Is there a setting i could set to limit memory usage by the exporter or is my server too weak to run pmm client? 

Which actually takes less memory footprint, using perfschema or slowlog? i commented slow query log in my.cnf . does it matter which is used in regards to memory taken by mysqld exporter?
Can i reduce the sample rate / upload rate / or something to reduce memory footprint to around 10% of system memory with the side effect of not getting the full picture in the pmm dashboard?

Thank you for the advice. Below are the settings for the database systems. (this is my first system that i try to manage, so i'm still learning on server management)

ProjectName: pmm-admin
Version: 2.8.0
PMMVersion: 2.8.0
Timestamp: 2020-06-25 13:53:24 (UTC)
FullCommit: 7d9e26cac82f694f1fef965fad9be862dbd7926c

I ran the following for setup:
sudo pmm-admin add mysql --query-source=perfschema --user=user --pass=pass --host=localhost --port=3306

pmm-admin list shows the following:
pmm_agent                   Connected  /agent_id/41cc2aa9-cbb5-42f6-a7db-9f15aa7ad045
node_exporter               Running    /agent_id/7b8c1697-932f-47d6-a81e-2bf82bcc3306
mysqld_exporter             Running    /agent_id/0de9dcb3-6810-4695-bc55-021429dd3632  /service_id/2f69a203-a7af-4af8-9eee-23228592f02d
mysql_perfschema_agent      Running    /agent_id/9667bf76-2501-4505-82a6-36f45c8a3cdf  /service_id/2f69a203-a7af-4af8-9eee-23228592f02d

my.cnf (mysql 8.0):
innodb_monitor_enable=all
performance_schema=ON
# slow query log commented
#log_output=file
#slow_query_log=ON
#long_query_time=0
#log_slow_admin_statements=ON
#log_slow_slave_statements=ON

[innodb]
innodb_buffer_pool_size = 3200M # (adjust value here, 50%-70% of total RAM)
innodb_log_file_size = 128M
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit=1


Comments

  • tyasatyasa Current User Role Poster
    for additional note, i just remove (stop) mysql service using the command

    sudo pmm-admin remove mysql vultr.guest-mysql

    And the memory of my system is back to normal which usually looks like this:
    top - 03:31:13 up 15 days, 11:08,  1 user,  load average: 1.86, 3.21, 3.54
    Tasks:  83 total,   1 running,  82 sleeping,   0 stopped,   0 zombie
    %Cpu(s): 13.6 us,  5.6 sy,  0.0 ni, 79.1 id,  1.7 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem :  3880184 total,  2374220 free,  1081108 used,   424856 buff/cache
    KiB Swap:  4194300 total,  3686652 free,   507648 used.  2523920 avail Mem

      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
     2933 mysql     20   0 2141744 979804   5028 S  37.9 25.3  25:37.32 mysqld

  • alexey.palazhchenkoalexey.palazhchenko PMM Tech Lead Percona Staff Role
    Hi. That sounds like a severe problem. May you please create a bug in our Jira at https://jira.percona.com/projects/PMM? And please also attach a memory profile file. You can create it with `curl http://127.0.0.1:42001/debug/pprof/heap > heap.pb.gz` where 42001 is a mysqld_exporter port. That file will not contain any sensitive information.
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.