Hi,
I finally got pmm to work after opening up port 42001 42002 on my client (where the mysql db is located)
While looking at the new metrics coming in, i was surprised to see memory available down to 4%. (usually around 40-50% free)
Running top on the database server, i see this:
3066 root 20 0 2825676 2.5g 1240 S 1.3 68.0 4:41.44 mysqld_exporter
thats 68% memory held by mysqld_exporter and it stays there.
Tasks: 84 total, 1 running, 83 sleeping, 0 stopped, 0 zombie%Cpu(s): 1.9 us, 5.5 sy, 0.0 ni, 65.5 id, 26.7 wa, 0.0 hi, 0.2 si, 0.2 stKiB Mem : 3880184 total, 111948 free, 3665160 used, 103076 buff/cacheKiB Swap: 4194300 total, 3543804 free, 650496 used. 40916 avail Mem
Noted that the server is on the weak side of 4GB memory. but the server is a new startup and rarely goes beyond 50% of memory usage yet.
Any advice on this? Is there a setting i could set to limit memory usage by the exporter or is my server too weak to run pmm client?
Which actually takes less memory footprint, using perfschema or slowlog? i commented slow query log in my.cnf . does it matter which is used in regards to memory taken by mysqld exporter?
Can i reduce the sample rate / upload rate / or something to reduce memory footprint to around 10% of system memory with the side effect of not getting the full picture in the pmm dashboard?
Thank you for the advice. Below are the settings for the database systems. (this is my first system that i try to manage, so i’m still learning on server management)
ProjectName: pmm-adminVersion: 2.8.0PMMVersion: 2.8.0Timestamp: 2020-06-25 13:53:24 (UTC)FullCommit: 7d9e26cac82f694f1fef965fad9be862dbd7926c
I ran the following for setup:
sudo pmm-admin add mysql --query-source=perfschema --user=user --pass=pass --host=localhost --port=3306
pmm-admin list shows the following:
pmm_agent Connected /agent_id/41cc2aa9-cbb5-42f6-a7db-9f15aa7ad045node_exporter Running /agent_id/7b8c1697-932f-47d6-a81e-2bf82bcc3306mysqld_exporter Running /agent_id/0de9dcb3-6810-4695-bc55-021429dd3632 /service_id/2f69a203-a7af-4af8-9eee-23228592f02dmysql_perfschema_agent Running /agent_id/9667bf76-2501-4505-82a6-36f45c8a3cdf /service_id/2f69a203-a7af-4af8-9eee-23228592f02d
my.cnf (mysql 8.0):
innodb_monitor_enable=allperformance_schema=ON
# slow query log commented
#log_output=file#slow_query_log=ON#long_query_time=0#log_slow_admin_statements=ON#log_slow_slave_statements=ON
[innodb]innodb_buffer_pool_size = 3200M # (adjust value here, 50%-70% of total RAM)innodb_log_file_size = 128Minnodb_flush_method = O_DIRECTinnodb_flush_log_at_trx_commit=1
for additional note, i just remove (stop) mysql service using the command
sudo pmm-admin remove mysql vultr.guest-mysql
And the memory of my system is back to normal which usually looks like this:
top - 03:31:13 up 15 days, 11:08, 1 user, load average: 1.86, 3.21, 3.54Tasks: 83 total, 1 running, 82 sleeping, 0 stopped, 0 zombie%Cpu(s): 13.6 us, 5.6 sy, 0.0 ni, 79.1 id, 1.7 wa, 0.0 hi, 0.0 si, 0.0 stKiB Mem : 3880184 total, 2374220 free, 1081108 used, 424856 buff/cacheKiB Swap: 4194300 total, 3686652 free, 507648 used. 2523920 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2933 mysql 20 0 2141744 979804 5028 S 37.9 25.3 25:37.32 mysqld
Hi. That sounds like a severe problem. May you please create a bug in our Jira at https://jira.percona.com/projects/PMM? And please also attach a memory profile file. You can create it with curl http://127.0.0.1:42001/debug/pprof/heap > heap.pb.gz where 42001 is a mysqld_exporter port. That file will not contain any sensitive information.