# MySQL memory creeping up in PXC node **URL:** \ **Category:** Percona XtraDB Cluster 8.x\ **Created:** [September 16, 2022, 7:39pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519 "2022-09-16T19:39:23Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![Dba1](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dba1/32/24223_2.png) [@Dba1](https://forums.percona.com/u/Dba1)\ **Post date:** [September 16, 2022, 7:39pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/1 "2022-09-16T19:39:23Z")
Greetings great minds I have a node that’s part of a 4 nodes galera cluster . I have allocated 16GB innodb buffer pool size and innodb log buffer size is 4GB. The total RAM in this box is 23GB. Recently using TOP I found out that the mysqld’s memory usage creeps up from 78% to 85% and now 93%. I would like to find out why is the memory usage growing when I absolutely have nothing going on in the process( mysql processlist) the only process is just one insert That’s all. Note: this node is not the even the busiest. No little or no traffics directed to it. Any suggestion would be helpful. I have done some few searches as wel ---
**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\ **Post date:** [September 20, 2022, 8:08pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/2 "2022-09-20T20:08:19Z")
Hi @Dba1 , thanks for posting to the Percona forums! Are you using PMM against this instance? If so there is a Memory section on the MySQL Instance Summary dashboard that may indicate where the memory is being allocated. There is also the Node Summary dashboard that will show actual memory allocation (vs VIRT). Please share with us your dashboards when you can and we can give them a review. Thanks! ---
**Author:** ![Dba1](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dba1/32/24223_2.png) [@Dba1](https://forums.percona.com/u/Dba1)\ **Post date:** [September 20, 2022, 8:28pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/3 "2022-09-20T20:28:49Z")
Than you Mr.Coburn, Unfortunately I am not using PMM in my environment for one reason my organization cannot reach to an external repos. Is there any better way of installing PMM and it client. For the memory leak I was able to solve it. Thank you ---
**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\ **Post date:** [September 28, 2022, 2:31pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/4 "2022-09-28T14:31:52Z")
Hi @Dba1 Did you try the quickstart? [Install Percona Monitoring and Management](https://www.percona.com/software/pmm/quickstart#) If you can download the docker container + pmm2-client package then you won’t need to connect later to external repositories ---
**Author:** ![Dba1](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/dba1/32/24223_2.png) [@Dba1](https://forums.percona.com/u/Dba1)\ **Post date:** [September 28, 2022, 2:42pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/5 "2022-09-28T14:42:29Z")
Hello Michael thanks a lot visiting this post again. For one reason or another which I believe it was proxy configurations I can’t even pull the docker image, also is there anyway I won’t need to install PMM client on all my database server. ---
**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\ **Post date:** [September 28, 2022, 2:58pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/6 "2022-09-28T14:58:39Z")
Hi @Dba1 Once you have PMM Server set up then you can avoid installing anything on the clients by using the Remote MySQL Monitoring feature > **[MySQL - Percona Monitoring and Management](https://docs.percona.com/percona-monitoring-and-management/setting-up/client/mysql.html#with-the-user-interface)** > > Documentation ---
**Author:** ![Ralph\_Moser](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ralph_moser/32/7966_2.png) [@Ralph\_Moser](https://forums.percona.com/u/Ralph_Moser)\ **Post date:** [October 3, 2022, 10:30pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/7 "2022-10-03T22:30:57Z")
Hello, I have the same problem with 8.0.29. We’re evaluating version 8 because we’re planning to switch from 5.7. So I fired up one node in bootstrap mode and tried to restore a dump of our 5.7 DB on it. It’s around 400GB. The InnoDB buffer size is set to 6GB but while restoring with myloader htop looks like this: ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/9/9a345496df8cc81292ca6894cf006c7607b6f9cf.png) The memory usage increases until the OOM killer strikes. I have PMM up and running if you need any metrics. ---
**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\ **Post date:** [October 4, 2022, 2:46pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/8 "2022-10-04T14:46:16Z")
Hi @Ralph_Moser , thanks for participating! What I am looking for in PMM is the following: > [@Michael\_Coburn](#): > > there is a Memory section on the MySQL Instance Summary dashboard that may indicate where the memory is being allocated. There is also the Node Summary dashboard that will show actual memory allocation (vs VIRT). ---
**Author:** ![Ralph\_Moser](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ralph_moser/32/7966_2.png) [@Ralph\_Moser](https://forums.percona.com/u/Ralph_Moser)\ **Post date:** [October 8, 2022, 4:29pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/9 "2022-10-08T16:29:04Z")
@Michael_Coburn it looks like a memory leak. Sorry I have to split this into three posts because I’m a new user which can’t embed more than one media item… HTOP: ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/b/bb5cd1d10c659e43e6496b566eea78681b5d5462.png) ---
**Author:** ![Ralph\_Moser](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ralph_moser/32/7966_2.png) [@Ralph\_Moser](https://forums.percona.com/u/Ralph_Moser)\ **Post date:** [October 8, 2022, 4:29pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/10 "2022-10-08T16:29:34Z")
PMM: ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/8/83585abe1adbbbca8eafe1a8e23969bd7927ffa8.png) ---
**Author:** ![Ralph\_Moser](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ralph_moser/32/7966_2.png) [@Ralph\_Moser](https://forums.percona.com/u/Ralph_Moser)\ **Post date:** [October 8, 2022, 4:30pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/11 "2022-10-08T16:30:06Z")
PMM: ![image](https://us1.discourse-cdn.com/flex019/uploads/percona1/original/2X/1/16dff12c630a9be8ced2f0b2c5bd994bc4899685.png) ---
**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\ **Post date:** [October 10, 2022, 5:10pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/12 "2022-10-10T17:10:22Z")
Thanks @Ralph_Moser and @Dba1 - I’ve posted to Percona internal Slack seeking additional information in order to troubleshoot this, then I’ll update this thread 👍 ---
**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\ **Post date:** [October 10, 2022, 6:08pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/13 "2022-10-10T18:08:47Z")
Hi @Ralph_Moser and @Dba1 - We have a confirmed bug related to memory usage, my recommendation is to follow this issue in JIRA so you’ll see when it hits a release and then you can test the version [https://jira.percona.com/browse/PXC-4051](https://jira.percona.com/browse/PXC-4051) ---
**Author:** ![Kamil\_Holubicki](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kamil_holubicki/32/5693_2.png) [@Kamil\_Holubicki](https://forums.percona.com/u/Kamil_Holubicki)\ **Post date:** [October 18, 2022, 12:40pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/14 "2022-10-18T12:40:50Z")
Hi @Ralph_Moser , If I understand correctly you have one node and try to load data into it. In such a case, please try setting ```auto set global wsrep_trx_fragment_unit = 'rows'; set global wsrep_trx_fragment_size = 100000; ``` prior to `myloader` usage, or try using `--queries-per-transaction` parameter of `myloader` to limit the transaction size. ---
**Author:** ![Ralph\_Moser](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ralph_moser/32/7966_2.png) [@Ralph\_Moser](https://forums.percona.com/u/Ralph_Moser)\ **Post date:** [October 18, 2022, 1:09pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/15 "2022-10-18T13:09:11Z")
Hi @Kamil_Holubicki Yes, you did. I tried enabling streaming replication with the statements you provided but sadly it doesn’t change anything. Memory usage is still growing steadily. On our 5.7 test cluster restoring mydumper dumps to one node always worked perfectly without any special setup. (Restore on a single node, added the other two nodes, SST, done) ---
**Author:** ![Ralph\_Moser](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ralph_moser/32/7966_2.png) [@Ralph\_Moser](https://forums.percona.com/u/Ralph_Moser)\ **Post date:** [October 18, 2022, 1:23pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/16 "2022-10-18T13:23:13Z")
I think wasn’t patient enough. Memory usage now stabilized at around 14.5G. Is this some new behavior in Galera 4 which makes it necessary to enable streaming? ---
**Author:** ![Kamil\_Holubicki](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kamil_holubicki/32/5693_2.png) [@Kamil\_Holubicki](https://forums.percona.com/u/Kamil_Holubicki)\ **Post date:** [October 18, 2022, 1:27pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/17 "2022-10-18T13:27:28Z")
@Ralph_Moser Could you post the exact myloader command line you are using? If possible please try experimenting with following settings: 1. reduce wsrep\_trx\_fragment\_size to 10000 rows or even less 2. use 1 thread when using myloader ( --threads=1) 3. limit myloader transaction size (–queries-per-transaction parameter) 4. use jemalloc [Enabling jemalloc on Percona Server for MySQL 5.7 and 8.0 Series - Percona Database Performance Blog](https://www.percona.com/blog/2020/12/02/enabling-jemalloc-on-percona-server-for-mysql-5-7-and-8-0-series/) ---
**Author:** ![Ralph\_Moser](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ralph_moser/32/7966_2.png) [@Ralph\_Moser](https://forums.percona.com/u/Ralph_Moser)\ **Post date:** [October 18, 2022, 1:37pm UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/18 "2022-10-18T13:37:25Z")
jemalloc is already enabled. 😉 Like I wrote before it stabilized. I just wasn’t waiting long enough. Thanks for all the help! The only thing I’m still asking myself is why this didn’t happen with 5.7. ---
**Author:** ![Kamil\_Holubicki](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/kamil_holubicki/32/5693_2.png) [@Kamil\_Holubicki](https://forums.percona.com/u/Kamil_Holubicki)\ **Post date:** [October 19, 2022, 8:57am UTC](https://forums.percona.com/t/mysql-memory-creeping-up-in-pxc-node/17519/19 "2022-10-19T08:57:29Z")
Hi @Ralph_Moser , I’ve just checked with PXC 5.7.39 and to be honest I observe the behavior very similarly to 8.0. If you’ve got a reproducible test case that shows the problem/behavior difference, please create a ticket in [jira.percona.com](http://jira.percona.com) providing all needed details (steps to reproduce, config, etc)