Percona Server 8.4-lts memory leakage

MySQL & MariaDB
1

Hello everyone!
I decided to install a new MySQL 8.4 server for our new projects…
Specifically, a Percona server, in group replication mode, not an extradb cluster…

I discovered that when idle, the server constantly consumes memory, not stopping until it’s visited by an OOM…
Initially, the servers were configured without jemalloc, but when the problem was discovered, the jemalloc library was used… But that didn’t help…

  1. The server is idle, not processing traffic;
  2. It has one nearly empty database;
  3. Two ProxySQL nodes are connected to the server;
  4. No special kernel settings were made, only HugePages were disabled.
# cat /etc/system-release
Oracle Linux Server release 9.6
# yum list installed | grep -E "(percona|jemalloc)"
jemalloc.x86_64                           5.2.1-2.el9                         @OL9_EPEL
percona-icu-data-files.x86_64             8.4.6-6.1.el9                       @Percona_Server_8.4-lts
percona-server-client.x86_64              8.4.6-6.1.el9                       @Percona_Server_8.4-lts
percona-server-server.x86_64              8.4.6-6.1.el9                       @Percona_Server_8.4-lts
percona-server-shared.x86_64              8.4.6-6.1.el9                       @Percona_Server_8.4-lts
percona-telemetry-agent.x86_64            1.0.3-4.el9                         @Percona_Telemetry
percona-xtrabackup-84.x86_64              8.4.0-2.1.el9                       @Percona_Tools
# lsof | grep $(pidof mysqld) | grep jemal
mysqld    211441                   mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211463 ib_io_ibu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211464 ib_io_rd-  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211465 ib_io_rd-  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211466 ib_io_rd-  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211467 ib_io_rd-  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211468 ib_io_wr-  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211469 ib_io_wr-  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211470 ib_io_wr-  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211471 ib_io_wr-  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211472 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211473 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211474 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211475 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211476 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211477 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211478 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211479 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211480 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211481 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211482 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211483 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211484 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211485 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211486 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211487 ib_pg_flu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211488 ib_srv_mo  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211489 ib_log_ch  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211490 ib_log_fl  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211491 ib_log_fl  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211492 ib_log_wr  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211493 ib_log_wr  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211494 ib_log_fi  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211499 ib_srv_lo  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211500 ib_srv_er  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211501 ib_buf_re  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211502 ib_src_ma  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211503 ib_dict_s  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211504 ib_fts_op  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211505 xpl_worke  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211506 xpl_worke  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211507 xpl_accep  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211509 gr_handle  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211510 gr_handle  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211516 ib_buf_du  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211517 ib_clone_  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211518 ib_srv_pu  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211520 evt_sched  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211521 sig_handl  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211522 xpl_accep  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211524 gtid_zip   mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211525 gcs_ext_l  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211526 gcs_engin  mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2
mysqld    211441 211527 gr_apply   mysql  mem       REG              252,2   563216   17786308 /usr/lib64/libjemalloc.so.2

# cat /etc/my.cnf
# This file controlled by ANSIBLE
# Ansible managed: my.cnf.j2 modified on 2025-10-09 20:56:17 by root

[mysqld]

datadir         = /var/lib/mysql
socket          = /var/lib/mysql/mysql.sock
port            = 3306
tmpdir          = /tmp
log-error       = /var/log/mysqld.log
pid-file        = /var/run/mysqld/mysqld.pid
symbolic-links  = 0

percona_telemetry_disable = ON

skip-name-resolve

plugin-load-add = validate_password.so
plugin-load-add = group_replication.so
plugin-load-add = mysql_clone.so

mysql_native_password                    = ON
default_password_lifetime                = 0

wait_timeout       = 300
max_connections    = 512
max_allowed_packet = 16M

join_buffer_size       = 1M
table_open_cache       = 32K
thread_cache_size      = 16
table_definition_cache = 2000

tmp_table_size         = 256M
max_heap_table_size    = 256M

key_buffer_size                = 256M
myisam_recover_options         = FORCE,BACKUP

innodb_data_home_dir           = /var/lib/mysql/
innodb_log_group_home_dir      = /var/lib/mysql/
innodb_buffer_pool_size        = 8G
innodb_log_buffer_size         = 16M
innodb_flush_method            = O_DIRECT
innodb_file_per_table          = 1
innodb_buffer_pool_instances   = 16
innodb_log_file_size           = 256M
innodb_buffer_pool_load_at_startup = OFF
innodb_buffer_pool_dump_at_shutdown = OFF

long_query_time = 5
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow_queries.log

gtid_mode                                     = ON
enforce_gtid_consistency                      = ON
binlog_checksum                               = NONE
report_port                                   = 3306
replica_sql_verify_checksum                   = ON
super_read_only                               = ON
group_replication_group_name                  = <redacted>
group_replication_group_seeds                 = <redacted>:33061,<redacted>:33061,<redacted>:33061
group_replication_single_primary_mode         = ON
group_replication_start_on_boot               = ON
group_replication_bootstrap_group             = OFF
group_replication_paxos_single_leader         = ON
group_replication_clone_threshold             = 100000
group_replication_exit_state_action           = READ_ONLY
group_replication_preemptive_garbage_collection = ON
server_id                                     = <redacted>
group_replication_local_address               = <redacted>:33061
log-bin = sql-bin
binlog_format = ROW
binlog_expire_logs_seconds = 172800

disabled_storage_engines       = MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY
max_binlog_size                = 1G

###########################################################################
[mysqld_safe]
pid-file  = /var/run/mysqld/mysqld.pid
socket    = /var/lib/mysql/mysql.sock
nice      = 0

Here’s what it looks like over a 3-days uptime:

image
image2444×401 112 KB

2

And here’s what the performance_schema tells us over the 3-day uptime period, all events are generally normal, except for these three:

  • memory/mysqld_openssl/openssl_malloc
  • memory/sql/String::value
  • memory/sql/THD::main_mem_root

image
image1971×1087 484 KB

What could this be if not a leak?

There is another server without replication configured, the behavior is exactly the same…

3

Hi Alexey,

We have an internal ticket related to this issue. After investigation, it was concluded that the problem is caused by the percona-mysql-shell package. However, I don’t see this package among the ones installed on your server.

Could you please confirm whether the percona-mysql-shell package is used in your environment?

Thank you.

4

No, i don’t use this package and it was not installed on the server:

# yum list | grep mysql-shell
percona-mysql-shell.x86_64                                                               8.4.6-1.el9                                         Percona_Server_8.4-lts
percona-mysql-shell-debuginfo.x86_64                                                     8.4.6-1.el9                                         Percona_Server_8.4-lts
percona-mysql-shell-debugsource.x86_64                                                   8.4.6-1.el9                                         Percona_Server_8.4-lts
5

I’d also like to note that similar behavior for two metrics is present on the community MySQL, but it doesn’t affect overall memory consumption.

image
image2450×858 299 KB

6

Hello!

I increased the maximum server memory to 24GB from 12 to see if there would be a limit of increasing consumption…
The server still isn’t processing requests…

Results:

  1. After starting the server with an InnoDB_buffer_pool of 8GB, the server starts up consuming just over 9GB of memory, then gradually begins consuming memory at a rate of about 10MB per hour, with another burst of about 90MB of memory consumed approximately every 11 hours and 20 minutes (40,800 seconds).
  2. After a week, the server’s memory consumption reaches 11.7GB, and seems to be stabilizing…

image
image1851×318 8.74 KB

image
image1851×347 25.1 KB

It looks like we can’t give the server more than 65% of the server’s memory for innodb pool without causing an OOM…

By the way, the openssl_malloc, String::value, and THD::main_mem_root event metrics haven’t changed their behavior, they’ve been growing and continue to do so, at the same pace…

image
image1851×347 26.5 KB

I’ve now changed the settings for another server, where I’ve set the InnoDB pool to only 2GB. We’ll see if the consumption changes beyond that…