There is serious memory leak

MySQL & MariaDB Percona Server for MySQL 8.0
1

Download Percona-Server-8.0.27-18-Linux.x86_64.glibc2.17-minimal.tar.gz from percona.com,

there are serious memory leak with lot’s of monitor.

there is jemalloc under Percona-Server-8.0.27/lib/mysql/lib/libjemalloc.so.1 by default and percona use it by default, I am not sure the version of this jemalloc, but mysql run out memory after startup one day.

I download source jemalloc-5.2.1 from official website, change LD_PRELOAD variable and deleted percona default jemalloc, everything goes well now.

1 Like
2

Hello @azuryy,
I checked with our engineering software team on this and there isn’t any other reports of memory leaks. Do you have any sort of performance analysis from PMM during your leak that you can share?

1 Like
3

I think it might be related, I have experienced (on multiple servers) memory leaks, but more specifically on INSERT actions. The time it usually took to synchronise a database from MS access to Percona MySQL has gone up 10 fold. I’ve worked with the developers of the synchronisation software, but the problem seems not to be there, but mainly on the INSERT actions.

1 Like
4

@kalahariresearch,
There can be many reasons why you are experiencing poor INSERT performance. What we really need to see if performance graphs from MySQL and the OS while you are seeing issues in order to determine the root cause.

1 Like
5

@matthewb
Please see below.
I’ll also post my mysql config, the weird thing is that without changing anything, the performance dropped after the update.

image
image808×150 8.14 KB 

[mysql]

# CLIENT #
#port                           = 3306
#socket                         = /var/run/mysqld/mysqld.sock

[mysqld]

# GENERAL #
#user                           = mysql
default_storage_engine         = InnoDB
#socket                         = /var/run/mysqld/mysqld.sock
#pid_file                       = /var/run/mysqld/mysqld.pid

# SAFETY #
max_allowed_packet             = 16M
max_connect_errors             = 1000000
skip_name_resolve
#sql_mode                       = STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY
sql_mode =
sysdate_is_now                 = 1
innodb                         = FORCE
innodb_strict_mode             = 1


# BINARY LOGGING #
#log_bin                        = /var/log/mysql/mysql-bin
expire_logs_days               = 14
sync_binlog                    = 1

# CACHES AND LIMITS #
tmp_table_size                 = 32M
max_heap_table_size            = 32M
#query_cache_type               = 0
#query_cache_size               = 0
max_connections                = 500
thread_cache_size              = 50
open_files_limit               = 65535
table_definition_cache         = 1024
table_open_cache               = 2048

# INNODB #
innodb_flush_method            = O_DIRECT
innodb_log_files_in_group      = 2
innodb_log_file_size           = 1536M
innodb_flush_log_at_trx_commit = 0
innodb_file_per_table          = 1
innodb_buffer_pool_size        = 8000M
innodb_buffer_pool_instances   = 1
innodb_read_io_threads         = 64
innodb_write_io_threads        = 64
innodb_fast_shutdown = 0

# LOGGING #
#log_error                      = /var/log/mysql/mysql-error.log
#log_queries_not_using_indexes  = 1
#slow_query_log                 = 1
#slow_query_log_file            = /var/log/mysql/mysql-slow.log
1 Like
6

Since I can only post one picture, here the performance graphs.

image
image1036×673 39.1 KB

1 Like
7

Thanks @matthewb ,

I don’t have perf analysis now, but I remember clearly is that, command ‘free -m’ to see, available memory only 600MB+(innodb buffer pool 96GB) after running mysqld 10 hours, and during these 10 hours, I’ve noticed the system memory was consumed increasingly.

I only do the following:

  1. download jemalloc-5.2.1 source code and compile/install it to /usr/lib/.
  2. stop mysqld
  3. delete $Percona/lib/mysql/lib/libjemalloc.so.1
  4. export LD_PRELOAD to /usr/lib/libjemalloc.so.2

then, using ‘free -m’ command to check system memory consuming, it’s normal(avalable memory 87G) after running mysqld 10 hours.

1 Like
8

One suggestion to Percona is upgrade jemalloc to 5.2.x which bundled with Percona MySQL server and update tar ball on the website.

1 Like
9

@matthewb, would you be able to give me feedback on the posted config and performance charts?

1 Like
10

Your config looks very normal. Unfortunately, your performance charts don’t show anything related to memory usage of the OS. I can see many other things in your top screenshot (php, postgresql, nodejs, etc) All of those are competing with MySQL for memory. You may not be seeing a leak, you may just be seeing competition for memory. Increase the amount of memory on the OS by 4-6GB and see if this is still an issue.

1 Like