Mariadb keeps crashing due to OOM

MySQL & MariaDB MariaDB Server & Utilities
1

Mariadb keeps crashing due to OOM couple or more time in a week after upgrade from MariaDB 5.5 to MariaDB 10.0.

We had upgraded one of the databases from MariaDB 5.5 to MariaDB 10.0 and ever since the upgrade the mysqld process keeps getting restarted by the OS due to out of memory issue.
The innodb_buffer_pool_size is 4GB out of around 8GB RAM.

OS version-
NAME=“Red Hat Enterprise Linux Server”
VERSION=“7.9 (Maipo)”

MARIADB version - mysql Ver 15.1 Distrib 10.0.33-MariaDB, for Linux (x86_64) using readline 5.1

The memory consumption after a restart is as below.

                total        used        free      shared  buff/cache   available

Mem: 7821 2969 3720 2 1131 4593
Swap: 2047 538 1509

Then the “memory used” gradually starts to increase and gets filled and consumes the swap and finally kills the mysqld again. I am not sure what is causing the crash, and i am trying to find a fix for the OOM issue.

Contents of cnf file.

[mysqld]
datadir = /data/db
innodb_buffer_pool_size = 4G
socket = /data/socket/mysql.sock
server-id = 1
log-bin = /data/log-bin/mysql-bin
expire_logs_days = 14

symbolic-links = 0
max_allowed_packet = 1024M
max_connections = 500
max_connect_errors = 10000
innodb_log_file_size =256M

-# Slow query log
slow_query_log = 1
long_query_time = 5
slow_query_log_file = /data/log/slow.queries.log

-# this is only for embedded server
[embedded]

-# This group is only read by MariaDB-5.5 servers.
-# If you use the same .cnf file for MariaDB of different versions,
-# use this group for options that older servers don’t understand
[mysqld-5.5]

-# These two groups are only read by MariaDB servers, not by MySQL.
-# If you use the same .cnf file for MySQL and MariaDB,
-# you can put MariaDB-only options here
[mariadb]

[mariadb-5.5]

2

Hi @youniz05,
Try enabling jemalloc. Install it using your OS package manager, find the path, then edit the mariadb systemd unit file and add the following:

ExecStartPre=/bin/sh -c "systemctl set-environment LD_PRELOAD=/usr/lib64/jemalloc.so"

Restart mariadb and check that jemalloc was loaded. Watch the RAM usage.

3

Thanks @matthewb Should i also disable Transparent huge pages? Currently i have that enabled.

4

THP hasn’t ever been an issue with MySQL as I recall, so you can leave it enabled. You also only have 8GB of RAM so I doubt THP is even doing anything.

1 Like
5

Thanks. Will enable jemalloc and see how it goes!

6

Can you check the output of the below query?

SELECT ( @@key_buffer_size
+ @@query_cache_size
+ @@innodb_buffer_pool_size
+ @@innodb_additional_mem_pool_size
+ @@innodb_log_buffer_size
+ @@max_connections * ( @@read_buffer_size
+ @@read_rnd_buffer_size
+ @@sort_buffer_size
+ @@join_buffer_size
+ @@binlog_cache_size
+ @@thread_stack
+ @@tmp_table_size )
) / (1024 * 1024 * 1024) AS MAX_MEMORY_GB;
7

±--------------+
| MAX_MEMORY_GB |
±--------------+
| 13.3264 |

8

You confirgured mariadb with 14 GB while you have only 8 GB. Either you need to increase the RAM on your node or tweak mariadb variables So that it can’t be used beyond 7 GB. 1 GB can reserve for OS things.

9

Well… Configured to potentially used up to 14GB of memory. Very likely it will never get close to that. Many of these buffers are allocated only if needed.

10

Are you using any monitoring system to monitor the node(OS level)?

11

Hi @youniz05,

You “recently” upgraded to “10.0.33-MariaDB” which was released on “30 Oct 2017” and is obsolete. Is there a specific reason for that choice? Why not to choose latest version (10.0.38) in that series (still old but if that’s what you want)?

Apart from adjusting jemalloc suggested by Matthew,

  • try lowering the bufferpool size and see if the crashes are controlled! Observe if that’s exhibiting memory leak (with no hope of fixture on older version).
  • upgrade to latest version in that series

PS: If you’re not using any mariadb specific features, what keeps you from migrating to Percona Server 5.7 and the migrate to Percona Server 8.0?

Thanks,
Kedar

1 Like
12

@youniz05 Agreed with @kedarpercona, As a first step, tweak down the parameters to monitor further.

and plan for the upgrade.

if it’s too frequent then you can adjust OOMScoreAdjust in service file as well.

13

We had the bufferpool size as 128MB and still ended up with the same error. For now we are trying to enable jemalloc. Will keep you posted.

14

Hi @matthewb Even after loading jemalloc library, i still see the malloc library as ‘system’ instead of ‘jemalloc’. Is that ok?

MariaDB [(none)]> SHOW GLOBAL VARIABLES LIKE ‘version_malloc_library’;
±-----------------------±-------+
| Variable_name | Value |
±-----------------------±-------+
| version_malloc_library | system |
±-----------------------±-------+

● rh-mariadb100-mariadb.service - MariaDB 10.0 database server
Loaded: loaded (/etc/systemd/system/rh-mariadb100-mariadb.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2025-04-15 19:05:53 AEST; 1min 19s ago
Process: 12379 ExecStopPost=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
Process: 12670 ExecStartPost=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
Process: 12490 ExecStartPost=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
Process: 12482 ExecStartPre=/bin/sh -c sudo systemctl set-environment LD_PRELOAD=/usr/lib64/libjemalloc.so.1 (code=exited, status=0/SUCCESS)
Process: 12442 ExecStartPre=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Process: 12414 ExecStartPre=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Process: 12408 ExecStartPre=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /usr/bin/scl_enabled rh-mariadb100 (code=exited, status=0/SUCCESS)
Main PID: 12489 (mysqld_safe)
Memory: 384.3M
CGroup: /system.slice/rh-mariadb100-mariadb.service
├─12489 /bin/sh /opt/rh/rh-mariadb100/root/usr/bin/mysqld_safe --basedir=/opt/rh/rh-mariadb100/root/usr
└─12637 /opt/rh/rh-mariadb100/root/usr/libexec/mysqld --basedir=/opt/rh/rh-mariadb100/root/usr --datadir=/data/db --plugin-dir=/opt/rh/rh-mariadb100/root/usr/lib64/mysql/plugi