Mariadb keeps crashing due to OOM

youniz05 · April 1, 2025, 11:14am

Mariadb keeps crashing due to OOM couple or more time in a week after upgrade from MariaDB 5.5 to MariaDB 10.0.

We had upgraded one of the databases from MariaDB 5.5 to MariaDB 10.0 and ever since the upgrade the mysqld process keeps getting restarted by the OS due to out of memory issue.
The innodb_buffer_pool_size is 4GB out of around 8GB RAM.

OS version-
NAME=“Red Hat Enterprise Linux Server”
VERSION=“7.9 (Maipo)”

MARIADB version - mysql Ver 15.1 Distrib 10.0.33-MariaDB, for Linux (x86_64) using readline 5.1

The memory consumption after a restart is as below.

                total        used        free      shared  buff/cache   available

Mem: 7821 2969 3720 2 1131 4593
Swap: 2047 538 1509

Then the “memory used” gradually starts to increase and gets filled and consumes the swap and finally kills the mysqld again. I am not sure what is causing the crash, and i am trying to find a fix for the OOM issue.

Contents of cnf file.

[mysqld]
datadir = /data/db
innodb_buffer_pool_size = 4G
socket = /data/socket/mysql.sock
server-id = 1
log-bin = /data/log-bin/mysql-bin
expire_logs_days = 14

symbolic-links = 0
max_allowed_packet = 1024M
max_connections = 500
max_connect_errors = 10000
innodb_log_file_size =256M

-# Slow query log
slow_query_log = 1
long_query_time = 5
slow_query_log_file = /data/log/slow.queries.log

-# this is only for embedded server
[embedded]

-# This group is only read by MariaDB-5.5 servers.
-# If you use the same .cnf file for MariaDB of different versions,
-# use this group for options that older servers don’t understand
[mysqld-5.5]

-# These two groups are only read by MariaDB servers, not by MySQL.
-# If you use the same .cnf file for MySQL and MariaDB,
-# you can put MariaDB-only options here
[mariadb]

[mariadb-5.5]

matthewb · April 1, 2025, 6:09pm

Hi @youniz05,
Try enabling jemalloc. Install it using your OS package manager, find the path, then edit the mariadb systemd unit file and add the following:

ExecStartPre=/bin/sh -c "systemctl set-environment LD_PRELOAD=/usr/lib64/jemalloc.so"

Restart mariadb and check that jemalloc was loaded. Watch the RAM usage.

youniz05 · April 2, 2025, 4:32am

Thanks @matthewb Should i also disable Transparent huge pages? Currently i have that enabled.

matthewb · April 2, 2025, 11:36am

THP hasn’t ever been an issue with MySQL as I recall, so you can leave it enabled. You also only have 8GB of RAM so I doubt THP is even doing anything.

youniz05 · April 3, 2025, 3:26am

Thanks. Will enable jemalloc and see how it goes!

ROHIT_KHURANA · April 3, 2025, 2:23pm

Can you check the output of the below query?

SELECT ( @@key_buffer_size
+ @@query_cache_size
+ @@innodb_buffer_pool_size
+ @@innodb_additional_mem_pool_size
+ @@innodb_log_buffer_size
+ @@max_connections * ( @@read_buffer_size
+ @@read_rnd_buffer_size
+ @@sort_buffer_size
+ @@join_buffer_size
+ @@binlog_cache_size
+ @@thread_stack
+ @@tmp_table_size )
) / (1024 * 1024 * 1024) AS MAX_MEMORY_GB;

youniz05 · April 4, 2025, 4:04am

ROHIT_KHURANA:

SELECT ( @@key_buffer_size
+ @@query_cache_size
+ @@innodb_buffer_pool_size
+ @@innodb_additional_mem_pool_size
+ @@innodb_log_buffer_size
+ @@max_connections * ( @@read_buffer_size
+ @@read_rnd_buffer_size
+ @@sort_buffer_size
+ @@join_buffer_size
+ @@binlog_cache_size
+ @@thread_stack
+ @@tmp_table_size )
) / (1024 * 1024 * 1024) AS MAX_MEMORY_GB;

±--------------+
| MAX_MEMORY_GB |
±--------------+
| 13.3264 |

ROHIT_KHURANA · April 4, 2025, 5:11am

You confirgured mariadb with 14 GB while you have only 8 GB. Either you need to increase the RAM on your node or tweak mariadb variables So that it can’t be used beyond 7 GB. 1 GB can reserve for OS things.

yves.trudeau · April 4, 2025, 3:26pm

Well… Configured to potentially used up to 14GB of memory. Very likely it will never get close to that. Many of these buffers are allocated only if needed.

ROHIT_KHURANA · April 9, 2025, 2:17pm

Are you using any monitoring system to monitor the node(OS level)?

kedarpercona · April 12, 2025, 7:18am

Hi @youniz05,

You “recently” upgraded to “10.0.33-MariaDB” which was released on “30 Oct 2017” and is obsolete. Is there a specific reason for that choice? Why not to choose latest version (10.0.38) in that series (still old but if that’s what you want)?

Apart from adjusting jemalloc suggested by Matthew,

try lowering the bufferpool size and see if the crashes are controlled! Observe if that’s exhibiting memory leak (with no hope of fixture on older version).
upgrade to latest version in that series

PS: If you’re not using any mariadb specific features, what keeps you from migrating to Percona Server 5.7 and the migrate to Percona Server 8.0?

Thanks,
Kedar

ROHIT_KHURANA · April 14, 2025, 2:10am

@youniz05 Agreed with @kedarpercona, As a first step, tweak down the parameters to monitor further.

and plan for the upgrade.

if it’s too frequent then you can adjust OOMScoreAdjust in service file as well.

youniz05 · April 15, 2025, 7:06am

We had the bufferpool size as 128MB and still ended up with the same error. For now we are trying to enable jemalloc. Will keep you posted.

youniz05 · April 15, 2025, 9:11am

Hi @matthewb Even after loading jemalloc library, i still see the malloc library as ‘system’ instead of ‘jemalloc’. Is that ok?

● rh-mariadb100-mariadb.service - MariaDB 10.0 database server
Loaded: loaded (/etc/systemd/system/rh-mariadb100-mariadb.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2025-04-15 19:05:53 AEST; 1min 19s ago
Process: 12379 ExecStopPost=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
Process: 12670 ExecStartPost=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
Process: 12490 ExecStartPost=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
Process: 12482 ExecStartPre=/bin/sh -c sudo systemctl set-environment LD_PRELOAD=/usr/lib64/libjemalloc.so.1 (code=exited, status=0/SUCCESS)
Process: 12442 ExecStartPre=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)
Process: 12414 ExecStartPre=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-check-socket (code=exited, status=0/SUCCESS)
Process: 12408 ExecStartPre=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /usr/bin/scl_enabled rh-mariadb100 (code=exited, status=0/SUCCESS)
Main PID: 12489 (mysqld_safe)
Memory: 384.3M
CGroup: /system.slice/rh-mariadb100-mariadb.service
├─12489 /bin/sh /opt/rh/rh-mariadb100/root/usr/bin/mysqld_safe --basedir=/opt/rh/rh-mariadb100/root/usr
└─12637 /opt/rh/rh-mariadb100/root/usr/libexec/mysqld --basedir=/opt/rh/rh-mariadb100/root/usr --datadir=/data/db --plugin-dir=/opt/rh/rh-mariadb100/root/usr/lib64/mysql/plugi

youniz05 · May 2, 2025, 4:00am

I have tried increasing the RAM to 16GB which increased duration between the restart, earlier it restarted once in 2 days after the change it is restarting once in 5days. So is upgrade the solution to fix this?
Though loading the jemalloc library when querying version_malloc_library the value still says ‘system’ instead of jemalloc.

          total        used        free      shared  buff/cache   available

Mem: 15885 8517 1100 14 6267 7022
Swap: 2047 220 1827

● rh-mariadb100-mariadb.service - MariaDB 10.0 database server
Loaded: loaded (/etc/systemd/system/rh-mariadb100-mariadb.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2025-04-15 19:05:53 AEST; 1min 19s ago
Process: 12379 ExecStopPost=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-wait-stop (code=exited, status=0/SUCCESS)
Process: 12670 ExecStartPost=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-check-upgrade (code=exited, status=0/SUCCESS)
Process: 12490 ExecStartPost=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-wait-ready $MAINPID (code=exited, status=0/SUCCESS)
Process: 12482 ExecStartPre=/bin/sh -c sudo systemctl set-environment LD_PRELOAD=/usr/lib64/libjemalloc.so.1 (code=exited, status=0/SUCCESS)
Process: 12442 ExecStartPre=/usr/bin/scl enable $RH_MARIADB100_SCLS_ENABLED – /opt/rh/rh-mariadb100/root/usr/libexec/mysql-prepare-db-dir %n (code=exited, status=0/SUCCESS)

matthewb · May 2, 2025, 4:20pm

@youniz05 Looks like it is different for MariaDB. Check out their official instructions for editing systemd unit files: Using MariaDB with TCMalloc or jemalloc - MariaDB Knowledge Base

Topic		Replies	Views
MariaDB crashing MySQL & MariaDB	12	1993	March 19, 2021
My mysql server has started crashing intermittently MySQL & MariaDB mysql	2	52	February 7, 2025
Innodb high memory load make mysql crash [help] Other MySQL® Questions	2	639	May 28, 2008
slowly growing memory usage Other MySQL® Questions	4	878	July 19, 2013
Performance Degradation Problem Other MySQL® Questions	4	896	June 27, 2013

Mariadb keeps crashing due to OOM

Related topics