MySQL 5.7 (Percona) — InnoDB RSS 75GB on 78GB server, 19GB unaccounted nr+sd mmap regions

1

Environment

OS:              Debian GNU/Linux 11 (bullseye)
Kernel:          Linux (glibc 2.31-13+deb11u7)
CPU cores:       16
Total RAM:       78 GB
Swap:            None
MySQL:           Percona Server 5.7.39-42-log
Uptime:          ~298 days

Problem Statement

mysqld RSS is 75.4GB on a 78GB server with only 69MB available memory and no swap configured. After accounting for the configured innodb_buffer_pool_size of 52GB, approximately 19GB remains unaccounted for in InnoDB internal mmap regions with MAP_NORESERVE flag (nr+sd in smaps VmFlags).

System Variables

SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
+-------------------------+-------------+
| innodb_buffer_pool_size | 55834574848 |   -- 52GB configured
+-------------------------+-------------+

SHOW VARIABLES LIKE 'innodb_buffer_pool_instances';
+------------------------------+-------+
| innodb_buffer_pool_instances | 32    |
+------------------------------+-------+

SHOW VARIABLES LIKE 'innodb_buffer_pool_chunk_size';
+-------------------------------+-----------+
| innodb_buffer_pool_chunk_size | 134217728 |  -- 128MB
+-------------------------------+-----------+

SELECT
    @@innodb_buffer_pool_size / @@innodb_buffer_pool_chunk_size AS total_chunks,
    @@innodb_buffer_pool_size / @@innodb_buffer_pool_instances / 
    @@innodb_buffer_pool_chunk_size AS chunks_per_instance;
+--------------+---------------------+
| total_chunks | chunks_per_instance |
+--------------+---------------------+
|     416.0000 |         13.00000000 |
+--------------+---------------------+

SHOW VARIABLES LIKE 'innodb_adaptive_hash_index%';
+----------------------------------+-------+
| innodb_adaptive_hash_index       | OFF   |
| innodb_adaptive_hash_index_parts | 8     |
+----------------------------------+-------+

SHOW VARIABLES LIKE 'innodb_undo%';
+--------------------------+-------+
| innodb_undo_directory    | ./    |
| innodb_undo_log_encrypt  | OFF   |
| innodb_undo_logs         | 128   |
| innodb_undo_tablespaces  | 0     |
+--------------------------+-------+

SHOW VARIABLES LIKE 'innodb_purge_threads';
+----------------------+-------+
| innodb_purge_threads | 4     |
+----------------------+-------+

SHOW VARIABLES LIKE 'innodb_read_io_threads';
+------------------------+-------+
| innodb_read_io_threads | 8     |
+------------------------+-------+

SHOW VARIABLES LIKE 'innodb_write_io_threads';
+-------------------------+-------+
| innodb_write_io_threads | 8     |
+-------------------------+-------+

SHOW VARIABLES LIKE 'tmp_table_size';
+----------------+-----------+
| tmp_table_size | 536870912 |  -- 512MB
+----------------+-----------+

SHOW VARIABLES LIKE 'max_heap_table_size';
+---------------------+-----------+
| max_heap_table_size | 536870912 |  -- 512MB
+---------------------+-----------+

SHOW VARIABLES LIKE 'max_connections';
+-----------------+-------+
| max_connections | 5000  |
+-----------------+-------+

SHOW VARIABLES LIKE 'thread_stack';
+---------------+--------+
| thread_stack  | 262144 |  -- 256KB
+---------------+--------+

SHOW VARIABLES LIKE 'innodb_log_buffer_size';
+------------------------+-----------+
| innodb_log_buffer_size | 134217728 |  -- 128MB
+------------------------+-----------+

SHOW VARIABLES LIKE '%buffer_size%';
+-------------------------+-----------+
| bulk_insert_buffer_size | 8388608   |
| innodb_log_buffer_size  | 134217728 |
| innodb_sort_buffer_size | 1048576   |
| join_buffer_size        | 262144    |
| key_buffer_size         | 8388608   |
| myisam_sort_buffer_size | 8388608   |
| read_buffer_size        | 131072    |
| read_rnd_buffer_size    | 262144    |
| sort_buffer_size        | 262144    |
+-------------------------+-----------+

SHOW VARIABLES LIKE 'performance_schema';
+--------------------+-------+
| performance_schema | ON    |
+--------------------+-------+

OS Settings

# Memory
free -mh
               total        used        free      shared  buff/cache   available
Mem:            78Gi        77Gi       484Mi       0.0Ki       266Mi        69Mi
Swap:             0B          0B          0B

# THP setting
cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

# mysqld RSS
cat /proc/$(pidof mysqld)/status | grep VmRSS
VmRSS: 79060188 kB   (~75.4 GB)

# glibc version
ldd --version | head -1
ldd (Debian GLIBC 2.31-13+deb11u7) 2.31

# jemalloc
ldd $(which mysqld) | grep -E 'jemalloc|malloc|tcmalloc'
(no output — mysqld using plain glibc malloc)

# CPU cores
nproc
16

# Uptime
mysqladmin status | grep Uptime
Uptime: 25744293  Threads: 131  Questions: 6211793746
Max_used_connections: 475

Buffer Pool Status

SELECT
    SUM(POOL_SIZE)          AS total_pages,
    SUM(FREE_BUFFERS)       AS free_pages,
    SUM(DATABASE_PAGES)     AS used_pages,
    SUM(FREE_BUFFERS)*100.0/SUM(POOL_SIZE) AS free_pct,
    SUM(POOL_SIZE)*16384/1024/1024/1024    AS total_gb
FROM information_schema.INNODB_BUFFER_POOL_STATS;

+-------------+------------+------------+----------+-----------------+
| total_pages | free_pages | used_pages | free_pct | total_gb        |
+-------------+------------+------------+----------+-----------------+
|     3407456 |      32767 |    3374689 |  0.96163 | 51.993652343750 |
+-------------+------------+------------+----------+-----------------+
-- Buffer pool is 99% utilized

-- Per instance (all 32 identical):
POOL_SIZE: 106483 pages = 1664MB per instance
FREE_BUFFERS: 1024 pages = 16MB free per instance

InnoDB Engine Status

BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 58124664832
Dictionary memory allocated 20552198
Internal hash tables (constant factor + variable factor)
    Adaptive hash index 883923648 	(883889984 + 33664)
    Page hash           1727144 (buffer pool 0 only)
    Dictionary cache    241524694 	(220972496 + 20552198)
    File system         1894792 	(812272 + 1082520)
    Lock system         143943400 	(143819576 + 123824)
    Recovery system     0 	(0 + 0)
Buffer pool size   3407456
Buffer pool size, bytes 55827759104
Free buffers       32768
Database pages     3374688
Old database pages 1245088
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1189086697, not young 12182634025
0.00 youngs/s, 0.00 non-youngs/s
Pages read 109154374, created 92663, written 154845768
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 11 / 1000 not 3 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3374688, unzip_LRU len: 0
I/O sum[13152]:cur[32], unzip sum[0]:cur[0]
----------------------

Performance Schema Memory

SELECT sys.format_bytes(SUM(CURRENT_NUMBER_OF_BYTES_USED)) AS total_perf_schema_mem
FROM performance_schema.memory_summary_global_by_event_name
WHERE EVENT_NAME LIKE 'memory/performance_schema/%';

+-----------------------+
| total_perf_schema_mem |
+-----------------------+
| 713.81 MiB            |
+-----------------------+

-- Non-P_S memory tracked:
SELECT sys.format_bytes(SUM(CURRENT_NUMBER_OF_BYTES_USED)) AS total_mysql_mem
FROM performance_schema.memory_summary_global_by_event_name;
+-----------------+
| total_mysql_mem |
+-----------------+
| 2.76 GiB        |  -- P_S instruments not fully enabled, InnoDB not tracked
+-----------------+

Tablespace Sizes

SELECT NAME, SPACE, ROUND(FILE_SIZE/1024/1024, 0) AS file_mb
FROM information_schema.INNODB_SYS_TABLESPACES
ORDER BY FILE_SIZE DESC LIMIT 5;

+----------------------------+-------+---------+
| NAME                       | SPACE | file_mb |
+----------------------------+-------+---------+
| warehouse/inventory_items  |     7 |   66112 |  -- 64.5GB
| warehouse/fulfilment_items |     6 |   37248 |  -- 36.4GB
+----------------------------+-------+---------+
-- Two tables alone = 100GB, larger than 52GB buffer pool

smaps Analysis

# Total RSS breakdown
cat /proc/$(pidof mysqld)/smaps_rollup
Rss:            79105456 kB   (~75.4 GB)
Anonymous:      79097912 kB   (~75.4 GB — almost entirely anonymous)
AnonHugePages:  ~34 GB        (THP promoted, cosmetic not causal)

# Memory by vmflag classification
python3 smaps_classifier.py

=== Complete Memory Classification ===
glibc_arena_dirty (ac+sd)  virt=56621MB  count=1    RSS=56,290 MB  ← Buffer pool
innodb_mmap (nr+sd)        virt=64MB     count=151  RSS= 8,870 MB  ← Unknown
innodb_mmap (nr+sd)        virt=63MB     count=144  RSS= 8,374 MB  ← Unknown
innodb_mmap (nr+sd)        virt=127MB    count=9    RSS=   861 MB
innodb_mmap (nr+sd)        virt=128MB    count=5    RSS=   493 MB
glibc heap  (ac+sd)        virt=422MB    count=1    RSS=   414 MB
glibc heap  (ac+sd)        virt=328MB    count=1    RSS=   328 MB
innodb_mmap (nr+sd)        virt=191MB    count=1    RSS=   169 MB
innodb_mmap (nr+sd)        virt=136MB    count=1    RSS=   136 MB
glibc heap  (ac+sd)        virt=132MB    count=1    RSS=   132 MB
glibc heap  (ac+sd)        virt=131MB    count=1    RSS=   131 MB
+ many smaller regions

Total RSS shown: 77,238 MB  (~75.4 GB) ✓ matches VmRSS

# Size distribution of nr+sd regions
64MB  × 151 regions  =  9,664 MB virtual,  8,870 MB RSS
63MB  × 144 regions  =  9,072 MB virtual,  8,374 MB RSS
127MB ×   9 regions  =  1,143 MB virtual,    861 MB RSS
128MB ×   5 regions  =    640 MB virtual,    493 MB RSS
+ smaller             =  3,197 MB
─────────────────────────────────────────
Total nr+sd:           23,616 MB virtual, 19,356 MB RSS

# Math
23,616 MB / 32 instances = 738 MB overhead per instance
738 MB / 128 MB chunk   =  5.8 MB per chunk
23,616 MB / 53,248 MB   =  0.444 ratio to buffer pool size

The Core Question for the Forum

We have 542 total nr+sd regions consuming 19,356 MB RSS that are clearly InnoDB internal mmap allocations (MAP_NORESERVE signature), but we cannot identify which specific InnoDB subsystem allocates them.

Key observations:

285 dominant regions of exactly 63-64MB virtual size = 18,099 MB
285 / 32 instances = 8.9  (not a clean number)
285 / 416 chunks   = 0.68 (not a clean number)
Total overhead / buffer pool = 0.444 ratio
Overhead per instance = 738 MB
Overhead per chunk    = 56.7 MB

Specific questions:

  1. How can I figure out where is 19GB consumed?
  2. Which InnoDB subsystem in Percona 5.7 allocates memory via mmap with MAP_NORESERVE in ~64MB chunks?
  3. Is a 44% overhead ratio above innodb_buffer_pool_size expected for 32 instances with 128MB chunk size?
  4. Would reducing innodb_buffer_pool_instances from 32 to 8 reduce this overhead proportionally, or is it per-page (variable) rather than per-instance (fixed)?
  5. Is there a way to identify these regions without debug symbols on a live production server?

Additional Context

  • mysqld is using plain glibc malloc (no jemalloc, no tcmalloc)
  • THP is set to always (known Percona anti-recommendation)
  • Buffer pool is 99% utilized — cannot reduce buffer pool size without impacting hit rate
2

Hi @sachin_japate,
Check a few more sys tables:

sys.memory_by_thread_by_current_bytes
sys.memory_global_by_current_bytes
sys.memory_global_total
sys.user_summary

Remember that in 5.7, tmp_table_size is a per-table limitation. 200 connections, each creating a 400MB temporary table would need to allocate 78GB of additional memory. Temp tables are not deallocated until after the connection closes. Ensure that connections are properly closing and that you are not using any connection pooling.

And of course, 5.7 is incredibly dead software. Even MySQL 8 is now dead. You should make effort to upgrade to 8.4, which has bugs fixed, security issues patched, and a new temp table engine.