Environment
OS: Debian GNU/Linux 11 (bullseye)
Kernel: Linux (glibc 2.31-13+deb11u7)
CPU cores: 16
Total RAM: 78 GB
Swap: None
MySQL: Percona Server 5.7.39-42-log
Uptime: ~298 days
Problem Statement
mysqld RSS is 75.4GB on a 78GB server with only 69MB available memory and no swap configured. After accounting for the configured innodb_buffer_pool_size of 52GB, approximately 19GB remains unaccounted for in InnoDB internal mmap regions with MAP_NORESERVE flag (nr+sd in smaps VmFlags).
System Variables
SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
+-------------------------+-------------+
| innodb_buffer_pool_size | 55834574848 | -- 52GB configured
+-------------------------+-------------+
SHOW VARIABLES LIKE 'innodb_buffer_pool_instances';
+------------------------------+-------+
| innodb_buffer_pool_instances | 32 |
+------------------------------+-------+
SHOW VARIABLES LIKE 'innodb_buffer_pool_chunk_size';
+-------------------------------+-----------+
| innodb_buffer_pool_chunk_size | 134217728 | -- 128MB
+-------------------------------+-----------+
SELECT
@@innodb_buffer_pool_size / @@innodb_buffer_pool_chunk_size AS total_chunks,
@@innodb_buffer_pool_size / @@innodb_buffer_pool_instances /
@@innodb_buffer_pool_chunk_size AS chunks_per_instance;
+--------------+---------------------+
| total_chunks | chunks_per_instance |
+--------------+---------------------+
| 416.0000 | 13.00000000 |
+--------------+---------------------+
SHOW VARIABLES LIKE 'innodb_adaptive_hash_index%';
+----------------------------------+-------+
| innodb_adaptive_hash_index | OFF |
| innodb_adaptive_hash_index_parts | 8 |
+----------------------------------+-------+
SHOW VARIABLES LIKE 'innodb_undo%';
+--------------------------+-------+
| innodb_undo_directory | ./ |
| innodb_undo_log_encrypt | OFF |
| innodb_undo_logs | 128 |
| innodb_undo_tablespaces | 0 |
+--------------------------+-------+
SHOW VARIABLES LIKE 'innodb_purge_threads';
+----------------------+-------+
| innodb_purge_threads | 4 |
+----------------------+-------+
SHOW VARIABLES LIKE 'innodb_read_io_threads';
+------------------------+-------+
| innodb_read_io_threads | 8 |
+------------------------+-------+
SHOW VARIABLES LIKE 'innodb_write_io_threads';
+-------------------------+-------+
| innodb_write_io_threads | 8 |
+-------------------------+-------+
SHOW VARIABLES LIKE 'tmp_table_size';
+----------------+-----------+
| tmp_table_size | 536870912 | -- 512MB
+----------------+-----------+
SHOW VARIABLES LIKE 'max_heap_table_size';
+---------------------+-----------+
| max_heap_table_size | 536870912 | -- 512MB
+---------------------+-----------+
SHOW VARIABLES LIKE 'max_connections';
+-----------------+-------+
| max_connections | 5000 |
+-----------------+-------+
SHOW VARIABLES LIKE 'thread_stack';
+---------------+--------+
| thread_stack | 262144 | -- 256KB
+---------------+--------+
SHOW VARIABLES LIKE 'innodb_log_buffer_size';
+------------------------+-----------+
| innodb_log_buffer_size | 134217728 | -- 128MB
+------------------------+-----------+
SHOW VARIABLES LIKE '%buffer_size%';
+-------------------------+-----------+
| bulk_insert_buffer_size | 8388608 |
| innodb_log_buffer_size | 134217728 |
| innodb_sort_buffer_size | 1048576 |
| join_buffer_size | 262144 |
| key_buffer_size | 8388608 |
| myisam_sort_buffer_size | 8388608 |
| read_buffer_size | 131072 |
| read_rnd_buffer_size | 262144 |
| sort_buffer_size | 262144 |
+-------------------------+-----------+
SHOW VARIABLES LIKE 'performance_schema';
+--------------------+-------+
| performance_schema | ON |
+--------------------+-------+
OS Settings
# Memory
free -mh
total used free shared buff/cache available
Mem: 78Gi 77Gi 484Mi 0.0Ki 266Mi 69Mi
Swap: 0B 0B 0B
# THP setting
cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
# mysqld RSS
cat /proc/$(pidof mysqld)/status | grep VmRSS
VmRSS: 79060188 kB (~75.4 GB)
# glibc version
ldd --version | head -1
ldd (Debian GLIBC 2.31-13+deb11u7) 2.31
# jemalloc
ldd $(which mysqld) | grep -E 'jemalloc|malloc|tcmalloc'
(no output — mysqld using plain glibc malloc)
# CPU cores
nproc
16
# Uptime
mysqladmin status | grep Uptime
Uptime: 25744293 Threads: 131 Questions: 6211793746
Max_used_connections: 475
Buffer Pool Status
SELECT
SUM(POOL_SIZE) AS total_pages,
SUM(FREE_BUFFERS) AS free_pages,
SUM(DATABASE_PAGES) AS used_pages,
SUM(FREE_BUFFERS)*100.0/SUM(POOL_SIZE) AS free_pct,
SUM(POOL_SIZE)*16384/1024/1024/1024 AS total_gb
FROM information_schema.INNODB_BUFFER_POOL_STATS;
+-------------+------------+------------+----------+-----------------+
| total_pages | free_pages | used_pages | free_pct | total_gb |
+-------------+------------+------------+----------+-----------------+
| 3407456 | 32767 | 3374689 | 0.96163 | 51.993652343750 |
+-------------+------------+------------+----------+-----------------+
-- Buffer pool is 99% utilized
-- Per instance (all 32 identical):
POOL_SIZE: 106483 pages = 1664MB per instance
FREE_BUFFERS: 1024 pages = 16MB free per instance
InnoDB Engine Status
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 58124664832
Dictionary memory allocated 20552198
Internal hash tables (constant factor + variable factor)
Adaptive hash index 883923648 (883889984 + 33664)
Page hash 1727144 (buffer pool 0 only)
Dictionary cache 241524694 (220972496 + 20552198)
File system 1894792 (812272 + 1082520)
Lock system 143943400 (143819576 + 123824)
Recovery system 0 (0 + 0)
Buffer pool size 3407456
Buffer pool size, bytes 55827759104
Free buffers 32768
Database pages 3374688
Old database pages 1245088
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1189086697, not young 12182634025
0.00 youngs/s, 0.00 non-youngs/s
Pages read 109154374, created 92663, written 154845768
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 999 / 1000, young-making rate 11 / 1000 not 3 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3374688, unzip_LRU len: 0
I/O sum[13152]:cur[32], unzip sum[0]:cur[0]
----------------------
Performance Schema Memory
SELECT sys.format_bytes(SUM(CURRENT_NUMBER_OF_BYTES_USED)) AS total_perf_schema_mem
FROM performance_schema.memory_summary_global_by_event_name
WHERE EVENT_NAME LIKE 'memory/performance_schema/%';
+-----------------------+
| total_perf_schema_mem |
+-----------------------+
| 713.81 MiB |
+-----------------------+
-- Non-P_S memory tracked:
SELECT sys.format_bytes(SUM(CURRENT_NUMBER_OF_BYTES_USED)) AS total_mysql_mem
FROM performance_schema.memory_summary_global_by_event_name;
+-----------------+
| total_mysql_mem |
+-----------------+
| 2.76 GiB | -- P_S instruments not fully enabled, InnoDB not tracked
+-----------------+
Tablespace Sizes
SELECT NAME, SPACE, ROUND(FILE_SIZE/1024/1024, 0) AS file_mb
FROM information_schema.INNODB_SYS_TABLESPACES
ORDER BY FILE_SIZE DESC LIMIT 5;
+----------------------------+-------+---------+
| NAME | SPACE | file_mb |
+----------------------------+-------+---------+
| warehouse/inventory_items | 7 | 66112 | -- 64.5GB
| warehouse/fulfilment_items | 6 | 37248 | -- 36.4GB
+----------------------------+-------+---------+
-- Two tables alone = 100GB, larger than 52GB buffer pool
smaps Analysis
# Total RSS breakdown
cat /proc/$(pidof mysqld)/smaps_rollup
Rss: 79105456 kB (~75.4 GB)
Anonymous: 79097912 kB (~75.4 GB — almost entirely anonymous)
AnonHugePages: ~34 GB (THP promoted, cosmetic not causal)
# Memory by vmflag classification
python3 smaps_classifier.py
=== Complete Memory Classification ===
glibc_arena_dirty (ac+sd) virt=56621MB count=1 RSS=56,290 MB ← Buffer pool
innodb_mmap (nr+sd) virt=64MB count=151 RSS= 8,870 MB ← Unknown
innodb_mmap (nr+sd) virt=63MB count=144 RSS= 8,374 MB ← Unknown
innodb_mmap (nr+sd) virt=127MB count=9 RSS= 861 MB
innodb_mmap (nr+sd) virt=128MB count=5 RSS= 493 MB
glibc heap (ac+sd) virt=422MB count=1 RSS= 414 MB
glibc heap (ac+sd) virt=328MB count=1 RSS= 328 MB
innodb_mmap (nr+sd) virt=191MB count=1 RSS= 169 MB
innodb_mmap (nr+sd) virt=136MB count=1 RSS= 136 MB
glibc heap (ac+sd) virt=132MB count=1 RSS= 132 MB
glibc heap (ac+sd) virt=131MB count=1 RSS= 131 MB
+ many smaller regions
Total RSS shown: 77,238 MB (~75.4 GB) ✓ matches VmRSS
# Size distribution of nr+sd regions
64MB × 151 regions = 9,664 MB virtual, 8,870 MB RSS
63MB × 144 regions = 9,072 MB virtual, 8,374 MB RSS
127MB × 9 regions = 1,143 MB virtual, 861 MB RSS
128MB × 5 regions = 640 MB virtual, 493 MB RSS
+ smaller = 3,197 MB
─────────────────────────────────────────
Total nr+sd: 23,616 MB virtual, 19,356 MB RSS
# Math
23,616 MB / 32 instances = 738 MB overhead per instance
738 MB / 128 MB chunk = 5.8 MB per chunk
23,616 MB / 53,248 MB = 0.444 ratio to buffer pool size
The Core Question for the Forum
We have 542 total nr+sd regions consuming 19,356 MB RSS that are clearly InnoDB internal mmap allocations (MAP_NORESERVE signature), but we cannot identify which specific InnoDB subsystem allocates them.
Key observations:
285 dominant regions of exactly 63-64MB virtual size = 18,099 MB
285 / 32 instances = 8.9 (not a clean number)
285 / 416 chunks = 0.68 (not a clean number)
Total overhead / buffer pool = 0.444 ratio
Overhead per instance = 738 MB
Overhead per chunk = 56.7 MB
Specific questions:
- How can I figure out where is 19GB consumed?
- Which InnoDB subsystem in Percona 5.7 allocates memory via
mmapwithMAP_NORESERVEin ~64MB chunks? - Is a 44% overhead ratio above
innodb_buffer_pool_sizeexpected for 32 instances with 128MB chunk size? - Would reducing
innodb_buffer_pool_instancesfrom 32 to 8 reduce this overhead proportionally, or is it per-page (variable) rather than per-instance (fixed)? - Is there a way to identify these regions without debug symbols on a live production server?
Additional Context
- mysqld is using plain glibc malloc (no jemalloc, no tcmalloc)
- THP is set to
always(known Percona anti-recommendation) - Buffer pool is 99% utilized — cannot reduce buffer pool size without impacting hit rate