# MySQL 5.7 (Percona) — InnoDB RSS 75GB on 78GB server, 19GB unaccounted nr+sd mmap regions **URL:** \ **Category:** Uncategorized\ **Created:** [June 27, 2026, 7:00am UTC](https://forums.percona.com/t/mysql-5-7-percona-innodb-rss-75gb-on-78gb-server-19gb-unaccounted-nr-sd-mmap-regions/40986 "2026-06-27T07:00:33Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![sachin\_japate](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sachin_japate/32/5668_2.png) [@sachin\_japate](https://forums.percona.com/u/sachin_japate)\ **Post date:** [June 27, 2026, 7:00am UTC](https://forums.percona.com/t/mysql-5-7-percona-innodb-rss-75gb-on-78gb-server-19gb-unaccounted-nr-sd-mmap-regions/40986/1 "2026-06-27T07:00:33Z")
### Environment ```auto OS: Debian GNU/Linux 11 (bullseye) Kernel: Linux (glibc 2.31-13+deb11u7) CPU cores: 16 Total RAM: 78 GB Swap: None MySQL: Percona Server 5.7.39-42-log Uptime: ~298 days ``` ### Problem Statement mysqld RSS is 75.4GB on a 78GB server with only 69MB available memory and no swap configured. After accounting for the configured `innodb_buffer_pool_size` of 52GB, approximately 19GB remains unaccounted for in InnoDB internal mmap regions with `MAP_NORESERVE` flag (`nr+sd` in smaps VmFlags). ### System Variables ```auto SHOW VARIABLES LIKE 'innodb_buffer_pool_size'; +-------------------------+-------------+ | innodb_buffer_pool_size | 55834574848 | -- 52GB configured +-------------------------+-------------+ SHOW VARIABLES LIKE 'innodb_buffer_pool_instances'; +------------------------------+-------+ | innodb_buffer_pool_instances | 32 | +------------------------------+-------+ SHOW VARIABLES LIKE 'innodb_buffer_pool_chunk_size'; +-------------------------------+-----------+ | innodb_buffer_pool_chunk_size | 134217728 | -- 128MB +-------------------------------+-----------+ SELECT @@innodb_buffer_pool_size / @@innodb_buffer_pool_chunk_size AS total_chunks, @@innodb_buffer_pool_size / @@innodb_buffer_pool_instances / @@innodb_buffer_pool_chunk_size AS chunks_per_instance; +--------------+---------------------+ | total_chunks | chunks_per_instance | +--------------+---------------------+ | 416.0000 | 13.00000000 | +--------------+---------------------+ SHOW VARIABLES LIKE 'innodb_adaptive_hash_index%'; +----------------------------------+-------+ | innodb_adaptive_hash_index | OFF | | innodb_adaptive_hash_index_parts | 8 | +----------------------------------+-------+ SHOW VARIABLES LIKE 'innodb_undo%'; +--------------------------+-------+ | innodb_undo_directory | ./ | | innodb_undo_log_encrypt | OFF | | innodb_undo_logs | 128 | | innodb_undo_tablespaces | 0 | +--------------------------+-------+ SHOW VARIABLES LIKE 'innodb_purge_threads'; +----------------------+-------+ | innodb_purge_threads | 4 | +----------------------+-------+ SHOW VARIABLES LIKE 'innodb_read_io_threads'; +------------------------+-------+ | innodb_read_io_threads | 8 | +------------------------+-------+ SHOW VARIABLES LIKE 'innodb_write_io_threads'; +-------------------------+-------+ | innodb_write_io_threads | 8 | +-------------------------+-------+ SHOW VARIABLES LIKE 'tmp_table_size'; +----------------+-----------+ | tmp_table_size | 536870912 | -- 512MB +----------------+-----------+ SHOW VARIABLES LIKE 'max_heap_table_size'; +---------------------+-----------+ | max_heap_table_size | 536870912 | -- 512MB +---------------------+-----------+ SHOW VARIABLES LIKE 'max_connections'; +-----------------+-------+ | max_connections | 5000 | +-----------------+-------+ SHOW VARIABLES LIKE 'thread_stack'; +---------------+--------+ | thread_stack | 262144 | -- 256KB +---------------+--------+ SHOW VARIABLES LIKE 'innodb_log_buffer_size'; +------------------------+-----------+ | innodb_log_buffer_size | 134217728 | -- 128MB +------------------------+-----------+ SHOW VARIABLES LIKE '%buffer_size%'; +-------------------------+-----------+ | bulk_insert_buffer_size | 8388608 | | innodb_log_buffer_size | 134217728 | | innodb_sort_buffer_size | 1048576 | | join_buffer_size | 262144 | | key_buffer_size | 8388608 | | myisam_sort_buffer_size | 8388608 | | read_buffer_size | 131072 | | read_rnd_buffer_size | 262144 | | sort_buffer_size | 262144 | +-------------------------+-----------+ SHOW VARIABLES LIKE 'performance_schema'; +--------------------+-------+ | performance_schema | ON | +--------------------+-------+ ``` ### OS Settings ```auto # Memory free -mh total used free shared buff/cache available Mem: 78Gi 77Gi 484Mi 0.0Ki 266Mi 69Mi Swap: 0B 0B 0B # THP setting cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never # mysqld RSS cat /proc/$(pidof mysqld)/status | grep VmRSS VmRSS: 79060188 kB (~75.4 GB) # glibc version ldd --version | head -1 ldd (Debian GLIBC 2.31-13+deb11u7) 2.31 # jemalloc ldd $(which mysqld) | grep -E 'jemalloc|malloc|tcmalloc' (no output — mysqld using plain glibc malloc) # CPU cores nproc 16 # Uptime mysqladmin status | grep Uptime Uptime: 25744293 Threads: 131 Questions: 6211793746 Max_used_connections: 475 ``` ### Buffer Pool Status ```auto SELECT SUM(POOL_SIZE) AS total_pages, SUM(FREE_BUFFERS) AS free_pages, SUM(DATABASE_PAGES) AS used_pages, SUM(FREE_BUFFERS)*100.0/SUM(POOL_SIZE) AS free_pct, SUM(POOL_SIZE)*16384/1024/1024/1024 AS total_gb FROM information_schema.INNODB_BUFFER_POOL_STATS; +-------------+------------+------------+----------+-----------------+ | total_pages | free_pages | used_pages | free_pct | total_gb | +-------------+------------+------------+----------+-----------------+ | 3407456 | 32767 | 3374689 | 0.96163 | 51.993652343750 | +-------------+------------+------------+----------+-----------------+ -- Buffer pool is 99% utilized -- Per instance (all 32 identical): POOL_SIZE: 106483 pages = 1664MB per instance FREE_BUFFERS: 1024 pages = 16MB free per instance ``` ### InnoDB Engine Status ```auto BUFFER POOL AND MEMORY ---------------------- Total large memory allocated 58124664832 Dictionary memory allocated 20552198 Internal hash tables (constant factor + variable factor) Adaptive hash index 883923648 (883889984 + 33664) Page hash 1727144 (buffer pool 0 only) Dictionary cache 241524694 (220972496 + 20552198) File system 1894792 (812272 + 1082520) Lock system 143943400 (143819576 + 123824) Recovery system 0 (0 + 0) Buffer pool size 3407456 Buffer pool size, bytes 55827759104 Free buffers 32768 Database pages 3374688 Old database pages 1245088 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 1189086697, not young 12182634025 0.00 youngs/s, 0.00 non-youngs/s Pages read 109154374, created 92663, written 154845768 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 999 / 1000, young-making rate 11 / 1000 not 3 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 3374688, unzip_LRU len: 0 I/O sum[13152]:cur[32], unzip sum[0]:cur[0] ---------------------- ``` ### Performance Schema Memory ```auto SELECT sys.format_bytes(SUM(CURRENT_NUMBER_OF_BYTES_USED)) AS total_perf_schema_mem FROM performance_schema.memory_summary_global_by_event_name WHERE EVENT_NAME LIKE 'memory/performance_schema/%'; +-----------------------+ | total_perf_schema_mem | +-----------------------+ | 713.81 MiB | +-----------------------+ -- Non-P_S memory tracked: SELECT sys.format_bytes(SUM(CURRENT_NUMBER_OF_BYTES_USED)) AS total_mysql_mem FROM performance_schema.memory_summary_global_by_event_name; +-----------------+ | total_mysql_mem | +-----------------+ | 2.76 GiB | -- P_S instruments not fully enabled, InnoDB not tracked +-----------------+ ``` ### Tablespace Sizes ```auto SELECT NAME, SPACE, ROUND(FILE_SIZE/1024/1024, 0) AS file_mb FROM information_schema.INNODB_SYS_TABLESPACES ORDER BY FILE_SIZE DESC LIMIT 5; +----------------------------+-------+---------+ | NAME | SPACE | file_mb | +----------------------------+-------+---------+ | warehouse/inventory_items | 7 | 66112 | -- 64.5GB | warehouse/fulfilment_items | 6 | 37248 | -- 36.4GB +----------------------------+-------+---------+ -- Two tables alone = 100GB, larger than 52GB buffer pool ``` ### smaps Analysis ```auto # Total RSS breakdown cat /proc/$(pidof mysqld)/smaps_rollup Rss: 79105456 kB (~75.4 GB) Anonymous: 79097912 kB (~75.4 GB — almost entirely anonymous) AnonHugePages: ~34 GB (THP promoted, cosmetic not causal) # Memory by vmflag classification python3 smaps_classifier.py === Complete Memory Classification === glibc_arena_dirty (ac+sd) virt=56621MB count=1 RSS=56,290 MB ← Buffer pool innodb_mmap (nr+sd) virt=64MB count=151 RSS= 8,870 MB ← Unknown innodb_mmap (nr+sd) virt=63MB count=144 RSS= 8,374 MB ← Unknown innodb_mmap (nr+sd) virt=127MB count=9 RSS= 861 MB innodb_mmap (nr+sd) virt=128MB count=5 RSS= 493 MB glibc heap (ac+sd) virt=422MB count=1 RSS= 414 MB glibc heap (ac+sd) virt=328MB count=1 RSS= 328 MB innodb_mmap (nr+sd) virt=191MB count=1 RSS= 169 MB innodb_mmap (nr+sd) virt=136MB count=1 RSS= 136 MB glibc heap (ac+sd) virt=132MB count=1 RSS= 132 MB glibc heap (ac+sd) virt=131MB count=1 RSS= 131 MB + many smaller regions Total RSS shown: 77,238 MB (~75.4 GB) ✓ matches VmRSS # Size distribution of nr+sd regions 64MB × 151 regions = 9,664 MB virtual, 8,870 MB RSS 63MB × 144 regions = 9,072 MB virtual, 8,374 MB RSS 127MB × 9 regions = 1,143 MB virtual, 861 MB RSS 128MB × 5 regions = 640 MB virtual, 493 MB RSS + smaller = 3,197 MB ───────────────────────────────────────── Total nr+sd: 23,616 MB virtual, 19,356 MB RSS # Math 23,616 MB / 32 instances = 738 MB overhead per instance 738 MB / 128 MB chunk = 5.8 MB per chunk 23,616 MB / 53,248 MB = 0.444 ratio to buffer pool size ``` ### The Core Question for the Forum We have **542 total `nr+sd` regions** consuming **19,356 MB RSS** that are clearly InnoDB internal mmap allocations (MAP\_NORESERVE signature), but we cannot identify which specific InnoDB subsystem allocates them. Key observations: ```auto 285 dominant regions of exactly 63-64MB virtual size = 18,099 MB 285 / 32 instances = 8.9 (not a clean number) 285 / 416 chunks = 0.68 (not a clean number) Total overhead / buffer pool = 0.444 ratio Overhead per instance = 738 MB Overhead per chunk = 56.7 MB ``` **Specific questions:** 1. How can I figure out where is 19GB consumed? 2. Which InnoDB subsystem in Percona 5.7 allocates memory via `mmap` with `MAP_NORESERVE` in ~64MB chunks? 3. Is a 44% overhead ratio above `innodb_buffer_pool_size` expected for 32 instances with 128MB chunk size? 4. Would reducing `innodb_buffer_pool_instances` from 32 to 8 reduce this overhead proportionally, or is it per-page (variable) rather than per-instance (fixed)? 5. Is there a way to identify these regions without debug symbols on a live production server? ### Additional Context - mysqld is using plain **glibc malloc** (no jemalloc, no tcmalloc) - THP is set to `always` (known Percona anti-recommendation) - Buffer pool is **99% utilized** — cannot reduce buffer pool size without impacting hit rate ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [July 15, 2026, 1:15pm UTC](https://forums.percona.com/t/mysql-5-7-percona-innodb-rss-75gb-on-78gb-server-19gb-unaccounted-nr-sd-mmap-regions/40986/2 "2026-07-15T13:15:35Z")
Hi @sachin_japate, Check a few more sys tables: ``` sys.memory_by_thread_by_current_bytes sys.memory_global_by_current_bytes sys.memory_global_total sys.user_summary ``` Remember that in 5.7, tmp\_table\_size is a per-table limitation. 200 connections, each creating a 400MB temporary table would need to allocate 78GB of additional memory. Temp tables are not deallocated until after the connection closes. Ensure that connections are properly closing and that you are not using any connection pooling. And of course, 5.7 is incredibly dead software. Even MySQL 8 is now dead. You should make effort to upgrade to 8.4, which has bugs fixed, security issues patched, and a new temp table engine. ---
**Author:** ![sachin\_japate](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/sachin_japate/32/5668_2.png) [@sachin\_japate](https://forums.percona.com/u/sachin_japate)\ **Post date:** [July 28, 2026, 3:02am UTC](https://forums.percona.com/t/mysql-5-7-percona-innodb-rss-75gb-on-78gb-server-19gb-unaccounted-nr-sd-mmap-regions/40986/3 "2026-07-28T03:02:59Z")
Hey @matthewb , Thanks for the suggestion. Based on our analysis it turned out to be caused by - **glibc arena retention** : freed memory that glibc’s 128-arena allocator held in per-arena free lists rather than returning to the OS. This was proven by a live `malloc_trim` experiment that reclaimed ~15 GB from allocator heaps while leaving the InnoDB buffer pool provably untouched. `malloc_trim` proved to be only a temporary band-aid — the arenas re-grew under load within hours. We then switched the mysqld process to use **jemalloc,** a **dirty-page decay** mechanism of jemalloc: freed pages are returned to the OS on a timer (default 10 s) via `madvise`. Where glibc hoards freed memory in per-arena free lists indefinitely, jemalloc continuously reclaims it — structurally preventing the retention. jemalloc’s decay-based return has one catch: **by default it is _lazy_ — it only runs on an arena that is actively being allocated from.** Mechanically: on free(), pages become **“dirty”** (freed, still resident, kept briefly for cheap reuse). jemalloc purges aged dirty pages by running its decay routine — but that routine is triggered by **allocation activity on that arena**. An arena that goes **idle** stops triggering it, so its dirty pages sit resident **forever** , long past the decay deadline. The timer says “purge,” but nothing checks the timer. The **background\_thread** is a dedicated thread that runs the decay routine on **all** arenas — including idle ones — on a schedule. It is the _only_ thing that purges quiet arenas. It is **off by default** (opt-in), and, importantly, **enabling it via MALLOC\_CONF does not reliably spawn it on glibc-based systems** (jemalloc initializes before pthread\_create is usable). The before/after below ( **dim-2** ) is the clearest illustration of what the background thread does: the same allocator, same live working set, with the thread off vs on.