Version:5.1.60-12.5
Currently have a system that is using up almost 50G of memory usage, we are more or less expecting it to take up closer to 40G.
I have calculated what I can from SHOW ENGINE INNODB STATUS and the global variables and status output, but cant seem to find where the memory went.
It seemed to rise up gradually over at least a week (from the output of SAR)
I have gathered pt-summary,pt-mysql-summary, meminfo, and innodb output for the purposes of diagnostics.
Here is the output of meminfo
$ cat /proc/meminfo
MemTotal: 49447676 kB
MemFree: 646504 kB
Buffers: 271340 kB
Cached: 742204 kB
SwapCached: 2923228 kB
Active: 47088424 kB
Inactive: 1432660 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 49447676 kB
LowFree: 646504 kB
SwapTotal: 8388600 kB
SwapFree: 343988 kB
Dirty: 108 kB
Writeback: 0 kB
AnonPages: 45193080 kB
Mapped: 23292 kB
Slab: 116260 kB
PageTables: 113000 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 33112436 kB
Committed_AS: 59970984 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 268736 kB
VmallocChunk: 34359469315 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
Here is the ps output:
mysql 5346 8.9 91.3 53300120 45152588 ? Sl Jan27 7987:51 /usr/local/mysql-5.1.60/sbin/mysqld --defaults-extra-file=/etc/my-3306.cnf --basedir=/usr/local/mysql --datadir=/var/lib/mysql_data/3306 --user=mysql --log-error=/var/lib/mysql_logs/3306/error.log --open-files-limit=8192 --pid-file=/var/run/mysql/3306.pid --socket=/var/run/mysql/3306.sock --port=3306
Shows us as using almost 50G. However Innodb is only using
Total memory allocated 39655047168; in additional pool allocated 0
Internal hash tables (constant factor + variable factor)
Adaptive hash index 617273808 (611993944 + 5279864)
Page hash 38250584
Dictionary cache 420153435 (152999984 + 267153451)
File system 12351904 (82672 + 12269232)
Lock system 95636480 (95624824 + 11656)
Recovery system 0 (0 + 0)
Threads 411640 (406936 + 4704)
Dictionary memory allocated 267153451
Buffer pool size 2359295
Buffer pool size, bytes 38654689280
Free buffers 1
Database pages 3922659
Old database pages 1447992
Modified db pages 49
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 45206245, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 43531033, created 27926938, written 129329407
0.00 reads/s, 0.00 creates/s, 11.45 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 3922659, unzip_LRU len: 392265
I/O sum[1485]:cur[0], unzip sum[38]:cur[0]
There seems to be a wide gap there in whats currently being used and what mysql thinks its using.
We have no MyISAM except for the mysql.* of course.
ODD SIDE NOTE: Most of our innodb tables are using compression@8k.
I have gathered up plenty of other diagnostic information but not sure what else to post.
Can someone point me in the right direction of where to look ?