Mysql Master eat up swap space and doesn't release it ... why?

MySQL & MariaDB Other MySQL® Questions
1

I am running a mysql 5.5 master
on a dedicated mysql master host.
it has 26Gb swap space and now alert on it’s 93% used
it was 24GB a week ago and I added a tmp file a 2gb and now it’s eaten up as well.
Swapoff/on doesn’t work cause it isn’t enough free mem avaliable.
What can I do ? ( besides reboot the master and wait until it all chewed up again )
What causes this ? bug in 5.5 ? memory leak ?
thx

swapoff -a; sleep 2; swapon -a
swapoff: /dev/sda3: swapoff failed: Cannot allocate memory

| innodb_buffer_pool_size | 30064771072

cat /proc/sys/vm/swappiness
1

cat /proc/sys/vm/dirty_ratio
20

RAM_ Size HD_Size HD_Used HD_Free Used% Speed Cache HT MySQL OSbits Linux_Version Model
49408748 689G 307G 348G 47% 2.40GHz 12288 16 5.5.32 x86_64 CentOS6.9(Final) R610f

DB_Type SwapSize SwapUsed SwapFree Usage% Cpu_1m Cpu_5m Cpu15m
master 26869752 25055192 1814560 93.247 2.23 2.30 2.30

free -m
total used free shared buffers cached
Mem: 48250 47649 600 0 8 101
-/+ buffers/cache: 47539 710
Swap: 26239 24699 1540

./find-out-what-is-using-your-swap.sh
…
Overall swap used: 25019700 kB

kB pid name

24816896 7087 mysqld
145968 4636 messaging_local
28228 4311 puppet
7016 3226 snmpd
2444 30865 rsyslogd
2420 7746 nscd
2308 7789 nslcd

2

I have similar thing with 5.7. The binary logs are being cached in the RAM, and then naturally in swap. Tuning helped a bit (which you already have done):
vm.swappiness = 1

As well as keeping buffer pool below max memory limit to allow some space in RAM for binary logs.

However, I didn’t manage to address the issue completely. Namely, I couldn’t find a way to make mysql push away the logs from the RAM that are older than certain age or make it reclaim memory used by older logs as new logs arrive (i.e. restrict amount of RAM allowed for binlog use). A couple of things that can be used as a workaround:

  • reduce amount of logs stored (expire_logs_days)
  • clear system caches: sync; echo 3 > /proc/sys/vm/drop_caches

Another thing that comes to my mind is performance_schema, but it wasn’t hogging on RAM until 5.7, I think.