I have similar thing with 5.7. The binary logs are being cached in the RAM, and then naturally in swap. Tuning helped a bit (which you already have done):
vm.swappiness = 1
As well as keeping buffer pool below max memory limit to allow some space in RAM for binary logs.
However, I didn’t manage to address the issue completely. Namely, I couldn’t find a way to make mysql push away the logs from the RAM that are older than certain age or make it reclaim memory used by older logs as new logs arrive (i.e. restrict amount of RAM allowed for binlog use). A couple of things that can be used as a workaround:
- reduce amount of logs stored (expire_logs_days)
- clear system caches: sync; echo 3 > /proc/sys/vm/drop_caches
Another thing that comes to my mind is performance_schema, but it wasn’t hogging on RAM until 5.7, I think.