Good day all
We are running Percona server 5.7 on a server with 128Gb of Memory and 32Gb of Swap Space.
We have had a lot of issues with the memory filling up and using up swap space on a daily basis requiring the database to be restarted daily in order to release the memory.
After adding some graphs, monitoring and physical testing, we found that the problem seems to be related to mysqldump.
The memory usage is fairly stable , increasing little bits until innodb_buffer_pool is full, however the moment we start a backup, the memory usage spikes and uses up all physical memory as well as Swap.
The symptoms we found was that the backup uses up all memory and starts to swap, however once the backup completes, the memory is not released at all and the database server eventually runs out of memory and kills mysql.
We have tried removing the compression of the backups as well using the --quick option but to no avail.
The size of the databases on the server totals about 350Gb in total, however we run backups on a per database setup and the biggest database being ±130Gb in size.
Is there anything else we can perhaps look at in order to try and troubleshoot why the mysqldump processes are not releasing the memory?
Any suggestions would be greatly appreciated.