Disk usage unaccounted for with binlog files

1

This is kind of a head scratcher for me, so I’m hoping someone has some suggestions on where to look next to solve a mystery.

I have a Percona/MySQL system that ran out of disk space. The data folder is about 35GB, binlog space capped with binlog_space_limit at 15G, OS and all other system data takes up around 4-5GB. The system is using a 60GB disk. When I logged into the system df showed the drive as 100% used but there were only 11 full size (1G) binlog files. By calculation the system should have had nearly 10GB free. Maybe 8-9GB if the current binlog file was close to the 1GB max (it wasn’t). I shut down MySQL, changed binlog_space_limit to 3G and restarted MySQL. 9 binlog files were purged but over 14GB of disk space not showed as available.

Where I’m stuck is that it seems like there is some hidden usage associated with the binlog files that I hadn’t accounted for. I don’t mind if the usage is legitimate but I do need to be able to account for it so that we don’t run out of space again. Given the minimal changes I made, I figure it has to be something in the shutdown or in the binlog purge that freed up the extra space, I’m just out of ideas on what it could be.

2

@trevor-ottoconnect

9 binlog files were purged but over 14GB of disk space not showed as available.

It could be the case, file descriptor not closed yet. Please share the output of below command.

lsof | grep mysql | grep deleted

Did you tried restarting the database if it releases the disk or not ? Are any replica connected to this node ?

Can you share the binary log details (both from file or client) along with the Index file output ?

du -sh /var/lib/mysql/binlog.*
show binary logs;
cat /var/lib/mysql/binlog.index

Did you see similar behaviour with manual purging also ?
PURGE BINARY LOGS TO '<mysql.bin.001>’

Can you pls share the below set values as well ?

mysql> select @@expire_logs_days;
mysql> select @@binlog_expire_logs_seconds;