I was trying to make MySQL backup using xtrabackup and I kept getting following error:
7f9c33d0a720 InnoDB: Operating system error number 24 in a file operation.
InnoDB: Error number 24 means ‘Too many open files’.
InnoDB: Some operating system error numbers are described at
InnoDB: Error: could not open single-table tablespace file ./optiweber2/top_keyword_urls#P#p61.ibd
InnoDB: We do not continue the crash recovery, because the table may become
InnoDB: corrupt if we cannot apply the log records in the InnoDB log to it.
InnoDB: To fix the problem and start mysqld:
InnoDB: 1) If there is a permission problem in the file and mysqld cannot
InnoDB: open the file, you should modify the permissions.
InnoDB: 2) If the table is not needed, or you can restore it from a backup,
InnoDB: then you can remove the .ibd file, and InnoDB will do a normal
InnoDB: crash recovery and ignore that table.
InnoDB: 3) If the file system or the disk is broken, and you cannot remove
InnoDB: the .ibd file, you can set innodb_force_recovery > 0 in my.cnf
InnoDB: and force InnoDB to continue crash recovery here.
innobackupex: Error: The xtrabackup child process has died at /usr/bin/innobackupex line 2672.
However, the issue is resolved when, I simply increase the ‘open-files-limit’ in my.cnf from
open-files-limit = 131072
open-files-limit = 151072
Please note, MySQL has not picked up these changes, its merely editing my.cnf file.
And, my systems open file hard limit is set to 131072 as well.
Excerpt of /etc/security/limits.conf
- soft nofile 131072
- hard nofile 131072
~ # ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 118674
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 131072
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 118674
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
The solution did not make sense. Posting it here to get some insight.