File name ./ib_logfile0 InnoDB: File operation call: 'open' returned OS error 71.

=== Problem solved. See last comment. ===

Like others, I have encountered an error with innobackupex stating cannot find path specified. I tried both version 2.1.7 and 2.0.8 with similar results. I am running MySQL 5.6.15 on RHEL 6.4. I have not used innobackupex on this machine before, although I have used it against 5.5 dbs on other machines.

2.1.7 reported

xtrabackup_56 version 2.1.7 for MySQL server 5.6.15 Linux (x86_64) (revision id: 721)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
2014-02-18 15:55:31 7f38abdd7720 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: File ./ib_logfile0: ‘open’ returned OS error 71. Cannot continue operation
innobackupex: Error: The xtrabackup child process has died at /usr/bin/innobackupex line 2648.

2.0.8 reported

xtrabackup_56 version 2.0.8 for MySQL server 5.6.10 Linux (x86_64) (revision id: 587)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
2014-02-18 15:59:33 7ffe84979720 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2014-02-18 15:59:33 7ffe84979720 InnoDB: File name ./ib_logfile0
2014-02-18 15:59:33 7ffe84979720 InnoDB: File operation call: ‘open’ returned OS error 71.
2014-02-18 15:59:33 7ffe84979720 InnoDB: Cannot continue operation.
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 386.

The /etc/my.cnf file

[mysqld56]

list temporary variables or settings here

log_warnings = 2

skip-slave-start
basedir = /usr/local/mysql56/
binlog-checksum = NONE
binlog_format = MIXED
connect_timeout = 60
datadir = /db01/mysql56/
default-storage-engine = InnoDB
event_scheduler = ON
expire_logs_days = 7
group_concat_max_len = 99999
innodb_autoextend_increment=64
innodb_buffer_pool_size = 5G
innodb_data_home_dir = /db01/mysql56
innodb_file_format = barracuda
innodb_file_per_table
innodb_flush_log_at_trx_commit = 1
innodb_io_capacity = 200
innodb_log_file_size = 64M
innodb_log_files_in_group=3
innodb_log_group_home_dir=/db01/mysql56/
innodb_purge_threads = 1
innodb_stats_on_metadata= 0
join_buffer_size = 4000000
key_buffer_size = 256000000
local-infile = 0
log-bin = mysql-bin
log-error = /var/log/mysqld56.log
log-queries-not-using-indexes
log-slave-updates
long-query-time = 5
master-info-repository = TABLE
max_allowed_packet = 32M
max_connect_errors = 9999
max_connections = 400
max_heap_table_size = 64000000
myisam-recover = FORCE,BACKUP
mysqladmin = /usr/local/mysql56/bin/mysqladmin
mysqld = /usr/local/mysql56/bin/mysqld
net_read_timeout = 120
old_passwords = 0
open_files_limit = 4410
performance_schema = 1
pid-file = /var/run/mysqld/mysqld56.pid
plugin_dir = /opt/mysql56/plugins
port = 3306
query_cache_size = 16777216
query_cache_type = ON
relay-log = /db01/mysql56/relay-bin
relay-log-index = /db01/mysql56/relay-bin.index
relay-log-info-file = /db01/mysql56/relay-bin.info
secure_auth = OFF
server-id = 3002
slave_net_timeout = 60
slow_query_log
slow_query_log_file = /var/log/mysqld56-slow.log
socket = /var/run/mysqld/mysql.sock
sort_buffer_size = 192000
symbolic-links = 0
sync_binlog = 1
table_open_cache = 2000
table_open_cache = 2000
thread_cache_size = 108
tmp_table_size = 64000000
user = mysql

Hope you can suggest how I can get the backup to run.

Thank you!
-dale

Forgot to mention, this is on RHEL6. Everywhere else I am running MySQL 5.5 on RHEL5 and not encountering any problem with Xtrabackup/Innobackupex. Very satisfied with the product, excepting this little hiccup. :slight_smile:

Can you please mention all your steps of the backup. Also, what’s the output of

$ xtrabackup --print-defaults

By the way, the only difference i can see is innodb_log_files_in_group. Xtrabackup took it as innodb_log_files_in_group = 2 while in your my.cnf it set at innodb_log_files_in_group = 3. We need all your steps of backup to drill down the problem. Either it’s happen on during taking backup or at apply-log stage.

The output displays no arguments.

xtrabackup --print-defaults

xtrabackup would have been started with the following arguments:

Ran this command:
innobackupex --user=‘xxxxx’ --socket=/var/run/mysqld/mysql.sock --password=‘xxxxx’ --defaults-group=mysqld56 /db01/perconabackups

Output of run:

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

140220 13:49:46 innobackupex: Starting mysql with options: --password=xxxxxxxx --user=‘xxxxx’ --socket=’/var/run/mysqld/mysql.sock’ --unbuffered –
140220 13:49:46 innobackupex: Connected to database with mysql child process (pid=29490)
140220 13:49:52 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints “completed OK!”.

innobackupex: Using mysql Ver 14.14 Distrib 5.1.69, for redhat-linux-gnu (x86_64) using readline 5.1
innobackupex: Using mysql server version Copyright © 2000, 2013, Oracle and/or its affiliates. All rights reserved.

innobackupex: Created backup directory /db01/perconabackups/2014-02-20_13-49-52
140220 13:49:52 innobackupex: Starting mysql with options: --password=xxxxxxxx --user=‘xxxxx’ --socket=’/var/run/mysqld/mysql.sock’ --unbuffered –
140220 13:49:52 innobackupex: Connected to database with mysql child process (pid=29513)
140220 13:49:54 innobackupex: Connection to database server closed

140220 13:49:54 innobackupex: Starting ibbackup with command: xtrabackup_56 --defaults-group=“mysqld56” --backup --suspend-at-end --target-dir=/db01/perconabackups/2014-02-20_13-49-52 --tmpdir=/tmp
innobackupex: Waiting for ibbackup (pid=29519) to suspend
innobackupex: Suspend file ‘/db01/perconabackups/2014-02-20_13-49-52/xtrabackup_suspended’

xtrabackup_56 version 2.0.8 for MySQL server 5.6.10 Linux (x86_64) (revision id: 587)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
2014-02-20 13:49:54 7fb4bac75720 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2014-02-20 13:49:54 7fb4bac75720 InnoDB: File name ./ib_logfile0
2014-02-20 13:49:54 7fb4bac75720 InnoDB: File operation call: ‘open’ returned OS error 71.
2014-02-20 13:49:54 7fb4bac75720 InnoDB: Cannot continue operation.
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 386.

Mirfan,

I also opened Bug #1282270 if you want to note anything there.

Thank you!
-dale

Found the problem in my install. The /etc/my.cnf owner was root, group root, permissions 640. Changed ownership of file to mysql, left group root and left permissions alone.

The hint was that you pointed out that the log file count was 2, not 3 as in my.cnf. Looked at /etc/my.cnf (ls -l /etc/my.cnf) and compared with system where innobackupex worked. Realized I had not set ownership properly.

Sorry for the interruption!

I have the same issues when working with mariadb 10 on Centos 6.5(Master)
I use xtrabackup to dump db from it to 2 Ubuntu 14(mariadb 10 - Slave )
I have already set :
-rwxrwxrwx 1 mysql root 5088 Oct 28 10:01 /etc/my.cnf
But no lucky

Can you please share the command you run to took backup along with backup log and my.cnf file

Yes,
On Master(Centos 6.5 - Mariadb 10.0)

innobackupex --user=root --password=password --stream=xbstream ./ |ssh root@10.10.10.220 "xbstream -x -C /var/lib/mysql"

and the error :

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

141028 10:20:05 innobackupex: Starting mysql with options: --password=xxxxxxxx --user='root' --unbuffered --
141028 10:20:05 innobackupex: Connected to database with mysql child process (pid=2712)
141028 10:20:11 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".

innobackupex: Using mysql Ver 15.1 Distrib 10.0.14-MariaDB, for Linux (x86_64) using readline 5.1
Warning: World-writable config file '/etc/my.cnf' is ignored
innobackupex: Using mysql server version Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.

Warning: World-writable config file '/etc/my.cnf' is ignored
innobackupex: Created backup directory /root
141028 10:20:11 innobackupex: Starting mysql with options: --password=xxxxxxxx --user='root' --unbuffered --
xbstream: Can't create/write to file './backup-my.cnf' (Errcode: 17)
xbstream: failed to create file.
141028 10:20:11 innobackupex: Connected to database with mysql child process (pid=2739)
141028 10:20:13 innobackupex: Connection to database server closed

141028 10:20:13 innobackupex: Starting ibbackup with command: xtrabackup_56 --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp --tmpdir=/tmp --stream=xbstream
innobackupex: Waiting for ibbackup (pid=2746) to suspend
innobackupex: Suspend file '/tmp/xtrabackup_suspended'

Warning: World-writable config file '/etc/my.cnf' is ignored
xtrabackup_56 version 2.0.8 for MySQL server 5.6.10 Linux (x86_64) (revision id: 587)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
2014-10-28 10:20:13 7f8ac7bc5720 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2014-10-28 10:20:13 7f8ac7bc5720 InnoDB: File name ./ib_logfile0
2014-10-28 10:20:13 7f8ac7bc5720 InnoDB: File operation call: 'open' returned OS error 71.
2014-10-28 10:20:13 7f8ac7bc5720 InnoDB: Cannot continue operation.
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 386.

After review i saw the problem is :


xbstream: Can't create/write to file './backup-my.cnf' (Errcode: 17)


root@cluster02:/home/itech# ll /var/lib/mysql/
total 12
drwxr-xr-x 2 mysql mysql 4096 Oct 29 03:00 ./
drwxr-xr-x 42 root root 4096 Oct 28 22:08 ../
-rw-r----- 1 root root 238 Oct 29 03:00 backup-my.cnf

But after i chown the /var/lib/mysql on the Slave from root:root to mysql:mysql


root@cluster02:/home/itech# chown -R mysql:mysql /var/lib/mysql/

But still error event i chmod -R 777 /var/lib/mysql

When i switched a Ubuntu node to became the master and the backup was successfull

xtrabackup: Stopping log copying thread.
..>> log scanned up to (1616807)

xtrabackup: Creating suspend file '/tmp/xtrabackup_log_copied' with pid '2235'
xtrabackup: Transaction log of lsn (1616807) to (1616807) was copied.
141029 03:23:34 innobackupex: All tables unlocked

innobackupex: Backup created in directory '/home/itech'
innobackupex: MySQL binlog position: filename 'mariadb-bin.000013', position 328
141029 03:23:34 innobackupex: Connection to database server closed
141029 03:23:34 innobackupex: completed OK!

Here is th my.cnf on Master(Centos)

[root@master01 ~]# cat /etc/my.cnf
# Example MariaDB config file for medium systems.
#
# This is for a system with little memory (32M - 64M) where MariaDB plays
# an important part, or systems up to 128M where MariaDB is used together with
# other programs (such as a web server)
#
# MariaDB programs look for option files in a set of
# locations which depend on the deployment platform.
# You can copy this option file to one of those
# locations. For information about these locations, do:
# 'my_print_defaults --help' and see what is printed under
# Default options are read from the following files in the given order:
# More information at: http://dev.mysql.com/doc/mysql/en/option-files.html
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the "--help" option.

# The following options will be passed to all MariaDB clients
[client]
#password = your_password
port = 3306
socket = /var/lib/mysql/mysql.sock

# Here follows entries for some specific programs

# The MariaDB server
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 1M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M

# Point the following paths to different dedicated disks
#tmpdir = /tmp/

# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (via the "enable-named-pipe" option) will render mysqld useless!
# 
#skip-networking

# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin

# binary logging format - mixed recommended
binlog_format=mixed

# required unique id between 1 and 2^32 - 1
# defaults to 1 if master-host is not set
# but will not function as a master if omitted
server-id = 1
bind-address = 0.0.0.0
log_bin = /var/log/mysql/mariadb-bin
sync_binlog = 0
expire_logs_days = 10
slave_compressed_protocol = 1
#transaction-isolation = READ COMMITED

# Replication Slave (comment out master section to use this)
#
# To configure this host as a replication slave, you can choose between
# two methods :
#
# 1) Use the CHANGE MASTER TO command (fully described in our manual) -
# the syntax is:
#
# CHANGE MASTER TO MASTER_HOST=<host>, MASTER_PORT=<port>,
# MASTER_USER=<user>, MASTER_PASSWORD=<password> ;
#
# where you replace <host>, <user>, <password> by quoted strings and
# <port> by the master's port number (3306 by default).
#
# Example:
#
# CHANGE MASTER TO MASTER_HOST='125.564.12.1', MASTER_PORT=3306,
# MASTER_USER='joe', MASTER_PASSWORD='secret';
#
# OR
#
# 2) Set the variables below. However, in case you choose this method, then
# start replication for the first time (even unsuccessfully, for example
# if you mistyped the password in master-password and the slave fails to
# connect), the slave will create a master.info file, and any later
# change in this file to the variables' values below will be ignored and
# overridden by the content of the master.info file, unless you shutdown
# the slave server, delete master.info and restart the slaver server.
# For that reason, you may want to leave the lines below untouched
# (commented) and instead use CHANGE MASTER TO (see above)
#
# required unique id between 2 and 2^32 - 1
# (and different from the master)
# defaults to 2 if master-host is set
# but will not function as a slave if omitted
#server-id = 2
#
# The replication master for this slave - required
#master-host = <hostname>
#
# The username the slave will use for authentication when connecting
# to the master - required
#master-user = <username>
#
# The password the slave will authenticate with when connecting to
# the master - required
#master-password = <password>
#
# The port the master is listening on.
# optional - defaults to 3306
#master-port = <port>
#
# binary logging - not required for slaves, but recommended
#log-bin=mysql-bin

# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /var/lib/mysql
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /var/lib/mysql
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 5M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates

[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

I think the problem is that the directory structures are not the same