xtrabackup failure: Inappropriate ioctl for device

Hi there,

I’m running Percona mysql Server version: 5.5.23-55-log Percona Server (GPL), Release 25.3 and xtrabackup version 2.0.0 for Percona Server 5.1.59 unknown-linux-gnu (x86_64) (revision id: undefined).

I have 5 database servers with this setup, and 4 are completing their backups fine, and one is failing with the error:

Failed to stream xtrabackup_binary: Inappropriate ioctl for device at /usr/bin/innobackupex-1.5.1 line 301.

Throughout the log of the backup, there’s also errors with exit code 141 like this:
innobackupex-1.5.1: Ignoring nonexistent file ‘my_db_staging/password_resets.frm’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.

I am running all 5 backups with the command: /usr/bin/innobackupex-1.5.1 --defaults-file=/etc/mysql/my.cnf --user=my_user --password=my_pass --throttle=40 --stream=xbstream ./ | pv -q -L40m | su backups -c ‘ssh backups@10.54.168.16 “xbstream -x -C /backups/mysql-9/0-days-ago”’

Does anyone know what might cause this error or how I should go about troubleshooting it?

Thank you,

Tim.

A truncated copy of the end of the backup log:

log scanned up to (1708839913493)
log scanned up to (1708840156643)
log scanned up to (1708840595827)
log scanned up to (1708841052489)
log scanned up to (1708841365198)
log scanned up to (1708841828552)
log scanned up to (1708842195574)
[01] …done

121105 18:37:24 innobackupex-1.5.1: Continuing after ibbackup has suspended
121105 18:37:24 innobackupex-1.5.1: Starting mysql with options: --defaults-file=‘/etc/mysql/my.cnf’ --password=xxxxxxxx --user=‘my_user’ --unbuffered –
121105 18:37:24 innobackupex-1.5.1: Connected to database with mysql child process (pid=13585)
121105 18:37:26 innobackupex-1.5.1: Starting to lock all tables…

log scanned up to (1708842531794)
log scanned up to (1708842747386)
121105 18:37:36 innobackupex-1.5.1: All tables locked and flushed to disk

121105 18:37:36 innobackupex-1.5.1: Starting to backup .frm, .MRG, .MYD, .MYI,
innobackupex-1.5.1: .TRG, .TRN, .ARM, .ARZ, .CSM, .CSV and .opt files in
innobackupex-1.5.1: subdirectories of ‘/var/lib/mysql’
innobackupex-1.5.1: Backing up files ‘/var/lib/mysql/beta_billing/.{frm,MYD,MYI,MRG,TRG,TRN,ARM, ARZ,CSM,CSV,opt,par}’ (10 files)
innobackupex-1.5.1: Backing up files '/var/lib/mysql/my_db_staging/
.{frm,MYD,MYI,MRG,TRG,TRN,ARM ,ARZ,CSM,CSV,opt,par}’ (54 files)
xbstream: Can’t create/write to file ‘./my_db_staging/user_opt_outs.ibd’ (Errcode: 17)
xbstream: failed to create file.
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘my_db_staging/user_opt_outs.ibd’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘my_db_staging/subscriptions.frm’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘my_db_staging/subscription_plans.frm’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘my_db_staging/billing_transactions.frm’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘my_db_staging/password_resets.frm’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘my_db_staging/tolk_phrases.frm’.
innobackupex-1.5.1: Backing up files ‘/var/lib/mysql/mysql/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM ,CSV,opt,par}’ (68 files)
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘mysql/db.frm’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘mysql/procs_priv.MYD’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘mysql/plugin.MYD’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.

innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘mysql/tables_priv.MYI’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘mysql/proxies_priv.MYD’.
innobackupex-1.5.1: Backing up file ‘/var/lib/mysql/event_tracker/events.frm’
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘event_tracker/events.frm’.
innobackupex-1.5.1: Backing up file ‘/var/lib/mysql/event_tracker/db.opt’
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘event_tracker/db.opt’.
innobackupex-1.5.1: Backing up files ‘/var/lib/mysql/performance_schema/*.{frm,MYD,MYI,MRG,TRG,TR N,ARM,ARZ,CSM,CSV,opt,par}’ (18 files)
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘performance_schema/cond_instances.frm’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.

innobackupex-1.5.1: Ignoring nonexistent file ‘performance_schema/events_waits_summary_global_by_event_nam e.frm’.
innobackupex-1.5.1: xbstream -c returned with exit code 141.
innobackupex-1.5.1: Ignoring nonexistent file ‘performance_schema/rwlock_instances.frm’.
121105 18:37:37 innobackupex-1.5.1: Finished backing up .frm, .MRG, .MYD, .MYI, .TRG, .TRN, .ARM, .ARZ, .CSV, .CSM and .opt files

innobackupex-1.5.1: Resuming ibbackup

xtrabackup: The latest check point (for incremental): ‘1708472789748’
xtrabackup: Stopping log copying thread.
.>> log scanned up to (1708842747978)

121105 18:37:39 innobackupex-1.5.1: All tables unlocked
121105 18:37:39 innobackupex-1.5.1: Connection to database server closed

innobackupex-1.5.1: Backup created in directory ‘/usr/local/sbin’
innobackupex-1.5.1: MySQL binlog position: filename ‘mysql-bin.005115’, position 76986073
Failed to stream xtrabackup_binary: Inappropriate ioctl for device at /usr/bin/innobackupex-1.5.1 line 301.

I just confirmed it is actually xtrabackup_55 that’s being run - not an xtrabackup that targets 5.1. This is in the log at the start of the backup:

121105 17:36:10 innobackupex-1.5.1: Starting ibbackup with command: xtrabackup_55 --defaults-file=“/etc/mysql/my.cnf” --backup --suspend-at-end --target-dir=/tmp --throttle=40 --stream=xbstream
innobackupex-1.5.1: Waiting for ibbackup (pid=26889) to suspend
innobackupex-1.5.1: Suspend file ‘/tmp/xtrabackup_suspended’

xtrabackup_55 version 2.0.0 for Percona Server 5.5.16 Linux (x86_64) (revision id: undefined)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 524288000
121105 17:36:10 InnoDB: Using Linux native AIO
xtrabackup: using O_DIRECT
121105 17:36:10 InnoDB: Warning: allocated tablespace 92, old maximum was 9

log scanned up to (1708590976624)
[01] Streaming ./ibdata1
[01] …done

This problem was solved by upgrading from xtrabackup 2.0.0 to 2.0.3.

About a month ago I added a new table which has opt as part of the table name. Xtrabackup 2.0.0 has a bug where files with opt in the name are backed up twice, causing a problem and several other files to be missed, resulting in the log you see above. This Xtrabackup bug was fixed in 2.0.1.