On linux (debian 9/stretch), running Percona for MySQL (Ver 14.14 Distrib 5.5.62-38.14) and innobackupex (2.3.10 based on MySQL server 5.6.24). When I try to run:
/usr/bin/innobackupex --stream=tar --user=someuser --password=apassword --safe-slave-backup --slave-info --databases="db1 db2 ... " /tmp | /bin/gzip - | /usr/bin/aws s3 cp - s3://somewhere/...
200303 09:00:31 innobackupex: Starting the backup operation
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".
200303 09:00:33 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;port=3306;mysql_socket=/var/run/mysqld/mysqld.sock' as 'backupuser' (using password: YES).
200303 09:00:33 version_check Connected to MySQL server
200303 09:00:33 version_check Executing a version check against the server...
# A software update is available:
200303 09:00:36 version_check Done.
200303 09:00:36 Connecting to MySQL server host: localhost, user: backupuser, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
Using server version 5.5.62-38.14-log
/usr/bin/innobackupex version 2.3.10 based on MySQL server 5.6.24 Linux (x86_64) (revision id: bd0d4403f36)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: open files limit requested 20000, set to 20000
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 268435456
200303 09:00:36 >> log scanned up to (139458147831)
xtrabackup: Generating a list of tablespaces
Skipping db: ./mysql-misc
Skipping db: ./test
Skipping db: ./mysql@002dmisc
Skipping db: ./mysql
Skipping db: ./performance_schema
Skipping db: ./quartz
Skipping db: ./gameiom_asia
200303 09:00:36 [01] Streaming ./ibdata1
/usr/bin/innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
archive_write_data() failed: (null) (errno = 0)
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
I have run this through strace, and it is clear that it fails when trying to pipe into gzip:
8819 fadvise64(6, 0, 10485760, POSIX_FADV_DONTNEED) = 0
8819 write(1, "\227\360i-\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 m\301\225\322\0\10\0\0\0\36\255E"..., 10485760) = 1241088
8819 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=8811, si_uid=0} ---
8819 write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 9244672) = -1 EPIPE (Broken pipe)
8819 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=8811, si_uid=0} ---
8819 write(2, "\7", 1) = 1
8819 write(2, "/usr/bin/innobackupex", 21) = 21
8819 write(2, ": ", 2) = 2
8819 write(2, "Error writing file 'UNOPENED' (E"..., 57) = 57
8819 write(2, "\n", 1) = 1
8819 write(2, "archive_write_data() failed: (nu"..., 48) = 48
8819 munmap(0x7fb4fc29b000, 10506240) = 0
8819 close(6) = 0
8819 write(2, "[01] xtrabackup: Error: xtraback"..., 59) = 59
8819 write(2, "[01] xtrabackup: Error: failed t"..., 49) = 49
8819 exit_group(1 <unfinished ...>
8811 <... select resumed> <unfinished ...>) = ?
8819 <... exit_group resumed>) = ?
I also tried running innobackupex with streaming, and got a stream of binary on the screen - as expected, so the program clearly works. However, I can run other pipelines from the commandline without error; furhtermore, rebooting the server makes no difference. This problem suddenly arose in the last 4 days - my thought is that there must be something in the evironment that causes this, but I can’t think what it would be. What is the best way to troubleshoot this?
