innobackupex: Errcode: 32 - Broken pipe

On linux (debian 9/stretch), running Percona for MySQL (Ver 14.14 Distrib 5.5.62-38.14) and innobackupex (2.3.10 based on MySQL server 5.6.24). When I try to run:

/usr/bin/innobackupex --stream=tar --user=someuser --password=apassword --safe-slave-backup --slave-info --databases="db1 db2 ... " /tmp | /bin/gzip - | /usr/bin/aws s3 cp - s3://somewhere/...
200303 09:00:31 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".

200303 09:00:33 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;port=3306;mysql_socket=/var/run/mysqld/mysqld.sock' as 'backupuser' (using password: YES).
200303 09:00:33 version_check Connected to MySQL server
200303 09:00:33 version_check Executing a version check against the server...

# A software update is available:
200303 09:00:36 version_check Done.
200303 09:00:36 Connecting to MySQL server host: localhost, user: backupuser, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
Using server version 5.5.62-38.14-log
/usr/bin/innobackupex version 2.3.10 based on MySQL server 5.6.24 Linux (x86_64) (revision id: bd0d4403f36)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: open files limit requested 20000, set to 20000
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 268435456
200303 09:00:36 >> log scanned up to (139458147831)
xtrabackup: Generating a list of tablespaces
Skipping db: ./mysql-misc
Skipping db: ./test
Skipping db: ./mysql@002dmisc
Skipping db: ./mysql
Skipping db: ./performance_schema
Skipping db: ./quartz
Skipping db: ./gameiom_asia
200303 09:00:36 [01] Streaming ./ibdata1
/usr/bin/innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
archive_write_data() failed: (null) (errno = 0)
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.

I have run this through strace, and it is clear that it fails when trying to pipe into gzip:


8819 fadvise64(6, 0, 10485760, POSIX_FADV_DONTNEED) = 0
8819 write(1, "\227\360i-\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 m\301\225\322\0\10\0\0\0\36\255E"..., 10485760) = 1241088
8819 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=8811, si_uid=0} ---
8819 write(1, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 9244672) = -1 EPIPE (Broken pipe)
8819 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=8811, si_uid=0} ---
8819 write(2, "\7", 1) = 1
8819 write(2, "/usr/bin/innobackupex", 21) = 21
8819 write(2, ": ", 2) = 2
8819 write(2, "Error writing file 'UNOPENED' (E"..., 57) = 57
8819 write(2, "\n", 1) = 1
8819 write(2, "archive_write_data() failed: (nu"..., 48) = 48
8819 munmap(0x7fb4fc29b000, 10506240) = 0
8819 close(6) = 0
8819 write(2, "[01] xtrabackup: Error: xtraback"..., 59) = 59
8819 write(2, "[01] xtrabackup: Error: failed t"..., 49) = 49
8819 exit_group(1 <unfinished ...>
8811 <... select resumed> <unfinished ...>) = ?
8819 <... exit_group resumed>) = ?

I also tried running innobackupex with streaming, and got a stream of binary on the screen - as expected, so the program clearly works. However, I can run other pipelines from the commandline without error; furhtermore, rebooting the server makes no difference. This problem suddenly arose in the last 4 days - my thought is that there must be something in the evironment that causes this, but I can’t think what it would be. What is the best way to troubleshoot this?

Hi j4nd3r53n is it possible you are running into one of these known issues? https://www.percona.com/doc/percona-xtrabackup/2.3/known_issues.html

Although it’s not at recovery stage, I notice a reference to the zlib library. Did you update anything in the past days? Or are there any other logs appeared? Here’s another suggestion about space https://jira.percona.com/browse/PXB-1731?jql=text%20~%20%22backup%20broken%20pipe%22

You probably already realize that 5.5. is EoL (see this table https://www.percona.com/services/policies/percona-software-platform-lifecycle) so we’d very much encourage you to look at upgrading ASAP. We’re also deprecating innobackupex…

Hi Lorraine, it turned out, after much troubleshooting, that the aws client at the far end of the pipeline had been corrupted somehow; when it died, the whole pipeline collapsed, of course. I should have thought as far as checking that from the beginning.

Hey no problem at all, thank you for updating us with this, appreciated :slight_smile: