# innobackupex sporadic failures **URL:** \ **Category:** Percona XtraBackup\ **Created:** [April 22, 2013, 8:09am UTC](https://forums.percona.com/t/innobackupex-sporadic-failures/2639 "2013-04-22T08:09:56Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![tgoodaire](https://avatars.discourse-cdn.com/v4/letter/t/ecd19e/32.png) [@tgoodaire](https://forums.percona.com/u/tgoodaire)\ **Post date:** [April 22, 2013, 8:09am UTC](https://forums.percona.com/t/innobackupex-sporadic-failures/2639/1 "2013-04-22T08:09:56Z")
I’m running into a problem where innobackup is failing, sometimes. My backup script is running innobackupex like this: innobackupex --stream=tar --user=bkp\_user --password=xxxxx ./ 2\>\> /mnt/backup/logs/backup\_2013-04-20.log | pigz -c -p 20 -6 - \> /mnt/backup/backup\_2013-04-20.tar.gz This works most of the time, but sometimes it fails on me. When it fails, it reports one of the following messages: innobackupex: Created backup directory /root tar: -: Cannot write: Broken pipe tar: Error is not recoverable: exiting now innobackupex: Error: Failed to stream ‘backup-my.cnf’: Inappropriate ioctl for device at /usr/bin/innobackupex line 381. [01] Streaming /data/db/schemas/ibdata innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 381. How can I troubleshoot this, or gather additional information about the failures? ---
**Author:** ![revin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/revin/32/824_2.png) [@revin](https://forums.percona.com/u/revin)\ **Post date:** [April 24, 2013, 11:40pm UTC](https://forums.percona.com/t/innobackupex-sporadic-failures/2639/2 "2013-04-24T23:40:08Z")
> [@tgoodaire;9817](#): > > I’m running into a problem where innobackup is failing, sometimes. My backup script is running innobackupex like this: innobackupex --stream=tar --user=bkp\_user --password=xxxxx ./ 2\>\> /mnt/backup/logs/backup\_2013-04-20.log | pigz -c -p 20 -6 - \> /mnt/backup/backup\_2013-04-20.tar.gz This works most of the time, but sometimes it fails on me. When it fails, it reports one of the following messages: innobackupex: Created backup directory /root tar: -: Cannot write: Broken pipe tar: Error is not recoverable: exiting now innobackupex: Error: Failed to stream ‘backup-my.cnf’: Inappropriate ioctl for device at /usr/bin/innobackupex line 381. [01] Streaming /data/db/schemas/ibdata innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 381. How can I troubleshoot this, or gather additional information about the failures? Hello, have you checked that you are not running out of disk space around this time? Not only on the destination backup directory, check also you tmpdir setting because this is where xtrabackup\_logfile is saved before being streamed. How long does it take to complete the backup? If you have Cacti or something else, maybe you can get a quick glimpse of disk usage. ---
**Author:** ![tgoodaire](https://avatars.discourse-cdn.com/v4/letter/t/ecd19e/32.png) [@tgoodaire](https://forums.percona.com/u/tgoodaire)\ **Post date:** [April 25, 2013, 12:44pm UTC](https://forums.percona.com/t/innobackupex-sporadic-failures/2639/3 "2013-04-25T12:44:50Z")
I don’t think that I’m running out of disk space. Nagios hasn’t sent out any alerts and the disk usage looks ok. We haven’t set tmpdir, so it should be the system default of /tmp. /tmp is 30GB with only 2 MB used and the backup job fails in 7 seconds. It very strange that this is just failing sometimes. Full log is here: Running backup at Thu Apr 25 16:30:01 UTC 2013 InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona Ireland Ltd 2009-2012. All Rights Reserved. This software is published under the GNU GENERAL PUBLIC LICENSE Version 2, June 1991. 130425 16:30:01 innobackupex: Starting mysql with options: --defaults-file=‘/etc/mysql/my.cnf’ --password=xxxxxxxx --user=‘bkpuser’ --host=‘localhost’ --unbuffered – 130425 16:30:01 innobackupex: Connected to database with mysql child process (pid=14098) 130425 16:30:07 innobackupex: Connection to database server closed IMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints “completed OK!”. innobackupex: Using mysql Ver 14.14 Distrib 5.1.63, for debian-linux-gnu (x86\_64) using readline 6.1 innobackupex: Using mysql server version Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved. innobackupex: Created backup directory /root tar: -: Cannot write: Broken pipe tar: Error is not recoverable: exiting now innobackupex: Error: Failed to stream ‘backup-my.cnf’: Inappropriate ioctl for device at /usr/bin/innobackupex line 381. innobackupex command failed at Thu Apr 25 16:30:07 UTC 2013 ---
**Author:** ![Wurstteppich](https://avatars.discourse-cdn.com/v4/letter/w/7993a0/32.png) [@Wurstteppich](https://forums.percona.com/u/Wurstteppich)\ **Post date:** [April 27, 2013, 7:06am UTC](https://forums.percona.com/t/innobackupex-sporadic-failures/2639/4 "2013-04-27T07:06:55Z")
I’ve the same problem and wondering how to fix that. It happens on the donor node and i’ve even tried to move the hardcoded /tmp path to a different path, but have the same problem there. The new path /var/lib/mysql got all permissions for the mysql user ready. ---
**Author:** ![revin](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/revin/32/824_2.png) [@revin](https://forums.percona.com/u/revin)\ **Post date:** [May 14, 2013, 11:42pm UTC](https://forums.percona.com/t/innobackupex-sporadic-failures/2639/5 "2013-05-14T23:42:26Z")
@tgoodaire, I would suggest to simplify the command first, to see if any parts are failing. As you can see here I have removed the pigz and pipe to stderr i.e. innobackupex --stream=tar --user=bkp\_user --password=xxxxx ./ | cat - \> /mnt/backup/backup\_2013-04-20.tar.gz ---
**Author:** ![tgoodaire](https://avatars.discourse-cdn.com/v4/letter/t/ecd19e/32.png) [@tgoodaire](https://forums.percona.com/u/tgoodaire)\ **Post date:** [May 31, 2013, 3:03pm UTC](https://forums.percona.com/t/innobackupex-sporadic-failures/2639/6 "2013-05-31T15:03:38Z")
For now, I’ve worked around the problem by doing a simple bash loop to retry the backup three times if it fails. So far it’s been working ok. If it fails once, the second attempt works. It’s still a mystery to me why it’s failing sometimes.