innobackupex sporadic failures

MySQL & MariaDB Percona XtraBackup
1

I’m running into a problem where innobackup is failing, sometimes. My backup script is running innobackupex like this:

innobackupex --stream=tar --user=bkp_user --password=xxxxx ./ 2>> /mnt/backup/logs/backup_2013-04-20.log | pigz -c -p 20 -6 - > /mnt/backup/backup_2013-04-20.tar.gz

This works most of the time, but sometimes it fails on me. When it fails, it reports one of the following messages:

innobackupex: Created backup directory /root
tar: -: Cannot write: Broken pipe
tar: Error is not recoverable: exiting now
innobackupex: Error: Failed to stream ‘backup-my.cnf’: Inappropriate ioctl for device at /usr/bin/innobackupex line 381.

[01] Streaming /data/db/schemas/ibdata
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 381.

How can I troubleshoot this, or gather additional information about the failures?

2

Hello,

have you checked that you are not running out of disk space around this time? Not only on the destination backup directory, check also you tmpdir setting because this is where xtrabackup_logfile is saved before being streamed. How long does it take to complete the backup? If you have Cacti or something else, maybe you can get a quick glimpse of disk usage.

3

I don’t think that I’m running out of disk space. Nagios hasn’t sent out any alerts and the disk usage looks ok. We haven’t set tmpdir, so it should be the system default of /tmp. /tmp is 30GB with only 2 MB used and the backup job fails in 7 seconds. It very strange that this is just failing sometimes.

Full log is here:

Running backup at Thu Apr 25 16:30:01 UTC 2013

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Ireland Ltd 2009-2012. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

130425 16:30:01 innobackupex: Starting mysql with options: --defaults-file=‘/etc/mysql/my.cnf’ --password=xxxxxxxx --user=‘bkpuser’ --host=‘localhost’ --unbuffered –
130425 16:30:01 innobackupex: Connected to database with mysql child process (pid=14098)
130425 16:30:07 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints “completed OK!”.

innobackupex: Using mysql Ver 14.14 Distrib 5.1.63, for debian-linux-gnu (x86_64) using readline 6.1
innobackupex: Using mysql server version Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.

innobackupex: Created backup directory /root
tar: -: Cannot write: Broken pipe
tar: Error is not recoverable: exiting now
innobackupex: Error: Failed to stream ‘backup-my.cnf’: Inappropriate ioctl for device at /usr/bin/innobackupex line 381.
innobackupex command failed at Thu Apr 25 16:30:07 UTC 2013

4

I’ve the same problem and wondering how to fix that. It happens on the donor node and i’ve even tried to move the hardcoded /tmp path to a different path, but have the same problem there. The new path /var/lib/mysql got all permissions for the mysql user ready.

5

@tgoodaire,

I would suggest to simplify the command first, to see if any parts are failing. As you can see here I have removed the pigz and pipe to stderr i.e.

innobackupex --stream=tar --user=bkp_user --password=xxxxx ./ | cat - > /mnt/backup/backup_2013-04-20.tar.gz

6

For now, I’ve worked around the problem by doing a simple bash loop to retry the backup three times if it fails. So far it’s been working ok. If it fails once, the second attempt works. It’s still a mystery to me why it’s failing sometimes.