# Sudden Failure of Automated Backups **URL:** \ **Category:** Percona XtraBackup\ **Created:** [February 13, 2017, 5:19pm UTC](https://forums.percona.com/t/sudden-failure-of-automated-backups/5412 "2017-02-13T17:19:37Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Jaws12](https://avatars.discourse-cdn.com/v4/letter/j/b487fb/32.png) [@Jaws12](https://forums.percona.com/u/Jaws12)\ **Post date:** [February 13, 2017, 5:19pm UTC](https://forums.percona.com/t/sudden-failure-of-automated-backups/5412/1 "2017-02-13T17:19:37Z")
Greetings! Automated Xtrabackups for our MySQL Percona Cluster suddenly seem to have stopped working without us changing anything specific in the backup procedures/normal operating processes of our cluster. In the error log, there are entries like this: 2017-02-13 02:00:02,007 - INFO - Pre-process for backup: Ensure the backup directory is present and empty… 2017-02-13 02:00:02,023 - INFO - Running innobackupex for the running instance… 2017-02-13 02:00:02,025 - INFO - … xtrabackup still running, check [/home/percona/backup.log/backup.log] ^Gxbstream: Can’t create/write to file ‘././backup-my.cnf’ (Errcode: 17 - File exists) xbstream: failed to create file. 2017-02-13 02:00:09,033 - ERROR - Caught this exception in Manager: Xtrabackup failed - please see log at [/home/percona/backup.log/backup.log] 2017-02-13 02:00:09,033 - INFO - Clean up any subprocesses before exit… The backup.log states: 170213 02:00:02 innobackupex: Starting ibbackup with command: xtrabackup --defaults-extra-file=“/home/percona/.my.cnf” --defaults-group=“mysqld” --backup --suspend-at-end --target-dir=/tmp --innodb\_data\_file\_path=“ibdata1:12M:autoextend” --tmpdir=/tmp --extra-lsndir=‘/tmp’ --stream=xbstream innobackupex: Waiting for ibbackup (pid=5292) to suspend innobackupex: Suspend file ‘/tmp/xtrabackup\_suspended\_2’ xtrabackup version 2.2.12 based on MySQL server 5.6.24 Linux (x86\_64) (revision id: 8726828) xtrabackup: uses posix\_fadvise(). xtrabackup: cd to /var/lib/mysql xtrabackup: open files limit requested 65535, set to 1024 xtrabackup: using the following InnoDB configuration: xtrabackup: innodb\_data\_home\_dir = ./ xtrabackup: innodb\_data\_file\_path = ibdata1:12M:autoextend xtrabackup: innodb\_log\_group\_home\_dir = ./ xtrabackup: innodb\_log\_files\_in\_group = 2 xtrabackup: innodb\_log\_file\_size = 4294967296 xtrabackup: using ALL\_O\_DIRECT > > log scanned up to (12969935228387) > > xtrabackup: Generating a list of tablespaces > > [01] Streaming ./ibdata1 > > ^Gxtrabackup: Error writing file ‘UNOPENED’ (Errcode: 32 - Broken pipe) > > xb\_stream\_write\_data() failed. > > ^Gxtrabackup: Error writing file ‘UNOPENED’ (Errcode: 32 - Broken pipe) > > [01] xtrabackup: Error: xtrabackup\_copy\_datafile() failed. > > [01] xtrabackup: Error: failed to copy datafile. > > innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2711 > > main::wait\_for\_ibbackup\_file\_create(‘/tmp/xtrabackup\_suspended\_2’) called at /usr/bin/innobackupex line 2731 > > main::wait\_for\_ibbackup\_suspend(‘/tmp/xtrabackup\_suspended\_2’) called at /usr/bin/innobackupex line 1984 > > main::backup() called at /usr/bin/innobackupex line 1609 > > innobackupex: Error: The xtrabackup child process has died at /usr/bin/innobackupex line 2711. The only thing I can think of that would have changed over time is the size of the database backups, as of course the database is always growing. Could the backup buffer sizes need to be increased? More information: our backups stream daily from the secondary node of our 3 node cluster to another server where the backups are kept and then a test restore is done from this backup server to another final server instance to verify the backups restore correctly. Furthermore, our binlogs are constantly also streamed from this secondary node of the 3 node cluster the final restore server instance (the binlogs have not failed and continue to stream properly according to files present on the final restore server). Topology Summary: SqlCluster Node1/2/3: Node 2 Streams Backups Daily to → Backup Keeper Server which restores to → Final Restore server Node 2 also Streams constant binlogs to → Final Restore server Any suggestions on what might be causing this sudden backup failure? The cluster has been running without issue otherwise so it is perplexing. There is also plenty of free space given the amount of data we are backing up/testing restores on. Thanks in advance and have a great day! 🙂