Database page corruption detected

xtrabackup fails after attempting to write backup of large datafile. The ibd file in the data directory is 123gigs. Can anyone give me some advice or direction here?

Header of backup log:

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oyand Percona Inc 2009-2012. All Rights Reserved.This software is published underthe GNU GENERAL PUBLIC LICENSE Version 2, June 1991.120905 00:00:01 innobackupex: Starting mysql with options: --unbuffered --120905 00:00:01 innobackupex: Connected to database with mysql child process (pid=9788)120905 00:00:08 innobackupex: Connection to database server closedIMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints “completed OK!”.innobackupex: Using mysql Ver 14.14 Distrib 5.5.27, for Linux (x86_64) using readline 5.1innobackupex: Using mysql server version Copyright © 2000, 2011, Oracle and/or its affiliates. All rights reserved.innobackupex: Created backup directory /home/db/backup/dbname/xtrabackup/2012-09-05_00-00-08120905 00:00:08 innobackupex: Starting mysql with options: --unbuffered --120905 00:00:08 innobackupex: Connected to database with mysql child process (pid=9815)120905 00:00:10 innobackupex: Connection to database server closed120905 00:00:10 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-group=“mysqld” --backup --suspend-at-end --target-dir=/home/db/backup/dbname/xtrabackup/2012-09-05_00-00-08 --parallel=4innobackupex: Waiting for ibbackup (pid=9822) to suspendinnobackupex: Suspend file '/home/db/backup/dbname/xtrabackup/2012-09-05_00-00-08/xtrabackup_suspended’xtrabackup_55 version 2.0.2 for Percona Server 5.5.16 Linux (x86_64) (revision id: 461)xtrabackup: uses posix_fadvise().xtrabackup: cd to /home/db/dataxtrabackup: Target instance is assumed as followings.xtrabackup: innodb_data_home_dir = /home/db/data/innodb/xtrabackup: innodb_data_file_path = ibdata1:10M:autoextendxtrabackup: innodb_log_group_home_dir = /home/db/data/innodb/xtrabackup: innodb_log_files_in_group = 2xtrabackup: innodb_log_file_size = 268435456120905 0:00:10 InnoDB: Using Linux native AIOxtrabackup: using O_DIRECT

Error Output:

[01] xtrabackup: Database page corruption detected at page 3999937. retrying…[01] xtrabackup: Database page corruption detected at page 3999937. retrying…[01] xtrabackup: Database page corruption detected at page 3999937. retrying…[01] xtrabackup: Database page corruption detected at page 3999937. retrying…[01] xtrabackup: Database page corruption detected at page 3999937. retrying…[01] xtrabackup: Database page corruption detected at page 3999937. retrying…[01] xtrabackup: Database page corruption detected at page 3999937. retrying…[01] xtrabackup: Database page corruption detected at page 3999937. retrying…[01] xtrabackup: Database page corruption detected at page 3999937. retrying…[01] xtrabackup: Error: 10 retries resulted in fail. File ./dbname/tablename.ibd seems to be corrupted.[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.[01] xtrabackup: Error: failed to copy datafile.innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 374.

Does it fail w/o parallel copy (remove --parallel)? I also have tables that are 100GB-200GB+ and would fail. This was on an pre 2.0 version though. I also had an issue where tables were being truncated while the copy was running, but believe this has been fixed for quite some time, just never got around to trying it again. I have been living with LVM snapshots for the time being.

Are you using “barracuda” InnoDB compression by chance? Try checking the value of innodb_file_format (mysql> show variables like ‘innodb_file_format’:wink:

If so:

https://groups.google.com/forum/?fromgroups=#!topic/percona- discussion/GR8GP9gLCBI

https://bugs.launchpad.net/percona-xtrabackup/+bug/498660/

Is it possible your tables are actually corrupted? What does the error log of your actual MySQL instance say? Here’s an article on recovering from InnoDB corruption:

http://www.mysqlperformanceblog.com/2008/07/04/recovering-in nodb-table-corruption/