# Xtrabackup 8.0.31 reports Database page corruption with fresh mysql-community 8.0.31 database

**URL:** <https://forums.percona.com/t/xtrabackup-8-0-31-reports-database-page-corruption-with-fresh-mysql-community-8-0-31-database/21938>\
**Category:** Percona XtraBackup\
**Created:** [May 8, 2023, 11:16pm UTC](https://forums.percona.com/t/xtrabackup-8-0-31-reports-database-page-corruption-with-fresh-mysql-community-8-0-31-database/21938 "2023-05-08T23:16:17Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![murray.atkinson](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@murray.atkinson](https://forums.percona.com/u/murray.atkinson)\
**Post date:** [May 8, 2023, 11:16pm UTC](https://forums.percona.com/t/xtrabackup-8-0-31-reports-database-page-corruption-with-fresh-mysql-community-8-0-31-database/21938/1 "2023-05-08T23:16:17Z")

</div>

Hi,

We are in the process of moving from MySQL 5.7 to MySQL 8 and have run into an issue with xtrabackup - it complains about page corruptions when copying ibdata1.

The database is on a test machine and was initialized using mysqld and then databases & tables created using mysqldump from another test MySQL 8 database with no data. At this point xtrabackup successfully created a backup.

I then loaded a small amount of data into some of the databases, again using mysqldump. After the data load, xtrabackup failed to create a backup when trying to copy ibdata1.

MySQL is working without errors, adding more data & querying hasn’t showed up any issues or resolved xtrabackups failure to copy ibdata1.

Any suggestions on what to try would be greatly appreciated.

System Setup:  
Debian Buster 10.13  
percona-xtrabackup-80:  
Installed: 8.0.31-24-1.buster  
Candidate: 8.0.32-26-1.buster  
mysql-community-server:  
Installed: 8.0.31-1debian10  
Candidate: 8.0.33-1debian10

Command used:  
sudo -u mysql xtrabackup --defaults-group-suffix=1 --backup --user=admin --password=\* --target-dir=/srv/mysql\_backups/for\_mysqld2.bak

xrtabackup output:  
2023-05-09T09:17:19.387497+10:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/var/lib/mysql/notused --tmpdir=/var/lib/mysql/tmp/notused --innodb\_file\_per\_table=1 --innodb\_log\_buffer\_size=16M --innodb\_log\_file\_size=16M --server-id=549358661 --datadir=/var/lib/mysql/tirtlnet --tmpdir=/var/lib/mysql/tmp/tirtlnet --log\_bin=/home/mysql/tirtlnet/binlog/mysql-bin.log --innodb\_undo\_directory=/var/lib/mysql/undo/tirtlnet --innodb\_log\_file\_size=512M --innodb\_io\_capacity=6000 --innodb\_buffer\_pool\_size=256M  
2023-05-09T09:17:19.387874+10:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --port=3306 --socket=/var/run/mysqld/mysqld.sock --backup=1 --user=admin --password=\* --target-dir=/srv/mysql\_backups/for\_mysqld2.bak  
xtrabackup version 8.0.31-24 based on MySQL server 8.0.31 Linux (x86\_64) (revision id: f0754edb)  
230509 09:17:19 version\_check Connecting to MySQL server with DSN ‘dbi:mysql:;mysql\_read\_default\_group=xtrabackup;port=3306;mysql\_socket=/var/run/mysqld/mysqld.sock’ as ‘admin’ (using password: YES).  
230509 09:17:19 version\_check Connected to MySQL server  
230509 09:17:19 version\_check Executing a version check against the server…  
230509 09:17:19 version\_check Done.  
2023-05-09T09:17:19.449524+10:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: admin, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock  
2023-05-09T09:17:19.452692+10:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.0.31  
2023-05-09T09:17:19.454722+10:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK INSTANCE FOR BACKUP …  
2023-05-09T09:17:19.456673+10:00 0 [Note] [MY-011825] [Xtrabackup] uses posix\_fadvise().  
2023-05-09T09:17:19.456704+10:00 0 [Note] [MY-011825] [Xtrabackup] cd to /var/lib/mysql/tirtlnet  
2023-05-09T09:17:19.456723+10:00 0 [Note] [MY-011825] [Xtrabackup] open files limit requested 0, set to 1024  
2023-05-09T09:17:19.456751+10:00 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration:  
2023-05-09T09:17:19.456760+10:00 0 [Note] [MY-011825] [Xtrabackup] innodb\_data\_home\_dir = .  
2023-05-09T09:17:19.456763+10:00 0 [Note] [MY-011825] [Xtrabackup] innodb\_data\_file\_path = ibdata1:12M:autoextend  
2023-05-09T09:17:19.456780+10:00 0 [Note] [MY-011825] [Xtrabackup] innodb\_log\_group\_home\_dir = ./  
2023-05-09T09:17:19.456787+10:00 0 [Note] [MY-011825] [Xtrabackup] innodb\_log\_files\_in\_group = 2  
2023-05-09T09:17:19.456793+10:00 0 [Note] [MY-011825] [Xtrabackup] innodb\_log\_file\_size = 536870912  
2023-05-09T09:17:19.457887+10:00 0 [Note] [MY-011825] [Xtrabackup] inititialize\_service\_handles suceeded  
2023-05-09T09:17:19.524080+10:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: admin, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock  
2023-05-09T09:17:19.526846+10:00 0 [Note] [MY-011825] [Xtrabackup] Redo Log Archiving is not set up.  
2023-05-09T09:17:19.636272+10:00 1 [Note] [MY-011825] [Xtrabackup] \>\> log scanned up to (159102699)  
2023-05-09T09:17:19.849323+10:00 0 [Note] [MY-011825] [Xtrabackup] Generating a list of tablespaces  
2023-05-09T09:17:19.849613+10:00 0 [Note] [MY-011825] [Xtrabackup] Generating a list of tablespaces  
2023-05-09T09:17:19.850198+10:00 0 [Note] [MY-012204] [InnoDB] Scanning ‘./’  
2023-05-09T09:17:19.854945+10:00 0 [Note] [MY-012204] [InnoDB] Scanning ‘/var/lib/mysql/undo/tirtlnet/’  
2023-05-09T09:17:19.855034+10:00 0 [Note] [MY-012208] [InnoDB] Completed space ID check of 2 files.  
2023-05-09T09:17:19.855140+10:00 0 [Warning] [MY-012091] [InnoDB] Allocated tablespace ID 490 for vic@002droads/wim\_axle, old maximum was 0  
2023-05-09T09:17:19.863771+10:00 0 [Note] [MY-013252] [InnoDB] Using undo tablespace ‘/var/lib/mysql/undo/tirtlnet/undo\_001’.  
2023-05-09T09:17:19.864218+10:00 0 [Note] [MY-013252] [InnoDB] Using undo tablespace ‘/var/lib/mysql/undo/tirtlnet/undo\_002’.  
2023-05-09T09:17:19.864718+10:00 0 [Note] [MY-012910] [InnoDB] Opened 2 existing undo tablespaces.  
2023-05-09T09:17:19.864999+10:00 2 [Note] [MY-011825] [Xtrabackup] Copying ./tirtlnet/orientation.ibd to /srv/mysql\_backups/for\_mysqld2.bak/tirtlnet/orientation.ibd  
2023-05-09T09:17:19.865252+10:00 2 [Note] [MY-011825] [Xtrabackup] Done: Copying ./tirtlnet/orientation.ibd to /srv/mysql\_backups/for\_mysqld2.bak/tirtlnet/orientation.ibd  
2023-05-09T09:17:19.870660+10:00 2 [Note] [MY-011825] [Xtrabackup] Copying ./vic@002ddoj/voi\_site.ibd to /srv/mysql\_backups/for\_mysqld2.bak/vic@002ddoj/voi\_site.ibd  
2023-05-09T09:17:19.871534+10:00 2 [Note] [MY-011825] [Xtrabackup] Done: Copying ./vic@002ddoj/voi\_site.ibd to /srv/mysql\_backups/for\_mysqld2.bak/vic@002ddoj/voi\_site.ibd  
2023-05-09T09:17:19.873837+10:00 2 [Note] [MY-011825] [Xtrabackup] Copying ./ibdata1 to /srv/mysql\_backups/for\_mysqld2.bak/ibdata1  
2023-05-09T09:17:19.887136+10:00 2 [Note] [MY-011825] [Xtrabackup] Database page corruption detected at page 256, retrying…  
2023-05-09T09:17:19.993844+10:00 2 [Note] [MY-011825] [Xtrabackup] Database page corruption detected at page 256, retrying…  
2023-05-09T09:17:20.102886+10:00 2 [Note] [MY-011825] [Xtrabackup] Database page corruption detected at page 256, retrying…  
2023-05-09T09:17:20.212189+10:00 2 [Note] [MY-011825] [Xtrabackup] Database page corruption detected at page 256, retrying…  
2023-05-09T09:17:20.321017+10:00 2 [Note] [MY-011825] [Xtrabackup] Database page corruption detected at page 256, retrying…  
2023-05-09T09:17:20.429678+10:00 2 [Note] [MY-011825] [Xtrabackup] Database page corruption detected at page 256, retrying…  
2023-05-09T09:17:20.538434+10:00 2 [Note] [MY-011825] [Xtrabackup] Database page corruption detected at page 256, retrying…  
2023-05-09T09:17:20.636696+10:00 1 [Note] [MY-011825] [Xtrabackup] \>\> log scanned up to (159102699)  
2023-05-09T09:17:20.647886+10:00 2 [Note] [MY-011825] [Xtrabackup] Database page corruption detected at page 256, retrying…  
2023-05-09T09:17:20.756374+10:00 2 [Note] [MY-011825] [Xtrabackup] Database page corruption detected at page 256, retrying…  
2023-05-09T09:17:20.864979+10:00 2 [ERROR] [MY-011825] [Xtrabackup] failed to read page after 10 retries. File ./ibdata1 seems to be corrupted.  
2023-05-09T09:17:20.875691+10:00 2 [ERROR] [MY-011825] [Xtrabackup] xtrabackup\_copy\_datafile() failed  
2023-05-09T09:17:20.875802+10:00 2 [ERROR] [MY-011825] [Xtrabackup] failed to copy datafile ./ibdata1  
2023-05-09T09:17:21.637187+10:00 1 [Note] [MY-011825] [Xtrabackup] \>\> log scanned up to (159102699)

I have tried running innochecksum as suggested in other forums posts:

sudo -u mysql innochecksum ibdata1  
Fail: page 256 invalid  
Exceeded the maximum allowed checksum mismatch count::0

---

<div class="post-metadata">

**Author:** ![Marcelo\_Altmann](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/marcelo_altmann/32/11299_2.png) [@Marcelo\_Altmann](https://forums.percona.com/u/Marcelo_Altmann)\
**Post date:** [May 9, 2023, 12:17am UTC](https://forums.percona.com/t/xtrabackup-8-0-31-reports-database-page-corruption-with-fresh-mysql-community-8-0-31-database/21938/2 "2023-05-09T00:17:21Z")

</div>

Hi @murray.atkinson .

Based on the output of innochecksum this is clearly a data corruption. Xtrabackup validates page checksums while it’s copying the data to avoid corrupted backups.

Track, where the corruption has happened in the first place, is rather a difficult task. Since this is a test I would advise you to start over the process.

---

<div class="post-metadata">

**Author:** ![murray.atkinson](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@murray.atkinson](https://forums.percona.com/u/murray.atkinson)\
**Post date:** [May 9, 2023, 6:20am UTC](https://forums.percona.com/t/xtrabackup-8-0-31-reports-database-page-corruption-with-fresh-mysql-community-8-0-31-database/21938/3 "2023-05-09T06:20:12Z")

</div>

Hi Marcelo,

thanks for your quick reply. This was my 2nd attempt. It’s curious that MySQL doesn’t report any errors but xtrabackup does.

I’ve since tried upgrading mysql & xtrabackup to 8.0.32 on the server. MySQL upgraded without issue and I can still make use of the server. without issues.  
xtrabackup still reports the same page corruption error with ibdata1.

I will try a clean mysql database with 8.0.32 and see what happens.

Thanks,

Murray.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [May 9, 2023, 1:59pm UTC](https://forums.percona.com/t/xtrabackup-8-0-31-reports-database-page-corruption-with-fresh-mysql-community-8-0-31-database/21938/4 "2023-05-09T13:59:13Z")

</div>

@murray.atkinson,  
InnoDB only checks pages when they are loaded into memory (ie: SELECT). It is perfectly possible for MySQL/InnoDB to not report any corruption errors if those pages are never accessed.  
This is different from xtrabackup which must read every single page in order to take a backup and therefor looks at every checksum.  
In theory, if you created a logical backup using mysqldump or MyDumper, you would see page corruption messages in your mysql log since both tools access every page during the dump process.

---

<div class="post-metadata">

**Author:** ![murray.atkinson](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@murray.atkinson](https://forums.percona.com/u/murray.atkinson)\
**Post date:** [May 10, 2023, 1:35am UTC](https://forums.percona.com/t/xtrabackup-8-0-31-reports-database-page-corruption-with-fresh-mysql-community-8-0-31-database/21938/5 "2023-05-10T01:35:18Z")

</div>

Hi Matthew,

thanks for your clarification about MySQL’s behavior.

I am trying to track down which step in our initialisation process for setting up new MySQL servers is producing this corruption.

I currently suspect it happens after a reboot and running of some of our initilialisation steps for our multi\_mysql setup.

I have tried creating a new empty MySQL server which PXB successfully backs up. Then shudown MySQL, shutdown the VirtualBox VM, take a snapshot, start the VM which starts MySQL and then PXB reports a corrupted ibdata1 file.

Interestingly, running mysqldump --all-databases on the MySQL server succeeds without any errors.

Anyway, I will continue to great a minimal system & track down what event it producing this ibdata1 file corruption.

Again, thank you for you comments.

Murray

---

<div class="post-metadata">

**Author:** ![murray.atkinson](https://avatars.discourse-cdn.com/v4/letter/m/cab0a1/32.png) [@murray.atkinson](https://forums.percona.com/u/murray.atkinson)\
**Post date:** [May 10, 2023, 5:01am UTC](https://forums.percona.com/t/xtrabackup-8-0-31-reports-database-page-corruption-with-fresh-mysql-community-8-0-31-database/21938/6 "2023-05-10T05:01:54Z")

</div>

I have confirmed that my issue is somehow related to rebooting my VirtualBox VM. With MySQL shutdown (no mysqld processes running). Taking the md5sum of ibdata1 & mysql.ibd then shutdown -h 0 of the VM, then starting the VM without MySQL starting the md5sums of the ibdata1 & mysql.ibd have changed but the mtime of the files havn’t changed.

Something very strange happening, so more investigation required.

Again, thanks for you advice.

Murray.
