Xtrabackup: Database page corruption detected at page

MySQL & MariaDB Percona XtraBackup
1

Hello Team,
What is the fix for already reported issue. I am also facing the issue no luck after performed innochecksum to fix the corrupted file. I am getting same error when tried the same thing in different servers in my infra. I don’t think so the issue related to table corruption. Can someone pls help me on this.

230704 09:24:56 [01] Copying ./ibdata1 to /var/tmp/backupxtra/ibdata1

[01] xtrabackup: Database page corruption detected at page 192, retrying...

[01] xtrabackup: Database page corruption detected at page 192, retrying...

[01] xtrabackup: Database page corruption detected at page 192, retrying...

[01] xtrabackup: Database page corruption detected at page 192, retrying...

230704 09:24:57 >> log scanned up to (24491496343)

[01] xtrabackup: Database page corruption detected at page 192, retrying...

[01] xtrabackup: Database page corruption detected at page 192, retrying...

[01] xtrabackup: Database page corruption detected at page 192, retrying...

[01] xtrabackup: Database page corruption detected at page 192, retrying...

[01] xtrabackup: Database page corruption detected at page 192, retrying...

[01] xtrabackup: Error: failed to read page after 10 retries. File ./ibdata1 seems to be corrupted.

[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.

[01] xtrabackup: Error: failed to copy datafile.

230704 09:24:58 >> log scanned up to (24491496343)

innochecksum -p 192 /store/vc/ibdata1

Fail: page 192 invalid

Exceeded the maximum allowed checksum mismatch count::0

mysql --version
mysql  Ver 8.0.23-14 for Linux on x86_64 (Percona Server (GPL), Release '14', Revision '3558242')

xtrabackup --version
xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)
2

Hi @jegana,

Can you try using appropriate xtrabackup for mysql Ver 8.0.23-14. I see you’re using xtrabackup 8.0.25.

Thanks,
K

3

@kedarpercona Thanks for your reply. Should the versions of both MySQL and XtraBackup be identical

4

sure @jegana, that’s correct.

5

But in this link mentioned that xtrabackup version can be equal or higher than source server(mysql) version. In my case I am having higher version of xtrabackup than mysql version.

mysql --version
mysql  Ver 8.0.23-14 for Linux on x86_64 (Percona Server (GPL), Release '14', Revision '3558242')

xtrabackup --version
xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)
6

Tried both with same version even though facing the same issue.

mysql --version
mysql  Ver 8.0.31-23 for Linux on x86_64 (Percona Server (GPL), Release '23', Revision '71449379')

xtrabackup --version
xtrabackup version 8.0.31-24 based on MySQL server 8.0.31 Linux (x86_64) (revision id: f0754edb)
7

Hello @jegana

It appears that you’re trying to create a backup.

  • Can you provide with the backup command you’re using?
  • Also can you verify mysql error logs for no issues? Is your database OK? (I ask this to confirm if your database is not already corrupted, if so, backup is not going to work)

Also, innochecksum to fix the corrupted file → this is not right. Innochecksum will only print the checksum for innodb file.

Thanks,
K.

8

Hello @kedarpercona

It is newly build server so I believe that there will be no chances for database corrupted. I have used the following commend to take the backup and got below message in mysql error.log.

xtrabackup --backup --target-dir=/var/tmp/backupxtra -u root

2023-07-05T09:34:15.383470Z 464 [Warning] [MY-013712] [Server] No suitable ‘keyring_component_metadata_query’ service implementation found to fulfill the request.

9

Hi @jegana,
So your error from the log is known bug and is harmless
WRT what you’re trying to achieve.

Also mysql is client and that could be of different version than the server mysqld. Can you confirm the server version?
mysql> SELECT VERSION();

About the error File ./ibdata1 seems to be corrupted can you confirm if your backup --target-dir=/var/tmp/backupxtra has sufficient space? Could you try changing that path?

Thanks,
K.

10

Hey @kedarpercona

Both the mysql client and mysqld versions are same. I have sufficient space for target-dir and tried with different target dir as well but the results are same.

Can you pls suggest me how to fix the corrupted table.

mysqld version

mysql> SELECT VERSION();

+-----------+

| VERSION() |

+-----------+

| 8.0.33-25 |

+-----------+

1 row in set (0.00 sec)

mysql version

mysql --version
mysql  Ver 8.0.33-25 for Linux on x86_64 (Percona Server (GPL), Release '25', Revision '60c9e2c5')
11

@jegana
Check and let us know if you see any corruption message on the source MySQL error log.

Other possible options to test and fix the issue:

  • Do you have a replica? If yes, try taking xtrabackup from it and see if it works.
  • If the database size is small, you can take a logical full backup of databases and restore to fix filesystem-level corruption.
12

Hi @jegana

This message tells you that page 192 is invalid. Xtrabackup is just reporting the same.
The only way to overcome this is to attempt a logical backup (mysqldump) and restore on a brand new instance.

1 Like
13

Hey @Marcelo_Altmann

Sure. I will give a try and share you how it is working.

1 Like
14

I don’t encounter the reported error when my application runs on unencrypted AWS EBS volume, but I do encounter the reported error when my application runs on an encrypted AWS EBS volume. Any reason the Xtrabackup is working differently on encrypted and unencrypted volumes.

15

FYI: Xtrabackup - Issue with encrypted volume

16

We did further checking using innochecksum and the output is as below.
innochecksum -f /store2/ibdata1
Unknown file format flags: 18432
Please note that innochecksum per ibd files are working fine and it is failing only for ibdata1 file. Any clue on this file
format flags?