Unable to restore from point-in-time physical backup

MySQL & MariaDB Percona Server for MySQL 8.0
1

Hi,

Trying to do a test backup and restore from a point-in-time storage snapshot and Percona MySQL 8.0 is failing to start with the following error:

2025-03-21T01:26:13.004557Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2025-03-21T01:26:15.355486Z 0 [ERROR] [MY-013183] [InnoDB] Assertion failure: fil0fil.cc:7982:err == DB_SUCCESS thread 140551824635456
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
2025-03-21T01:26:15Z UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
BuildID[sha1]=a32bfce59229782139dde88004d464c063c13240
Server Version: 8.0.37-29 Percona Server (GPL), Release 29, Revision 30dc4e71

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0 thread_stack 0x100000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x15e94f1]
/usr/sbin/mysqld(print_fatal_signal(int)+0x3e7) [0xde1447]
/usr/sbin/mysqld(my_server_abort()+0x6d) [0xde14cd]
/usr/sbin/mysqld(my_abort()+0xe) [0x15ded2e]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x16f) [0x17bc71f]
/usr/sbin/mysqld(fil_aio_wait(unsigned long)+0x230) [0x18d3de0]
/usr/sbin/mysqld() [0x1706dca]
/usr/sbin/mysqld() [0x17076b9]
/lib64/libstdc++.so.6(+0xdbad4) [0x7fd4d4647ad4]
/lib64/libc.so.6(+0x897f2) [0x7fd4d42f87f2]
/lib64/libc.so.6(+0x10e880) [0x7fd4d437d880]
Please help us make Percona Server better by reporting any
bugs at …

MySQL version:
mysql Ver 8.0.37-29 for Linux on x86_64 (Percona Server (GPL), Release 29, Revision 30dc4e71)

/etc/my.cnf

[mysqld]
datadir = /data/mysql
socket = /var/lib/mysql/mysql.sock

log-error = /var/log/mysqld.log
pid-file = /var/run/mysqld/mysqld.pid
auto_increment_increment = 2
auto_increment_offset = 1
binlog_transaction_compression = ON

innodb_dedicated_server = ON
innodb_io_capacity = 5000
innodb_io_capacity_max = 10000
innodb_redo_log_capacity = 32G

gtid_mode = ON
enforce_gtid_consistency = ON
read_only = ON
replicate-wild-ignore-table = mysql.%
server_id = 10

VM Instance size is 32 cores, 48 GB Mem, DB size is roughly 230G.

Any help that would point to the cause of that assertion failure would be greatly appreciated.
Thanks

2

What exactly does this mean? Please provide details on how the backup was taken, what tools were used, what commands, output of those commands, all the steps, etc.

3

Apologies, missed that in my post.

Using Veeam Backup and Restore via vSphere to take a backup of the entire virtual machine. It snapshots the virtual machine disk at a point in time for consistency, similar to how one would use an LVM snapshot to create a backup.

Thanks

Edit: For Backup & Restore steps just followed standard Veeam Backup & Restore:

4

We would need to see some logs from the backup process itself. If MySQL is not flush-locked during the backup, then partially written pages could be snap’d by Veeam, and then those corrupt pages get restored.

You should be configuring “MySQL backups”, not “VM” backups.

5

I see.

I was under the impression Veeam MySQL backups used the same storage snapshot mechanism.

Our current issue is that we lost the primary mysql DB due to a host crash and are currently running on the replica. I’m trying to create a copy of the data but not having much success so far.

It looks like the backup mechanism for MySQL 8.0 changed because when I try to backup via xtrabackup or mysqldump, it causes “Waiting for table metadata lock” on application processes which impacts prod.

Would you be able to recommend a way I can backup this DB in order to rebuild our replica?

6

Do you have any MyISAM tables in use? If every table is InnoDB, you should be able to take a hot, non-locking, backup with Xtrabackup. If you want to use mysqldump, try enabling --single-transaction or use mydumper as a better alternative to mysqldump.

7

All the tables are InnoDB.

If every table is InnoDB, you should be able to take a hot, non-locking, backup with Xtrabackup

That was my understanding as well, alternatively, the process could be killing throughput. Our production instance is 1.3TB in size…

8

You should not see this message when using Xtrabackup. How exactly are you executing PXB?
You could try --no-lock.

9

Hi,
If there are MyISAM tables, you could try running xtrabackup with --rsync (it’s undocumented, but it is there) - I have just recently had great success with mixed tablespace backups using --rsync. Even on large datasets the lock time is only a couple of seconds. For our loads that is ok, your mileage may vary.

Cheers,
Liz

10

Hi @liz.rea, above he says all tables are InnoDB. Also, the --rsync flag is a fully documented option.