XtraBackup prepare - ib_redo0 size is not a multiple of innodb_page_size

MySQL & MariaDB Percona XtraBackup
1

Hello,

We are running multiple Percona MySQL servers (8.0.x versions) and are backing them up regurarly. Some of the preparation of full backups on instances are failing (not all time, but some), giving us the following error message:

2024-10-03T12:05:14.733958-00:00 0 [Note] [MY-012201] [InnoDB] Checked 15055 files
2024-10-03T12:05:14.753836-00:00 0 [Note] [MY-012208] [InnoDB] Completed space ID check of 15057 files.
2024-10-03T12:05:14.762477-00:00 0 [Note] [MY-012955] [InnoDB] Initializing buffer pool, total size = 128.000000M, instances = 1, chunk size =128.000000M
2024-10-03T12:05:14.795773-00:00 0 [Note] [MY-012957] [InnoDB] Completed initialization of buffer pool
2024-10-03T12:05:14.805786-00:00 0 [Note] [MY-011951] [InnoDB] page_cleaner coordinator priority: -20
2024-10-03T12:05:14.805964-00:00 0 [Note] [MY-011954] [InnoDB] page_cleaner worker priority: -20
2024-10-03T12:05:14.806082-00:00 0 [Note] [MY-011954] [InnoDB] page_cleaner worker priority: -20
2024-10-03T12:05:14.806118-00:00 0 [Note] [MY-011954] [InnoDB] page_cleaner worker priority: -20
2024-10-03T12:05:14.936301-00:00 0 [ERROR] [MY-012962] [InnoDB] The redo log file ./#innodb_redo/#ib_redo0 size 8397824 is not a multiple of innodb_page_size
2024-10-03T12:05:14.936360-00:00 0 [ERROR] [MY-012930] [InnoDB] Plugin initialization aborted at srv0start.cc[1926] with error Generic error.
2024-10-03T12:05:14.937827-00:00 0 [ERROR] [MY-011825] [Xtrabackup] innodb_init(): Error occured
2024-10-03T12:05:14.938452-00:00 0 [ERROR] [MY-012962] [InnoDB] The redo log file ./#innodb_redo/#ib_redo0 size 8397824 is not a multiple of innodb_page_size
2024-10-03T12:05:14.938469-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Unable to validate the redo log files at the end of prepare

Version info:
xtrabackup version 8.0.35-30 based on MySQL server 8.0.35 Linux (x86_64) (revision id: 6beb4b49)
mysql Ver 8.0.35-27 for Linux on x86_64 (Percona Server (GPL), Release 27, Revision 2f8eeab2)
The backups are going to NFS mount.

[Actual workaround is to restart the full backup (xtrabackup --backup), then run prepare again (xtrabackup --prepare)]

What can be wrong?

Thanks,
Bence

2

8397824/(16*1024) = 512.5625

Yep. That’s odd. What is your innodb_redo_log_capacity set to? Make sure it is a multiple of 16K. After that, try taking a backup locally and prepare locally. If that works, perhaps there is an issue going to NFS.

3

The actual server has 512M configured for innodb_redo_log_capacity.
I’ve run test towards NFS and local targets:

  • run backup-test towards NFS storage: 6 failed out of 14
  • run backup-test towards local storage: 0 failed out of 91

So for now it seems to be an NFS problem. As we don’t have enough space locally, we must use NFS.
My question: can #innodb_redo folder be separated from the backup itself during the backup-prepare stages?

4

During the backup process, the redo logs can be in a different location. PXB will store its own copy of them with the backup.

5

PXB will store its own copy of them with the backup.

Well, that’s the point: seems that during --prepare some times #innodb_redo folder contents are messed up on NFS folder. The question in more detailed format:

  • we want to have the backup on NFS
  • we want to do prepare there (same folder)
  • can we separate #innodb_redo folder from the backup/prepare folder to have it in local disk?

Example command:

xtrabackup-debug \
  --defaults-file="/etc/my_${_db_instance}.cnf" \
  --socket=/var/lib/mysql1/${_db_instance}/mysql.sock \
  --user="backup" --password="***" \
  --parallel=2 \
  --prepare \
  --target-dir="${FOLDER_ON_NFS}" \
  --debug="d:f:F:L:n:P:t:i:o,/tmp/prepare.trace"

So is it possible to have #innodb_redo folder not with the backup, but in another folder (on local disk)?

6

Looks like PXB can read innodb parameters from my.cnf:

You might be able to create a [xtrabackup] section in my.cnf and specify innodb_log_group_home_dir. You could also simply try xtrabackup --innodb_log_group_home_dir=/local/disk/path ...

7

Hello,

I’ve tried many parameters and configuration settings, but couldn’t manage to get #innodb_redo folder out of the prepared data directory. The only change was that #innodb_temp could be replaced.
I’ve updated the --prepare command (took out defaults and credentials to avoid collisions):

xtrabackup-debug \
  --parallel=2 \
  --prepare \
  --target-dir="<DB_BACKUP_DIR>" \
  --redo-log-arch-dir="<LOCAL_DIR>/${_redo_log_arch_dir}" \
  --innodb-data-home-dir="<LOCAL_DIR>/${_innodb_data_home_dir}" \
  --innodb-undo-directory="<LOCAL_DIR>/${_innodb_undo_directory}" \
  --innodb-log-group-home-dir="<LOCAL_DIR>/${_innodb_log_group_home_dir}" \
  --temp-tablespaces-dir="<LOCAL_DIR>/${_temp_tablespaces_dir}" \
  --fifo-dir="<LOCAL_DIR>/${_fifo_dir}" \
  --debug="d:f:F:L:n:P:t:i:o,<LOCAL_DIR>/prepare.trace"

The only folder which has content in it during the prepare was the one set with --temp-tablespaces-dir

#innodb_redo is created in the data directory - this can be seen in the debug/trace output as well:

T@0: xtrabackup-debug:      my_lib.cc:   303:    1: >my_stat
T@0: xtrabackup-debug:      my_lib.cc:   305:    1: | my: path: './#innodb_redo'  stat_area: 0x7fff4d89cb70  MyFlags: 0
T@0: xtrabackup-debug:      my_lib.cc:   314:    1: | error: Got errno: 2 from stat
T@0: xtrabackup-debug:      my_lib.cc:    1: <my_stat
T@0: xtrabackup-debug:    my_mkdir.cc:    45:    1: >my_mkdir
T@0: xtrabackup-debug:    my_mkdir.cc:    46:    1: | enter: dir: ./#innodb_redo
T@0: xtrabackup-debug:    my_mkdir.cc:    1: <my_mkdir

So question remained open: is it possible to have #innodb_redo folder not with the backup, but in another folder (on local disk)?

8

I see lots of references to this parameter in the code:

This could be a bug, OR, you can’t override the parameters on the CLI. Try changing this parameter in my.cnf for both [mysqld] section, and [xtrabackup] section and see if PXB picks it up. You are in uncharted waters, so there will be lots of testing on your side for this unusual setup.

9

I’ve done some testing.
As MySQL documentation shows, that parameter should do what we want:

I’ve tried to set that parameter using the following methods:

  • command line parameter to xtrabackup command (--innodb-log-group-home-dir)
  • through .cnf file:
    • [xtrabackup] block, innodb_log_group_home_dir parameter
    • [mysqld] block, innodb_log_group_home_dir parameter

The latest test was run with the following .cnf and command:

xtrabackup-debug \
  --defaults-file="${MAIN_DIR}/xtrabackup-mysql.cnf" \
  --parallel=2 \
  --prepare \
  --target-dir="${_backup_baseDir}" \
  --innodb-log-group-home-dir="<LOCAL_DIR>/${_innodb_log_group_home_dir}" \
  --debug="d:f:F:L:n:P:t:i:o,<LOCAL_DIR>/prepare.trace"

[xtrabackup]
innodb_log_group_home_dir           = <LOCAL_DIR>/innodb_log_group_home_dir

[mysqld]
innodb_log_group_home_dir           = <LOCAL_DIR>/innodb_log_group_home_dir

Neither of them helped, so it seems that PXB doesn’t pick up that parameter.
What type of test should I run?

Can we say that this is a bug in Percona Xtrabackup?

10

Looks like it. Please open a bug report at https://jira.percona.com and include as much detail as you can, including a repeatable test case with all parameters that you are using. Our engineering team will take a look.

11

Looks like it. Please open a bug report at https://jira.percona.com and include as much detail as you can, including a repeatable test case with all parameters that you are using. Our engineering team will take a look.

I’ve opened the ticket: [PXB-3392] xtrabackup doesn’t pick up --innodb-log-group-home-dir config parameter - Percona JIRA (atlassian.net)
Is there anything I can do about it or from this point Percona Team will assign that ticket to somebody?

Thanks for the help!

12

That’s all you can do, unless you can fix the code yourself :smiley: Our engineering team will take a look at it, someone will attempt to reproduce it and confirm, then it will be prioritized for an engineer to take a look.

1 Like