Xtrabackup not pulling all innodb settings

MySQL & MariaDB Percona XtraBackup
1

Forgive me if this is in the wrong forum.

I have a MariaDB Galera cluster using xtrabackup to do the new node initialization (SST script.)

We have a pretty unique setup. We are running mariaDB using ZFS file system. We have tuned ZFS for both data and log files - each having their own file system.

after the initial transfer of data xtrabackup is run with --apply-logs. It is creating ib_logfile’s in the data directory instead of the logFiles directory.

my server.cnf
…
innodb_log_group_home_dir = /database/logFiles
innodb_data_home_dir = /database/data
innodb_log_file_size=2000M
innodb_log_files_in_group=2
innodb_use_native_aio=FALSE
…

NOTE: my /etc/my.cnf just has:
!includedir /etc/my.cnf.d

but that does not seem to get picked up by xtrabackup so i have to specify the full /etc/my.cnf.d/server.cnf

the apply log portion of the transfer:

130408 15:14:52 innobackupex: Restarting xtrabackup with command: xtrabackup_55 --defaults-file=“/etc/my.cnf.d/server.cnf” --defaults-group=“mysqld” --prepare --target-dir=/database/data --tmpdir=/database/tmp/
for creating ib_logfile*

xtrabackup_55 version 2.0.6 for Percona Server 5.5.16 Linux (x86_64) (revision id: 521)
xtrabackup: cd to /database/data
xtrabackup: This target seems to be already prepared.
xtrabackup: notice: xtrabackup_logfile was already used to ‘–prepare’.
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 2097152000
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 2097152000
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
130408 15:14:52 InnoDB: The InnoDB memory heap is disabled
130408 15:14:52 InnoDB: Mutexes and rw_locks use GCC atomic builtins
130408 15:14:52 InnoDB: Compressed tables use zlib 1.2.3
130408 15:14:52 InnoDB: Initializing buffer pool, size = 100.0M
130408 15:14:52 InnoDB: Completed initialization of buffer pool
130408 15:14:52 InnoDB: Log file ./ib_logfile0 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile0 size to 2000 MB
InnoDB: Database physically writes the file full: wait…
InnoDB: Progress in MB: 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000
130408 15:16:41 InnoDB: Log file ./ib_logfile1 did not exist: new to be created
InnoDB: Setting log file ./ib_logfile1 size to 2000 MB
InnoDB: Database physically writes the file full: wait…
InnoDB: Progress in MB: 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000
130408 15:18:26 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130408 15:18:26 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Last MySQL binlog file position 0 17819, file name ./mysql-bin.000027
130408 15:18:26 InnoDB: Waiting for the background threads to start
130408 15:18:27 Percona XtraDB (grrrrr link removed. bah!!!) 1.1.8-20.1 started; log sequence number 137716748

so it is picking up the size and number of log files… it just isn’t putting them in the right spot.

we have also had an issue with it not picking up the aio settings, but i think that is in the Galera xtrabackup script, not xtrabackup itself.

the problem is that when MariaDB starts and tries to apply changes since the process started it always fails. As best as we can guess it is because it is pointing to the wrong ib_logfiles.

we worked around this by forcing maria to put the log files in the same directory, then once synced, shut down the server, move the log files, change the location in the config and restart the server. it does a quick catch-up and it is all set. but this is kind of a pain.

any help would be appreciated.

thanks.

2

According to your input, it would seem that xtrabackup SST is ignoring --innodb-log-group-home-dir when it is not on my.cnf. The cause should be isolated - for example, running and restoring with xtrabackup manually and/or setting it temporarily on that file. I am assuming that you have the exact same configuration on both nodes, don’t you?

Once done, I would recommend you to search for an issue/report one for xtrabackup (if it fails manually) here: [url]https://bugs.launchpad.net/percona-xtrabackup[/url] or to MariaDB Cluster bugs.

3

It is ignoring it at all times. even when i point to the config file directly containing that parameter it still creates the innodb log files in the wrong directory.

130408 15:14:52 innobackupex: Restarting xtrabackup with command: xtrabackup_55 --defaults-file=“/etc/my.cnf.d/server.cnf” --defaults-group=“mysqld” --prepare --target-dir=/database/data --tmpdir=/database/tmp/

for creating ib_logfile*
…

xtrabackup: innodb_log_group_home_dir = ./

In the original log you can see that xtrabackup is launched pointing to the server.cnf as it’s defaults-file, and yet when it spits out it’s run parameters it is not using innodb-log-group-home.

server.cnf contains:

[mysqld]


socket = /database/mysql.sock


port = 3306


datadir = /database/data


tmpdir = /database/tmp/


innodb_log_group_home_dir = /database/logFiles

I have also seen this behavior outside of the SST/clustering environment. Then, i could remove the log files and restart MariaDB and it would create new ones and there weren’t any transactions going on to cause an issue.

4

One more piece of info…

This is only occurring during the prepare portion. The initial backup sees the logs just fine.

5

This is a known bug with the xtrabackup SST script - no known workaround so far except to consolidate everything on a single datadir, stop mysql after SST update configuration and move the files manually to the intended locations and restart the node again.

[url]https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1098566[/url]

Or do a manual SST instead [url]http://www.mysqlperformanceblog.com/2012/08/02/avoiding-sst-when-adding-new-percona-xtradb-cluster-node/[/url]

Of course, you can also engage Percona services to get this bug fixed.

6

Thanks for the response. I’ll broach the subject of a bug bounty with the money people.

For now, we do have a work around - doing what you said. I’m just happy it works at all. :wink: