Problems with applying incremental logs.

Hello
I am having problems, getting the incremental backup to apply to the full backup.

Created some backups for the test with the followoing commands.

innobackupex --no-timestamp --user=bkpuser --password=… --no-lock /storage/backups/prodmydb/fullbackup
innobackupex --no-timestamp --user=bkpuser --password=… --no-lock /storage/backups/prodmydb/incremental --incremental --incremental-basedir=/storage/backups/prodmydb/fullbackup
innobackupex --no-timestamp --user=bkpuser --password=… --no-lock /storage/backups/prodmydb/incremental2 --incremental --incremental-basedir=/storage/backups/prodmydb/incremental/

Then i want to apply the log to prepare the backup the first one the fullbackup applies fine as you can see below, however the when i want to apply the first incremental throws an error which is in the bottom of the post.

It is the same error with or without --no-lock

I followed the howto here [url]https://www.percona.com/doc/percona-xtrabackup/2.4/innobackupex/incremental_backups_innobackupex.html[/url]

System is a mysql 5.7.21 galera cluster 2 nodes and a garbd running on Centos 7.

Any help would be appreciated


innobackupex --apply-log --redo-only /storage/backups/prodmydb/fullbackup/

xtrabackup: recognized server arguments: --innodb_checksum_algorithm=crc32 --innodb_log_checksum_algorithm=strict_crc32 --innodb_data_file_path=ibdata1:12M:autoextend --innodb_log_files_in_group=2 --innodb_log_file_size=50331648 --innodb_fast_checksum=0 --innodb_page_size=16384 --innodb_log_block_size=512 --innodb_undo_directory=./ --innodb_undo_tablespaces=0 --server-id=1 --redo-log-version=1
xtrabackup: recognized client arguments: --innodb_checksum_algorithm=crc32 --innodb_log_checksum_algorithm=strict_crc32 --innodb_data_file_path=ibdata1:12M:autoextend --innodb_log_files_in_group=2 --innodb_log_file_size=50331648 --innodb_fast_checksum=0 --innodb_page_size=16384 --innodb_log_block_size=512 --innodb_undo_directory=./ --innodb_undo_tablespaces=0 --server-id=1 --redo-log-version=1
180830 10:19:17 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints “completed OK!”.

innobackupex version 2.4.12 based on MySQL server 5.7.19 Linux (x86_64) (revision id: 170eb8c)
xtrabackup: cd to /storage/backups/prodmydb/fullbackup/
xtrabackup: This target seems to be not prepared yet.
InnoDB: Number of pools: 1
xtrabackup: xtrabackup_logfile detected: size=52232192, start_lsn=(763360013256)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = .
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 52232192
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = .
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 52232192
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: PUNCH HOLE support available
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Uses event mutexes
InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
InnoDB: Compressed tables use zlib 1.2.7
InnoDB: Number of pools: 1
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
InnoDB: Completed initialization of buffer pool
InnoDB: page_cleaner coordinator priority: -20
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 763360013256
InnoDB: Doing recovery: scanned up to log sequence number 763365255680 (11%)
InnoDB: Doing recovery: scanned up to log sequence number 763370498560 (22%)
InnoDB: Doing recovery: scanned up to log sequence number 763375741440 (33%)
InnoDB: Doing recovery: scanned up to log sequence number 763380984320 (45%)
InnoDB: Doing recovery: scanned up to log sequence number 763386227200 (56%)
InnoDB: Doing recovery: scanned up to log sequence number 763391470080 (67%)
InnoDB: Doing recovery: scanned up to log sequence number 763396712960 (79%)
InnoDB: Doing recovery: scanned up to log sequence number 763401955840 (90%)
InnoDB: Doing recovery: scanned up to log sequence number 763406449503 (100%)
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Doing recovery: scanned up to log sequence number 763360733696 (1%)
InnoDB: Doing recovery: scanned up to log sequence number 763365976576 (12%)
InnoDB: Doing recovery: scanned up to log sequence number 763371219456 (24%)
InnoDB: Doing recovery: scanned up to log sequence number 763376462336 (35%)
InnoDB: Doing recovery: scanned up to log sequence number 763381705216 (46%)
InnoDB: Starting an apply batch of log records to the database…
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Doing recovery: scanned up to log sequence number 763386948096 (58%)
InnoDB: Doing recovery: scanned up to log sequence number 763392190976 (69%)
InnoDB: Doing recovery: scanned up to log sequence number 763397433856 (80%)
InnoDB: Doing recovery: scanned up to log sequence number 763402676736 (91%)
InnoDB: Doing recovery: scanned up to log sequence number 763406449503 (100%)
InnoDB: 2 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
InnoDB: Trx id counter is 1832808704
InnoDB: Starting an apply batch of log records to the database…
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: page_cleaner: 1000ms intended loop took 10704ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)

xtrabackup: Recovered WSREP position: 7955d49d-1c6b-11e8-81b7-17b39a3815ae:1167901398
xtrabackup: starting shutdown with innodb_fast_shutdown = 1
InnoDB: Starting shutdown…
InnoDB: Shutdown completed; log sequence number 763406453345
InnoDB: Number of pools: 1
180830 10:19:30 completed OK!


innobackupex --apply-log --redo-only /storage/backups/prodmydb/fullbackup/ --incremental-dir=/storage/backups/prodmydb/incremental

xtrabackup: recognized server arguments: --innodb_checksum_algorithm=crc32 --innodb_log_checksum_algorithm=strict_crc32 --innodb_data_file_path=ibdata1:12M:autoextend --innodb_log_files_in_group=2 --innodb_log_file_size=50331648 --innodb_fast_checksum=0 --innodb_page_size=16384 --innodb_log_block_size=512 --innodb_undo_directory=./ --innodb_undo_tablespaces=0 --server-id=1 --redo-log-version=1
xtrabackup: recognized client arguments: --innodb_checksum_algorithm=crc32 --innodb_log_checksum_algorithm=strict_crc32 --innodb_data_file_path=ibdata1:12M:autoextend --innodb_log_files_in_group=2 --innodb_log_file_size=50331648 --innodb_fast_checksum=0 --innodb_page_size=16384 --innodb_log_block_size=512 --innodb_undo_directory=./ --innodb_undo_tablespaces=0 --server-id=1 --redo-log-version=1
180830 10:38:49 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints “completed OK!”.

innobackupex version 2.4.12 based on MySQL server 5.7.19 Linux (x86_64) (revision id: 170eb8c)
incremental backup from 763397681068 is enabled.
xtrabackup: cd to /storage/backups/prodmydb/fullbackup/
xtrabackup: This target seems to be already prepared with --apply-log-only.
InnoDB: Number of pools: 1
InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open /storage/backups/prodmydb/incremental//xtrabackup_logfile. will try to find.
xtrabackup: ‘ib_logfile0’ seems to be ‘xtrabackup_logfile’. will retry.
xtrabackup: xtrabackup_logfile detected: size=27394048, start_lsn=(763484196290)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = /storage/backups/prodmydb/incremental/
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 27394048
xtrabackup: Generating a list of tablespaces
InnoDB: Allocated tablespace ID 30 for boliawallboard/buttons, old maximum was 0
xtrabackup: page size for /storage/backups/prodmydb/incremental//ibdata1.delta is 16384 bytes
Applying /storage/backups/prodmydb/incremental//ibdata1.delta to ./ibdata1…
InnoDB: Retry attempts for reading partial data failed.
InnoDB: Tried to read 16384 bytes at offset 0, but was only able to read 0
InnoDB: Operating system error number 22 in a file operation.
InnoDB: Error number 22 means ‘Invalid argument’
InnoDB: Some operating system error numbers are described at [url]http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html[/url]
InnoDB: File (unknown): ‘read’ returned OS error 122. Cannot continue operation
InnoDB: Cannot continue operation.