MariaDB 5.5.29, Xtrabackup 2.0.5 on el5 - all from repo. Works fine on several servers with smaller datasets, but fails on the big one (350G):
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 995545447, file name mysql-bin.001309
InnoDB: and relay log file
InnoDB: position 0 995545731, file name ./mysql-relay-bin.002034
InnoDB: Last MySQL binlog file position 0 276207520, file name ./mysql-bin.000005
130417 10:18:35 InnoDB: Waiting for the background threads to start
130417 10:18:36 Percona XtraDB ([url]http://www.percona.com[/url]) 1.1.8-20.1 started; log sequence number 13758730250258
130417 10:18:38 InnoDB: Assertion failure in thread 1191156032 in file xtrabackup.c line 1632
InnoDB: Failing assertion: cset == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to [url]http://bugs.mysql.com[/url].
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: [url]http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html[/url]
InnoDB: about forcing recovery.
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 381.
Any suggestions what to check? Workarounds may be?
Thanks!
Partial error log:
…
…
…
InnoDB: Doing recovery: scanned up to log sequence number 309111833600 (88 %)
InnoDB: Doing recovery: scanned up to log sequence number 309114524155 (88 %)
130419 17:33:34 InnoDB: Starting an apply batch of log records to the database…
InnoDB: Progress in percents: 0 1 InnoDB: Probable data corruption on page 1662
InnoDB: Original record PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 32
0: len 4; hex 00000043; asc C;;
1: len 1; hex 00; asc ;;
2: len 4; hex 000047a0; asc G ;;
3: len 22; hex 00000001860300048000860800088000860800088000; asc ;;
4: len 4; hex 81757165; asc uqe;;
5: len 8; hex 8000013e1e903b68; asc > ;h;;
6: len 8; hex 800000000042e94f; asc B O;;
InnoDB: on that page.
InnoDB: Cannot find the dir slot for record PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 0
0: len 4; hex 00000043; asc C;;
1: len 1; hex 00; asc ;;
2: len 4; hex 000047ce; asc G ;;
3: len 22; hex 00000001860300048000860800088000860800088000; asc ;;
4: len 4; hex 81828972; asc r;;
5: len 8; hex 8000013e1ea8e7d0; asc > ;;
6: len 8; hex 800000000042e9e9; asc B ;;
InnoDB: on that page!
130419 17:33:36 InnoDB: Page dump in ascii and hex (16384 bytes):
…
…
…
InnoDB: End of page dump
130419 17:33:36 InnoDB: Page checksum 2139845666 (32bit_calc: 3413430305), prior-to-4.0.14-form checksum 705155133
InnoDB: stored checksum 700663176, prior-to-4.0.14-form stored checksum 71
InnoDB: Page lsn 71 3951049913, low 4 bytes of lsn at page end 3951049913
InnoDB: Page number (if stored to page already) 1662,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 18446744069414584320
InnoDB: (index “CLUST_IND” of table “SYS_IBUF_TABLE”)
130419 17:33:36 InnoDB: Assertion failure in thread 140549583533824 in file page0page.c line 153
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to [URL=“http://bugs.mysql.com”]http://bugs.mysql.com[/URL=“http://bugs.mysql.com”].
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: [URL=“MySQL :: MySQL 8.0 Reference Manual :: 15.21.3 Forcing InnoDB Recovery”]http://dev.mysql.com/doc/refman/5.5/…-recovery.html[/URL=“MySQL :: MySQL 8.0 Reference Manual :: 15.21.3 Forcing InnoDB Recovery”]
InnoDB: about forcing recovery.
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 381.
I do encounter the same problem, like DmitryK - however, I do encounter it on MySQL instead of mariaDB.
First of all, my versions:
MySQL 5.5.27
Percona XtraBackup 2.0.6
What’s strange in my case, is the fact that the backups for two of the three instances on my backup node can be restored without any hassle - but the third, and largest one ( about 1.2TB data-dir / 437GB zipped backup ) ultimately fails with the above mentioned assertion failure:
130426 9:04:02 InnoDB: Assertion failure in thread 139909354481408 in file xtrabackup.c line 1601
InnoDB: Failing assertion: cset == 0
I am using the following statements to create and restore the backup