Not the answer you need?
Register and ask your own question!

Xtrabackup is failing once log is applied

DmitryKDmitryK EntrantCurrent User Role Beginner
MariaDB 5.5.29, Xtrabackup 2.0.5 on el5 - all from repo. Works fine on several servers with smaller datasets, but fails on the big one (350G):


InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 995545447, file name mysql-bin.001309
InnoDB: and relay log file
InnoDB: position 0 995545731, file name ./mysql-relay-bin.002034
InnoDB: Last MySQL binlog file position 0 276207520, file name ./mysql-bin.000005
130417 10:18:35 InnoDB: Waiting for the background threads to start
130417 10:18:36 Percona XtraDB (http://www.percona.com) 1.1.8-20.1 started; log sequence number 13758730250258
130417 10:18:38 InnoDB: Assertion failure in thread 1191156032 in file xtrabackup.c line 1632
InnoDB: Failing assertion: cset == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 381.



Any suggestions what to check? Workarounds may be?
Thanks!

Comments

  • gmousegmouse Mod Squad Inactive User Role Beginner
    Do you use special features of MariaDB that are incompatible with standard InnoDB?
  • DmitryKDmitryK Entrant Current User Role Beginner
    Not really, only default ones. Not using area tables ether.
  • harveyzhharveyzh Entrant Current User Role Beginner
    I also have a problem.


    Packages info:

    Percona-Server-server-55-5.5.29-rel29.4.401.rhel6.x86_64
    percona-xtrabackup-2.0.5-499.rhel6.x86_64


    Commands:

    # cat /data/backup/mysql/2013-04-19_11-04-34/xtrabackup_checkpoints
    backup_type = full-prepared
    from_lsn = 0
    to_lsn = 308184144593
    last_lsn = 308882154381

    # innobackupex --apply-log --redo-only --use-memory=16G /data/backup/mysql/2013-04-19_11-04-34/ --incremental-dir=/data/backup/mysql/2013-04-19_13-42-17

    Partial error log:
    ......
    ......
    ......
    InnoDB: Doing recovery: scanned up to log sequence number 309111833600 (88 %)
    InnoDB: Doing recovery: scanned up to log sequence number 309114524155 (88 %)
    130419 17:33:34 InnoDB: Starting an apply batch of log records to the database...
    InnoDB: Progress in percents: 0 1 InnoDB: Probable data corruption on page 1662
    InnoDB: Original record PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 32
    0: len 4; hex 00000043; asc C;;
    1: len 1; hex 00; asc ;;
    2: len 4; hex 000047a0; asc G ;;
    3: len 22; hex 00000001860300048000860800088000860800088000; asc ;;
    4: len 4; hex 81757165; asc uqe;;
    5: len 8; hex 8000013e1e903b68; asc > ;h;;
    6: len 8; hex 800000000042e94f; asc B O;;

    InnoDB: on that page.
    InnoDB: Cannot find the dir slot for record PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 0
    0: len 4; hex 00000043; asc C;;
    1: len 1; hex 00; asc ;;
    2: len 4; hex 000047ce; asc G ;;
    3: len 22; hex 00000001860300048000860800088000860800088000; asc ;;
    4: len 4; hex 81828972; asc r;;
    5: len 8; hex 8000013e1ea8e7d0; asc > ;;
    6: len 8; hex 800000000042e9e9; asc B ;;

    InnoDB: on that page!
    130419 17:33:36 InnoDB: Page dump in ascii and hex (16384 bytes):
    ......
    ......
    ......
    InnoDB: End of page dump
    130419 17:33:36 InnoDB: Page checksum 2139845666 (32bit_calc: 3413430305), prior-to-4.0.14-form checksum 705155133
    InnoDB: stored checksum 700663176, prior-to-4.0.14-form stored checksum 71
    InnoDB: Page lsn 71 3951049913, low 4 bytes of lsn at page end 3951049913
    InnoDB: Page number (if stored to page already) 1662,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
    InnoDB: Page may be an index page where index id is 18446744069414584320
    InnoDB: (index "CLUST_IND" of table "SYS_IBUF_TABLE")
    130419 17:33:36 InnoDB: Assertion failure in thread 140549583533824 in file page0page.c line 153
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.5/...-recovery.html
    InnoDB: about forcing recovery.
    innobackupex: Error:
    innobackupex: ibbackup failed at /usr/bin/innobackupex line 381.
  • jscheuflerjscheufler Entrant Current User Role Beginner
    I do encounter the same problem, like DmitryK - however, I do encounter it on MySQL instead of mariaDB.

    First of all, my versions:
    MySQL 5.5.27
    Percona XtraBackup 2.0.6

    What's strange in my case, is the fact that the backups for two of the three instances on my backup node can be restored without any hassle - but the third, and largest one ( about 1.2TB data-dir / 437GB zipped backup ) ultimately fails with the above mentioned assertion failure:
    130426  9:04:02  InnoDB: Assertion failure in thread 139909354481408 in file xtrabackup.c line 1601
    InnoDB: Failing assertion: cset == 0
    

    I am using the following statements to create and restore the backup

    Creation:
    innobackupex --slave-info --safe-slave-backup --stream=tar ./ | pigz -p 5 | split -d -b 100G - $FILENAME
    

    Restore:
    cat 2013-04-23--11-51-59.xtrabackup.tar.gz* | pigz -dc | tar xif - -C /export/backup-tmp/
    

    Any advice would be appreciated.

    Greetings,

    Jan
  • DmitryKDmitryK Entrant Current User Role Beginner
    I replaced MariaDB with Percona 5.5.30-30.2 and installed xtrabackup 2.0.6 - same problem with normal (full, no streaming) backup!
Sign In or Register to comment.

MySQL, InnoDB, MariaDB and MongoDB are trademarks of their respective owners.
Copyright ©2005 - 2020 Percona LLC. All rights reserved.