# Xtrabackup --prepare crash with federate database

**URL:** <https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386>\
**Category:** Percona XtraBackup\
**Created:** [May 7, 2021, 7:14pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386 "2021-05-07T19:14:38Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Figueroa](https://avatars.discourse-cdn.com/v4/letter/f/ce73a5/32.png) [@Figueroa](https://forums.percona.com/u/Figueroa)\
**Post date:** [May 7, 2021, 7:14pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/1 "2021-05-07T19:14:38Z")

</div>

We have few backup of db running but when we try to restore one backup with federated failed. I try from master and slave intance . and both have same issue to restore in other server.

# xtrabackup --prepare --target-dir=/backup/database

xtrabackup: recognized server arguments: --innodb\_checksum\_algorithm=crc32 --innodb\_log\_checksums=1 --innodb\_data\_file\_path=ibdata1:12M:autoextend --innodb\_log\_files\_in\_group=2 --innodb\_log\_file\_size=50331648 --innodb\_page\_size=16384 --innodb\_undo\_directory=./ --innodb\_undo\_tablespaces=2 --server-id=21 --innodb\_log\_checksums=ON --innodb\_redo\_log\_encrypt=0 --innodb\_undo\_log\_encrypt=0  
xtrabackup: recognized client arguments: --prepare=1 --target-dir=/backup/database  
xtrabackup version 8.0.23-16 based on MySQL server 8.0.23 Linux (x86\_64) (revision id: 934bc8f)  
xtrabackup: cd to /backup/database  
xtrabackup: This target seems to be not prepared yet.  
Number of pools: 1  
xtrabackup: xtrabackup\_logfile detected: size=8388608, start\_lsn=(4792455009376)  
xtrabackup: using the following InnoDB configuration for recovery:  
xtrabackup: innodb\_data\_home\_dir = .  
xtrabackup: innodb\_data\_file\_path = ibdata1:12M:autoextend  
xtrabackup: innodb\_log\_group\_home\_dir = .  
xtrabackup: innodb\_log\_files\_in\_group = 1  
xtrabackup: innodb\_log\_file\_size = 8388608  
xtrabackup: using the following InnoDB configuration for recovery:  
xtrabackup: innodb\_data\_home\_dir = .  
xtrabackup: innodb\_data\_file\_path = ibdata1:12M:autoextend  
xtrabackup: innodb\_log\_group\_home\_dir = .  
xtrabackup: innodb\_log\_files\_in\_group = 1  
xtrabackup: innodb\_log\_file\_size = 8388608  
xtrabackup: Starting InnoDB instance for recovery.  
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)  
PUNCH HOLE support available  
Uses event mutexes  
GCC builtin \_\_atomic\_thread\_fence() is used for memory barrier  
Compressed tables use zlib 1.2.11  
Number of pools: 1  
Using CPU crc32 instructions  
Directories to scan ‘./’  
Scanning ‘./’  
Completed space ID check of 111 files.  
Initializing buffer pool, total size = 128.000000M, instances = 1, chunk size =128.000000M  
Completed initialization of buffer pool  
page\_cleaner coordinator priority: -20  
page\_cleaner worker priority: -20  
page\_cleaner worker priority: -20  
page\_cleaner worker priority: -20  
The log sequence number 4788139721173 in the system tablespace does not match the log sequence number 4792455009376 in the ib\_logfiles!  
Database was not shutdown normally!  
Starting crash recovery.  
Starting to parse redo log at lsn = 4792455009323, whereas checkpoint\_lsn = 4792455009376 and start\_lsn = 4792455009280  
Doing recovery: scanned up to log sequence number 4792456451303  
Log background threads are being started…  
Applying a batch of 811 redo log records …  
10%  
20%  
30%  
40%  
50%  
60%  
70%  
80%  
90%  
100%  
Apply batch completed!  
Using undo tablespace ‘./undo\_001’.  
Using undo tablespace ‘./undo\_002’.  
Opened 2 existing undo tablespaces.  
GTID recovery trx\_no: 376009489  
Creating shared tablespace for temporary tables  
Setting file ‘./ibtmp1’ size to 12 MB. Physically writing the file full; Please wait …  
File ‘./ibtmp1’ size is now 12 MB.  
Scanning temp tablespace dir:‘./#innodb\_temp/’  
Created 128 and tracked 128 new rollback segment(s) in the temporary tablespace. 128 are now active.  
8.0.23 started; log sequence number 4792456451303  
Page cleaner took 5338ms to flush 0 and evict 0 pages  
Allocated tablespace ID 9221686 for sportsDB/tbPeriodDescriptionsByLeague, old maximum was 0  
InnoDB: Assertion failure: dict0dict.cc:1213:table2 == nullptr  
InnoDB: thread 140693118307456InnoDB: We intentionally generate a memory trap.  
InnoDB: Submit a detailed bug report to [Log in with Atlassian account](https://jira.percona.com/projects/PXB).  
InnoDB: If you get repeated assertion failures or crashes, even  
InnoDB: immediately after the mysqld startup, there may be  
InnoDB: corruption in the InnoDB tablespace. Please refer to  
InnoDB: [http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html](http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html)  
InnoDB: about forcing recovery.  
16:51:04 UTC - mysqld got signal 6 ;  
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.  
Thread pointer: 0x55e57e460820  
Attempting backtrace. You can use the following information to find out  
where mysqld died. If you see no messages after this, something went  
terribly wrong…  
stack\_bottom = 7ffdad3f8c60 thread\_stack 0x46000  
xtrabackup(my\_print\_stacktrace(unsigned char const\*, unsigned long)+0x41) [0x55e57ba9c271]  
xtrabackup(handle\_fatal\_signal+0x313) [0x55e57a6d4523]  
/lib/x86\_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7ff5abf843c0]  
/lib/x86\_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7ff5ab5d918b]  
/lib/x86\_64-linux-gnu/libc.so.6(abort+0x12b) [0x7ff5ab5b8859]  
xtrabackup(+0xdce77a) [0x55e57a0e477a]  
xtrabackup(dict\_table\_add\_to\_cache(dict\_table\_t\*, unsigned long, mem\_block\_info\_t\*)+0x11c) [0x55e57a884b2c]  
xtrabackup(dd\_table\_create\_on\_dd\_obj(dd::Table const\*, dd::Partition const\*, std::\_\_cxx11::basic\_string\<char, std::char\_traits, Stateless\_allocator\<char, dd::String\_type\_alloc, My\_free\_functor\> \> const\*, bool, unsigned int)+0x1d30) [0x55e57a8afa80]  
xtrabackup(+0x159b0c0) [0x55e57a8b10c0]  
xtrabackup(dd\_table\_load\_on\_dd\_obj(dd::cache::Dictionary\_client\*, unsigned int, dd::Table const&, dict\_table\_t\*&, THD\*, std::\_\_cxx11::basic\_string\<char, std::char\_traits, Stateless\_allocator\<char, dd::String\_type\_alloc, My\_free\_functor\> \> const\*, bool)+0xff) [0x55e57a8b120f]  
xtrabackup(dict\_load\_tables\_from\_space\_id(unsigned int, THD\*, trx\_t\*)+0x445) [0x55e57a182305]  
xtrabackup(+0xe6cf87) [0x55e57a182f87]  
xtrabackup(+0xe6e1e4) [0x55e57a1841e4]  
xtrabackup(main+0x136e) [0x55e57a13c05e]  
/lib/x86\_64-linux-gnu/libc.so.6(\_\_libc\_start\_main+0xf3) [0x7ff5ab5ba0b3]  
xtrabackup(\_start+0x2e) [0x55e57a16c40e]

Trying to get some variables.  
Some pointers may be invalid and cause the dump to abort.  
Query (0): Connection ID (thread ID): 0  
Status: NOT\_KILLED

## Please report a bug at [Log in with Atlassian account](https://jira.percona.com/projects/PXB)

INFO:

[mysqld@db2]  
pid-file = /var/run/mysqld/mysql\_db2.pid  
socket = /var/run/mysqld/mysql\_db2.sock  
datadir = /var/lib/mysql\_db2  
log-error = /var/log/mysql\_db2/error.log  
log-bin = /var/log/mysql\_db2/mysql-bin.log  
port = 3307  
server\_id = 41  
mysqlx=0  
skip-slave-start  
replicate\_ignore\_db = ppm

log-error-verbosity=1  
long\_query\_time = 10  
slow\_query\_log = ON  
slow\_query\_log\_file = /var/log/mysql\_db2/db2-slow.log

innodb\_buffer\_pool\_size = 32G  
innodb\_buffer\_pool\_instances = 16  
symbolic-links=0

event-scheduler = ON  
max\_connections = 4096  
group\_concat\_max\_len = 2048  
transaction-isolation = READ-COMMITTED  
innodb\_print\_all\_deadlocks  
federated  
table\_open\_cache = 32768  
open\_files\_limit = 65536  
#skip-slave-start

max\_allowed\_packet = 256M  
sort\_buffer\_size = 2097152

How I make the backups:

xtrabackup --backup --user=lb --password=XXXX --compress --compress-threads=32 --parallel=32 --target-dir=/backup/db2 --slave-info --safe-slave-backup --datadir=/var/lib/mysql\_db2 --socket=/var/run/mysqld/mysql\_db2.sock

and descompress don’t have any error  
xtrabackup --decompress --remove-original --parallel=16 --decompress-threads=16 --target-dir=/backup/db2

if error is federated database, how I excluse that database (dbfed) from backup? if federate don’t cause the error how I can know what is the error?

thanks for anyhelp

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [May 7, 2021, 8:23pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/2 "2021-05-07T20:23:12Z")

</div>

Hello @Figueroa

> **[Percona Software Documentation](https://docs.percona.com/percona-xtrabackup/8.0/xtrabackup_bin/xbk_option_reference.html)**
>
> .container-box-download { justify-content: flex-start; } .container-button-learn-more { margin-bottom: 10px; } .logo-server-home { background: url(/img/product-logos/sprite.png)

Report back if that solves your issue.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [May 12, 2021, 2:07pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/4 "2021-05-12T14:07:32Z")

</div>

Just to check, you are using xtrabackup-8 with MySQL 8, correct? Is this table `sportsDB/tbPeriodDescriptionsByLeague` innodb? Can you run an OPTIMIZE TABLE on it and try the backup again?

---

<div class="post-metadata">

**Author:** ![Figueroa](https://avatars.discourse-cdn.com/v4/letter/f/ce73a5/32.png) [@Figueroa](https://forums.percona.com/u/Figueroa)\
**Post date:** [May 12, 2021, 8:00pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/5 "2021-05-12T20:00:32Z")

</div>

# xtrabackup -v

xtrabackup version 8.0.23-16 based on MySQL server 8.0.23 Linux (x86\_64) (revision id: 934bc8f)

# mysql -V

mysql Ver 8.0.21-12 for Linux on x86\_64 (Percona Server (GPL), Release ‘12’, Revision ‘7ddfdfe’)

# 

yes, that table is innodb

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [May 12, 2021, 9:08pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/6 "2021-05-12T21:08:45Z")

</div>

And what happens after you optimize the table and run a backup? Still crashes? If so, I highly recommend you perform a logical dump of the table, drop the table, recreate it and import the dump. Your backup should work then with xb.

---

<div class="post-metadata">

**Author:** ![Figueroa](https://avatars.discourse-cdn.com/v4/letter/f/ce73a5/32.png) [@Figueroa](https://forums.percona.com/u/Figueroa)\
**Post date:** [May 19, 2021, 5:17am UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/7 "2021-05-19T05:17:05Z")

</div>

Yes, it still crashing, How I make logical dump of the table? I supposed it is mysqldump

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [May 19, 2021, 2:10pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/8 "2021-05-19T14:10:40Z")

</div>

Yes, use mysqldump or better, [mydumper](https://github.com/maxbube/mydumper)

---

<div class="post-metadata">

**Author:** ![Figueroa](https://avatars.discourse-cdn.com/v4/letter/f/ce73a5/32.png) [@Figueroa](https://forums.percona.com/u/Figueroa)\
**Post date:** [May 19, 2021, 5:50pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/9 "2021-05-19T17:50:36Z")

</div>

about the --databases-exclude=  
I try it “–databases-exclude=dbfed,dbfed2” but it still adding the DB.  
are we need put in special format?

---

<div class="post-metadata">

**Author:** ![Figueroa](https://avatars.discourse-cdn.com/v4/letter/f/ce73a5/32.png) [@Figueroa](https://forums.percona.com/u/Figueroa)\
**Post date:** [May 20, 2021, 9:08pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/10 "2021-05-20T21:08:08Z")

</div>

We restore that DB and we still have de issue.  
I don’t think is relate to table tbPeriodDescriptionsByLeague. Maybe is relate to:

InnoDB: Assertion failure: dict0dict.cc:1213:table2 == nullptr  
InnoDB: thread 140693118307456InnoDB: We intentionally generate a memory trap.

Beacuse I see the message  
“Allocated tablespace ID 9221686 for sportsDB/tbPeriodDescriptionsByLeague, old maximum was 0”  
in other servers when I make the backup.

if I try backup one by one of all DBs in the instance, it is working

xtrabackup --backup --user=… --databases=db1  
xtrabackup --backup --user=… --databases=db2  
xtrabackup --backup --user=… --databases=db3

but if I try full backup I have same issue

xtrabackup --backup …

InnoDB: Assertion failure: dict0dict.cc:1213:table2 == nullptr  
InnoDB: thread 140693118307456InnoDB: We intentionally generate a memory trap.  
InnoDB: Submit a detailed bug report to [https://jira.percona.com/projects/PXB](https://jira.percona.com/projects/PXB).  
InnoDB: If you get repeated assertion failures or crashes, even  
InnoDB: immediately after the mysqld startup, there may be  
InnoDB: corruption in the InnoDB tablespace. Please refer to  
InnoDB: [MySQL :: MySQL 8.0 Reference Manual :: 15.21.3 Forcing InnoDB Recovery](http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html)  
InnoDB: about forcing recovery.

I am worried that is bug in mysql or error scheme, impact the future.

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [May 20, 2021, 9:35pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/11 "2021-05-20T21:35:10Z")

</div>

Unfortunately, I don’t think I can assist further over forum posts. If this is indeed a bug, you’ll need to submit a ticket at [https://jira.percona.com](https://jira.percona.com) but be aware that team will need a repeatable test case and possibly a copy of your table space in order to diagnose the issue.

---

<div class="post-metadata">

**Author:** ![Figueroa](https://avatars.discourse-cdn.com/v4/letter/f/ce73a5/32.png) [@Figueroa](https://forums.percona.com/u/Figueroa)\
**Post date:** [May 20, 2021, 9:58pm UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/12 "2021-05-20T21:58:31Z")

</div>

thanks for you help, Do you think is a bug? and not that the some files, tables are corrupt

---

<div class="post-metadata">

**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\
**Post date:** [May 21, 2021, 12:12am UTC](https://forums.percona.com/t/xtrabackup-prepare-crash-with-federate-database/10386/13 "2021-05-21T00:12:54Z")

</div>

Well, xtrabackup is telling us there is some type of corruption. But you said you dropped the table and re-imported it and it’s still causing an issue. There is probably corruption somewhere in the ibdata1 file then. The error says “table2” so I’m not sure if that is an internal pointer or if you have an actual table called “table2”.
