Hi!,
Iam running an 5 Node Multi-Instances Galera Cluster since 2 years now.
SST is encrypted xtrabackup. The Donor is doing nothing but backup and sst.
I did a minor Update to 5.5.39-36.0-55-log / wsrep_25.11.r4023
The biggest change was the Update from xtrabackup 2.1 to xtrabackup 2.2.
Normally i let the node re-sync the content after an update. So i delete the grastate.dat.
After starting and successfully SST the Log started to fill up with:
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
150210 12:57:21 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
150210 12:57:21 InnoDB: Error: page 2 log sequence number 44208161331
Endless scrolling until hard-killing it.
After stopping the Node, cleaning the data-dir and restarting it the sst is completing fine and the error is gone.
This is happening only once per instance an node, so Debugging is really hard!
The same happend on the second node. First I deleted the ib_* Files. The Node recreated them and crashed again.
Deleting the table ibd Files did nothing, too.
Deleting the xtrabackup_* Files on the joiner fixes the error!
My current impression is that the new xtrabackup is not deleting all essential files. The old Version wipes all Content.
Does somebody knows something about this?
Greetz