I have a problem with a four node cluster after a SST Transfer when a node has failed.
The facts:
- Four node Percona XtraDB Cluster running version 5.5.29-23.7.2.389.rhel6 on x86_64
- Using CentOS 6.4
- Both methods tried (rsync and xtrabackup)
- Sometimes large imports will break cluster as well. One random node stops and after rejoin the same error as below is received.
- Credentials for the transfer are correct in the whole cluster and the transfer works
When a node has joined the cluster again it requests a transfer, because it is out of sync. As far as I can see the transfer succeeds and the node has been joined the cluster again. When I try to select something from the transferred database I receive the following errors:
130125 15:57:16 InnoDB: Error: page 1 log sequence number 63541754
InnoDB: is in the future! Current system log sequence number 1611701.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: for more information.
or
130125 15:58:32 InnoDB: Error: table xxxx.xxxx does not exist in the InnoDB internal
InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html
or
130125 16:21:38 InnoDB: Error: trying to open a table, but could not
InnoDB: open the tablespace file ‘./xxxx/xxxx.ibd’!
InnoDB: Have you moved InnoDB .ibd files around without using the
InnoDB: commands DISCARD TABLESPACE and IMPORT TABLESPACE?
InnoDB: It is also possible that this is a temporary table #sql…,
InnoDB: and MySQL removed the .ibd file for this.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
which can result in:
InnoDB: Error: tablespace id is 4955 in the data dictionary
InnoDB: but in file ./xxxx/xxxx.ibd it is 4991!
130125 16:22:18 InnoDB: Assertion failure in thread 140335641454336 in file fil0fil.c line 776
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
15:22:18 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/
key_buffer_size=8388608
read_buffer_size=2097152
max_used_connections=12
max_threads=151
thread_count=10
connection_count=10
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 628597 K bytes of memory
Hope that’s ok; if not, decrease some variables in the equation.
Thread pointer: 0xde7f370
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 7fa2700ede48 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7c83f5]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x6a1e04]
/lib64/libpthread.so.0(+0xf500)[0x7fa28eb5f500]
/lib64/libc.so.6(gsignal+0x35)[0x7fa28dd278a5]
/lib64/libc.so.6(abort+0x175)[0x7fa28dd29085]
/usr/sbin/mysqld[0x90267e]
/usr/sbin/mysqld[0x902799]
/usr/sbin/mysqld[0x90ea7d]
/usr/sbin/mysqld[0x90eb89]
/usr/sbin/mysqld[0x8e4629]
/usr/sbin/mysqld[0x878443]
/usr/sbin/mysqld[0x878e01]
/usr/sbin/mysqld[0x967978]
/usr/sbin/mysqld[0x9680c6]
/usr/sbin/mysqld[0x865977]
/usr/sbin/mysqld[0x848ab9]
/usr/sbin/mysqld(_Z15ha_delete_tableP3THDP10handlertonPKcS4_S4_b+0xae)[0x6a74de]
/usr/sbin/mysqld(_Z23mysql_rm_table_no_locksP3THDP10TABLE_LISTbbbb+0x824)[0x5f8474]
/usr/sbin/mysqld(_Z14mysql_rm_tableP3THDP10TABLE_LISTcc+0x15d)[0x5f8b7d]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0xafd)[0x59425d]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x333)[0x598de3]
/usr/sbin/mysqld[0x5997a0]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1a82)[0x59c072]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x59c499]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x15f)[0x63710f]
/usr/sbin/mysqld(handle_one_connection+0x51)[0x6372d1]
/lib64/libpthread.so.0(+0x7851)[0x7fa28eb57851]
/lib64/libc.so.6(clone+0x6d)[0x7fa28dddd11d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fa0a8004d70): is an invalid pointer
Connection ID (thread ID): 29
Status: NOT_KILLED
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
I know these are all InnoDB errors which are the same if you hard copy files from one server to another, but I can’t figure it out why this happens in the cluster version.
Hope someone can help me out.
If more info is needed, please ask. It can be provided