Percona Server 5.5.16 crashes

Does anyone know, if this is a bug? We have upgraded 2 slave machines to Percona Server 5.5 from standard MySQL 5.1.53 and suddenly the server stability has gone away. We have 1 crash on both machines in the last 24 hours.

The output:

This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=104857600
read_buffer_size=131072
max_used_connections=37
max_threads=2000
thread_count=16
connection_count=15
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4478321 K
bytes of memory
Hope that’s ok; if not, decrease some variables in the equation.

Thread pointer: 0x2abd74082140
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 0x49d500e0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x7ce749]
/usr/sbin/mysqld(handle_segfault+0x379)[0x500de9]
/lib64/libpthread.so.0[0x3f5560eb10]
/usr/lib64/mysql/plugin/ha_sphinx.so(_ZN9ha_sphinx7ConnectEP Kct+0x123)[0x2abd59e706a3]
/usr/lib64/mysql/plugin/ha_sphinx.so(_ZN9ha_sphinx10ConnectA PIEPKci+0x47)[0x2abd59e707b7]
/usr/lib64/mysql/plugin/ha_sphinx.so(_ZN9ha_sphinx10index_re adEPhPKhj16ha_rkey_function+0x2da)[0x2abd59e75eca]
/usr/sbin/mysqld[0x59d2f0]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x66) [0x5a6dd6]
/usr/sbin/mysqld[0x5a9070]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x12a8)[0x5b8128]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTj R4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_sel ect_lex_unitP13st_select_lex+0x1a9)[0x5b9589]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_result m+0x1ce)[0x5b9ede]
/usr/sbin/mysqld[0x57727d]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2505)[0x57 9c55]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x38 c)[0x57dadc]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3 THDPcj+0x148e)[0x57ef8e]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x106)[0x57f326]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x111)[0x 61a0a1]
/usr/sbin/mysqld(handle_one_connection+0x54)[0x61a194]
/lib64/libpthread.so.0[0x3f5560673d]
/lib64/libc.so.6(clone+0x6d)[0x3f54ed44bd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x2abd6c910a80): is an invalid pointer
Connection ID (thread ID): 144499
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
mysqld: /home/buildbot/slaves/percona-server-51-12/CentOS-5-x86-RPM_ CentOS_5_x86_64/work/BUILD/Percona-Server-5.5.16-rel22.0/Per cona-Server-5.5.16-rel22.0/mysys/my_new.cc:52: int _cxa_pure_virtual(): Assertion ! "Aborted: pure virtual method called."' failed. Fatal signal 6 while backtracing mysqld: /home/buildbot/slaves/percona-server-51-12/CentOS-5-x86-RPM_ CentOS_5_x86_64/work/BUILD/Percona-Server-5.5.16-rel22.0/Per cona-Server-5.5.16-rel22.0/mysys/my_new.cc:52: int __cxa_pure_virtual(): Assertion ! “Aborted: pure virtual method called.”’ failed.
mysqld: /home/buildbot/slaves/percona-server-51-12/CentOS-5-x86-RPM
CentOS_5_x86_64/work/BUILD/Percona-Server-5.5.16-rel22.0/Per cona-Server-5.5.16-rel22.0/mysys/my_new.cc:52: int _cxa_pure_virtual(): Assertion ! "Aborted: pure virtual method called."' failed. mysqld: /home/buildbot/slaves/percona-server-51-12/CentOS-5-x86-RPM_ CentOS_5_x86_64/work/BUILD/Percona-Server-5.5.16-rel22.0/Per cona-Server-5.5.16-rel22.0/mysys/my_new.cc:52: int __cxa_pure_virtual(): Assertion ! “Aborted: pure virtual method called.”’ failed.
mysqld: /home/buildbot/slaves/percona-server-51-12/CentOS-5-x86-RPM
CentOS_5_x86_64/work/BUILD/Percona-Server-5.5.16-rel22.0/Per cona-Server-5.5.16-rel22.0/mysys/my_new.cc:52: int __cxa_pure_virtual(): Assertion `! “Aborted: pure virtual method called.”’ failed.
111115 17:12:39 mysqld_safe Number of processes running now: 0
111115 17:12:39 mysqld_safe mysqld restarted
111115 17:12:39 [Note] Flashcache bypass: disabled
111115 17:12:39 [Note] Flashcache setup error is : ioctl failed

111115 17:12:39 [Note] Plugin ‘FEDERATED’ is disabled.
/usr/sbin/mysqld: Table ‘./mysql/plugin’ is marked as crashed and should be repaired
/usr/sbin/mysqld: Table ‘plugin’ is marked as crashed and should be repaired
111115 17:12:39 [Warning] Checking table: ‘./mysql/plugin’
111115 17:12:39 [ERROR] 1 client is using or hasn’t closed the table properly
111115 17:12:39 InnoDB: The InnoDB memory heap is disabled
111115 17:12:39 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111115 17:12:39 InnoDB: Compressed tables use zlib 1.2.3
111115 17:12:39 InnoDB: Using Linux native AIO
111115 17:12:39 InnoDB: Initializing buffer pool, size = 70.0G
111115 17:12:43 InnoDB: Completed initialization of buffer pool
111115 17:12:43 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 6056753489614
111115 17:12:43 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
InnoDB: Doing recovery: scanned up to log sequence number 6056758732288
InnoDB: Doing recovery: scanned up to log sequence number 6056763975168
InnoDB: Doing recovery: scanned up to log sequence number 6056769218048

If you need any other information, please just let me know.

Thank you, Vojtech Kurka

Hmm, I’m not C developer, but the stack trace might point to some problem in the sphinxSE plugin, am I right?

That’s a crash in the SphinxSE storage engine plugin.

Thank you Baron!

I’ve filed a bug report: http://sphinxsearch.com/bugs/view.php?id=983

HarryPoll: The Sphinx bug is already fixed, so you just need to compile SphinxSE with the latest SVN source. We have been running it in production for a week and it’s pretty stable.

HarryPoll was a spammer; I’ve deleted that message.