We are using Percona 5.6.23-72.1 on Centos 7.0.1406 64bit (with TokuDB plugin) and migrated the tablespace from MySQL 5.5.27.
( 1 )
We see frequent table marked as crashed after OPTIMIZE, CHECK, REPAIR command (MyISAM)
Log entry:
2015-03-28 05:03:28 123161 [ERROR] Got an error from thread_id=7373, /mnt/workspace/percona-server-5.6-redhat-binary/label_exp/centos7-64/rpmbuild/BUILD/percona-server-5.6.23-72.1/storage/myisam/ha_myisam.cc:910
2015-03-28 05:03:29 123161 [ERROR] MySQL thread id 7373, OS thread handle 0x7f4f663bd700, query id 653385 192.168.0.. * Checking table *** multiple tables ***
2015-03-28 05:03:30 123161 [ERROR] /usr/sbin/mysqld: Table '**’ is marked as crashed and should be repaired
Fixing the table with REPAIR TABLE works for most cases - in some cases REPAIR TABLE fails and we are using myisamcheck, which works well.
( 2 )
In some cases we don’t see the behaviour in (1) , but we see a server crash (only for multiple tables in CHECK TABLE or REPAIR TABLE). Error log:
05:06:41 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at [URL]System Dashboard - Percona JIRA
key_buffer_size=8589934592
read_buffer_size=2097152
max_used_connections=261
max_threads=5002
thread_count=135
connection_count=135
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2641195005 K bytes of memory
Hope that’s ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f4edb468000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 7f4f23dcad40 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x8d9b8b]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0x658ab1]
/lib64/libpthread.so.0(+0xf130)[0x7f524e348130]
/usr/sbin/mysqld(_ZNK7handler22ha_statistic_incrementEM17sys tem_status_vary+0xc)[0x59a6ac]
/usr/sbin/mysqld(_ZN7handler16ha_external_lockEP3THDi+0x33)[0x5a02b3]
/usr/sbin/mysqld(_Z17mysql_lock_tablesP3THDPP5TABLEjj+0x75a)[0x7d5cea]
/usr/sbin/mysqld(_Z11lock_tablesP3THDP10TABLE_LISTjj+0x530)[0x690180]
/usr/sbin/mysqld(_Z20open_and_lock_tablesP3THDP10TABLE_LISTb jP19Prelocking_strategy+0xa2)[0x696e52]
/usr/sbin/mysqld[0x812853]
/usr/sbin/mysqld(_ZN20Sql_cmd_repair_table7executeEP3THD+0xc 8)[0x8140b8]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x16f0)[0x6dbd40]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5e 8)[0x6e1618]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3 THDPcj+0xfc8)[0x6e2d78]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x172)[0x6afdf2]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6afef0]
/usr/sbin/mysqld(pfs_spawn_thread+0x143)[0x9119b3]
/lib64/libpthread.so.0(+0x7df3)[0x7f524e340df3]
/lib64/libc.so.6(clone+0x6d)[0x7f524c9b91ad]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f4ede41d010): is an invalid pointer
Connection ID (thread ID): 7367
Status: NOT_KILLED
You may download the Percona Server operations manual by visiting
[URL]http://www.percona.com/software/percona-server/[/URL]. You may find information
in the manual which will help you identify the cause of the crash.
150328 06:06:42 mysqld_safe Transparent huge pages are already set to: never.
150328 06:06:42 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2015-03-28 06:06:43 0 [Warning] The syntax ‘pre-4.1 password hash’ is deprecated and will be removed in a future release. Please use post-4.1 password hash instead.
2015-03-28 06:06:43 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2015-03-28 06:06:44 124453 [Note] Plugin ‘FEDERATED’ is disabled.
2015-03-28 06:06:44 124453 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-03-28 06:06:44 124453 [Note] InnoDB: The InnoDB memory heap is disabled
2015-03-28 06:06:44 124453 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-03-28 06:06:44 124453 [Note] InnoDB: Memory barrier is not used
2015-03-28 06:06:44 124453 [Note] InnoDB: Compressed tables use zlib 1.2.3
2015-03-28 06:06:44 124453 [Note] InnoDB: Using Linux native AIO
2015-03-28 06:06:44 124453 [Note] InnoDB: Using CPU crc32 instructions
2015-03-28 06:06:44 124453 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-03-28 06:06:44 124453 [Note] InnoDB: Completed initialization of buffer pool
2015-03-28 06:06:44 124453 [Note] InnoDB: Highest supported file format is Barracuda.
2015-03-28 06:06:44 124453 [Note] InnoDB: The log sequence numbers 1626265 and 1626265 in ibdata files do not match the log sequence number 1626285 in the ib_logfiles!
2015-03-28 06:06:44 124453 [Note] InnoDB: Database was not shutdown normally!
2015-03-28 06:06:44 124453 [Note] InnoDB: Starting crash recovery.
2015-03-28 06:06:44 124453 [Note] InnoDB: Reading tablespace information from the .ibd files…
2015-03-28 06:06:51 124453 [Note] InnoDB: Restoring possible half-written data pages
2015-03-28 06:06:51 124453 [Note] InnoDB: from the doublewrite buffer…
2015-03-28 06:06:51 124453 [Note] InnoDB: 128 rollback segment(s) are active.
2015-03-28 06:06:51 124453 [Note] InnoDB: Waiting for purge to start
2015-03-28 06:06:51 124453 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.23-72.1 started; log sequence number 1626285
Sat Mar 28 06:06:51 2015 TokuFT recovery starting in env /var/lib/mysql/
Sat Mar 28 06:06:51 2015 TokuFT recovery scanning backward from 140611
Sat Mar 28 06:06:51 2015 TokuFT recovery bw_end_checkpoint at 140611 timestamp 1427519174964095 xid 140607 (bw_newer)
Sat Mar 28 06:06:51 2015 TokuFT recovery bw_begin_checkpoint at 140607 timestamp 1427519174964044 (bw_between)
Sat Mar 28 06:06:51 2015 TokuFT recovery turning around at begin checkpoint 140607 time 51
Sat Mar 28 06:06:51 2015 TokuFT recovery starts scanning forward to 140611 from 140607 left 4 (fw_between)
Sat Mar 28 06:06:51 2015 TokuFT recovery closing 2 dictionaries
Sat Mar 28 06:06:51 2015 TokuFT recovery making a checkpoint
Sat Mar 28 06:06:51 2015 TokuFT recovery done
2015-03-28 06:06:51 124453 [Note] Recovering after a crash using mysql-bin
2015-03-28 06:06:52 124453 [Note] Starting crash recovery…
2015-03-28 06:06:52 124453 [Note] Crash recovery finished.
2015-03-28 06:06:52 124453 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2015-03-28 06:06:52 124453 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2015-03-28 06:06:52 124453 [Note] Server hostname (bind-address): ‘0.0.0.0’; port: 3306
2015-03-28 06:06:52 124453 [Note] - ‘0.0.0.0’ resolves to ‘0.0.0.0’;
2015-03-28 06:06:52 124453 [Note] Server socket created on IP: ‘0.0.0.0’.
2015-03-28 06:06:52 124453 [Note] Event Scheduler: Loaded 0 events
2015-03-28 06:06:52 124453 [Note] /usr/sbin/mysqld: ready for connections.
Version: ‘5.6.23-72.1-log’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Percona Server (GPL), Release 72.1, Revision 0503478
Any suggestions?
Edit: We could reproduce the same issue on a slave with CentOS 6.5 64bit.
Thanks,
Ralf