Hi,
Few days back I had faced MySQL v5.5.27 abnormal & random crash.
I am facing this issue since long time.
I am using Sphinx v2.1.3, with MySQL for Full Text Searching.
This installation was made using source code method.
Please find below error log for the same.
170603 06:36:38 mysqld_safe mysqld restarted
170603 6:36:38 [Warning] The syntax ‘–log-slow-queries’ is deprecated and will be removed in a future release. Please use ‘–slow-query-log’/‘–slow-query-log-file’ instead.
170603 6:36:38 InnoDB: The InnoDB memory heap is disabled
170603 6:36:38 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170603 6:36:38 InnoDB: Compressed tables use zlib 1.2.3
170603 6:36:38 InnoDB: Initializing buffer pool, size = 2.0G
170603 6:36:38 InnoDB: Completed initialization of buffer pool
170603 6:36:38 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
170603 6:36:38 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
InnoDB: Last MySQL binlog file position 0 53033350, file name /data03/Binlog/mysql-bin.000377
170603 6:37:15 InnoDB: Waiting for the background threads to start
170603 6:37:16 InnoDB: 1.1.8 started; log sequence number 429497866330
170603 6:37:17 [Note] Server hostname (bind-address): ‘0.0.0.0’; port: 3306
170603 6:37:17 [Note] - ‘0.0.0.0’ resolves to ‘0.0.0.0’;
170603 6:37:17 [Note] Server socket created on IP: ‘0.0.0.0’.
170603 6:37:17 [Note] Event Scheduler: Loaded 0 events
170603 6:37:34 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: ‘5.5.27-log’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Source distribution
170603 6:37:34 [ERROR] /usr/local/mysql/bin/mysqld: Table ‘./xxxx_mdm/mdo_catalog’ is marked as crashed and should be repaired
170603 6:37:34 [Warning] Checking table: ‘./xxxx_mdm/mdo_catalog’
170603 6:39:45 [ERROR] /usr/local/mysql/bin/mysqld: Table ‘./xxxx_mdm/FIELDLIST’ is marked as crashed and should be repaired
170603 6:39:45 [Warning] Checking table: ‘./xxxx_mdm/FIELDLIST’
170603 7:00:01 [Warning] IP address ‘x.x.x.x’ could not be resolved: Name or service not known
170603 10:20:37 [Warning] IP address ‘x.x.x.x’ could not be resolved: Name or service not known
170603 11:00:10 [Warning] IP address ‘x.x.x.x’ could not be resolved: Name or service not known
170605 6:59:48 [Warning] IP address ‘x.x.x.x’ could not be resolved: Name or service not known
*** glibc detected *** /usr/local/mysql/bin/mysqld: malloc(): smallbin double linked list corrupted: 0x00007f3e265dca50 ***
*** glibc detected *** /usr/local/mysql/bin/mysqld: malloc(): smallbin double linked list corrupted: 0x00007f3e265dca50 ***
*** glibc detected *** /usr/local/mysql/bin/mysqld: malloc(): smallbin double linked list corrupted: 0x00007f3e265dca50 ***
*** glibc detected *** /usr/local/mysql/bin/mysqld: malloc(): smallbin double linked list corrupted: 0x00007f3e265dca50 ***
*** glibc detected *** /usr/local/mysql/bin/mysqld: malloc(): smallbin double linked list corrupted: 0x00007f3e265dca50 ***
*** glibc detected *** /usr/local/mysql/bin/mysqld: malloc(): smallbin double linked list corrupted: 0x00007f3e265dca50 ***
*** glibc detected *** /usr/local/mysql/bin/mysqld: malloc(): smallbin double linked list corrupted: 0x00007f3e265dca50 ***
*** glibc detected *** /usr/local/mysql/bin/mysqld: malloc(): smallbin double linked list corrupted: 0x00007f3e265dca50 ***
======= Backtrace: =========
/lib64/libc.so.6[0x35b4675e66]
/lib64/libc.so.6[0x35b4679c3f]
/lib64/libc.so.6(__libc_malloc+0x71)[0x35b467a6b1]
======= Backtrace: =========
======= Backtrace: =========
======= Backtrace: =========
======= Backtrace: =========
======= Backtrace: =========
======= Backtrace: =========
======= Backtrace: =========
/usr/local/mysql/bin/mysqld(my_malloc+0x32)[0x752262]
/lib64/libc.so.6[0x35b4675e66]
/lib64/libc.so.6[0x35b4675e66]
/lib64/libc.so.6[0x35b4679c3f]
/lib64/libc.so.6(__libc_malloc+0x71)[0x35b467a6b1]
/usr/local/mysql/bin/mysqld(my_malloc+0x32)[0x752262]
/lib64/libc.so.6[0x35b4675e66]
/lib64/libc.so.6[0x35b4679c3f]
/lib64/libc.so.6(__libc_malloc+0x71)[0x35b467a6b1]
/usr/local/mysql/bin/mysqld(my_malloc+0x32)[0x752262]
/lib64/libc.so.6[0x35b4675e66]
/usr/local/mysql/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x176)[0x5f70d6]
/usr/local/mysql/bin/mysqld(handle_one_connection+0x4a)[0x5f713a]
/lib64/libpthread.so.0[0x35b4a079d1]
/lib64/libc.so.6(clone+0x6d)[0x35b46e88fd]
======= Memory map: ========
00400000-00c88000 r-xp 00000000 fd:03 9699660 /usr/local/mysql/bin/mysqld
00e87000-00f2e000 rw-p 00887000 fd:03 9699660 /usr/local/mysql/bin/mysqld
00f2e000-08a31000 rw-p 00000000 00:00 0 [heap]
35b4200000-35b4220000 r-xp 00000000 fd:00 5497 /lib64/ld-2.12.so
35b441f000-35b4420000 r–p 0001f000 fd:00 5497 /lib64/ld-2.12.so
000-7f401a486000 rw-p 00000000 00:00 0
7f424c042000-7f424ef48000 rw-p 00000000 00:00 0
7f424ef48000-7f424ef54000 r-xp 00000000 fd:00 5641 /lib64/libnss_files-2.12.so
7f424ef54000-7f424f154000 —p 0000c000 fd:00 5641 /lib64/libnss_files-2.12.so
7f424f154000-7f424f155000 r–p 0000c000 fd:00 5641 /lib64/libnss_files-2.12.so
7f424f155000-7f424f156000 rw-p 0000d000 fd:00 5641 /lib64/libnss_files-2.12.so
7fff0b427000-7fff0b43c000 rw-p 00000000 00:00 0 [stack]
7fff0b558000-7fff0b559000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
/usr/local/mysql/bin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x671)[0x70bd51]
/usr/local/mysql/bin/mysqld(_Z21mysql_execute_commandP3THD+0x1488)[0x5679a8]
/usr/local/mysql/bin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x1c)[0x707bcc]
/lib64/libpthread.so.0[0x35b4a079d1]
/lib64/libc.so.6(clone+0x6d)[0x35b46e88fd]
======= Memory map: ========
00400000-00c88000 r-xp 00000000 fd:03 9699660 /usr/local/mysql/bin/mysqld
00e87000-00f2e000 rw-p 00887000 fd:03 9699660 /usr/local/mysql/bin/mysqld
00f2e000-08a31000 rw-p 00000000 00:00 0 [heap]
35b4200000-35b4220000 r-xp 00000000 fd:00 5497 /lib64/ld-2.12.so
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
170622 06:07:39 mysqld_safe Number of processes running now: 0
170622 06:07:39 mysqld_safe mysqld restarted
170622 6:07:39 [Warning] The syntax ‘–log-slow-queries’ is deprecated and will be removed in a future release. Please use ‘–slow-query-log’/‘–slow-query-log-file’ instead.
170622 6:07:40 InnoDB: The InnoDB memory heap is disabled
170622 6:07:40 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170622 6:07:40 InnoDB: Compressed tables use zlib 1.2.3
170622 6:07:40 InnoDB: Initializing buffer pool, size = 2.0G
170622 6:07:40 InnoDB: Completed initialization of buffer pool
170622 6:07:40 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 444455479017
170622 6:07:40 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
InnoDB: Doing recovery: scanned up to log sequence number 444455483107
170622 6:08:19 InnoDB: Starting an apply batch of log records to the database…
InnoDB: Progress in percents: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 53033350, file name /data03/Binlog/mysql-bin.000377
170622 6:08:19 InnoDB: Waiting for the background threads to start
170622 6:08:20 InnoDB: 1.1.8 started; log sequence number 444455483107
170622 6:08:20 [Note] Server hostname (bind-address): ‘0.0.0.0’; port: 3306
170622 6:08:20 [Note] - ‘0.0.0.0’ resolves to ‘0.0.0.0’;
170622 6:08:20 [Note] Server socket created on IP: ‘0.0.0.0’.
170622 6:08:21 [Note] Event Scheduler: Loaded 0 events
170622 6:08:38 [Note] /usr/local/mysql/bin/mysqld: ready for connections.
Version: ‘5.5.27-log’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Source distribution
170622 6:08:38 [Warning] IP address ‘x.x.x.x’ could not be resolved: Name or service not known
170622 6:08:38 [ERROR] /usr/local/mysql/bin/mysqld: Table ‘./xxxx_mdm/mdo_catalog’ is marked as crashed and should be repaired
170622 6:08:38 [Warning] Checking table: ‘./xxxx_mdm/mdo_catalog’
170622 6:08:38 [ERROR] /usr/local/mysql/bin/mysqld: Table ‘./xxxx_mdm/mdo_varcompany’ is marked as crashed and should be repaired
170622 6:08:38 [Warning] Checking table: ‘./xxxx_mdm/mdo_varcompany’
170622 21:01:32 [Warning] Checking table: ‘./xxxx_006418/price_dandh’
170622 21:01:34 [ERROR] /usr/local/mysql/bin/mysqld: Table ‘./xxxx_006418/price_digitek’ is marked as crashed and should be repaired
170622 21:01:34 [Warning] Checking table: ‘./xxxx_006418/price_digitek’
170622 21:01:34 [ERROR] /usr/local/mysql/bin/mysqld: Table ‘./xxxx_006418/price_blackbox’ is marked as crashed and should be repaired
170622 21:01:34 [Warning] Checking table: ‘./xxxx_006418/price_blackbox’
170623 2:30:01 [ERROR] /usr/local/mysql/bin/mysqld: Table ‘./xxxx_005797/price_ingram’ is marked as crashed and should be repaired
170623 2:30:01 [ERROR] /usr/local/mysql/bin/mysqld: Table ‘./xxxx_006085/price_ingram’ is marked as crashed and should be repaired