# mysql crashes with InnoDB: Assertion failure in thread 1459657024 in file srv/srv0srv.c line 2523

**URL:** <https://forums.percona.com/t/mysql-crashes-with-innodb-assertion-failure-in-thread-1459657024-in-file-srv-srv0srv-c-line-2523/1484>\
**Category:** Other MySQL® Questions\
**Created:** [August 25, 2010, 7:26am UTC](https://forums.percona.com/t/mysql-crashes-with-innodb-assertion-failure-in-thread-1459657024-in-file-srv-srv0srv-c-line-2523/1484 "2010-08-25T07:26:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![massoo](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/massoo/32/769_2.png) [@massoo](https://forums.percona.com/u/massoo)\
**Post date:** [August 25, 2010, 7:26am UTC](https://forums.percona.com/t/mysql-crashes-with-innodb-assertion-failure-in-thread-1459657024-in-file-srv-srv0srv-c-line-2523/1484/1 "2010-08-25T07:26:09Z")

</div>

Hi,

We are using Percona MySQL builds on CentOS-5.5. The details are as follows

Hardware: HP dc7900 with 16 GB RAM, Intel Core 2 Duo E8400/3Ghz/4GB RAM, 160GB HDD.

OS: CentOS release 5.5 (Final) / Linux [blr-cos-mdb01.digi.com](http://blr-cos-mdb01.digi.com) 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86\_64 x86\_64 x86\_64 GNU/Linux.

Percona MySQL:  
Percona-XtraDB-1.0.6-10.2-5.1.45-10.2.rhel5  
Percona-Server-server-51-5.1.47-rel11.1.51.rhel5  
Percona-XtraDB-1.0.3-5-5.1.34-5.rhel5  
Percona-Server-shared-compat-5.1.43-3  
Percona-Server-client-51-5.1.47-rel11.1.51.rhel5  
Percona-Server-test-51-5.1.47-rel11.1.51.rhel5  
Percona-Server-devel-51-5.1.47-rel11.1.51.rhel5  
Percona-Server-shared-51-5.1.47-rel11.1.51.rhel5

The MySQL Server crashes 3~4 times a days along with crashing the system totally by rendering it totally unresponsive. The ping /Magic-SysRq do not work at all. We have to do hard power recycling (plug out the power cable and plug in back after 30 secs).

The /var/log/mysqld.log has

| [B]Quote:[/B] |
| InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Error: semaphore wait has lasted \> 600 seconds InnoDB: We intentionally crash the server, because it appears to be hung. 100824 15:19:57 InnoDB: Assertion failure in thread 1459657024 in file srv/srv0srv.c line 2523 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to [URL]http://bugs.mysql.com.[/URL] InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: [URL="http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html"] http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html[/URL] InnoDB: about forcing recovery. 100824 15:19:57 - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.

key\_buffer\_size=33554432  
read\_buffer\_size=2097152  
max\_used\_connections=101  
max\_threads=500  
threads\_connected=100  
It is possible that mysqld could use up to  
key\_buffer\_size + (read\_buffer\_size + sort\_buffer\_size)\*max\_threads = 5158166 K  
bytes of memory  
Hope that’s ok; if not, decrease some variables in the equation.

thd: 0x0  
Attempting backtrace. You can use the following information to find out  
where mysqld died. If you see no messages after this, something went  
terribly wrong…  
stack\_bottom = (nil) thread\_stack 0x30000  
/usr/sbin/mysqld(my\_print\_stacktrace+0x2e)[0x86892e]  
/usr/sbin/mysqld(handle\_segfault+0x322)[0x5b0f72]  
/lib64/libpthread.so.0[0x35d480eb10]  
/lib64/libc.so.6(gsignal+0x35)[0x35d3c30265]  
/lib64/libc.so.6(abort+0x110)[0x35d3c31d10]  
/usr/sbin/mysqld[0x7914bb]  
/lib64/libpthread.so.0[0x35d480673d]  
/lib64/libc.so.6(clone+0x6d)[0x35d3cd3d1d]  
The manual page at [URL][http://dev.mysql.com/doc/mysql/en/crashing.html[/URL]](http://dev.mysql.com/doc/mysql/en/crashing.html%5B/URL%5D) contains  
information that should help you find out what is causing the crash.

The “–memlock” argument, which was enabled, uses system calls that are  
unreliable and unstable on some operating systems and operating-system  
versions (notably, some versions of Linux). This crash could be due to use  
of those buggy OS calls. You should consider whether you really need the  
“–memlock” parameter and/or consult the OS distributer about “mlockall”  
bugs.  
100824 18:22:30 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql/data  
100824 18:22:30 [Note] Plugin ‘FEDERATED’ is disabled.  
InnoDB: The InnoDB memory heap is disabled  
InnoDB: Mutexes and rw\_locks use GCC atomic builtins  
InnoDB: Compressed tables use zlib 1.2.3  
100824 18:22:32 InnoDB: highest supported file format is Barracuda.  
InnoDB: Log scan progressed past the checkpoint lsn 51613706496  
100824 18:22:32 InnoDB: Database was not shut down normally!  
InnoDB: Starting crash recovery.  
InnoDB: Reading tablespace information from the .ibd files…  
InnoDB: Restoring possible half-written data pages from the doublewrite  
InnoDB: buffer…  
InnoDB: Doing recovery: scanned up to log sequence number 51613709094  
InnoDB: 2 transaction(s) which must be rolled back or cleaned up  
InnoDB: in total 9 row operations to undo  
InnoDB: Trx id counter is 5989A00  
100824 18:22:46 InnoDB: Starting an apply batch of log records to the database…  
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99  
InnoDB: Apply batch completed  
InnoDB: Last MySQL binlog file position 0 57735, file name /var/lib/mysql/binlogs/mtrak32-bin.000123  
InnoDB: Starting in background the rollback of uncommitted transactions  
100824 18:22:47 InnoDB: Rolling back trx with id 5989818, 6 rows to undo  
100824 18:22:47 Percona XtraDB ([http://www.percona.com](http://www.percona.com)) 1.0.8-11.1 started; log sequence number 51613709094  
100824 18:22:47 [Note] Recovering after a crash using /var/lib/mysql/binlogs/mtrak32-bin  
100824 18:22:47 [Note] Starting crash recovery…  
100824 18:22:47 [Note] Crash recovery finished.  
InnoDB: Rolling back of trx id 5989818 completed  
100824 18:22:48 InnoDB: Rolling back trx with id 5989817, 3 rows to undo  
InnoDB: Rolling back of trx id 5989817 completed  
100824 18:22:48 InnoDB: Rollback of non-prepared transactions completed  
100824 18:22:53 [Note] Event Scheduler: Loaded 0 events  
100824 18:22:53 [Note] /usr/sbin/mysqld: ready for connections.  
Version: ‘5.1.47-rel11.1-log’ socket: ‘/var/lib/mysql/mysql.sock’ port: 3306 Percona Server (GPL), 11.1, Revision 51  
100824 18:27:57 [ERROR] /usr/sbin/mysqld: Table ‘./mobiq/jms\_messages’ is marked as crashed and should be repaired  
100824 18:27:57 [ERROR] /usr/sbin/mysqld: Table ‘./mobiq/jms\_messages’ is marked as crashed and should be repaired  
100824 18:27:57 [ERROR] /usr/sbin/mysqld: Table ‘./mobiq/jms\_messages’ is marked as crashed and should be repaired  
100824 18:27:57 [ERROR] /usr/sbin/mysqld: Table ‘./mobiq/jms\_messages’ is marked as crashed and should be repaired

 |

 and the /var/log/messages has 

| [B]Quote:[/B] |
| Aug 24 18:05:35 blr-cos-mdb01 kernel: BUG: soft lockup - CPU#0 stuck for 10s! [mysqld:5365] Aug 24 18:05:35 blr-cos-mdb01 kernel: CPU 0: Aug 24 18:05:45 blr-cos-mdb01 kernel: Modules linked in: ipv6 xfrm\_nalgo crypto\_api hidp l2cap bluetooth lockd sunrpc cpufreq\_ondemand acpi\_cpufreq freq\_table dm\_multipath scsi\_dh video backlight sbs power\_meter hwmon i2c\_ec i2c\_core dell\_wmi wmi button battery asus\_acpi acpi\_memhotplug ac parport\_pc lp parport floppy snd\_hda\_intel sr\_mod tpm\_infineon snd\_seq\_dummy snd\_seq\_oss snd\_seq\_midi\_event snd\_seq cdrom tpm snd\_seq\_device snd\_pcm\_oss snd\_mixer\_oss tpm\_bios snd\_pcm e1000e snd\_timer shpchp serio\_raw snd\_page\_alloc snd\_hwdep pcspkr sg snd soundcore dm\_raid45 dm\_message dm\_region\_hash dm\_mem\_cache dm\_snapshot dm\_zero dm\_mirror dm\_log dm\_mod ahci libata sd\_mod scsi\_mod ext3 jbd uhci\_hcd ohci\_hcd ehci\_hcd Aug 24 18:05:45 blr-cos-mdb01 kernel: Pid: 5365, comm: mysqld Not tainted 2.6.18-194.3.1.el5 #1 Aug 24 18:05:45 blr-cos-mdb01 kernel: RIP: 0010:[] [] \_\_d\_lookup+0xe2/0xff Aug 24 18:05:45 blr-cos-mdb01 kernel: RSP: 0018:ffff8101a415fc88 EFLAGS: 00000282 Aug 24 18:05:45 blr-cos-mdb01 kernel: RAX: ffff8103c6c864c8 RBX: ffff8103c6c864c8 RCX: 0000000000000015 Aug 24 18:05:45 blr-cos-mdb01 kernel: RDX: 00000000000db1d6 RSI: ffff8101a415fd28 RDI: ffff8103c83274b0 Aug 24 18:05:45 blr-cos-mdb01 kernel: RBP: ffff810417bbf800 R08: 0000000000008001 R09: ffff81041747e5c0 Aug 24 18:05:45 blr-cos-mdb01 kernel: R10: ffff810188c8c580 R11: ffffffff8002c3e0 R12: ffff81040c0840c0 Aug 24 18:05:45 blr-cos-mdb01 kernel: R13: 0000000000000000 R14: ffff8101a394e348 R15: ffff8101a394e348 Aug 24 18:05:45 blr-cos-mdb01 kernel: FS: 00000000405c8940(0063) GS:ffffffff803ca000(0000) knlGS:0000000000000000 Aug 24 18:05:45 blr-cos-mdb01 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Aug 24 18:05:45 blr-cos-mdb01 kernel: CR2: 00002aace6224000 CR3: 0000000401db3000 CR4: 00000000000006e0 Aug 24 18:05:45 blr-cos-mdb01 kernel: Aug 24 18:05:45 blr-cos-mdb01 kernel: Call Trace: Aug 24 18:05:45 blr-cos-mdb01 kernel: [] \_\_d\_lookup+0xb0/0xff Aug 24 18:05:45 blr-cos-mdb01 kernel: [] do\_lookup+0x2c/0x1e6 Aug 24 18:05:45 blr-cos-mdb01 kernel: [] \_\_link\_path\_walk+0xa01/0xf42 Aug 24 18:05:45 blr-cos-mdb01 kernel: [] link\_path\_walk+0x42/0xb2 Aug 24 18:05:45 blr-cos-mdb01 kernel: [] do\_path\_lookup+0x275/0x2f1 Aug 24 18:05:45 blr-cos-mdb01 kernel: [] \_\_path\_lookup\_intent\_open+0x56/0x97 Aug 24 18:05:45 blr-cos-mdb01 kernel: [] open\_namei+0x73/0x6d5 Aug 24 18:05:45 blr-cos-mdb01 kernel: [] do\_page\_fault+0x4fe/0x874 Aug 24 18:05:45 blr-cos-mdb01 kernel: [] do\_filp\_open+0x1c/0x38 Aug 24 18:05:45 blr-cos-mdb01 kernel: [] \_atomic\_dec\_and\_lock+0x39/0x57 Aug 24 18:05:45 blr-cos-mdb01 kernel: [] do\_sys\_open+0x44/0xbe Aug 24 18:05:45 blr-cos-mdb01 kernel: [] tracesys+0xd5/0xe0 Aug 24 18:05:45 blr-cos-mdb01 kernel: Aug 24 18:22:23 blr-cos-mdb01 syslogd 1.4.1: restart. |

 My /etc/my.cnf is

| [B]Quote:[/B] |
| [client] port = 3306 socket = /var/lib/mysql/mysql.sock

[mysqld]  
port = 3306  
datadir = /var/lib/mysql/data  
socket = /var/lib/mysql/mysql.sock  
user=mysql  
back\_log = 50  
skip-name-resolve  
max\_connections = 500  
max\_connect\_errors = 10  
table\_cache = 4096  
max\_allowed\_packet = 64M  
innodb-status-file=1  
binlog\_cache\_size = 32M  
max\_heap\_table\_size = 64M  
lower\_case\_table\_names=1  
sort\_buffer\_size = 8M  
join\_buffer\_size = 8M  
thread\_cache\_size = 8  
thread\_concurrency = 8  
query\_cache\_size = 64M  
query\_cache\_limit = 64M  
ft\_min\_word\_len = 4  
memlock  
default\_table\_type = MYISAM  
thread\_stack = 192K  
transaction\_isolation = READ-COMMITTED  
max\_write\_lock\_count = 1  
net\_read\_timeout = 60  
tmp\_table\_size = 64M  
log\_bin = /var/lib/mysql/binlogs/mtrak32-bin  
log-bin-index = /var/lib/mysql/binlogs/mtrak32-bin.index  
expire\_logs\_days = 10  
max\_binlog\_size=100M  
max\_binlog\_cache\_size=32M  
binlog\_format=MIXED  
binlog-ignore-db=mysql  
binlog-ignore-db=test  
binlog-ignore-db=information\_schema  
relay\_log = /var/lib/mysql/binlogs/mysql-relay-bin  
relay\_log\_index = /var/lib/mysql/binlogs/mysql-relay-bin.index  
log\_warnings  
long\_query\_time = 2  
server-id = 1  
key\_buffer\_size = 32M  
read\_buffer\_size = 2M  
read\_rnd\_buffer\_size = 32M  
bulk\_insert\_buffer\_size = 64M  
myisam\_sort\_buffer\_size = 128M  
myisam\_max\_sort\_file\_size = 10G  
myisam\_repair\_threads = 10  
innodb\_file\_per\_table=1  
innodb\_additional\_mem\_pool\_size = 20M  
innodb\_buffer\_pool\_size = 8G  
innodb\_data\_file\_path = ibdata1:100M:autoextend  
innodb\_read\_io\_threads = 20  
innodb\_write\_io\_threads = 10  
innodb\_thread\_concurrency = 16  
innodb\_flush\_log\_at\_trx\_commit = 2  
innodb\_log\_file\_size = 256M  
innodb\_max\_dirty\_pages\_pct = 90  
innodb\_flush\_method = O\_DIRECT  
innodb\_autoextend\_increment = 1  
innodb\_open\_files = 4096  
sync-binlog=1  
log-slave-updates  
auto\_increment\_increment = 10  
auto\_increment\_offset = 1  
innodb\_io\_capacity = 2000  
innodb\_log\_files\_in\_group = 3  
innodb\_log\_buffer\_size = 64M  
innodb\_extra\_rsegments = 32  
innodb\_lock\_wait\_timeout = 120  
innodb\_table\_locks = 0

[mysqldump]  
quick  
max\_allowed\_packet = 64M

[mysql]  
no-auto-rehash

[isamchk]  
key\_buffer = 512M  
sort\_buffer\_size = 512M  
read\_buffer = 8M  
write\_buffer = 8M  
[myisamchk]  
key\_buffer = 512M  
sort\_buffer\_size = 512M  
read\_buffer = 8M  
write\_buffer = 8M

[mysqlhotcopy]  
interactive-timeout

[mysqld\_safe]  
open-files-limit = 10240  
log-error=/var/log/mysqld.log  
pid-file=/var/run/mysqld/mysqld.pid

 |

The /etc/security/limits.conf has 

| [B]Quote:[/B] |
| mysql soft nofile 24000 mysql hard nofile 32000 |

I googled and got a bug report [URL]http://bugs.mysql.com/bug.php?id=25645[/URL]

How to resolve this ?

Regards  
Shann

---

<div class="post-metadata">

**Author:** ![xaprb](https://avatars.discourse-cdn.com/v4/letter/x/49beb7/32.png) [@xaprb](https://forums.percona.com/u/xaprb)\
**Post date:** [August 26, 2010, 11:50am UTC](https://forums.percona.com/t/mysql-crashes-with-innodb-assertion-failure-in-thread-1459657024-in-file-srv-srv0srv-c-line-2523/1484/2 "2010-08-26T11:50:46Z")

</div>

I suggest you check your hardware thoroughly. It looks to me like the server is crashing MySQL, not the other way around.

---

<div class="post-metadata">

**Author:** ![massoo](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/massoo/32/769_2.png) [@massoo](https://forums.percona.com/u/massoo)\
**Post date:** [August 30, 2010, 6:55am UTC](https://forums.percona.com/t/mysql-crashes-with-innodb-assertion-failure-in-thread-1459657024-in-file-srv-srv0srv-c-line-2523/1484/3 "2010-08-30T06:55:35Z")

</div>

Issue resolved. BAD … BAD MEMORY, after removing one if the DIMM’s, the server no longer crashes.

I wonder how memtest passed the Memory !!!