Percona / Innodb crashed three times (found possible reason and fix)

What could be wrong here. I switched from Mysql 5.5.23 to Percona 5.5.23 and converted most of the tables from MyISAM to Innodb.

Since then Percona has stalled three times and every time I have had to do ‘kill -9 mysql’ so that I could start it again (‘mysql stop’ just keeps doing … …).

Here are the logs:

Version: ‘5.5.23-rel25.3’ socket: ‘/tmp/mysql.sock’ port: 0 Percona Server with XtraDB (GPL), Release rel25.3, Revision 240
InnoDB: Warning: a long semaphore wait:
–Thread 139679652914944 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0

=====================================
120523 21:00:18 INNODB MONITOR OUTPUT

Per second averages calculated from the last 36 seconds

BACKGROUND THREAD

srv_master_thread loops: 368042 1_second, 367727 sleeps, 36613 10_second, 2156 background, 2156 flush
srv_master_thread log flush and writes: 425253

SEMAPHORES


OS WAIT ARRAY INFO: reservation count 77805, signal count 105351
–Thread 139679652914944 has waited at dict0dict.c line 742 for 258.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679655577344 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679656908544 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679656642304 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679654778624 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139656811095808 has waited at buf0flu.c line 1481 for 256.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679652382464 has waited at buf0buf.c line 2527 for 256.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679661434624 has waited at buf0buf.c line 2527 for 256.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679663032064 has waited at buf0buf.c line 2527 for 256.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679665428224 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656747890432 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679657441024 has waited at buf0buf.c line 2527 for 253.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139656381863680 has waited at buf0buf.c line 2527 for 252.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679663830784 has waited at dict0dict.c line 742 for 252.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679664895744 has waited at dict0dict.c line 742 for 251.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679655843584 has waited at dict0dict.c line 742 for 250.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679662499584 has waited at dict0dict.c line 742 for 250.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679661967104 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1

–Thread 139679655311104 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679664363264 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679652116224 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679663564544 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679653447424 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679664629504 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679657707264 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679661168384 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679736440576 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679657973504 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679656376064 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679659570944 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679654512384 has waited at dict0dict.c line 742 for 246.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679652648704 has waited at dict0dict.c line 742 for 246.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679660369664 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679651849984 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679658505984 has waited at dict0dict.c line 742 for 243.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679735641856 has waited at buf0buf.c line 2527 for 243.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679657174784 has waited at buf0buf.c line 2527 for 243.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679664097024 has waited at dict0dict.c line 742 for 243.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679658239744 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1

–Thread 139679736174336 has waited at dict0dict.c line 742 for 240.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679735375616 has waited at dict0dict.c line 742 for 239.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679654246144 has waited at dict0dict.c line 742 for 239.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679734843136 has waited at dict0dict.c line 742 for 237.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679653181184 has waited at dict0dict.c line 742 for 235.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679663298304 has waited at dict0dict.c line 742 for 235.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139679661700864 has waited at dict0dict.c line 742 for 234.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656758646528 has waited at buf0buf.c line 2527 for 232.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139679660103424 has waited at dict0dict.c line 742 for 230.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656747624192 has waited at dict0dict.c line 742 for 228.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656747357952 has waited at dict0dict.c line 742 for 223.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656747091712 has waited at dict0dict.c line 742 for 222.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656746825472 has waited at dict0dict.c line 742 for 221.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656746559232 has waited at dict0dict.c line 742 for 219.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656746292992 has waited at dict0dict.c line 742 for 214.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656746026752 has waited at dict0dict.c line 742 for 207.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656745760512 has waited at dict0dict.c line 742 for 203.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656745494272 has waited at dict0dict.c line 742 for 202.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656745228032 has waited at dict0dict.c line 742 for 194.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656744961792 has waited at dict0dict.c line 742 for 194.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656744695552 has waited at dict0dict.c line 742 for 139.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656744429312 has waited at dict0dict.c line 742 for 139.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656744163072 has waited at dict0dict.c line 742 for 98.000 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656743896832 has waited at dict0dict.c line 742 for 39.000 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139656743630592 has waited at dict0dict.c line 742 for 17.000 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
Mutex spin waits 174645, rounds 933148, OS waits 8825
RW-shared spins 65088, rounds 1443040, OS waits 45489

RW-excl spins 15637, rounds 777875, OS waits 22758
Spin rounds per wait: 5.34 mutex, 22.17 RW-shared, 49.75 RW-excl

FILE I/O

I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o’s: 0, sync i/o’s: 0
Pending flushes (fsync) log: 0; buffer pool: 0
175995 OS file reads, 5893732 OS file writes, 793642 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s

INSERT BUFFER AND ADAPTIVE HASH INDEX

Ibuf: size 1, free list len 8868, seg size 8870, 995 merges
merged operations:
insert 12034, delete mark 756, delete 157
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 42499631, node heap has 8001 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s

LOG

Log sequence number 73408964340
Log flushed up to 73408964340
Last checkpoint at 73408964310
Max checkpoint age 433446544
Checkpoint age target 419901340
Modified age 30
Checkpoint age 30
0 pending log writes, 0 pending chkp writes
2419317 log i/o’s done, 0.00 log i/o’s/second

BUFFER POOL AND MEMORY

Total memory allocated 22062039040; in additional pool allocated 0
Internal hash tables (constant factor + variable factor)
Adaptive hash index 471089648 (339997048 + 131092600)
Page hash 21250552 (buffer pool 0 only)
Dictionary cache 90295032 (85000912 + 5294120)
File system 400352 (82672 + 317680)
Lock system 53141048 (53125256 + 15792)
Recovery system 0 (0 + 0)
Dictionary memory allocated 5294120

InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
–Thread 139679652914944 has waited at dict0dict.c line 742 for 272.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679655577344 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679656908544 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679656642304 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679654778624 has waited at buf0buf.c line 2527 for 271.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139656811095808 has waited at buf0flu.c line 1481 for 270.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679652382464 has waited at buf0buf.c line 2527 for 270.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679661434624 has waited at buf0buf.c line 2527 for 270.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679663032064 has waited at buf0buf.c line 2527 for 270.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679665428224 has waited at dict0dict.c line 742 for 267.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656747890432 has waited at dict0dict.c line 742 for 267.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1

InnoDB: Warning: a long semaphore wait:
–Thread 139679657441024 has waited at buf0buf.c line 2527 for 267.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139656381863680 has waited at buf0buf.c line 2527 for 266.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679663830784 has waited at dict0dict.c line 742 for 266.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679664895744 has waited at dict0dict.c line 742 for 265.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679655843584 has waited at dict0dict.c line 742 for 264.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679662499584 has waited at dict0dict.c line 742 for 264.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679661967104 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679655311104 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679664363264 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679652116224 has waited at dict0dict.c line 742 for 262.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679663564544 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679653447424 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679664629504 has waited at buf0buf.c line 2527 for 261.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679657707264 has waited at buf0buf.c line 2527 for 261.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679661168384 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679736440576 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679657973504 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1

InnoDB: Warning: a long semaphore wait:
–Thread 139679656376064 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679659570944 has waited at dict0dict.c line 742 for 261.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679654512384 has waited at dict0dict.c line 742 for 260.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679652648704 has waited at dict0dict.c line 742 for 260.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679660369664 has waited at buf0buf.c line 2527 for 260.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679651849984 has waited at dict0dict.c line 742 for 258.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679658505984 has waited at dict0dict.c line 742 for 257.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679735641856 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679657174784 has waited at buf0buf.c line 2527 for 257.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679664097024 has waited at dict0dict.c line 742 for 257.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679658239744 has waited at dict0dict.c line 742 for 256.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679736174336 has waited at dict0dict.c line 742 for 254.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679735375616 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679654246144 has waited at dict0dict.c line 742 for 253.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679734843136 has waited at dict0dict.c line 742 for 251.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679653181184 has waited at dict0dict.c line 742 for 249.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679663298304 has waited at dict0dict.c line 742 for 249.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679661700864 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1

InnoDB: Warning: a long semaphore wait:
–Thread 139656758646528 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679660103424 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656747624192 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
–Thread 139679652914944 has waited at dict0dict.c line 742 for 303.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679655577344 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679656908544 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679656642304 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679654778624 has waited at buf0buf.c line 2527 for 302.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139656811095808 has waited at buf0flu.c line 1481 for 301.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679652382464 has waited at buf0buf.c line 2527 for 301.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679661434624 has waited at buf0buf.c line 2527 for 301.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679663032064 has waited at buf0buf.c line 2527 for 301.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679665428224 has waited at dict0dict.c line 742 for 298.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656747890432 has waited at dict0dict.c line 742 for 298.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1

Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679657441024 has waited at buf0buf.c line 2527 for 298.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139656381863680 has waited at buf0buf.c line 2527 for 297.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679663830784 has waited at dict0dict.c line 742 for 297.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679664895744 has waited at dict0dict.c line 742 for 296.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679655843584 has waited at dict0dict.c line 742 for 295.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679662499584 has waited at dict0dict.c line 742 for 295.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679661967104 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679655311104 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679664363264 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679652116224 has waited at dict0dict.c line 742 for 293.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679663564544 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679653447424 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679664629504 has waited at buf0buf.c line 2527 for 292.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679657707264 has waited at buf0buf.c line 2527 for 292.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679661168384 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679736440576 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679657973504 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679656376064 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1

InnoDB: Warning: a long semaphore wait:
–Thread 139679659570944 has waited at dict0dict.c line 742 for 292.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679654512384 has waited at dict0dict.c line 742 for 291.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679652648704 has waited at dict0dict.c line 742 for 291.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679660369664 has waited at buf0buf.c line 2527 for 291.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679651849984 has waited at dict0dict.c line 742 for 289.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679658505984 has waited at dict0dict.c line 742 for 288.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679735641856 has waited at buf0buf.c line 2527 for 288.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679657174784 has waited at buf0buf.c line 2527 for 288.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139679664097024 has waited at dict0dict.c line 742 for 288.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679658239744 has waited at dict0dict.c line 742 for 287.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679736174336 has waited at dict0dict.c line 742 for 285.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679735375616 has waited at dict0dict.c line 742 for 284.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679654246144 has waited at dict0dict.c line 742 for 284.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679734843136 has waited at dict0dict.c line 742 for 282.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679653181184 has waited at dict0dict.c line 742 for 280.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679663298304 has waited at dict0dict.c line 742 for 280.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139679661700864 has waited at dict0dict.c line 742 for 279.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656758646528 has waited at buf0buf.c line 2527 for 277.00 seconds the semaphore:
S-lock on RW-latch at 0x2c647f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139679655577344) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628

InnoDB: Warning: a long semaphore wait:
–Thread 139679660103424 has waited at dict0dict.c line 742 for 275.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656747624192 has waited at dict0dict.c line 742 for 273.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656747357952 has waited at dict0dict.c line 742 for 268.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656747091712 has waited at dict0dict.c line 742 for 267.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656746825472 has waited at dict0dict.c line 742 for 266.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656746559232 has waited at dict0dict.c line 742 for 264.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656746292992 has waited at dict0dict.c line 742 for 259.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656746026752 has waited at dict0dict.c line 742 for 252.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656745760512 has waited at dict0dict.c line 742 for 248.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139656745494272 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2486ba78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0
InnoDB: ###### Diagnostic info printed to the standard error stream
120523 21:26:09 [Warning] Too many connections
120523 21:26:09 [Warning] Too many connections

120523 21:35:49 mysqld_safe Number of processes running now: 0
120523 21:35:49 mysqld_safe mysqld restarted
120523 21:35:49 [Note] Flashcache bypass: disabled
120523 21:35:49 [Note] Flashcache setup error is : ioctl failed

120523 21:35:49 [Note] Plugin ‘FEDERATED’ is disabled.
120523 21:35:49 InnoDB: The InnoDB memory heap is disabled
120523 21:35:49 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120523 21:35:49 InnoDB: Compressed tables use zlib 1.2.3
120523 21:35:49 InnoDB: Using Linux native AIO
120523 21:35:49 InnoDB: Initializing buffer pool, size = 20.0G
120523 21:35:50 InnoDB: Completed initialization of buffer pool
120523 21:35:50 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 73408964310
120523 21:35:50 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
InnoDB: Doing recovery: scanned up to log sequence number 73408964340
120523 21:35:52 InnoDB: Starting an apply batch of log records to the database…
InnoDB: Progress in percents: 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
120523 21:35:53 InnoDB: Waiting for the background threads to start
120523 21:35:54 Percona XtraDB (http://www.percona.com) 1.1.8-rel25.3 started; log sequence number 73408964340
120523 21:35:54 [Note] Event Scheduler: Loaded 0 events
120523 21:35:54 [Note] /usr/local/mysql/bin/mysqld: ready for connections.

Version: ‘5.5.23-rel25.3’ socket: ‘/tmp/mysql.sock’ port: 0 Percona Server with XtraDB (GPL), Release rel25.3, Revision 240
InnoDB: Warning: a long semaphore wait:
–Thread 140479082170112 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479079507712 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140502064256768 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479088293632 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479081903872 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479080572672 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479075780352 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479111403264 has waited at buf0flu.c line 1481 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0

=====================================
120603 21:00:07 INNODB MONITOR OUTPUT

Per second averages calculated from the last 30 seconds

BACKGROUND THREAD

srv_master_thread loops: 519884 1_second, 519864 sleeps, 51784 10_second, 2305 background, 2305 flush
srv_master_thread log flush and writes: 551646

SEMAPHORES

OS WAIT ARRAY INFO: reservation count 83453, signal count 157821
–Thread 140479082170112 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479079507712 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140502064256768 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479088293632 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479081903872 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479080572672 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479075780352 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479111403264 has waited at buf0flu.c line 1481 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479100913408 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479079241472 has waited at dict0dict.c line 742 for 246.00 seconds the semaphore:
Mutex at 0x24c5ea78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1

–Thread 140479077111552 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479078708992 has waited at buf0buf.c line 2527 for 246.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479088027392 has waited at buf0buf.c line 2527 for 245.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140502064789248 has waited at buf0buf.c line 2527 for 245.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479086163712 has waited at buf0buf.c line 2527 for 245.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479081371392 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
Mutex at 0x24c5ea78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 140479089624832 has waited at dict0dict.c line 742 for 244.00 seconds the semaphore:
Mutex at 0x24c5ea78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 140502065055488 has waited at dict0dict.c line 742 for 243.00 seconds the semaphore:
Mutex at 0x24c5ea78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 140479090157312 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
Mutex at 0x24c5ea78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 140479086429952 has waited at dict0dict.c line 742 for 242.00 seconds the semaphore:
Mutex at 0x24c5ea78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 140502004061952 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
Mutex at 0x24c5ea78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 140479074449152 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 140479084300032 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
Mutex at 0x24c5ea78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1

–Thread 140479071254272 has waited at dict0dict.c line 742 for 58.000 seconds the semaphore:
Mutex at 0x24c5ea78 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
Mutex spin waits 218766, rounds 734044, OS waits 8318
RW-shared spins 96974, rounds 1807098, OS waits 56729
RW-excl spins 22955, rounds 698339, OS waits 18061
Spin rounds per wait: 3.36 mutex, 18.63 RW-shared, 30.42 RW-excl

FILE I/O

I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o’s: 0, sync i/o’s: 0
Pending flushes (fsync) log: 0; buffer pool: 0
210627 OS file reads, 9171450 OS file writes, 1138010 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s

INSERT BUFFER AND ADAPTIVE HASH INDEX

Ibuf: size 1, free list len 8868, seg size 8870, 2975 merges
merged operations:
insert 12697, delete mark 15496, delete 493
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 42499631, node heap has 8787 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s

LOG

Log sequence number 82147830513
Log flushed up to 82147830513
Last checkpoint at 82147805141
Max checkpoint age 433446544
Checkpoint age target 419901340
Modified age 483
Checkpoint age 25372
0 pending log writes, 0 pending chkp writes
3583346 log i/o’s done, 0.00 log i/o’s/second

BUFFER POOL AND MEMORY

Total memory allocated 22062039040; in additional pool allocated 0
Internal hash tables (constant factor + variable factor)
Adaptive hash index 483967472 (339997048 + 143970424)
Page hash 21250552 (buffer pool 0 only)
Dictionary cache 89768970 (85000912 + 4768058)
File system 402992 (82672 + 320320)
Lock system 53138416 (53125256 + 13160)
Recovery system 0 (0 + 0)
Dictionary memory allocated 4768058

InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
–Thread 140479082170112 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479079507712 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140502064256768 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479088293632 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479081903872 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479080572672 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 140479075780352 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x30577f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 140479082170112) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628

120604 00:28:08 mysqld_safe Starting mysqld daemon with databases from /sdd/mysql
120604 0:28:08 [Note] Flashcache bypass: disabled
120604 0:28:08 [Note] Flashcache setup error is : ioctl failed

120604 0:28:08 [Note] Plugin ‘FEDERATED’ is disabled.
120604 0:28:08 InnoDB: The InnoDB memory heap is disabled
120604 0:28:08 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120604 0:28:08 InnoDB: Compressed tables use zlib 1.2.3
120604 0:28:08 InnoDB: Using Linux native AIO
120604 0:28:08 InnoDB: Initializing buffer pool, size = 20.0G
120604 0:28:09 InnoDB: Completed initialization of buffer pool
120604 0:28:09 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 82147805141
120604 0:28:09 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
InnoDB: Doing recovery: scanned up to log sequence number 82147830513
120604 0:28:11 InnoDB: Starting an apply batch of log records to the database…
InnoDB: Progress in percents: 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
120604 0:28:13 InnoDB: Waiting for the background threads to start
120604 0:28:14 Percona XtraDB (http://www.percona.com) 1.1.8-rel25.3 started; log sequence number 82147830513
120604 0:28:14 [Note] Event Scheduler: Loaded 0 events
120604 0:28:14 [Note] /usr/local/mysql/bin/mysqld: ready for connections.

(I made this a little bit shorter because of those similar lines there… Thread numer has waited… and so on…

Version: ‘5.5.23-rel25.3’ socket: ‘/tmp/mysql.sock’ port: 0 Percona Server with XtraDB (GPL), Release rel25.3, Revision 240
InnoDB: Warning: a long semaphore wait:
–Thread 139789856839424 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x27dd7f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139789856839424) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: Warning: a long semaphore wait:
–Thread 139789782632192 has waited at dict0dict.c line 742 for 241.00 seconds the semaphore:
Mutex at 0x2440b738 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
InnoDB: Warning: a long semaphore wait:
–Thread 139766884292352 has waited at buf0buf.c line 2527 for 241.00 seconds the semaphore:
S-lock on RW-latch at 0x27dd7f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139789856839424) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info:
InnoDB: Pending preads 0, pwrites 0

=====================================
120611 21:00:09 INNODB MONITOR OUTPUT

Per second averages calculated from the last 23 seconds

BACKGROUND THREAD

srv_master_thread loops: 655361 1_second, 655337 sleeps, 65197 10_second, 3784 background, 3784 flush
srv_master_thread log flush and writes: 698575

SEMAPHORES

OS WAIT ARRAY INFO: reservation count 119896, signal count 213227
–Thread 139789856839424 has waited at buf0buf.c line 2527 for 249.00 seconds the semaphore:
S-lock on RW-latch at 0x27dd7f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139789856839424) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139789782632192 has waited at dict0dict.c line 742 for 249.00 seconds the semaphore:
Mutex at 0x2440b738 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139766884292352 has waited at buf0buf.c line 2527 for 249.00 seconds the semaphore:
S-lock on RW-latch at 0x27dd7f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139789856839424) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139789780236032 has waited at buf0buf.c line 2527 for 248.00 seconds the semaphore:
S-lock on RW-latch at 0x27dd7f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139789856839424) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139789781567232 has waited at buf0buf.c line 2527 for 247.00 seconds the semaphore:
S-lock on RW-latch at 0x27dd7f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139789856839424) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0
Last time read locked in file buf0buf.c line 2527
Last time write locked in file /home/jenkins/workspace/percona-server-5.5-binaries/label_ex p/centos5-64/Percona-Server-5.5.23-rel25.3/storage/innobase/ buf/buf0lru.c line 628
–Thread 139789856573184 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2440b738 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139789785827072 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2440b738 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139789779437312 has waited at dict0dict.c line 742 for 247.00 seconds the semaphore:
Mutex at 0x2440b738 ‘&dict_sys->mutex’, lock var 1
waiters flag 1

–Thread 139766866614016 has waited at dict0dict.c line 742 for 226.00 seconds the semaphore:
Mutex at 0x2440b738 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139766865815296 has waited at dict0dict.c line 742 for 163.00 seconds the semaphore:
Mutex at 0x2440b738 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139766865549056 has waited at dict0dict.c line 742 for 49.000 seconds the semaphore:
Mutex at 0x2440b738 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
–Thread 139766866347776 has waited at dict0dict.c line 742 for 38.000 seconds the semaphore:
Mutex at 0x2440b738 ‘&dict_sys->mutex’, lock var 1
waiters flag 1
Mutex spin waits 2114760, rounds 1000421, OS waits 8770
RW-shared spins 148421, rounds 2962902, OS waits 89565
RW-excl spins 32041, rounds 834483, OS waits 21034
Spin rounds per wait: 0.47 mutex, 19.96 RW-shared, 26.04 RW-excl

FILE I/O

I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
ibuf aio reads: 0, log i/o’s: 0, sync i/o’s: 0
Pending flushes (fsync) log: 0; buffer pool: 0
213895 OS file reads, 11478013 OS file writes, 1496154 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s

INSERT BUFFER AND ADAPTIVE HASH INDEX

Ibuf: size 1, free list len 8868, seg size 8870, 2347 merges
merged operations:
insert 14321, delete mark 5444, delete 512
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 42499631, node heap has 13984 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s

LOG

Log sequence number 88320533337
Log flushed up to 88320533337
Last checkpoint at 88320533337
Max checkpoint age 433446544
Checkpoint age target 419901340
Modified age 0
Checkpoint age 0
0 pending log writes, 0 pending chkp writes
4361377 log i/o’s done, 0.00 log i/o’s/second

BUFFER POOL AND MEMORY

Total memory allocated 22062039040; in additional pool allocated 0
Internal hash tables (constant factor + variable factor)
Adaptive hash index 569115120 (339997048 + 229118072)
Page hash 21250552 (buffer pool 0 only)
Dictionary cache 87971625 (85000912 + 2970713)
File system 402992 (82672 + 320320)
Lock system 53140672 (53125256 + 15416)
Recovery system 0 (0 + 0)
Dictionary memory allocated 2970713
InnoDB: ###### Diagnostic info printed to the standard error stream
InnoDB: Warning: a long semaphore wait:
–Thread 139789856839424 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:
S-lock on RW-latch at 0x27dd7f8 ‘&buf_pool->page_hash_latch’
a writer (thread id 139789856839424) has reserved it in mode exclusive
number of readers 0, waiters flag 1, lock_word: 0

120611 21:52:29 mysqld_safe Starting mysqld daemon with databases from /sdd/mysql
120611 21:52:29 [Note] Flashcache bypass: disabled
120611 21:52:29 [Note] Flashcache setup error is : ioctl failed

120611 21:52:29 [Note] Plugin ‘FEDERATED’ is disabled.
120611 21:52:29 InnoDB: The InnoDB memory heap is disabled
120611 21:52:29 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120611 21:52:29 InnoDB: Compressed tables use zlib 1.2.3
120611 21:52:29 InnoDB: Using Linux native AIO
120611 21:52:29 InnoDB: Initializing buffer pool, size = 20.0G
120611 21:52:30 InnoDB: Completed initialization of buffer pool
120611 21:52:30 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
120611 21:52:30 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files…
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer…
120611 21:52:33 InnoDB: Waiting for the background threads to start
120611 21:52:34 Percona XtraDB (http://www.percona.com) 1.1.8-rel25.3 started; log sequence number 88320533337
120611 21:52:34 [Note] Event Scheduler: Loaded 0 events
120611 21:52:34 [Note] /usr/local/mysql/bin/mysqld: ready for connections.

How are your backups performed?
Could you have a SAN that takes snapshots of the disk?
Or a remote backupserver that locks files during backup?

I do not believe in coincidences and although there isn’t anything in the local crontab. Things like backups (which are very often executed by a remote server) and sometimes some applications contain grooming or monitoring/software upgrade/patch solutions that are network wide run regularly.

Good luck!

Hello!

Thanks for you suggestion - unfortunately this can’t be a backup issue - I only take backups at 04.00 am using innobackupex.

That specific cron task performs only vBulletin (discussion forum software) thread views counting. “This script updates thread views once per hour. This option is suggested for forums with large numbers of threads or users.”.

Of course this whole issue could be something else but it oddly coincides with that 56 minute mark (because that script runs every hour and these stalls have happened only at 8 pm 56min.

Hopefully somebody is able to go through those logs and explain what could cause this (I know that error is partly of because Innodb waits for something and then spits out that “–Thread 140479082170112 has waited at buf0buf.c line 2527 for 272.00 seconds the semaphore:” when it’s ‘tired’ of waiting. BUT of course the big thing is that why it stalls the whole mysql until I kill -9 it.