my database env:
MySQL Server | Percona-Server-5.5.33-rel31.1
# Percona Toolkit System Summary Report ######################
Date | 2017-08-30 11:46:02 UTC (local TZ: CST +0800)
Hostname | database9
Uptime | 370 days, 2:29, 2 users, load average: 0.08, 0.12, 0.16
System | Dell Inc.; PowerEdge R630; vNot Specified (Rack Mount Chassis)
Service Tag | FTD5DF2
Platform | Linux
Release | CentOS release 6.8 (Final)
Kernel | 2.6.32-642.4.2.el6.x86_64
Architecture | CPU = 64-bit, OS = 64-bit
Threading | NPTL 2.12
SELinux | Disabled
Virtualized | No virtualization detected
# Processor ##################################################
Processors | physical = 2, cores = 20, virtual = 40, hyperthreading = yes
Speeds | 40x2199.942
Models | 40xIntel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
Caches | 40x25600 KB
# Memory ################################################## ###
Total | 125.9G
last sunday, one of my database response to application suddenly very slow, and the mysql thread running and cpu load is very high, such as:
[HTML]top - 21:16:17 up 367 days, 4:00, 0 users, load average: 906.18, 623.71, 300.
Tasks: 1149 total, 3 running, 1146 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5%us, 0.3%sy, 0.0%ni, 99.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 132042600k total, 131641800k used, 400800k free, 0k buffers
Swap: 41929644k total, 3606772k used, 38322872k free, 24683400k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
38391 mysql 20 0 111g 99g 5636 D 3403.4 78.8 147443:28 mysqld
TS 1503839784.028255754 2017-08-27 21:16:24
MemTotal: 132042600 kB
MemFree: 435984 kB
Buffers: 0 kB
Cached: 24602272 kB
SwapCached: 383772 kB[/HTML]
and have many repeate message in error log, such as:
[HTML]Aug 27 21:00:00 database9 mysqld-3309: Memory status:
Aug 27 21:00:00 database9 mysqld-3309: Non-mmapped space allocated from system: 0
Aug 27 21:00:00 database9 mysqld-3309: Number of free chunks:#011#011#011 1
Aug 27 21:00:00 database9 mysqld-3309: Number of fastbin blocks:#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Number of mmapped regions:#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Space in mmapped regions:#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Maximum total allocated space:#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Space available in freed fastbin blocks: 0
Aug 27 21:00:00 database9 mysqld-3309: Total allocated space:#011#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Total free space:#011#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Top-most, releasable space:#011#011 0
Aug 27 21:00:00 database9 mysqld-3309: Estimated memory (with thread stack): 16515072
Aug 27 21:00:00 database9 mysqld-3309: Events status:
Aug 27 21:00:00 database9 mysqld-3309: LLA = Last Locked At LUA = Last Unlocked At
Aug 27 21:00:00 database9 mysqld-3309: WOC = Waiting On Condition DL = Data Locked
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: Event scheduler status:
Aug 27 21:00:00 database9 mysqld-3309: State : INITIALIZED
Aug 27 21:00:00 database9 mysqld-3309: Thread id : 0
Aug 27 21:00:00 database9 mysqld-3309: LLA : n/a:0
Aug 27 21:00:00 database9 mysqld-3309: LUA : n/a:0
Aug 27 21:00:00 database9 mysqld-3309: WOC : NO
Aug 27 21:00:00 database9 mysqld-3309: Workers : 0
Aug 27 21:00:00 database9 mysqld-3309: Executed : 0
Aug 27 21:00:00 database9 mysqld-3309: Data locked: NO
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: Event queue status:
Aug 27 21:00:00 database9 mysqld-3309: Element count : 0
Aug 27 21:00:00 database9 mysqld-3309: Data locked : NO
Aug 27 21:00:00 database9 mysqld-3309: Attempting lock : NO
Aug 27 21:00:00 database9 mysqld-3309: LLA : init_queue:139
Aug 27 21:00:00 database9 mysqld-3309: LUA : init_queue:149
Aug 27 21:00:00 database9 mysqld-3309: WOC : NO
Aug 27 21:00:00 database9 mysqld-3309: Next activation : never
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: Status information:
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: Current dir: /web/mysql/node3309/data/
Aug 27 21:00:00 database9 mysqld-3309: Running threads: 61 Stack size: 262144
Aug 27 21:00:00 database9 mysqld-3309: Current locks:
Aug 27 21:00:00 database9 mysqld-3309: lock: 0x7fcbe4c41e38:
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: lock: 0x7fcc35bf82c0: read
Aug 27 21:00:00 database9 mysqld-3309: read : 0x7fcc01689398 (29404920:2);
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: lock: 0x7fcc35bf8180: read
Aug 27 21:00:00 database9 mysqld-3309: read : 0x7fcc01687398 (29404920:2);
Aug 27 21:00:00 database9 mysqld-3309: lock: 0x7fcc35bf7f00: read
Aug 27 21:00:00 database9 mysqld-3309: read : 0x7fcd1bcc8398 (29404920:2);
Aug 27 21:00:00 database9 mysqld-3309:
Aug 27 21:00:00 database9 mysqld-3309: lock: 0x7fcc35bf7dc0: read
Aug 27 21:00:00 database9 mysqld-3309: read : 0x7fcc017b2398 (29404920:2); [/HTML]
it seems that mysql server receives many SIGHUP, but there is no operate to MySQL server by admin or application, I have no idea for this error message, thanks a lot for everyone help.