I have a server running centos7 and percona 5.6.27. From time to time, it will hang up, meaning that no queries will complete, no show commands will complete, etc. This happens until the server reaches max connections and it will not recover it self. Here is a pt-pmp and a gdb bt of the server in question. Nothing is logged to messages or dmsg.
OS: CentOS Linux release 7.2.1511 (Core)
Kernel: 3.10.0-327.22.2.el7.x86_64
output:
(gdb) bt #0 0x00007f0b03bb5b0c in waitpid () from /lib64/libc.so.6 #1 0x0000000000440994 in waitchld.isra.10 () #2 0x0000000000441c4c in wait_for () #3 0x000000000043392e in execute_command_internal () #4 0x000000000046f494 in parse_and_execute () #5 0x000000000042f1bb in execute_builtin.isra.2 () #6 0x0000000000431319 in execute_simple_command () #7 0x00000000004326ab in execute_command_internal () #8 0x00000000004324c0 in execute_command_internal () #9 0x0000000000432782 in execute_command_internal () #10 0x0000000000434cbf in execute_function.isra.7 () #11 0x0000000000431631 in execute_simple_command () #12 0x00000000004326ab in execute_command_internal () #13 0x00000000004324c0 in execute_command_internal () #14 0x0000000000433b5e in execute_command () #15 0x0000000000432484 in execute_command_internal () #16 0x0000000000433b5e in execute_command () #17 0x0000000000432484 in execute_command_internal () #18 0x0000000000433b5e in execute_command () #19 0x0000000000432484 in execute_command_internal () #20 0x0000000000433b5e in execute_command () #21 0x0000000000432484 in execute_command_internal () #22 0x0000000000433b5e in execute_command () #23 0x0000000000432484 in execute_command_internal () #24 0x0000000000433b5e in execute_command () #25 0x0000000000432484 in execute_command_internal () #26 0x0000000000433b5e in execute_command () #27 0x0000000000433c35 in execute_while_or_until () #28 0x0000000000431fe0 in execute_command_internal () #29 0x0000000000433b5e in execute_command () #30 0x000000000041e285 in reader_loop () #31 0x000000000041c8ee in main () (gdb) bt^CQuit (gdb) quit A debugging session is active. Inferior 1 [process 26560] will be detached. Quit anyway? (y or n) y Detaching from program: /usr/bin/bash, process 26560 [root@db101-prod /root]# pt-pmp -p 26560 Thu Mar 23 19:54:32 EDT 2017 1 waitpid(libc.so.6),waitchld.isra.10,wait_for,execute_command_internal,parse_and_execute,execute_builtin.isra.2,execute_simple_command,execute_command_internal,execute_command_internal,execute_command_internal,execute_function.isra.7,execute_simple_command,execute_command_internal,execute_command_internal,execute_command,execute_command_internal,execute_command,execute_command_internal,execute_command,execute_command_internal,execute_command,execute_command_internal,execute_command,execute_command_internal,execute_command,execute_command_internal,execute_command,execute_while_or_until,execute_command_internal,execute_command,reader_loop,main