Hi,
These messages I’ve got in /var/log/messages (see below).
At that moment PHP response increased at about 1/3, and MyISAM key_buf_bytes_used reduced significantly too.
Is it a soft or hard failure? How to fail over it?
CentOS Linux release 6.0
2.6.32-71.29.1.el6.x86_64
Percona-Server-server-55-5.5.16-rel22.0.189.rhel6.x86_64
Mar 22 10:41:48 ishopdb01 kernel: BUG: soft lockup - CPU#7 stuck for 17163091968s! [mysqld:21471]
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: CPU 7:
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: Pid: 21471, comm: mysqld Not tainted 2.6.32-71.29.1.el6.x86_64 #1 System x3650 M3 -[7945SBC]-
Mar 22 10:41:48 ishopdb01 kernel: RIP: 0010:[] [] sys_clock_gettime+0x1/0xe0
Mar 22 10:41:48 ishopdb01 kernel: RSP: 0018:ffff880464e35f78 EFLAGS: 00000293
Mar 22 10:41:48 ishopdb01 kernel: RAX: 00000000000000e4 RBX: 00007f932d3307e0 RCX: 00000000000000d0
Mar 22 10:41:48 ishopdb01 kernel: RDX: 000000000084247f RSI: 00007f932d330810 RDI: 0000000000000003
Mar 22 10:41:48 ishopdb01 kernel: RBP: ffffffff81013c8e R08: 00000000009d7c61 R09: 0000000000000a33
Mar 22 10:41:48 ishopdb01 kernel: R10: 00000000000000d0 R11: 0000000000000293 R12: ffffffff81092e9c
Mar 22 10:41:48 ishopdb01 kernel: R13: ffff880464e35ef8 R14: ffffffff81092998 R15: ffff880464e35ef8
Mar 22 10:41:48 ishopdb01 kernel: FS: 00007f932d335700(0000) GS:ffff880caa660000(0000) knlGS:0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 22 10:41:48 ishopdb01 kernel: CR2: 00007f861c67d000 CR3: 00000003da378000 CR4: 00000000000006e0
Mar 22 10:41:48 ishopdb01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 22 10:41:48 ishopdb01 kernel: Call Trace:
Mar 22 10:41:48 ishopdb01 kernel: [] ? system_call_fastpath+0x16/0x1b
Mar 22 10:41:48 ishopdb01 kernel: BUG: soft lockup - CPU#9 stuck for 17163091968s! [mysqld:21463]
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: CPU 9:
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: Pid: 21463, comm: mysqld Not tainted 2.6.32-71.29.1.el6.x86_64 #1 System x3650 M3 -[7945SBC]-
Mar 22 10:41:48 ishopdb01 kernel: RIP: 0033:[<000000000090e775>] [<000000000090e775>] 0x90e775
Mar 22 10:41:48 ishopdb01 kernel: RSP: 002b:00007f932d4af890 EFLAGS: 00000246
Mar 22 10:41:48 ishopdb01 kernel: RAX: 0000000000000000 RBX: 00007f932d4af920 RCX: 00007f87a7545300
Mar 22 10:41:48 ishopdb01 kernel: RDX: 0000000000000000 RSI: 00007f932d4b3dc0 RDI: 00007f932d4b3da0
Mar 22 10:41:48 ishopdb01 kernel: RBP: ffffffff81013c8e R08: 0000000000000000 R09: 0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000433
Mar 22 10:41:48 ishopdb01 kernel: R13: 00000000009d7c61 R14: 00000000009d7ea9 R15: 0000000000a3bbe9
Mar 22 10:41:48 ishopdb01 kernel: FS: 00007f932d4bd700(0000) GS:ffff8800282a0000(0000) knlGS:0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 22 10:41:48 ishopdb01 kernel: CR2: 00007f85605af000 CR3: 00000003da378000 CR4: 00000000000006e0
Mar 22 10:41:48 ishopdb01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 22 10:41:48 ishopdb01 kernel: Call Trace:
Mar 22 10:41:48 ishopdb01 kernel: BUG: soft lockup - CPU#10 stuck for 17163091968s! [mysqld:21489]
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: CPU 10:
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: Pid: 21489, comm: mysqld Not tainted 2.6.32-71.29.1.el6.x86_64 #1 System x3650 M3 -[7945SBC]-
Mar 22 10:41:48 ishopdb01 kernel: RIP: 0033:[<00000000006aaf90>] [<00000000006aaf90>] 0x6aaf90
Mar 22 10:41:48 ishopdb01 kernel: RSP: 002b:00007f932cfbf138 EFLAGS: 00000246
Mar 22 10:41:48 ishopdb01 kernel: RAX: 0000000000f49bd0 RBX: 00007f932cfbf2e0 RCX: 0000000001002f80
Mar 22 10:41:48 ishopdb01 kernel: RDX: 00007f8534509710 RSI: 0000000000000001 RDI: 00007f85d42908b8
Mar 22 10:41:48 ishopdb01 kernel: RBP: ffffffff81013c8e R08: 00007f85d433f538 R09: 0000000001040080
Mar 22 10:41:48 ishopdb01 kernel: R10: 00007f852c53bad0 R11: 00007f852c53bad5 R12: 0000000000000a0f
Mar 22 10:41:48 ishopdb01 kernel: R13: 00000000009d7c61 R14: 00000000009d7e97 R15: 00000000009cb39f
Mar 22 10:41:48 ishopdb01 kernel: FS: 00007f932cfc3700(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 22 10:41:48 ishopdb01 kernel: CR2: 00007f864455d498 CR3: 00000003da378000 CR4: 00000000000006e0
Mar 22 10:41:48 ishopdb01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 22 10:41:48 ishopdb01 kernel: Call Trace:
Mar 22 10:41:48 ishopdb01 kernel: BUG: soft lockup - CPU#14 stuck for 17163091968s! [mysqld:21408]
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: CPU 14:
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: Pid: 21408, comm: mysqld Not tainted 2.6.32-71.29.1.el6.x86_64 #1 System x3650 M3 -[7945SBC]-
Mar 22 10:41:48 ishopdb01 kernel: RIP: 0033:[<00000000008478e4>] [<00000000008478e4>] 0x8478e4
Mar 22 10:41:48 ishopdb01 kernel: RSP: 002b:00007f8726dc2740 EFLAGS: 00000206
Mar 22 10:41:48 ishopdb01 kernel: RAX: 00000000000139f6 RBX: 00007f8726dc2eb0 RCX: 000000004612b7f8
Mar 22 10:41:48 ishopdb01 kernel: RDX: 00007f8762bb93c0 RSI: 0000000000000000 RDI: 0000000002929568
Mar 22 10:41:48 ishopdb01 kernel: RBP: ffffffff81013c8e R08: 0000e2f73bbf7755 R09: 000006b730759f38
Mar 22 10:41:48 ishopdb01 kernel: R10: 0000000000000001 R11: 0000000000021b7f R12: 00007f8726dc2eb0
Mar 22 10:41:48 ishopdb01 kernel: R13: 000006b6c50e15f0 R14: 0000000000000000 R15: 000006b738e58a89
Mar 22 10:41:48 ishopdb01 kernel: FS: 00007f8726dc3700(0000) GS:ffff880caa6c0000(0000) knlGS:0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 22 10:41:48 ishopdb01 kernel: CR2: 00007f86d0472000 CR3: 00000003da378000 CR4: 00000000000006e0
Mar 22 10:41:48 ishopdb01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 22 10:41:48 ishopdb01 kernel: Call Trace:
Mar 22 10:41:48 ishopdb01 kernel: BUG: soft lockup - CPU#15 stuck for 17163091968s! [mysqld:21508]
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: CPU 15:
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: Pid: 21508, comm: mysqld Not tainted 2.6.32-71.29.1.el6.x86_64 #1 System x3650 M3 -[7945SBC]-
Mar 22 10:41:48 ishopdb01 kernel: RIP: 0033:[<00000000005beb87>] [<00000000005beb87>] 0x5beb87
Mar 22 10:41:48 ishopdb01 kernel: RSP: 002b:00007f932cc7e270 EFLAGS: 00000202
Mar 22 10:41:48 ishopdb01 kernel: RAX: 00007f85841d56e0 RBX: 00007f932cc7e300 RCX: 00007f85841d56c0
Mar 22 10:41:48 ishopdb01 kernel: RDX: 00007f85841d5630 RSI: 00007f85845c48f8 RDI: 00007f932cc7f720
Mar 22 10:41:48 ishopdb01 kernel: RBP: ffffffff81013c8e R08: 00007f85845c3d28 R09: 00007f85845c3600
Mar 22 10:41:48 ishopdb01 kernel: R10: 0000000000000000 R11: 00007f85845c3df0 R12: 00000000000012fb
Mar 22 10:41:48 ishopdb01 kernel: R13: 00000000009d3d69 R14: 00000000009d3eb9 R15: 00000000009d3cf4
Mar 22 10:41:48 ishopdb01 kernel: FS: 00007f932cc82700(0000) GS:ffff880caa6e0000(0000) knlGS:0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 22 10:41:48 ishopdb01 kernel: CR2: 00007f862c3ad000 CR3: 00000003da378000 CR4: 00000000000006e0
Mar 22 10:41:48 ishopdb01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 22 10:41:48 ishopdb01 kernel: Call Trace:
Mar 22 10:41:48 ishopdb01 kernel: BUG: soft lockup - CPU#0 stuck for 17163091968s! [mysqld:21520]
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: CPU 0:
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: Pid: 21520, comm: mysqld Not tainted 2.6.32-71.29.1.el6.x86_64 #1 System x3650 M3 -[7945SBC]-
Mar 22 10:41:48 ishopdb01 kernel: RIP: 0033:[<00000000006a2b9f>] [<00000000006a2b9f>] 0x6a2b9f
Mar 22 10:41:48 ishopdb01 kernel: RSP: 002b:00007f932ca64750 EFLAGS: 00000212
Mar 22 10:41:48 ishopdb01 kernel: RAX: 0000000000f460b0 RBX: 00007f932ca647a0 RCX: 0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: RDX: 0000000000000002 RSI: 0000000000000001 RDI: 00007f8560424490
Mar 22 10:41:48 ishopdb01 kernel: RBP: ffffffff81013c8e R08: 00007f856042429a R09: 0000000000000002
Mar 22 10:41:48 ishopdb01 kernel: R10: 0000000000000000 R11: 00007f932ed0be0f R12: 00000000000012fb
Mar 22 10:41:48 ishopdb01 kernel: R13: 00000000009d3d69 R14: 00000000009d3eb9 R15: 00000000009d3cf4
Mar 22 10:41:48 ishopdb01 kernel: FS: 00007f932ca67700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 22 10:41:48 ishopdb01 kernel: CR2: 00007f856077ff28 CR3: 00000003da378000 CR4: 00000000000006f0
Mar 22 10:41:48 ishopdb01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 22 10:41:48 ishopdb01 kernel: Call Trace:
Mar 22 10:41:48 ishopdb01 kernel: BUG: soft lockup - CPU#2 stuck for 17163091968s! [mysqld:21472]
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: CPU 2:
Mar 22 10:41:48 ishopdb01 kernel: Modules linked in: usb_storage bonding ipv6 xfs exportfs cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp sg ioatdma dca i7core_edac edac_core bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mod [last unloaded: scsi_wait_scan]
Mar 22 10:41:48 ishopdb01 kernel: Pid: 21472, comm: mysqld Not tainted 2.6.32-71.29.1.el6.x86_64 #1 System x3650 M3 -[7945SBC]-
Mar 22 10:41:48 ishopdb01 kernel: RIP: 0033:[<000000000057c100>] [<000000000057c100>] 0x57c100
Mar 22 10:41:48 ishopdb01 kernel: RSP: 002b:00007f932d2ff778 EFLAGS: 00000286
Mar 22 10:41:48 ishopdb01 kernel: RAX: 00007f8620157940 RBX: 00007f932d2ff7a0 RCX: 00000000000008ec
Mar 22 10:41:48 ishopdb01 kernel: RDX: 0000000000006f5b RSI: 000000004641d290 RDI: 00007f932d300520
Mar 22 10:41:48 ishopdb01 kernel: RBP: ffffffff81013c8e R08: 00000000000003e8 R09: 0000000001041960
Mar 22 10:41:48 ishopdb01 kernel: R10: 0d45534c4509090a R11: 00007f86200007e8 R12: 00000000000001f5
Mar 22 10:41:48 ishopdb01 kernel: R13: 0000000000000004 R14: 00007f932d3046a8 R15: 0000000001067460
Mar 22 10:41:48 ishopdb01 kernel: FS: 00007f932d304700(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 22 10:41:48 ishopdb01 kernel: CR2: 00007f868460f9e8 CR3: 00000003da378000 CR4: 00000000000006e0
Mar 22 10:41:48 ishopdb01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 22 10:41:48 ishopdb01 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 22 10:41:48 ishopdb01 kernel: Call Trace: