xtradb cluster 5.7.19-29.22-1.trusty query hangs on node

[Antonio_Falzarano]

Hi,
i recently migrate one cluster from galera codership 5.6 to xtradb cluster 5.7.19-29.22-1.trusty
Sometimes one node of my cluster (5 nodes) hangs on select and doesn’t receive query anymore.
show processlist and show innodb status works instead.
All select are stuck on sending data or statistics (query that usually duration was 5ms), if i kill them they stuck on killed status.
The only option is to stop and start mysql inside node. i didn’t find any lock on TRANSACTION section of innodb status

Maybe there is a bug with this version that generate an internal deadlock?

I attach my innodb status

Thank you for everyone that can help me to troubleshooting this problem

innodb_status_20180109.txt (144 KB)

[Antonio_Falzarano]

Nobody has had experience of my same problem ?

[Antonio_Falzarano]

Hi guys,
problem was appened again, anyone of Percona can help me to troubleshoot inside this problem?

[lorraine.pocklington]

Hi Antonio, you mention that you were attaching innodb status but can’t see anything here?
Just to confirm, you were running this application successfully in the past, but the problem has developed only since the migration?
If you have any logs or messages you can share, that might help inform the tech team. Thanks!

[Antonio_Falzarano]

Hi lorraine, yes my application running well on mysql 5.6 codership and issue of hanging not happening (we have multiple galera clusters where application calls),
We have migrate our first cluster from 5.6 codership to Xtradb 5.7, and it works well except this randomly hanging of one node, and i need to restart the node to became online again.

I link new innodb status log (exceed max size) and i have this node in maintenance status without restarting them, so it’s a good opportunity to troubleshoot this randomly problem.

https://pastebin.com/he84ZiPZ

Regards
Antonio

[Kenn_Takara]

Hi Antonio,

There have been several bugs fixed in the upcoming PXC 5.7.20 release that look similar to what you’re seeing.

https://jira.percona.com/browse/PXC-877
https://jira.percona.com/browse/PXC-878

Does this look similar to what you’re seeing?

Thanks,

[Antonio_Falzarano]

Hi Kenn Takara , i can view only PXC-877 and not have access to PXC-878, so about PXC-877 yes it is a similar problem except that i can execute show engine innodb status and global status.

When will be released 5.7.20 version?
Regards

[Kenn_Takara]

PXC-878 has the threads hanging and they show up as being in the “statistics” state in SHOW PROCESSLIST.
Here is the launchpad bug:
https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1722560

PXC 5.7.20 is due to be released soon, unfortunately I can’t give an exact date.