Hello,
So, we had this issue a few years back and I don’t actually know what the fix ever was. We have a 3 node cluster and all nodes are in sync then we get an influx of requests in a minute to our website and then for whatever reason 2 of the nodes fall out of the cluster. Simply stopping and starting mysql seems to bring things back to normal and cluster is synced up again. Just looking as to what to check for this behavior? I am not the Mysql admin or site admin but wanted to get some insight as what I can give them to check.
percona-xtradb-cluster-8.0.27-18.1.el7.x86_64
Thank you in advance!
Hi,
What would be helpful if you could show us the mysql error logs from all 3 nodes from the time when the issue was happening.
Thanks.
@Tibor_Korocz_Percona Checking for log files here. I’ll post back tomorrow with info. I’m not sure our timing is correct but I believe it is. Or is there a specific line i’m looking for in the error.log that would indicate the nodes sudden non sync status from cluster? when checking cluster status on the 2 nodes that dropped they showed “inconsistent” for wsrep_local_state_comment and cluster size 0 and no wsrep_incoming_addresses. The 1 node that was alive showed wsrep_local_state_comment at Synced and then incoming address of itself and cluster size of 1.
Hello,
Just checking if these logs show anything that would invoke a node to becoming “inconsistent” when viewing cluster status from the node? Maybe that doesn’t mean they are out of the cluster but rather being an inconsistent state will not show cluster status correctly? Looking for some smoking gun here to help prevent this from happening again.
Thank you!
I think I had wrong timing of the logs I had posted before so will repost here soon. We actually had this issue this morning and I have found the logs that hopefully will show what is happening and why. Thank you for your patience.