Verify Node after error during shutdown

[OERR]

I recently adopted 3 mysql servers and my back ground is in Oracle an mssql so please forgive my terminology or lack of knowledge of mysql. We recently shutdown three of our mysql nodes for maintenance. Nodes 2 and 3 shutdown just fine, but node 1 took 20+ minutes to shutdown with an error (sorry, I do not have the error message). These are synchronized nodes. My question is, what are the steps to:

1. Verify the database integrity on node 1
2. How do you join and sync node 1 to the remaining nodes
   [LIST]
   []Database client version: libmysql - 5.5.41
   []Server type: Percona Server
   [*]CentOS
   [/LIST]
   Thanks!

[jrivera]

From your description, I would assume all three nodes were gracefully stopped. Please check scenario 3 from this link - https://www.percona.com/blog/2014/09…a-pxc-cluster/ to determine which node is the most advanced and should be bootstrapped first.

So first, bootstrap the most advanced node (I will assume it is node 1 since it was the last node that was shutdown), then start the rest of the nodes normally.

[OERR]

Hi Jrivera,
Thank you for the straight forward article. So it should be ok to power up node one an rejoin it to the cluster. Also, where there be any impact if node 1 has been offline for more than 48 hours?

[jrivera]

So first you need to determine which node has the most advanced transactions. Then bootstrap it accordingly. Once it is up you will have a cluster of 1 node, you can then start another node and let it request for a state transfer with the first node that was bootstrapped. Do the same with the last node, if all is successful you should have a wsrep_cluster_size=3.
If one node (out of 3) has been offline for more than 48 hours, and your gcache.size is not big enough to keep more than 48 hrs of transactions then once you start up the offline node it will request for SST. Otherwise it will do an IST.