# 3 node cluster - gracefully shutdown ALL nodes - cluster fail to start

**URL:** <https://forums.percona.com/t/3-node-cluster-gracefully-shutdown-all-nodes-cluster-fail-to-start/3589>\
**Category:** Percona XtraDB Cluster 5.x\
**Created:** [July 2, 2014, 6:24am UTC](https://forums.percona.com/t/3-node-cluster-gracefully-shutdown-all-nodes-cluster-fail-to-start/3589 "2014-07-02T06:24:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielm](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@danielm](https://forums.percona.com/u/danielm)\
**Post date:** [July 2, 2014, 6:24am UTC](https://forums.percona.com/t/3-node-cluster-gracefully-shutdown-all-nodes-cluster-fail-to-start/3589/1 "2014-07-02T06:24:37Z")

</div>

Hi,

Ubuntu 12  
Percona Cluster 5.6

Selinux (not applied to Ubuntu), ApArmor - disabled

Does it true that due normal (gracefully) server shutdown  
(all nodes) I need to do

bootstrap-pxc

If true, how can I determine which node has latest data ?

For now I didn’t find any other method to bring cluster online.

Thanks in advanced…  
Daniel

---

<div class="post-metadata">

**Author:** ![jrivera](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jrivera/32/13_2.png) [@jrivera](https://forums.percona.com/u/jrivera)\
**Post date:** [July 3, 2014, 7:50am UTC](https://forums.percona.com/t/3-node-cluster-gracefully-shutdown-all-nodes-cluster-fail-to-start/3589/2 "2014-07-03T07:50:55Z")

</div>

Hi,

You don’t have to bootstrap each node after a graceful shutdown of all pxc nodes in a cluster. All you have to do is first identify the node with the most advanced data and then bootstrap that node and then the rest of the nodes you can start as usual.

You may refer to the following documentation:  
[url][Percona XtraDB Cluster](http://www.percona.com/doc/percona-xtradb-cluster/5.5/manual/restarting_nodes.html%5B/url%5D)  
[url][http://galeracluster.com/documentation-webpages/restartingacluster.html[/url]](http://galeracluster.com/documentation-webpages/restartingacluster.html%5B/url%5D)

---

<div class="post-metadata">

**Author:** ![danielm](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@danielm](https://forums.percona.com/u/danielm)\
**Post date:** [July 9, 2014, 2:14am UTC](https://forums.percona.com/t/3-node-cluster-gracefully-shutdown-all-nodes-cluster-fail-to-start/3589/3 "2014-07-09T02:14:22Z")

</div>

Hi,

Thanks a lot for answer. Intersiting thing in my case is  
all nodes have the same entry:

@wp01:/var/lib/mysql# cat grastate.dat

# GALERA saved state

version: 2.1  
uuid: e0e86b14-fdfe-11e3-95f7-de5f58d5c277  
seqno: -1  
cert\_index:

@wp02:/var/lib/mysql# cat grastate.dat

# GALERA saved state

version: 2.1  
uuid: e0e86b14-fdfe-11e3-95f7-de5f58d5c277  
seqno: -1  
cert\_index:

@wp03:/var/lib/mysql# cat grastate.dat

# GALERA saved state

version: 2.1  
uuid: e0e86b14-fdfe-11e3-95f7-de5f58d5c277  
seqno: -1  
cert\_index:

All nodes was closed by

#shutdown -h now

---

<div class="post-metadata">

**Author:** ![cetanhota](https://avatars.discourse-cdn.com/v4/letter/c/7cd45c/32.png) [@cetanhota](https://forums.percona.com/u/cetanhota)\
**Post date:** [July 21, 2014, 12:36pm UTC](https://forums.percona.com/t/3-node-cluster-gracefully-shutdown-all-nodes-cluster-fail-to-start/3589/4 "2014-07-21T12:36:37Z")

</div>

Since all your nodes are showing a seqno as -1 you will need to do a wsrep-recover to see which node has the most current data. You can use wsrep-recover like this.

mysqld --wsrep-recover

Mysql will startup and then shutdown it will record the needed information in your mysqld.log file.

Look at your mysql log file to see which machine has the highest seqno. This would be the node you want to bootstrap. In your log file you will see something like this:

140721 12:00:00 [Note] WSREP: Recovered position UUID:seqno