# trouble of starting the cluster after a crash **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [January 23, 2014, 9:55am UTC](https://forums.percona.com/t/trouble-of-starting-the-cluster-after-a-crash/3222 "2014-01-23T09:55:04Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![AdriannaXTS](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AdriannaXTS](https://forums.percona.com/u/AdriannaXTS)\ **Post date:** [January 23, 2014, 9:55am UTC](https://forums.percona.com/t/trouble-of-starting-the-cluster-after-a-crash/3222/1 "2014-01-23T09:55:04Z")
Hello; I had a crash on my donor node so I wanted to pass one of the machines in the cluster node as donor. The configuration file for the node that must pass donor is below: > [@](#): > > # The MySQL server > > [mysqld] > port = 3306 > socket = /var/lib/mysql/mysql.sock > #skip-external-locking > key\_buffer\_size = 384M > max\_allowed\_packet = 1M > table\_open\_cache = 512 > sort\_buffer\_size = 2M > read\_buffer\_size = 2M > read\_rnd\_buffer\_size = 8M > myisam\_sort\_buffer\_size = 64M > thread\_cache\_size = 8 > query\_cache\_size = 32M > > # Try number of CPU’s\*2 for thread\_concurrency > > thread\_concurrency = 8 > max\_connections=10000 > max\_connect\_errors=10000 > > #######################################################Configuration Percona > wsrep\_provider\_options=gmcast.listen\_addr=tcp://0.0.0.0:4567 > > wsrep\_cluster\_address=gcomm://node1,node2,node3 > > datadir=/var/lib/mysql > user=mysql > > # Path to Galera library > > wsrep\_provider=/usr/lib64/libgalera\_smm.so > > # In order for Galera to work correctly binlog format should be ROW > > binlog\_format=ROW > > # MyISAM storage engine has only experimental support > > default\_storage\_engine=InnoDB > > # This is a recommended tuning variable for performance > > innodb\_locks\_unsafe\_for\_binlog=1 > > # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera > > innodb\_autoinc\_lock\_mode=2 > > # Node #3 address > > wsrep\_node\_address=@ip\_node3 > > # SST method > > wsrep\_sst\_method=xtrabackup > > # Cluster name > > wsrep\_cluster\_name=cluster\_name > > # Authentication for SST method > > wsrep\_sst\_auth=“sstuser:PWD” > > wsrep\_sst\_donor= node3 > > server-id = 3 When I restart the new node donor without any parameters with / etc / init.d / mysql restart - wsrep-cluster-address = gcomm 😕 /, mysql starts without problem. Case 1: Now when I want to add other machines in the cluster.Et I get the following error: > [@](#): > > 140123 11:26:27 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql > 140123 11:26:27 mysqld\_safe WSREP: Running position recovery with --log\_error=/tmp/tmp.cOxHIxHCdo > 140123 11:26:33 mysqld\_safe WSREP: Recovered position c9a557c5-7eb7-11e3-0800-b30c642d54fb:631798 > 140123 11:26:33 [Note] WSREP: wsrep\_start\_position var submitted: ‘c9a557c5-7eb7-11e3-0800-b30c642d54fb:631798’ > 140123 11:26:33 [Note] WSREP: Read nil XID from storage engines, skipping position init > 140123 11:26:33 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ > 140123 11:26:33 [Note] WSREP: wsrep\_load(): Galera 2.5(r150) by Codership Oy \ loaded succesfully. > 140123 11:26:33 [Note] WSREP: Found saved state: c9a557c5-7eb7-11e3-0800-b30c642d54fb:-1 > 140123 11:26:33 [Note] WSREP: Reusing existing ‘/var/lib/mysql//galera.cache’. > 140123 11:26:33 [Note] WSREP: Passing config to GCS: base\_host = 10.xx.xx.xx; base\_port = 4567; cert.log\_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 1; gcs.fc\_limit = 16; gcs.fc\_master\_slave = NO; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = NO; gmcast.listen\_addr = tcp://0.0.0.0:4567; replicator.causal\_read\_timeout = PT30S; replicator.commit\_order = 3 > 140123 11:26:33 [Note] WSREP: Assign initial position for certification: 631798, protocol version: -1 > 140123 11:26:33 [Note] WSREP: wsrep\_sst\_grab() > 140123 11:26:33 [Note] WSREP: Start replication > 140123 11:26:33 [Note] WSREP: Setting initial position to c9a557c5-7eb7-11e3-0800-b30c642d54fb:631798 > 140123 11:26:33 [Note] WSREP: protonet asio version 0 > 140123 11:26:33 [Note] WSREP: backend: asio > 140123 11:26:33 [Note] WSREP: GMCast version 0 > 140123 11:26:33 [Note] WSREP: (d4fd9a88-8418-11e3-0800-e6447370e946, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 > 140123 11:26:33 [Note] WSREP: (d4fd9a88-8418-11e3-0800-e6447370e946, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 > 140123 11:26:33 [Note] WSREP: EVS version 0 > 140123 11:26:33 [Note] WSREP: PC version 0 > 140123 11:26:33 [Note] WSREP: gcomm: connecting to group ‘cluster\_name’, peer ‘node3:4567’ > 140123 11:26:33 [Warning] WSREP: (d4fd9a88-8418-11e3-0800-e6447370e946, ‘tcp://0.0.0.0:4567’) address ‘tcp://node3:4567’ points to own listening address, blacklisting > 140123 11:26:36 [Warning] WSREP: no nodes coming from prim view, prim not possible > 140123 11:26:36 [Note] WSREP: view(view\_id(NON\_PRIM,d4fd9a88-8418-11e3-0800-e6447370e946,1) memb { > d4fd9a88-8418-11e3-0800-e6447370e946, > } joined { > } left { > } partitioned { > }) > 140123 11:26:37 [Warning] WSREP: last inactive check more than PT1.5S ago, skipping check > 140123 11:27:06 [Note] WSREP: view((empty)) > 140123 11:27:06 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) > at gcomm/src/pc.cpp:connect():139 > 140123 11:27:06 [ERROR] WSREP: gcs/src/gcs\_core.c:gcs\_core\_open():195: Failed to open backend connection: -110 (Connection timed out) > 140123 11:27:06 [ERROR] WSREP: gcs/src/gcs.c:gcs\_open():1290: Failed to open channel ‘cluster\_name’ at ‘gcomm://node3:4567’: -110 (Connection timed out) > 140123 11:27:06 [ERROR] WSREP: gcs connect failed: Connection timed out > 140123 11:27:06 [ERROR] WSREP: wsrep::connect() failed: 6 > 140123 11:27:06 [ERROR] Aborting > > 140123 11:27:06 [Note] WSREP: Service disconnected. > 140123 11:27:07 [Note] WSREP: Some threads may fail to exit. > 140123 11:27:07 [Note] /usr/sbin/mysqld: Shutdown complete > > 140123 11:27:07 mysqld\_safe mysqld from pid file /var/lib/mysql/node3.pid ended Case 2: I start the donor machine and 2 other machines I add wsrep-cluster-address = gcomm 😕 / donor\_node on all machines but I get the following error: > [@](#): > > 140123 14:40:49 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql > 140123 14:40:49 mysqld\_safe WSREP: Running position recovery with --log\_error=/tmp/tmp.IxqyTq9fv5 > 140123 14:40:55 mysqld\_safe WSREP: Recovered position 693ba945-81de-11e3-0800-277c5cbcb35b:0 > 140123 14:40:55 [Note] WSREP: wsrep\_start\_position var submitted: ‘693ba945-81de-11e3-0800-277c5cbcb35b:0’ > 140123 14:40:55 [Note] WSREP: Read nil XID from storage engines, skipping position init > 140123 14:40:55 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ > 140123 14:40:55 [Note] WSREP: wsrep\_load(): Galera 2.5(r150) by Codership Oy \ loaded succesfully. > 140123 14:40:55 [Note] WSREP: Found saved state: 693ba945-81de-11e3-0800-277c5cbcb35b:-1 > 140123 14:40:55 [Note] WSREP: Reusing existing ‘/var/lib/mysql//galera.cache’. > 140123 14:40:55 [Note] WSREP: Passing config to GCS: base\_host = node1; base\_port = 4567; cert.log\_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 1; gcs.fc\_limit = 16; gcs.fc\_master\_slave = NO; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = NO; gmcast.listen\_addr = tcp://0.0.0.0:4567; replicator.causal\_read\_timeout = PT30S; replicator.commit\_order = 3 > 140123 14:40:55 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1 > 140123 14:40:55 [Note] WSREP: wsrep\_sst\_grab() > 140123 14:40:55 [Note] WSREP: Start replication > 140123 14:40:55 [Note] WSREP: Setting initial position to 693ba945-81de-11e3-0800-277c5cbcb35b:0 > 140123 14:40:55 [Note] WSREP: protonet asio version 0 > 140123 14:40:55 [Note] WSREP: backend: asio > 140123 14:40:55 [Note] WSREP: GMCast version 0 > 140123 14:40:55 [Note] WSREP: (fbbe9aa8-8433-11e3-0800-cbe33796b957, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 > 140123 14:40:55 [Note] WSREP: (fbbe9aa8-8433-11e3-0800-cbe33796b957, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 > 140123 14:40:55 [Note] WSREP: EVS version 0 > 140123 14:40:55 [Note] WSREP: PC version 0 > 140123 14:40:55 [Note] WSREP: gcomm: connecting to group ‘cluster\_name’, peer ‘10.128.26.154:4567’ > 140123 14:40:55 [Note] WSREP: declaring 3c8938b5-841d-11e3-0800-fac3799a7d7e stable > 140123 14:40:55 [Note] WSREP: Node 3c8938b5-841d-11e3-0800-fac3799a7d7e state prim > 140123 14:40:55 [Note] WSREP: view(view\_id(PRIM,3c8938b5-841d-11e3-0800-fac3799a7d7e,2) memb { > 3c8938b5-841d-11e3-0800-fac3799a7d7e, > fbbe9aa8-8433-11e3-0800-cbe33796b957, > } joined { > } left { > } partitioned { > }) > 140123 14:40:56 [Note] WSREP: gcomm: connected > 140123 14:40:56 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 > 140123 14:40:56 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) > 140123 14:40:56 [Note] WSREP: Opened channel ‘cluster\_name’ > 140123 14:40:56 [Note] WSREP: Waiting for SST to complete. > 140123 14:40:56 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 1, memb\_num = 2 > 140123 14:40:56 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. > 140123 14:40:56 [Note] WSREP: STATE EXCHANGE: sent state msg: fc0c0787-8433-11e3-0800-6c8d004124e1 > 140123 14:40:56 [Note] WSREP: STATE EXCHANGE: got state msg: fc0c0787-8433-11e3-0800-6c8d004124e1 from 0 (xts-priv-xtsinf-pp-percona3) > 140123 14:40:56 [Note] WSREP: STATE EXCHANGE: got state msg: fc0c0787-8433-11e3-0800-6c8d004124e1 from 1 (node1) > 140123 14:40:56 [Note] WSREP: Quorum results: > version = 2, > component = PRIMARY, > conf\_id = 1, > members = 1/2 (joined/total), > act\_id = 669294, > last\_appl. = -1, > protocols = 0/4/2 (gcs/repl/appl), > group UUID = c9a557c5-7eb7-11e3-0800-b30c642d54fb > 140123 14:40:56 [Note] WSREP: Flow-control interval: [23, 23] > 140123 14:40:56 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 669294) > 140123 14:40:56 [Note] WSREP: State transfer required: > Group state: c9a557c5-7eb7-11e3-0800-b30c642d54fb:669294 > Local state: 693ba945-81de-11e3-0800-277c5cbcb35b:0 > 140123 14:40:56 [Note] WSREP: New cluster view: global state: c9a557c5-7eb7-11e3-0800-b30c642d54fb:669294, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 2 > 140123 14:40:56 [Warning] WSREP: Gap in state sequence. Need state transfer. > 140123 14:40:58 [Note] WSREP: Running: ‘wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘node1’ --auth ‘sstuser:PWD’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘30106’’ > nc: Address already in use > tar: This does not look like a tar archive > tar: Exiting with failure status due to previous errors > 140123 14:40:58 [Note] WSREP: Prepared SST request: xtrabackup|node1:4444/xtrabackup\_sst > 140123 14:40:58 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. > 140123 14:40:58 [Note] WSREP: Assign initial position for certification: 669294, protocol version: 2 > 140123 14:40:58 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (693ba945-81de-11e3-0800-277c5cbcb35b) does not match group state UUID (c9a557c5-7eb7-11e3-0800-b30c642d54fb): 1 (Operation not permitted) > at galera/src/replicator\_str.cpp:prepare\_for\_IST():436. IST will be unavailable. > 140123 14:40:58 [ERROR] WSREP: Requesting state transfer failed: -113(No route to host) > 140123 14:40:58 [ERROR] WSREP: State transfer request failed unrecoverably: 113 (No route to host). Most likely it is due to inability to communicate with the cluster primary component. Restart required. > 140123 14:40:58 [Note] WSREP: Closing send monitor… > 140123 14:40:58 [Note] WSREP: Closed send monitor. > 140123 14:40:58 [Note] WSREP: gcomm: terminating thread > 140123 14:40:58 [Note] WSREP: gcomm: joining thread > 140123 14:40:58 [Note] WSREP: gcomm: closing backend > WSREP\_SST: [ERROR] Error while getting st data from donor node: 1, 2 (20140123 14:40:58.423) > 140123 14:40:58 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘node1’ --auth ‘sstuser:PWD’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘30106’: 32 (Broken pipe) > 140123 14:40:58 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. > 140123 14:40:58 [ERROR] WSREP: SST failed: 32 (Broken pipe) > 140123 14:40:58 [ERROR] Aborting > > 140123 14:40:59 [Note] WSREP: view(view\_id(NON\_PRIM,3c8938b5-841d-11e3-0800-fac3799a7d7e,2) memb { > fbbe9aa8-8433-11e3-0800-cbe33796b957, > } joined { > } left { > } partitioned { > 3c8938b5-841d-11e3-0800-fac3799a7d7e, > }) > 140123 14:40:59 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my\_idx = 0, memb\_num = 1 > 140123 14:40:59 [Note] WSREP: view((empty)) > 140123 14:40:59 [Note] WSREP: gcomm: closed > 140123 14:40:59 [Note] WSREP: Flow-control interval: [16, 16] > 140123 14:40:59 [Note] WSREP: Received NON-PRIMARY. > 140123 14:40:59 [Note] WSREP: Shifting PRIMARY → OPEN (TO: 669298) > 140123 14:40:59 [Note] WSREP: Received self-leave message. > 140123 14:40:59 [Note] WSREP: Flow-control interval: [0, 0] > 140123 14:40:59 [Note] WSREP: Received SELF-LEAVE. Closing connection. > 140123 14:40:59 [Note] WSREP: Shifting OPEN → CLOSED (TO: 669298) > 140123 14:40:59 [Note] WSREP: RECV thread exiting 0: Success > 140123 14:40:59 [Note] WSREP: recv\_thread() joined. > 140123 14:40:59 [Note] WSREP: Closing slave action queue. > 140123 14:40:59 [Note] WSREP: /usr/sbin/mysqld: Terminated. > 140123 14:40:59 mysqld\_safe mysqld from pid file /var/lib/mysql/node1.pid ended The configurations of the other two machines are as follows: > [@](#): > > wsrep\_provider\_options=“gmcast.listen\_addr=tcp://0.0.0.0:4567” > > wsrep\_cluster\_address=gcomm://node3:4567 > > datadir=/var/lib/mysql > > user=mysql > > # Path to Galera library > > wsrep\_provider=/usr/lib64/libgalera\_smm.so > > # In order for Galera to work correctly binlog format should be ROW > > binlog\_format=ROW > > # MyISAM storage engine has only experimental support > > default\_storage\_engine=InnoDB > > # This is a recommended tuning variable for performance > > innodb\_locks\_unsafe\_for\_binlog=1 > > # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera > > innodb\_autoinc\_lock\_mode=2 > > # Node #1 address > > wsrep\_node\_address=@ip\_node > > # SST method > > wsrep\_sst\_method=xtrabackup > > # Cluster name > > wsrep\_cluster\_name=cluster\_name > > # Authentication for SST method > > wsrep\_sst\_auth=“sstuser:MDP” > wsrep\_sst\_donor=node3 > > # Don’t listen on a TCP/IP port at all. This can be a security enhancement, > > # if all processes that need to connect to mysqld run on the same host. > > # All interaction with mysqld must be made via Unix sockets or named pipes. > > # Note that using this option without enabling named pipes on Windows > > # (via the “enable-named-pipe” option) will render mysqld useless! > > # > > #skip-networking > > # Replication Master Server (default) > > # binary logging is required for replication > > log-bin=mysql-bin