# rsync or xtrabackup in wsrep\_sst\_method? **URL:** \ **Category:** Percona XtraDB Cluster 5.x\ **Created:** [July 10, 2013, 10:22am UTC](https://forums.percona.com/t/rsync-or-xtrabackup-in-wsrep-sst-method/2805 "2013-07-10T10:22:29Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![AdriannaXTS](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AdriannaXTS](https://forums.percona.com/u/AdriannaXTS)\ **Post date:** [July 10, 2013, 10:22am UTC](https://forums.percona.com/t/rsync-or-xtrabackup-in-wsrep-sst-method/2805/1 "2013-07-10T10:22:29Z")
I just install xtrabdb cluster on 3 nodes , but I have some problems: -If I use xtrabackup the node 2 and 3 refuse to start. -If I use rsync , theses 2 nodes start but when I create a database in one the node , it is not replicated on the others. Someone please can help me to find out what is going wrong with my configuration? Thanks a lot. Here is the configuration I have in my.cnf node 1 wsrep\_provider\_options=“gmcast.listen\_addr=tcp://0.0.0.0:4567” wsrep\_cluster\_address=gcomm:// datadir=/var/lib/mysql user=mysql # Path to Galera library wsrep\_provider=/usr/lib64/libgalera\_smm.so # Cluster connection URL contains the IPs of node#1, node#2 and node#3 # In order for Galera to work correctly binlog format should be ROW binlog\_format=ROW # MyISAM storage engine has only experimental support default\_storage\_engine=InnoDB # This is a recommended tuning variable for performance innodb\_locks\_unsafe\_for\_binlog=1 # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb\_autoinc\_lock\_mode=2 # Node #1 address wsrep\_node\_address=@IP node 1 wsrep\_node\_name=node1 # SST method wsrep\_sst\_method=rsync # Cluster name wsrep\_cluster\_name=my\_cluster # Authentication for SST method wsrep\_sst\_auth=“sstuser:mdp” server-id = 1 node 2 wsrep\_provider\_options=“gmcast.listen\_addr=tcp://0.0.0.0:4567” wsrep\_cluster\_address=gcomm://@IP node1 datadir=/var/lib/mysql user=mysql # Path to Galera library wsrep\_provider=/usr/lib64/libgalera\_smm.so # In order for Galera to work correctly binlog format should be ROW binlog\_format=ROW # MyISAM storage engine has only experimental support default\_storage\_engine=InnoDB # This is a recommended tuning variable for performance innodb\_locks\_unsafe\_for\_binlog=1 # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb\_autoinc\_lock\_mode=2 # Node #1 address wsrep\_node\_address=@IP node2 # SST method wsrep\_sst\_method=rsync # Cluster name wsrep\_cluster\_name=my\_cluster # Authentication for SST method wsrep\_sst\_auth=“sstuser:mdp” server-id = 2 node3 wsrep\_provider\_options=“gmcast.listen\_addr=tcp://0.0.0.0:4567” wsrep\_cluster\_address=gcomm://@IP node1 datadir=/var/lib/mysql user=mysql # Path to Galera library wsrep\_provider=/usr/lib64/libgalera\_smm.so # In order for Galera to work correctly binlog format should be ROW binlog\_format=ROW # MyISAM storage engine has only experimental support default\_storage\_engine=InnoDB # This is a recommended tuning variable for performance innodb\_locks\_unsafe\_for\_binlog=1 # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera innodb\_autoinc\_lock\_mode=2 # Node #1 address wsrep\_node\_address=@IP node3 # SST method wsrep\_sst\_method=rsync # Cluster name wsrep\_cluster\_name=my\_cluster # Authentication for SST method wsrep\_sst\_auth=“sstuser:mdp” server-id = 3 So When I try to start the node 2 and 3 , here are the log I have : 130710 16:30:02 mysqld\_safe mysqld from pid file /var/lib/mysql/node2.pid ended 130710 16:39:08 mysqld\_safe Starting mysqld daemon with databases from /var/lib/mysql 130710 16:39:08 mysqld\_safe WSREP: Running position recovery with --log\_error=/tmp/tmp.RtwzeB83XD 130710 16:39:13 mysqld\_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1 130710 16:39:13 [Note] WSREP: wsrep\_start\_position var submitted: ‘00000000-0000-0000-0000-000000000000:-1’ 130710 16:39:13 [Note] WSREP: Read nil XID from storage engines, skipping position init 130710 16:39:13 [Note] WSREP: wsrep\_load(): loading provider library ‘/usr/lib64/libgalera\_smm.so’ 130710 16:39:13 [Note] WSREP: wsrep\_load(): Galera 2.5(r150) by Codership Oy \ loaded succesfully. 130710 16:39:13 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 130710 16:39:13 [Note] WSREP: Reusing existing ‘/var/lib/mysql//galera.cache’. 130710 16:39:13 [Note] WSREP: Passing config to GCS: base\_host = @IPnode2; base\_port = 4567; cert.log\_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep\_pages\_size = 0; gcache.mem\_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page\_size = 128M; gcache.size = 128M; gcs.fc\_debug = 0; gcs.fc\_factor = 1; gcs.fc\_limit = 16; gcs.fc\_master\_slave = NO; gcs.max\_packet\_size = 64500; gcs.max\_throttle = 0.25; gcs.recv\_q\_hard\_limit = 9223372036854775807; gcs.recv\_q\_soft\_limit = 0.25; gcs.sync\_donor = NO; replicator.causal\_read\_timeout = PT30S; replicator.commit\_order = 3 130710 16:39:13 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1 130710 16:39:13 [Note] WSREP: wsrep\_sst\_grab() 130710 16:39:13 [Note] WSREP: Start replication 130710 16:39:13 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1 130710 16:39:13 [Note] WSREP: protonet asio version 0 130710 16:39:13 [Note] WSREP: backend: asio 130710 16:39:13 [Note] WSREP: GMCast version 0 130710 16:39:13 [Note] WSREP: (7d9463fd-e96e-11e2-0800-2d863130a295, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567 130710 16:39:13 [Note] WSREP: (7d9463fd-e96e-11e2-0800-2d863130a295, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1 130710 16:39:13 [Note] WSREP: EVS version 0 130710 16:39:13 [Note] WSREP: PC version 0 130710 16:39:13 [Note] WSREP: gcomm: connecting to group ‘my\_cluster’, peer ‘@IPnode1:’ 130710 16:39:14 [Note] WSREP: declaring 43070462-e96e-11e2-0800-a8d5b7bb6865 stable 130710 16:39:14 [Note] WSREP: Node 43070462-e96e-11e2-0800-a8d5b7bb6865 state prim 130710 16:39:14 [Note] WSREP: view(view\_id(PRIM,43070462-e96e-11e2-0800-a8d5b7bb6865,2) memb { 43070462-e96e-11e2-0800-a8d5b7bb6865, 7d9463fd-e96e-11e2-0800-2d863130a295, } joined { } left { } partitioned { }) 130710 16:39:14 [Note] WSREP: gcomm: connected 130710 16:39:14 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 130710 16:39:14 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0) 130710 16:39:14 [Note] WSREP: Opened channel ‘my\_cluster’ 130710 16:39:14 [Note] WSREP: Waiting for SST to complete. 130710 16:39:14 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my\_idx = 1, memb\_num = 2 130710 16:39:14 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 130710 16:39:14 [Note] WSREP: STATE EXCHANGE: sent state msg: 7de25592-e96e-11e2-0800-6c7412230421 130710 16:39:14 [Note] WSREP: STATE EXCHANGE: got state msg: 7de25592-e96e-11e2-0800-6c7412230421 from 0 (node1) 130710 16:39:14 [Note] WSREP: STATE EXCHANGE: got state msg: 7de25592-e96e-11e2-0800-6c7412230421 from 1 (node2) 130710 16:39:14 [Note] WSREP: Quorum results: version = 2, component = PRIMARY, conf\_id = 1, members = 1/2 (joined/total), act\_id = 0, last\_appl. = -1, protocols = 0/4/2 (gcs/repl/appl), group UUID = bce764fb-e884-11e2-0800-f7e913b7323c 130710 16:39:14 [Note] WSREP: Flow-control interval: [23, 23] 130710 16:39:14 [Note] WSREP: Shifting OPEN → PRIMARY (TO: 0) 130710 16:39:14 [Note] WSREP: State transfer required: Group state: bce764fb-e884-11e2-0800-f7e913b7323c:0 Local state: 00000000-0000-0000-0000-000000000000:-1 130710 16:39:14 [Note] WSREP: New cluster view: global state: bce764fb-e884-11e2-0800-f7e913b7323c:0, view# 2: Primary, number of nodes: 2, my index: 1, protocol version 2 130710 16:39:14 [Warning] WSREP: Gap in state sequence. Need state transfer. 130710 16:39:16 [Note] WSREP: Running: ‘wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘@IPnode2’ --auth ‘sstuser:mdp’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘4209’’ 130710 16:39:16 [Note] WSREP: Prepared SST request: xtrabackup|@IPnode2:4444/xtrabackup\_sst 130710 16:39:16 [Note] WSREP: wsrep\_notify\_cmd is not defined, skipping notification. 130710 16:39:16 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2 130710 16:39:16 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (bce764fb-e884-11e2-0800-f7e913b7323c): 1 (Operation not permitted) at galera/src/replicator\_str.cpp:prepare\_for\_IST():436. IST will be unavailable. 130710 16:39:16 [Note] WSREP: Node 1 (node2) requested state transfer from ‘_any_’. Selected 0 (node1)(SYNCED) as donor. 130710 16:39:16 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 0) 130710 16:39:16 [Note] WSREP: Requesting state transfer: success, donor: 0 WSREP\_SST: [ERROR] xtrabackup process ended without creating ‘/var/lib/mysql//xtrabackup\_galera\_info’ (20130710 16:39:25.555) 130710 16:39:25 [ERROR] WSREP: Process completed with error: wsrep\_sst\_xtrabackup --role ‘joiner’ --address ‘@IPnode2’ --auth ‘sstuser:mdp’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘4209’: 32 (Broken pipe) 130710 16:39:25 [ERROR] WSREP: Failed to read uuid:seqno from joiner script. 130710 16:39:25 [ERROR] WSREP: SST failed: 32 (Broken pipe) 130710 16:39:25 [ERROR] Aborting 130710 16:39:25 [Warning] WSREP: 0 (node1): State transfer to 1 (node2) failed: -1 (Operation not permitted) 130710 16:39:25 [ERROR] WSREP: gcs/src/gcs\_group.c:gcs\_group\_handle\_join\_msg():719: Will never receive state. Need to abort. 130710 16:39:25 [Note] WSREP: gcomm: terminating thread 130710 16:39:25 [Note] WSREP: gcomm: joining thread 130710 16:39:25 [Note] WSREP: gcomm: closing backend 130710 16:39:26 [Note] WSREP: view(view\_id(NON\_PRIM,43070462-e96e-11e2-0800-a8d5b7bb6865,2) memb { 7d9463fd-e96e-11e2-0800-2d863130a295, } joined { } left { } partitioned { 43070462-e96e-11e2-0800-a8d5b7bb6865, }) 130710 16:39:26 [Note] WSREP: view((empty)) 130710 16:39:26 [Note] WSREP: gcomm: closed 130710 16:39:26 [Note] WSREP: gcomm: closed 130710 16:39:26 [Note] WSREP: /usr/sbin/mysqld: Terminated. 130710 16:39:26 mysqld\_safe mysqld from pid file /var/lib/mysql/node2.pid ended ---
**Author:** ![percona.jayj](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/percona.jayj/32/927_2.png) [@percona.jayj](https://forums.percona.com/u/percona.jayj)\ **Post date:** [July 12, 2013, 1:42pm UTC](https://forums.percona.com/t/rsync-or-xtrabackup-in-wsrep-sst-method/2805/2 "2013-07-12T13:42:55Z")
Check the innobackupex.backup.log in the datadir on the first node after the second fails with xtrabackup – this will tell you what error xtrabackup is facing. Did you execute the grants necessary for Xtrabackup? As far as with rsync – I’m not sure what’s happening. Can you confirm all nodes have joined the cluster from SHOW GLOBAL STATUS like ‘wsrep%’? ---
**Author:** ![AdriannaXTS](https://avatars.discourse-cdn.com/v4/letter/a/b487fb/32.png) [@AdriannaXTS](https://forums.percona.com/u/AdriannaXTS)\ **Post date:** [July 15, 2013, 4:29am UTC](https://forums.percona.com/t/rsync-or-xtrabackup-in-wsrep-sst-method/2805/3 "2013-07-15T04:29:50Z")
Hi , After updating the sstuser rights on mysql and adding the following option on my.cnf (wsrep\_sst\_donor= node1), everything gone well . All the nodes join the cluster , here is the result of the command --SHOW GLOBAL STATUS like ‘wsrep%’;-- : wsrep\_local\_state\_comment | Synced | | wsrep\_cert\_index\_size | 134145 | | wsrep\_causal\_reads | 0 | | wsrep\_incoming\_addresses | node1:3306,node2:3306,node3:3306 | | wsrep\_cluster\_conf\_id | 17 | | wsrep\_cluster\_size | 3 | | wsrep\_cluster\_state\_uuid | bce354fb-e884-11e2-0800-f7e913b7323c | | wsrep\_cluster\_status | Primary | | wsrep\_connected | ON | | wsrep\_local\_index | 2 | | wsrep\_provider\_name | Galera | | wsrep\_provider\_vendor | Codership Oy \ | | wsrep\_provider\_version | 2.5(r150) | | wsrep\_ready | ON Thanks a lots for your answer.