Cluster does not start

I have Percona-XtraDB-Cluster-server 5.6.21 installed on 3 nodes. The cluster used to work on 2 nodes, but I cannot bring it back up.

I have bootstrapped the cluster on node1. But node2 fails:

WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20150209 02:15:30.956)
grep: /var/lib/mysql//xtrabackup_checkpoints: No such file or directory
WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql/ (20150209 02:15:41.121)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20150209 02:15:41.122)
2015-02-09 02:15:41 2672050 [Warning] WSREP: 1.0 (node1.com): State transfer to 0.0 (node2.com) failed: -22 (Invalid argument)
2015-02-09 02:15:41 2672050 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():722: Will never receive state. Need to abort.
2015-02-09 02:15:41 2672050 [Note] WSREP: gcomm: terminating thread
2015-02-09 02:15:41 2672050 [Note] WSREP: gcomm: joining thread
2015-02-09 02:15:41 2672050 [Note] WSREP: gcomm: closing backend
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20150209 02:15:41.489)
2015-02-09 02:15:41 2672050 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘node2.com’ --auth user:pass’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘2672050’ ‘’ : 1 (Operation not permitted)
2015-02-09 02:15:41 2672050 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-02-09 02:15:41 2672050 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2015-02-09 02:15:41 2672050 [ERROR] Aborting

I don’t know how to debug this issue. Any idea?

I am having a similar issue with 5.6.21-70 on CentOS 7

NODE 1 - mysql-error.log

WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql//innobackup.backup.log (20150219 18:01:58.157)
WSREP_SST: [ERROR] Cleanup after exit with status:22 (20150219 18:01:58.158)
WSREP_SST: [INFO] Cleaning up temporary directories (20150219 18:01:58.161)
2015-02-19 18:01:58 10231 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.88.181:4444/xtrabackup_sst//1' --auth 'sstuser:xxxxx' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' '' --gtid '7cac0fcf-b83b-11e4-a754-ba12038dc0d4:2'
2015-02-19 18:01:58 10231 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.88.181:4444/xtrabackup_sst//1' --auth 'sstuser:xxxxx' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' '' --gtid '7cac0fcf-b83b-11e4-a754-ba12038dc0d4:2': 22 (Invalid argument)
2015-02-19 18:01:58 10231 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.88.181:4444/xtrabackup_sst//1' --auth 'sstuser:xxxxx' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' '' --gtid '7cac0fcf-b83b-11e4-a754-ba12038dc0d4:2'

NODE 1 - innobackup.backup.log

150219 18:01:58 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'sstuser' (using password: YES).
innobackupex: got a fatal error with the following stacktrace: at /usr//bin/innobackupex line 3004.
main::mysql_connect('abort_on_error', 1) called at /usr//bin/innobackupex line 1539
innobackupex: Error: Failed to connect to MySQL server: DBI connect(';mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock','sstuser',...) failed: Access denied for user 'sstuser'@'localhost' (using password: YES) at /usr//bin/innobackupex line 2988.

NODE 2 - mysql-error.log

WSREP_SST: [INFO] Cleaning the binlog directory /var/lib/mysql as well (20150219 18:01:48.073)
rm: cannot remove ‘/var/lib/mysql/*.index’: No such file or directory
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20150219 18:01:48.079)
2015-02-19 18:01:49 11547 [Note] WSREP: (5e5687d2, 'tcp://0.0.0.0:4567') turning message relay requesting off
grep: /var/lib/mysql//xtrabackup_checkpoints: No such file or directory
WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql/ (20150219 18:01:58.162)
2015-02-19 18:01:58 11547 [Warning] WSREP: 1.0 (Node1): State transfer to 0.0 (Node2) failed: -22 (Invalid argument)
2015-02-19 18:01:58 11547 [ERROR] WSREP: gcs/src/gcs_group.cpp:int gcs_group_handle_join_msg(gcs_group_t*, const gcs_recv_msg_t*)():722: Will never receive state. Need to abort.
2015-02-19 18:01:58 11547 [Note] WSREP: gcomm: terminating thread
2015-02-19 18:01:58 11547 [Note] WSREP: gcomm: joining thread
2015-02-19 18:01:58 11547 [Note] WSREP: gcomm: closing backend
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20150219 18:01:58.165)
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20150219 18:01:58.344)
2015-02-19 18:01:58 11547 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.88.181' --auth 'sstuser:xxxxx' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '11547' '' : 1 (Operation not permitted)
2015-02-19 18:01:58 11547 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-02-19 18:01:58 11547 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2015-02-19 18:01:58 11547 [ERROR] Aborting

As you can see it errors “Access denied for user ‘sstuser’@‘localhost’”
Is “sstuser” created ? Can you please share my.cnf on failed node. You may found this useful http://www.percona.com/doc/percona-xtradb-cluster/5.5/howtos/cenots_howto.html

Hi,

I am facing similar error as posted by jsobrier in intial post.

  • I am using below version of Percona XtraDB Cluster
  • sstuser has proper grants
  • Below is the error seen on 2nd node which I am trying to join with first donor node.
  • Log files on DONOR server shows

** innobackup.backup.log

** mysqld.log

What could be the issue here ?