SST failing please help

Hello,

When trying to join a new node into my bootstrapped cluster it fails. I am using mariadb 10 with galera.
The error i am seeing on the joining node is as follows:

WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/my.cnf --defaults-group=mysqld --no-version-check --move-back --force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log (20160422 11:02:42.807)
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20160422 11:02:42.881)
160422 11:02:42 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘107.xxx.xxx.xxx’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --defaults-group-suffix ‘’ --parent ‘15553’ ‘’ : 1 (Operation not permitted)
160422 11:02:42 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
160422 11:02:42 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
160422 11:02:42 [ERROR] Aborting

My config file on the bootstrapped node looks like this:

These groups are read by MariaDB server.

Use it for options that only the server (but not clients) should see

See the examples of server my.cnf files in /usr/share/mysql/

this is read by the standalone daemon and embedded servers

[server]

this is only for the mysqld standalone daemon

[mysqld]

max_connections = 1500
max_allowed_packet = 64M
max_connect_errors = 1000000
default_storage_engine = InnoDB

query_cache_type = 0
#query_cache_size = 64M

thread_cache_size = 75

sort_buffer_size = 2M
tmp_table_size = 32M

read_buffer_size = 128k
read_rnd_buffer_size = 256k
join_buffer_size = 128k

table_definition_cache = 400
table_open_cache = 400

max_heap_table_size = 32M

innodb_buffer_pool_size = 27G
innodb_flush_method = O_DIRECT
innodb_write_io_threads = 8
innodb_io_capacity = 1000
innodb_buffer_pool_instances = 40

* Galera-related settings

[galera]

this is only for embedded server

[embedded]

This group is only read by MariaDB servers, not by MySQL.

If you use the same .cnf file for MySQL and MariaDB,

you can put MariaDB-only options here

[mariadb]

This group is only read by MariaDB-10.0 servers.

If you use the same .cnf file for MariaDB of different versions,

use this group for options that older servers don’t understand

[mariadb-10.0]

binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
bind-address=0.0.0.0

datadir=/var/lib/mysql
innodb_log_file_size=100M
innodb_file_per_table
innodb_flush_log_at_trx_commit=2

wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_provider_options=“evs.keepalive_period = PT3S;evs.suspect_timeout = PT30S;evs.inactive_timeout = PT1M;evs.install_timeout = PT1M;gmcast.segment=1”
wsrep_cluster_address=“gcomm://107.xxx.xxx.xxx,107.xxx.xxx.xxx,107.xxx.xxx.xxx”
wsrep_cluster_name=‘ecs-orlando-staging’
wsrep_node_address='107.xxx.xxx.xxx
wsrep_node_name=‘ecs10-db.orl.rootleveltech.com
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=sst_user:secret

The joining node config looks like this

These groups are read by MariaDB server.

Use it for options that only the server (but not clients) should see

See the examples of server my.cnf files in /usr/share/mysql/

this is read by the standalone daemon and embedded servers

[server]

this is only for the mysqld standalone daemon

[mysqld]

max_connections = 1500
max_allowed_packet = 64M
max_connect_errors = 1000000
default_storage_engine = InnoDB

query_cache_type = 0
#query_cache_size = 64M

thread_cache_size = 75

sort_buffer_size = 2M
tmp_table_size = 32M

read_buffer_size = 128k
read_rnd_buffer_size = 256k
join_buffer_size = 128k

table_definition_cache = 400
table_open_cache = 400

max_heap_table_size = 32M

innodb_buffer_pool_size = 27G
innodb_flush_method = O_DIRECT
innodb_write_io_threads = 8
innodb_io_capacity = 1000
innodb_buffer_pool_instances = 40

* Galera-related settings

[galera]

this is only for embedded server

[embedded]

This group is only read by MariaDB servers, not by MySQL.

If you use the same .cnf file for MySQL and MariaDB,

you can put MariaDB-only options here

[mariadb]

This group is only read by MariaDB-10.0 servers.

If you use the same .cnf file for MariaDB of different versions,

use this group for options that older servers don’t understand

[mariadb-10.0]

binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
bind-address=0.0.0.0

datadir=/var/lib/mysql
innodb_log_file_size=100M
innodb_file_per_table
innodb_flush_log_at_trx_commit=2

wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_provider_options=“evs.keepalive_period = PT3S;evs.suspect_timeout = PT30S;evs.inactive_timeout = PT1M;evs.install_timeout = PT1M;gmcast.segment=1”
wsrep_cluster_address=“gcomm://107.xxx.xxx.xxx,107.xxx.xxx.xxx,107.xxx.xxx.xxx”
wsrep_cluster_name=‘ecs-orlando-staging’
wsrep_node_address=‘107.xxx.xxx.xxx’
wsrep_node_name=‘ecs13-db.orl.rootleveltech.com
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=sst_user:secret

I cant figure it out. Any help would be great;y appreciated

It seam like you don’t have the rights to replicate. Try adding this to all the nodes in the my.cfg
s3cret

then run this on the first node: