WSREP: SST failed: 1 (Operation not permitted)

Hi everybody,

I upgrade my nodes one by one in PXC 5.6 but I have a problem with my second node.
This error message is :
2014-07-04 10:01:31 11953 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.xx.xx’ --auth ‘:***’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --parent ‘11953’ ‘’ : 1 (Operation not permitted)
2014-07-04 10:01:31 11953 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2014-07-04 10:01:31 11953 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2014-07-04 10:01:31 11953 [ERROR] Aborting

So it can’t start, I don’t understand why because it’s ok for the others.

Thx for your help

Hi everybody,

I decide to create a new node with PXC 5.6 and now I have this error message :
2014-07-08 17:14:32 29001 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘192.168.60.12’ --auth ‘sst:*******’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/mysql/my.cnf’ --parent ‘29001’ ‘’ : 22 (Invalid argument)
2014-07-08 17:14:32 29001 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2014-07-08 17:14:32 29001 [ERROR] WSREP: SST failed: 22 (Invalid argument)
2014-07-08 17:14:32 29001 [ERROR] Aborting

I think there is a big bug with PXC 5.6

Thx for your help

Hi

I solve my second problem, it’s becaus I’ve forgotted to install qpress.
But I have always my first error message.
I want to know how can I disable the “Cleaning the existing datadir”.
When I start the second and third node, it’s removing all databases even mysql and performance_schema database

Thx

I want to know how can I disable the “Cleaning the existing datadir”.

I think this is not the error message and there happens normal SST.

This is normal using xbstream!
Use tar and this is no problem any more but then you don’t have compact, compress, … → less performance!

Hi everybody,

I have again this error with 2 nodes :
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20170418 18:59:02.528)
2017-04-18 18:59:02 140072523396864 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role ‘joiner’ --address ‘10.91.18.32’ --datadir ‘/data/mariadb/’ --parent ‘4872’ ‘’ : 1 (Operation not permitted)
2017-04-18 18:59:02 140072523396864 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2017-04-18 18:59:02 140075136813312 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2017-04-18 18:59:02 140075136813312 [ERROR] Aborting

Error in my_thread_global_end(): 1 threads didn’t exit

And in the innobackup.move.log :
170418 17:57:30 innobackupex: Starting the move-back operation

IMPORTANT: Please check that the move-back run completes successfully.
At the end of a successful move-back run innobackupex
prints “completed OK!”.

innobackupex version 2.4.7 based on MySQL server 5.7.13 Linux (x86_64) (revision id: 6f7a799)
Error: datadir must be specified.

My conf file is :

These groups are read by MariaDB server.

Use it for options that only the server (but not clients) should see

See the examples of server my.cnf files in /usr/share/mysql/

this is read by the standalone daemon and embedded servers

[server]

mysqld options MANDATORY for correct opration of the cluster

[mysqld_safe]
flush_cache = 1
numa_interleave = 1

[xtrabackup]
compress
parallel=8
compress-threads=8

2014-09-09

#rebuild-threads=8
#compact

[sst]
streamfmt=xbstream

this is only for the mysqld standalone daemon

[mysqld]

* Galera-related settings

[galera]

Mandatory settings

wsrep_provider=“/usr/lib64/galera/libgalera_smm.so”
wsrep_cluster_address=“gcomm://yy.yy.yyy.yyy”
wsrep_provider_options=“gcache.size=2G; evs.keepalive_period=PT3S; evs.inactive_check_period=PT10S; evs.suspect_timeout=PT1M; evs.inactive_timeout=PT1M; evs.install_timeout=PT1M; cert.log_conflicts=1”
#wsrep_provider_options=“gcache.size=2G; cert.log_conflicts=1”
binlog_format=row
wsrep_cluster_name=“prodealcenter_cluster”
wsrep_sst_method=xtrabackup-v2
wsrep_on=ON
wsrep_node_address=“xx.xx.xx.xx”
wsrep_node_name=“mdb02”

SST authentication string. This will be used to send SST to joining nodes.

Depends on SST method. For mysqldump method it is root:

wsrep_sst_auth=sst:zzzzzzzzzzz

Optional setting

#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0

this is only for embedded server

[embedded]

This group is only read by MariaDB servers, not by MySQL.

If you use the same .cnf file for MySQL and MariaDB,

you can put MariaDB-only options here

[mariadb]

This group is only read by MariaDB-10.1 servers.

If you use the same .cnf file for MariaDB of different versions,

use this group for options that older servers don’t understand

[mariadb-10.1]

* Basic Settings

user = mysql
#socket = /var/lib/mariadb/mariadb.sock
socket = /var/lib/mysql/mysql.sock
port = 3306
basedir = /usr
datadir = /data/mariadb
#datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
explicit_defaults_for_timestamp
#auto_increment_increment = 2
#auto_increment_offset = 2
character-set-server=utf8
collation-server=utf8_general_ci
pid-file=/data/mariadb/mariadb.pid

Instead of skip-networking the default is now to listen only on

localhost which is more compatible and is not less secure.

bind-address = xx.xx.xx.xx

* Fine Tuning General

thread_stack = 192K
tmp_table_size = 64M
thread_cache_size = 8
table_open_cache = 400
join_buffer_size=2M
key_buffer_size=50M
max_allowed_packet = 512M
skip-name-resolve
skip-host-cache
max_connections=200

* Fine Tuning InnoDB

default-storage-engine=innodb

innodb_file_format=Barracuda
innodb_file_format_max=Barracuda
innodb_flush_log_at_trx_commit=2
sync_binlog=0
innodb_io_capacity=1000
innodb_io_capacity_max=10000
innodb_log_file_size=256M
innodb_log_files_in_group=3
innodb_max_dirty_pages_pct=75
innodb_read_io_threads=16
innodb_write_io_threads=16
innodb_stats_on_metadata=0
innodb_support_xa=0
innodb_thread_concurrency=0
innodb_autoinc_lock_mode=2
innodb_buffer_pool_size=16G
innodb_buffer_pool_dump_at_shutdown=1
innodb_buffer_pool_load_at_startup=1
innodb_print_all_deadlocks=1

* Query Cache Configuration

query_cache_type = 1
query_cache_size = 256M
query_cache_limit = 2M

* Logging and Replication

Both location gets rotated by the cronjob.

Be aware that this log type is a performance killer.

As of 5.1 you can enable the log at runtime!

#general_log_file = /var/log/mariadb/mariadb.log
#general_log = 1

Here you can see queries with especially long duration

slow_query_log = 1
slow_query_log_file = /var/log/mariadb/mariadb-slow.log
long_query_time = 2
log-queries-not-using-indexes

#Error log
log-error=/var/log/mariadb/mariadb-error.log

The following can be used as easy to replay backup logs or for replication.

note: if you are setting up a replication slave, see README.Debian about

other settings you may need to change.

#server-id = 2
#log_bin = /var/log/mariadb/mariadb-bin.log
#expire_logs_days = 2
#max_binlog_size = 100M

* InnoDB

InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.

Read the manual for more InnoDB related options. There are many!

It seems that we need this for successful mysql-5.1 → mysql-5.6 upgrade

innodb_data_file_path=ibdata1:10M:autoextend

Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

Recommended in standard MySQL setup

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

I install all tools : percona-xtrabackup-24, qpress and socat
I disable iptables and selinux
But always the problem

Can u help me please

Thx

The config file lists the options (like the datadir) under the [mariadb] and [mariadb-10.1] sections, the xtrabackup-v2 SST scripts will not read the values from those sections. For PXC, those will have to be moved under [mysqld]

Hi Kenn,

Thx for your response but I use MariaDB.
I thought xtrabackup worked the same way for PXC. No ?

Kind regards

Hi Kenn,

Thx a lot I move my options in mysqld section and it’s working successfully