XtraDB Cluster with Xtrabackup - tar fails

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

I am currently trying to setup an XdraDB Cluster using the xtrabackup transfer method, however it just doesn’t work out for me. Here is what I get in the innobackup.backup.log of the donor:

innobackupex: Created backup directory /var/lib/mysql

tar: -: Cannot write: Broken pipe

tar: Error is not recoverable: exiting now

innobackupex: Error: Failed to stream ‘backup-my.cnf’: Inappropriate ioctl for device at /usr/local/mysql//bin/innobackupex line 381.

I’ve already changed the hardcoded value for the TMPDIR, however that doesn’t work either, tough the mysql user has full permissions on that directory. I’ve also specified tmpdir=/var/lib/mysql in the [mysqld] and [xtrabackup] section of the donors and joiners my.cnf file.

Any other ideas why the second node my fail to join the cluster?

I’ve compiled Percona-XtraDB-Cluster 5.5.30-23.7.4 and Percona-Xtrabackup-2.0.6 from the source on a Gentoo 64-bit system.

Here is the donors log:

130427 14:20:38 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1

130427 14:20:38 [Note] WSREP: STATE_EXCHANGE: sent state UUID: dea22da6-af34-11e2-0800-54545e34a8dc

130427 14:20:38 [Note] WSREP: STATE EXCHANGE: sent state msg: dea22da6-af34-11e2-0800-54545e34a8dc

130427 14:20:38 [Note] WSREP: STATE EXCHANGE: got state msg: dea22da6-af34-11e2-0800-54545e34a8dc from 0 (atlantia)

130427 14:20:38 [Note] WSREP: Quorum results:

version = 2,

component = PRIMARY,

conf_id = 6,

members = 1/1 (joined/total),

act_id = 7,

last_appl. = 0,

protocols = 0/4/2 (gcs/repl/appl),

group UUID = bda9abb9-ae84-11e2-0800-514b0920452c

130427 14:20:38 [Note] WSREP: Flow-control interval: [16, 16]

130427 14:20:38 [Note] WSREP: New cluster view: global state: bda9abb9-ae84-11e2-0800-514b0920452c:7, view# 7: Primary, number of nodes: 1, my index: 0, protocol version 2

130427 14:20:38 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

130427 14:20:38 [Note] WSREP: Assign initial position for certification: 7, protocol version: 2

WSREP_SST: [ERROR] innobackupex finished with error: 25. Check /var/lib/mysql//innobackup.backup.log (20130427 14:20:41.414)

130427 14:20:41 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup --role ‘donor’ --address ‘joiner.example.com:4444/xtrabackup_sst’ --auth ‘backup:password’ --socket '/var/run/mysqld$

130427 14:20:41 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role ‘donor’ --address ‘joiner.example.com:4444/xtrabackup_sst’ --auth ‘backup:.Riptide!’ --socket '/var/r$

130427 14:20:41 [Warning] WSREP: Could not find peer: d48f40ba-af34-11e2-0800-68c73681a738

130427 14:20:41 [Warning] WSREP: 0 (atlantia): State transfer to -1 (left the group) failed: -1 (Operation not permitted)

130427 14:20:41 [Note] WSREP: Shifting DONOR/DESYNCED → JOINED (TO: 7)

130427 14:20:41 [Note] WSREP: Member 0 (donor) synced with group.

130427 14:20:41 [Note] WSREP: Shifting JOINED → SYNCED (TO: 7)

130427 14:20:41 [Note] WSREP: Synchronized with group, ready for connections

130427 14:20:41 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

130427 14:20:43 [Note] WSREP: cleaning up d48f40ba-af34-11e2-0800-68c73681a738 (tcp://xxx.xxx.xxx.xxx:4567)

Any ideas?

2

Here is my my.cnf (the conf on the joiner looks the same except for the host-specific things of course):

[mysqld]

character-set-server = utf8

user = mysql

port = 60010

socket = /var/run/mysqld/mysqld.sock

pid-file = /var/run/mysqld/mysqld.pid

log-error = /var/log/mysql/mysqld.err

basedir = /usr/local/mysql

datadir = /var/lib/mysql

tmpdir = /var/lib/mysql

innodb-buffer-pool-size = 2757M

innodb-additional-mem-pool-size = 20M

innodb-log-buffer-size = 16M

innodb-flush-log-at-trx_commit = 0

innodb-file-per-table = 1

innodb_data_file_path = ibdata1:100M:autoextend

innodb_read_io_threads = 4

innodb_write_io_threads = 4

innodb-doublewrite = 1

innodb_log_file_size = 512M

innodb-log-files-in-group = 2

#innodb-buffer-pool-instances = 4

innodb-thread-concurrency = 0

#innodb-file-format = barracuda

innodb-flush-method = O_DIRECT

innodb_locks_unsafe_for_binlog = 1

innodb_autoinc_lock_mode = 2

innodb_stats_on_metadata = 0

engine-condition-pushdown = 1

collation-server = utf8_unicode_ci

init-connect = ‘SET NAMES utf8’

character-set-server = utf8

binlog_format = ROW

key_buffer_size = 24M

tmp_table_size = 64M

max_heap_table_size = 64M

max-allowed-packet = 512M

#sort-buffer-size = 512K

#read-buffer-size = 256K

#read-rnd-buffer-size = 512K

myisam-sort-buffer_size = 8M

memlock = 0

sysdate-is-now = 1

max-connections = 400

thread-cache-size = 512

query-cache-type = 0

query-cache-size = 0

table-open_cache = 1024

lower-case-table-names = 0

wsrep_provider = /usr/local/lib/libgalera_smm.so

wsrep_node_address = “donor.example.com”

wsrep_provider_options = "gcache.size=1024M; evs.keepalive_period=PT3S; evs.inactive_check_period=PT10S; evs.suspect_timeout=PT30S; evs.inactive_timeout=PT1M; evs.consensu$

wsrep_cluster_name = “my_sql_cluster”

wsrep_cluster_address = “gcomm://”

wsrep_node_name = “donor”

#wsrep_node_incoming_address =

wsrep_slave_threads = 1

#wsrep_dbug_option

wsrep_certify_nonPK = 1

#wsrep_data_home_dir =

wsrep_max_ws_rows = 131072

wsrep_max_ws_size = 1073741824

wsrep_debug = 0

wsrep_convert_LOCK_to_trx = 0

wsrep_retry_autocommit = 1

wsrep_auto_increment_control = 1

wsrep_replicate_myisam = 1

wsrep_drupal_282555_workaround = 0

wsrep_causal_reads = 0

#wsrep_notify_cmd =

wsrep_sst_method = xtrabackup

#wsrep_sst_receive_address =

wsrep_sst_auth = backup:password

#wsrep_sst_donor =

[mysql]

socket = /var/run/mysqld/mysqld.sock

default-character-set = utf8

[client]

socket = /var/run/mysqld/mysqld.sock

default-character-set = utf8

[mysqldump]

max-allowed-packet = 512M

default-character-set = utf8

[mysqld_safe]

pid-file = /var/run/mysqld/mysqld.pid

log-error = /var/log/mysql/mysqld.err

basedir = /usr/local/mysql

datadir = /var/lib/mysql

[xtrabackup]

tmpdir=/var/lib/mysql

3

Problem solved. Actually it wasn’t a problem on the donor, but on the joiner. The netcat version installed there doesn’t support the -d switch. I had to install the OpenBSD Netcat package for the script to work. Now everything syncs properly.