I’m setting up a 3 node cluster . The first node starts up fine with bootstrap , the second node fails to join the cluster
[root@mysql-test02 mysql]# /etc/init.d/mysql start
MySQL (Percona XtraDB Cluster) is not running, but lock fil[FAILED]lock/subsys/mysql) exists
Starting MySQL (Percona XtraDB Cluster)…State transfer in progress, setting sleep higher
…The server quit without updating PID file (/var/lib/mysql[FAILED]est02.bo3.e-dialog.com.pid).
MySQL (Percona XtraDB Cluster) server startup failed! [FAILED]
[root@mysql-test02 mysql]#
Here is the relevant information from the node 2 error log -
2014-12-03 16:02:09 51655 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role ‘joiner’ --address ‘10.100.7.28’ --auth ‘sstuser:s3cret’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘51655’ ‘’ ’
WSREP_SST: [INFO] Streaming with tar (20141203 16:02:10.030)
WSREP_SST: [INFO] Using socat as streamer (20141203 16:02:10.031)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | pv -f -i 10 -N joiner 2>>/tmp/sstl.progress | tar xfi - --recursive-unlink -h; RC=( ${PIPESTATUS[@]} ) (20141203 16:02:10.041)
2014-12-03 16:02:10 51655 [Note] WSREP: Prepared SST request: xtrabackup|10.100.7.28:4444/xtrabackup_sst
2014-12-03 16:02:10 51655 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-03 16:02:10 51655 [Note] WSREP: REPL Protocols: 6 (3, 2)
2014-12-03 16:02:10 51655 [Note] WSREP: Service thread queue flushed.
2014-12-03 16:02:10 51655 [Note] WSREP: Assign initial position for certification: 3, protocol version: 3
2014-12-03 16:02:10 51655 [Note] WSREP: Service thread queue flushed.
2014-12-03 16:02:10 51655 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (60c7c2b6-7b1d-11e4-9d8f-3723cbbcdb67): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
2014-12-03 16:02:10 51655 [Note] WSREP: Member 1.0 (mysql-test02.bo3.e-dialog.com) requested state transfer from ‘any’. Selected 0.0 (mysql-test01.bo3.e-dialog.com)(SYNCED) as donor.
2014-12-03 16:02:10 51655 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 3)
2014-12-03 16:02:10 51655 [Note] WSREP: Requesting state transfer: success, donor: 0
2014-12-03 16:02:12 51655 [Note] WSREP: (a4faecfe, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2014-12-03 16:02:13 51655 [Note] WSREP: 0.0 (mysql-test01.bo3.e-dialog.com): State transfer to 1.0 (mysql-test02.bo3.e-dialog.com) complete.
2014-12-03 16:02:13 51655 [Note] WSREP: Member 0.0 (mysql-test01.bo3.e-dialog.com) synced with group.
WSREP_SST: [ERROR] xtrabackup process ended without creating ‘/var/lib/mysql//xtrabackup_galera_info’ (20141203 16:02:13.684)
WSREP_SST: [INFO] Contents of datadir (20141203 16:02:13.685)
WSREP_SST: [INFO] -rw------- 1 mysql mysql 8820 Dec 3 13:09 /var/lib/mysql//mysql/columns_priv.frm
-rw------- 1 mysql mysql 0 Dec 3 13:09 /var/lib/mysql//mysql/columns_priv.MYD
-rw------- 1 mysql mysql 4096 Dec 3 13:09 /var/lib/mysql//mysql/columns_priv.MYI
-rw------- 1 mysql mysql 9582 Dec 3 13:09 /var/lib/mysql//mysql/db.frm
-rw------- 1 mysql mysql 880 Dec 3 13:09 /var/lib/mysql//mysql/db.MYD
-rw------- 1 mysql mysql 5120 Dec 3 13:09 /var/lib/mysql//mysql/db.MYI
-rw------- 1 mysql mysql 10223 Dec 3 13:09 /var/lib/mysql//mysql/event.frm
-rw------- 1 mysql mysql 0 Dec 3 13:09 /var/lib/mysql//mysql/event.MYD
.
.
.
-rw-rw---- 1 mysql mysql 13518 Dec 3 13:09 /var/lib/mysql//performance_schema/table_lock_waits_summary_by_table.frm
-rw-rw---- 1 mysql mysql 9201 Dec 3 13:09 /var/lib/mysql//performance_schema/threads.frm
-rw-rw---- 1 mysql mysql 8676 Dec 3 13:09 /var/lib/mysql//performance_schema/users.frm
-rw-rw---- 1 mysql mysql 0 Dec 3 16:02 /var/lib/mysql//test/db.opt (20141203 16:02:13.694)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20141203 16:02:13.698)
WSREP_SST: [INFO] Removing the sst_in_progress file (20141203 16:02:13.699)
2014-12-03 16:02:13 51655 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role ‘joiner’ --address ‘10.100.7.28’ --auth ‘sstuser:s3cret’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘51655’ ‘’ : 32 (Broken pipe)
2014-12-03 16:02:13 51655 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2014-12-03 16:02:13 51655 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2014-12-03 16:02:13 51655 [ERROR] Aborting
On node 1 the the innobackup.backup.log log looks clean -
[root@mysql-test01 mysql]# cat innobackup.backup.log
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p
141203 16:02:10 innobackupex: Connecting to MySQL server with DSN ‘dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_s ocket=/var/lib/mysql/mysql.sock’ as ‘sstuser’ (using password: YES).
141203 16:02:10 innobackupex: Connected to MySQL server
141203 16:02:10 innobackupex: Executing a version check against the server…
141203 16:02:10 innobackupex: Done.
141203 16:02:10 innobackupex: Starting the backup operation
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints “completed OK!”.
innobackupex: Using server version 5.6.21-70.1-56
innobackupex: Created backup directory /tmp
141203 16:02:10 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file=“/etc/my.cnf” --defaults-group=“mysqld” --backup --suspend-at-end --target-dir=/tmp --innodb_log_file_size=“50331648” --innodb_data_file_path=“ibdata1:12M:autoextend” --tmpdir=/tmp --extra-lsndir=‘/tmp’ --stream=tar
innobackupex: Waiting for ibbackup (pid=53974) to suspend
innobackupex: Suspend file ‘/tmp/xtrabackup_suspended_2’
xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: open files limit requested 0, set to 131072
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
log scanned up to (1626819)
xtrabackup: Generating a list of tablespaces
[01] Streaming ./ibdata1
[01] …done
[01] Streaming ./mysql/slave_worker_info.ibd
[01] …done
[01] Streaming ./mysql/slave_relay_log_info.ibd
[01] …done
[01] Streaming ./mysql/innodb_table_stats.ibd
[01] …done
[01] Streaming ./mysql/innodb_index_stats.ibd
[01] …done
[01] Streaming ./mysql/slave_master_info.ibd
[01] …done
log scanned up to (1626819)
xtrabackup: Creating suspend file ‘/tmp/xtrabackup_suspended_2’ with pid ‘53974’
141203 16:02:12 innobackupex: Continuing after ibbackup has suspended
141203 16:02:12 innobackupex: Executing LOCK TABLES FOR BACKUP…
141203 16:02:12 innobackupex: Backup tables lock acquired
141203 16:02:12 innobackupex: Starting to backup non-InnoDB tables and files
innobackupex: in subdirectories of ‘/var/lib/mysql/’
innobackupex: Backing up files ‘/var/lib/mysql//mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM ,CSV,opt,par}’ (74 files)
log scanned up to (1626819)
innobackupex: Backing up files ‘/var/lib/mysql//performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM ,CSV,opt,par}’ (53 files)
141203 16:02:12 innobackupex: Finished backing up non-InnoDB tables and files
141203 16:02:12 innobackupex: Executing LOCK BINLOG FOR BACKUP…
141203 16:02:12 innobackupex: Executing FLUSH ENGINE LOGS…
141203 16:02:12 innobackupex: Waiting for log copying to finish
xtrabackup: The latest check point (for incremental): ‘1626819’
xtrabackup: Stopping log copying thread.
.>> log scanned up to (1626819)
xtrabackup: Creating suspend file ‘/tmp/xtrabackup_log_copied’ with pid ‘53974’
xtrabackup: Transaction log of lsn (1626819) to (1626819) was copied.
141203 16:02:13 innobackupex: Executing UNLOCK BINLOG
141203 16:02:13 innobackupex: Executing UNLOCK TABLES
141203 16:02:13 innobackupex: All tables unlocked
innobackupex: Backup created in directory ‘/tmp’
141203 16:02:13 innobackupex: Connection to database server closed
innobackupex: You must use -i (–ignore-zeros) option for extraction of the tar stream.
141203 16:02:13 innobackupex: completed OK!
[root@mysql-test01 mysql]#
