Folks, how are you all? I need some help here, the story is as follows…
Some time ago I implemented a XtraDB Cluster using the packages on version 5.6.20-68.0-56-log. This cluster’s version has four nodes and it’s in production, running very well with no problems at all. I’ve got another one running version 5.5.39 with the same configuration files on all the four nodes and no problems to implement the cluster (considering out some variables presented below for 5.6).
On both scenarios, the setup sequence was the same: to get the Percona Repository installed on CentOS 6.5 machines, install the Percona-XtraDB-Cluster-56 and xtrabackup as well (just for a double check).
Configuration files were created with the following basic configs:
#: arquivo configuração, nó # 1
[mysqld]
user=mysql
server_id=1
datadir=/var/lib/mysql
log_bin=node01-bin
binlog_format=ROW
log_slave_updates=true
enforce_gtid_consistency=true
default_storage_engine=innodb
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
innodb_log_group_home_dir=/var/lib/mysql
innodb_log_files_in_group=2
innodb_log_file_size=2G
#: wsrep variables
wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_cluster_name=mycluster
wsrep_node_address=192.168.0.101
wsrep_node_name=node01
wsrep_cluster_address=gcomm://192.168.0.101:4567,192.168.0.102:4567,192.168.0.10 3:4567, 192.168.0.104:4567
wsrep_slave_threads=2
wsrep_sst_method=xtrabackup
wsrep_sst_auth=sst:123
#: arquivo configuração, nó # 2
[mysqld]
user=mysql
server_id=2
datadir=/var/lib/mysql
log_bin=node02-bin
binlog_format=ROW
log_slave_updates=true
enforce_gtid_consistency=true
default_storage_engine=innodb
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
innodb_log_group_home_dir=/var/lib/mysql
innodb_log_files_in_group=2
innodb_log_file_size=
#: wsrep variables
wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_cluster_name=mycluster
wsrep_node_address=192.168.0.102
wsrep_node_name=node02
wsrep_cluster_address=gcomm://192.168.0.101:4567,192.168.0.102:4567,192.168.0.10 3:4567, 192.168.0.104:4567
wsrep_slave_threads=2
wsrep_sst_method=xtrabackup
wsrep_sst_auth=sst:123
#: arquivo configuração, nó # 3
[mysqld]
user=mysql
server_id=3
datadir=/var/lib/mysql
log_bin=node03-bin
binlog_format=ROW
default_storage_engine=innodb
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
innodb_log_group_home_dir=/var/lib/mysql
innodb_log_files_in_group=2
innodb_log_file_size=
#: wsrep variables
wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_cluster_name=mycluster
wsrep_node_address=192.168.0.103
wsrep_node_name=node03
wsrep_cluster_address=gcomm://192.168.0.101:4567,192.168.0.102:4567,192.168.0.10 3:4567, 192.168.0.104:4567
wsrep_slave_threads=2
wsrep_sst_method=xtrabackup
wsrep_sst_auth=sst:123
#: arquivo de configuração, nó # 4
[mysqld]
user=mysql
datadir=/var/lib/mysql
server_id=4
log_bin=node04-bin
binlog_format=ROW
log_slave_updates
default_storage_engine=innodb
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
innodb_log_group_home_dir=/var/lib/mysql
innodb_log_files_in_group=2
innodb_log_file_size=
#: wsrep variables
wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_cluster_address=gcomm://192.168.0.101:4567,192.168.0.102:4567,192.168.0.10 3:4567, 192.168.0.104:4567
wsrep_cluster_name=mycluster
wsrep_node_name=node04
wsrep_node_address=192.168.0.104
wsrep_sst_method=xtrabackup
wsrep_sst_auth=sst:123
The cluster is running as I said…
$ mysql -u root -pxxxxxxx -e “show status like ‘wsrep_cluster_size’”
±-------------------±------+
| Variable_name | Value |
±-------------------±------+
| wsrep_cluster_size | 4 |
±-------------------±------+
So, with those configuration files, I started a new project, but now to test the version Ver 5.6.21-70.1-56. I simple copied all the configuration I have currently running very well as showed previously. The new version I’m testing is:
$ mysqld --version
mysqld Ver 5.6.21-70.1-56 for Linux on x86_64 (Percona XtraDB Cluster (GPL), Release rel70.1, Revision 938, WSREP version 25.8, wsrep_25.8.r4150)
The bootstrap was OK, but the second node is not joining the cluster even having the SST completing with an OK message at the end (gotten from the innobackup.backup.log).
$ sudo tail -f /var/lib/mysql/innobackup.backup.log
xtrabackup: Transaction log of lsn (1627264) to (1627264) was copied.
141212 16:48:55 innobackupex: Executing UNLOCK BINLOG
141212 16:48:55 innobackupex: Executing UNLOCK TABLES
141212 16:48:55 innobackupex: All tables unlocked
innobackupex: Backup created in directory ‘/tmp’
innobackupex: MySQL binlog position: filename ‘node01-bin.000002’, position 120
141212 16:48:55 innobackupex: Connection to database server closed
innobackupex: You must use -i (–ignore-zeros) option for extraction of the tar stream.
141212 16:48:55 innobackupex: completed OK!
The mysql error log on the second node says that the SST completed with errors:
WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (d4022a87-820a-11e4-bcf4-5793c1c5841f): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():456. IST will be unavailable.
2014-12-12 16:49:47 5698 [Note] WSREP: Member 1.0 (node02) requested state transfer from ‘any’. Selected 0.0 (node01)(SYNCED) as donor.
2014-12-12 16:49:47 5698 [Note] WSREP: Shifting PRIMARY → JOINER (TO: 5)
2014-12-12 16:49:47 5698 [Note] WSREP: Requesting state transfer: success, donor: 0
2014-12-12 16:49:47 5698 [Note] WSREP: (dff2f8f7, ‘tcp://0.0.0.0:4567’) turning message relay requesting off
2014-12-12 16:49:57 5698 [Note] WSREP: 0.0 (node01): State transfer to 1.0 (node02) complete.
WSREP_SST: [ERROR] xtrabackup process ended without creating ‘/var/lib/mysql//xtrabackup_galera_info’ (20141212 16:49:57.174)
[…]
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20141212 16:49:57.297)
WSREP_SST: [INFO] Removing the sst_in_progress file (20141212 16:49:57.326)
2014-12-12 16:49:57 5698 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role ‘joiner’ --address ‘192.168.0.102’ --auth ‘sst:123’ --datadir ‘/var/lib/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘5698’ ‘’ : 32 (Broken pipe)
2014-12-12 16:49:57 5698 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2014-12-12 16:49:57 5698 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2014-12-12 16:49:57 5698 [ERROR] Aborting
Just to get you guys aware, I’ve got the user sst with 123 as its password well created on MySQL:
$ mysql -u root -e “show grants for sst@localhost\G”
*************************** 1. row ***************************
Grants for sst@localhost: GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON . TO ‘sst’@‘localhost’ IDENTIFIED BY PASSWORD ‘*23AE809DDACAF96AF0FD78ED04B6A265E05AA257’
Trying to test manually the xtrabackup command as per the documentation to check if it’s doing the right thing, I’ve got the following:
$ innobackupex --user=sst --password=123 /tmp/
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
Get the latest version of Percona XtraBackup, documentation, and help resources:
[URL]Percona XtraBackup - MySQL Database Backup Software
141212 17:17:36 innobackupex: Connecting to MySQL server with DSN ‘dbi:mysql:;mysql_read_default_group=xtrabackup’ as ‘sst’ (using password: YES).
141212 17:17:36 innobackupex: Connected to MySQL server
141212 17:17:36 innobackupex: Executing a version check against the server…
Can’t use an undefined value as an ARRAY reference at /usr/bin/innobackupex line 1069.
The Xtrabackup packages I’ve got installed on these new machines:
$ xtrabackup --version
xtrabackup version 2.2.7 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
Any lights in here? Any advice?
