Hi all,
I’m trying to add a new node to my db cluster, and encountering an error I can’t solve.
Context:
The cluster currently has 3 active servers (nodes: db1, db3, db4) , 1 down with a filesystem issue (db2)
I’ve updated db1 and db3 config (new ip in the wsrep_cluster_address), but not db4 since it’s under heavy load at the moment (it’s taking db2 load).
So I’m trying to add a new node db5 to replace db2 with the following config:
[mysqld]
log-error = /var/log/mysql.log
wsrep_provider = /usr/lib64/libgalera_smm.so
wsrep_cluster_address = gcomm://xxx.xxx.xxx.3,xxx.xxx.xxx.6,xxx.xxx.xxx.7,xxx.xxx. xxx.23,xxx.xxx.xxx.54
binlog_format = ROW
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
log_slave_updates = 1
innodb_locks_unsafe_for_binlog = 1
innodb_buffer_pool_size = 12G
datadir = /db-data/mysql
server_id = 5
sql_mode = NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
wsrep_sst_method = xtrabackup
wsrep_cluster_name = live_cluster
wsrep_node_name = db5
log_bin = mysql-bin
wsrep_sst_auth = “user:password”
sort_buffer_size = 4M
join_buffer_size = 128M
expire_logs_days = 7
max_connections = 200
//Test: tried to specify a donor
wsrep_sst_donor = db1
An now the error I get when I start the server:
[…]
2015-05-20 14:59:35 4466 [Note] WSREP: 1.0 (db1): State transfer to 3.0 (db5) complete.
2015-05-20 14:59:35 4466 [Note] WSREP: Member 1.0 (db1) synced with group.
WSREP_SST: [INFO] Preparing the backup at /db-data/mysql/ (20150520 14:59:57.972)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20150520 14:59:57.975)
WSREP_SST: [ERROR] Cleanup after exit with status:2 (20150520 14:59:58.231)
WSREP_SST: [INFO] Removing the sst_in_progress file (20150520 14:59:58.233)
2015-05-20 14:59:58 4466 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role ‘joiner’ --address ‘xxx.xxx.xxx.54’ --auth 'user: password --datadir ‘/db-data/mysql/’ --defaults-file ‘/etc/my.cnf’ --parent ‘4466’ ‘’ : 2 (No such file or directory)
2015-05-20 14:59:58 4466 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-05-20 14:59:58 4466 [ERROR] WSREP: SST failed: 2 (No such file or directory)
2015-05-20 14:59:58 4466 [ERROR] Aborting
2015-05-20 15:00:00 4466 [Note] WSREP: Closing send monitor…
2015-05-20 15:00:00 4466 [Note] WSREP: Closed send monitor.
2015-05-20 15:00:00 4466 [Note] WSREP: gcomm: terminating thread
2015-05-20 15:00:00 4466 [Note] WSREP: gcomm: joining thread
2015-05-20 15:00:00 4466 [Note] WSREP: gcomm: closing backend
2015-05-20 15:00:00 4466 [Note] WSREP: gcomm: closed
2015-05-20 15:00:00 4466 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2015-05-20 15:00:00 4466 [Note] WSREP: Flow-control interval: [16, 16]
2015-05-20 15:00:00 4466 [Note] WSREP: Received NON-PRIMARY.
2015-05-20 15:00:00 4466 [Note] WSREP: Shifting JOINER -> OPEN (TO: 39533446)
2015-05-20 15:00:00 4466 [Note] WSREP: Received self-leave message.
2015-05-20 15:00:00 4466 [Note] WSREP: Flow-control interval: [0, 0]
2015-05-20 15:00:00 4466 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2015-05-20 15:00:00 4466 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 39533446)
2015-05-20 15:00:00 4466 [Note] WSREP: RECV thread exiting 0: Success
2015-05-20 15:00:00 4466 [Note] WSREP: recv_thread() joined.
2015-05-20 15:00:00 4466 [Note] WSREP: Closing replication queue.
2015-05-20 15:00:00 4466 [Note] WSREP: Closing slave action queue.
2015-05-20 15:00:00 4466 [Note] WSREP: Service disconnected.
2015-05-20 15:00:00 4466 [Note] WSREP: rollbacker thread exiting
2015-05-20 15:00:01 4466 [Note] WSREP: Some threads may fail to exit.
2015-05-20 15:00:01 4466 [Note] Binlog end
2015-05-20 15:00:01 4466 [Note] /usr/sbin/mysqld: Shutdown complete
Error in my_thread_global_end(): 1 threads didn’t exit
150520 15:00:06 mysqld_safe mysqld from pid file /db-data/mysql/db5.pid ended
Could anyone please give me a hand.
Thank you.