We have a 3 node cluster. One of the node which happened to be the bootstrapped node crash and we needed to reinstall. Is there a way to add it back to the cluster without bringing down the cluster? Thank you!
you can simply start mysql service ) service mysql start ( It will join cluster). If you get issues like PID file exists, you move PID file anf then restart mysql
Thanks Sanjay92! I did that but mysql failed to start with the below error in the logs
2015-11-23 22:58:46 6234 [Warning] WSREP: 0.0 (ip-10-196-43-250.ec2.internal): State transfer to 2.0 (ebppvmifsdb01a) failed: -255 (Unknown error 255)
2015-11-23 22:58:46 6234 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():731: Will never receive state. Need to abort.
I stopped Iptables on all the nodes to rule out communication issue. I did verified that the sst user exist on the donor and wsrep_sst_auth is set correctly on the joiner.
[COLOR=#4070A0]What is your setting for [COLOR=#4070A0]wsrep_sst_method ?
[COLOR=#4070A0]wsrep_sst_method = [COLOR=#4070A0]xtrabackup-v2 I have read some where it mention that there are some files under .sst folder so delete all files and then restart it, however I don’t know which location .sst folder exists.
We have
wsrep_sst_method=rsync
Cheers,
may be trying changing to [COLOR=#4070A0]
Tried changing to xtrabackup-v2 but no luck.
I think I was able to fix it though. The new rebuilt host has a different hostname than the previous. I changed the hostname to reflect the same as the previous and it joined the cluster. So the IP change didn’t matter as long as the hostname is the same. Is that an expected behavior?
Thanks Sanjay for the assist!