I had 3 node galera cluster running on RHEL6.5x64.
node1 - 10.22.5.8
node2 - 10.22.5.11
node3 - 10.22.75.185
Version on all nodes : mysqld Ver 5.5.23 for Linux on x86_64 (Percona XtraDB Cluster (GPL), wsrep_23.5.r333)
node1 /etc/my.cnf
== [MYSQLD]
user=mysql
basedir=/usr/
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
pid-file=mysqld.pid
port=3306
log-error=error.log
character_set_client=utf8
character_set_server=utf8
innodb-buffer-pool-size=8964M
innodb-flush-log-at-trx_commit=2
innodb-file-per-table=1
innodb-doublewrite=0
innodb_log_file_size=256M
innodb-log-files-in-group=3
innodb-thread-concurrency=0
innodb-flush-method = O_DIRECT
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
engine-condition-pushdown=1
binlog_format=ROW
key_buffer_size = 256M
max-allowed-packet = 128M
sort-buffer-size = 512K
read-buffer-size = 256K
read-rnd-buffer-size = 512K
myisam-sort-buffer_size = 8M
skip-name-resolve
memlock=0
sysdate-is-now=1
max-connections=512
thread-cache-size=512
query-cache-type = 0
query-cache-size = 0
table-open_cache=1024
lower-case-table-names=0
wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_node_address=10.22.5.8
wsrep_provider_options=“pc.ignore_sb = yes”
wsrep_cluster_name=“my_wsrep_cluster”
wsrep_cluster_address=“gcomm://10.22.75.185”
wsrep_node_name=10.22.5.8
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=1
wsrep_causal_reads=0
wsrep_sst_method=rsync
wsrep_sst_auth=root:passwd
wsrep_sst_donor=10.22.75.185
[MYSQL]
socket=/var/lib/mysql/mysql.sock
[client]
socket=/var/lib/mysql/mysql.sock
[MYSQLD_SAFE]
pid-file=mysqld.pid
log-error=error.log
basedir=/usr/
datadir=/var/lib/mysql
node2 /etc/my.cnf
== [MYSQLD]
user=mysql
basedir=/usr/
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
pid-file=mysqld.pid
port=3306
log-error=error.log
innodb-buffer-pool-size=8964M
innodb-flush-log-at-trx_commit=2
innodb-file-per-table=1
innodb-doublewrite=0
innodb_log_file_size=256M
innodb-log-files-in-group=3
innodb-thread-concurrency=0
innodb-flush-method = O_DIRECT
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
engine-condition-pushdown=1
binlog_format=ROW
key_buffer_size = 256M
max-allowed-packet = 128M
sort-buffer-size = 512K
read-buffer-size = 256K
read-rnd-buffer-size = 512K
myisam-sort-buffer_size = 8M
skip-name-resolve
memlock=0
sysdate-is-now=1
max-connections=512
thread-cache-size=512
query-cache-type = 0
query-cache-size = 0
table-open_cache=1024
lower-case-table-names=0
wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_node_address=10.22.5.11
wsrep_cluster_name=“my_wsrep_cluster”
wsrep_cluster_address=“gcomm://10.22.75.185”
wsrep_node_name=10.22.5.11
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=1
wsrep_causal_reads=0
wsrep_sst_method=rsync
wsrep_sst_receive_address=10.22.5.11
wsrep_sst_auth=root:passwd
wsrep_sst_donor=10.22.75.185
[MYSQL]
socket=/var/lib/mysql/mysql.sock
[client]
socket=/var/lib/mysql/mysql.sock
[MYSQLD_SAFE]
pid-file=mysqld.pid
log-error=error.log
basedir=/usr/
datadir=/var/lib/mysql
node3 /etc/my.cnf
== [MYSQLD]
user=mysql
basedir=/usr/
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
pid-file=mysqld.pid
port=3306
log-error=error.log
innodb-buffer-pool-size=8964M
innodb-flush-log-at-trx_commit=2
innodb-file-per-table=1
innodb-doublewrite=0
innodb_log_file_size=256M
innodb-log-files-in-group=3
innodb-thread-concurrency=0
innodb-flush-method = O_DIRECT
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
engine-condition-pushdown=1
binlog_format=ROW
key_buffer_size = 256M
max-allowed-packet = 16M
sort-buffer-size = 512K
read-buffer-size = 256K
read-rnd-buffer-size = 512K
myisam-sort-buffer_size = 8M
skip-name-resolve
memlock=0
sysdate-is-now=1
max-connections=512
thread-cache-size=512
query-cache-type = 0
query-cache-size = 0
table-open_cache=1024
lower-case-table-names=0
wsrep_provider=/usr/lib64/libgalera_smm.so
wsrep_node_address=10.22.75.185
wsrep_cluster_address=“gcomm://10.22.5.11”
wsrep_node_name=10.22.75.185
wsrep_slave_threads=1
wsrep_certify_nonPK=1
wsrep_debug=0
wsrep_convert_LOCK_to_trx=0
wsrep_retry_autocommit=1
wsrep_auto_increment_control=1
wsrep_drupal_282555_workaround=1
wsrep_causal_reads=0
wsrep_sst_method=rsync
wsrep_sst_auth=root:passwd
wsrep_sst_donor=10.22.5.11
[MYSQL]
socket=/var/lib/mysql/mysql.sock
[client]
socket=/var/lib/mysql/mysql.sock
[MYSQLD_SAFE]
pid-file=mysqld.pid
log-error=error.log
basedir=/usr/
datadir=/var/lib/mysql
I shutdown entire cluster because I had to decrease amount of ‘innodb-buffer-pool-size’ because mysqld was using to much SWAP space.
After changing this from most advanced node which was node1 I’ve boostrapped the cluster with below command
service mysql start --wsrep_cluster_address=“gcomm://”
Now cluster size is 1 and I can’t join other nodes (node2, node3) to the cluster, please observer below output from error.log of node2 during ‘service mysql start’ process.
/var/lib/mysql/error.log
== 150728 10:34:14 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150728 10:34:14 [Note] Flashcache bypass: disabled
150728 10:34:14 [Note] Flashcache setup error is : ioctl failed
150728 10:34:14 [Note] WSREP: Read nil XID from storage engines, skipping position init
150728 10:34:14 [Note] WSREP: wsrep_load(): loading provider library ‘/usr/lib64/libgalera_smm.so’
150728 10:34:14 [Note] WSREP: wsrep_load(): Galera 2.1dev(r109) by Codership Oy <info@codership.com> loaded succesfully.
150728 10:34:14 [Note] WSREP: Reusing existing ‘/var/lib/mysql//galera.cache’.
150728 10:34:14 [Note] WSREP: Passing config to GCS: base_host = 10.22.5.11; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
150728 10:34:14 [Note] WSREP: wsrep_sst_grab()
150728 10:34:14 [Note] WSREP: Start replication
150728 10:34:14 [Note] WSREP: Found saved state: 9ba720b3-8d53-11e1-0800-52407463bac7:304911180
150728 10:34:14 [Note] WSREP: Assign initial position for certification: 304911180, protocol version: -1
150728 10:34:14 [Note] WSREP: Setting initial position to 9ba720b3-8d53-11e1-0800-52407463bac7:304911180
150728 10:34:14 [Note] WSREP: protonet asio version 0
150728 10:34:14 [Note] WSREP: backend: asio
150728 10:34:14 [Note] WSREP: GMCast version 0
150728 10:34:14 [Note] WSREP: (6da55788-3503-11e5-0800-0de34d03910b, ‘tcp://0.0.0.0:4567’) listening at tcp://0.0.0.0:4567
150728 10:34:14 [Note] WSREP: (6da55788-3503-11e5-0800-0de34d03910b, ‘tcp://0.0.0.0:4567’) multicast: , ttl: 1
150728 10:34:14 [Note] WSREP: EVS version 0
150728 10:34:14 [Note] WSREP: PC version 0
150728 10:34:14 [Note] WSREP: gcomm: connecting to group ‘my_wsrep_cluster’, peer ‘10.22.75.185:’
150728 10:34:17 [Warning] WSREP: no nodes coming from prim view, prim not possible
150728 10:34:17 [Note] WSREP: view(view_id(NON_PRIM,6da55788-3503-11e5-0800-0de34d03910b,1) memb {
6da55788-3503-11e5-0800-0de34d03910b,
} joined {
} left {
} partitioned {
})
150728 10:34:17 [Note] WSREP: gcomm: connected
150728 10:34:17 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
150728 10:34:17 [Note] WSREP: Shifting CLOSED → OPEN (TO: 0)
150728 10:34:17 [Note] WSREP: Opened channel ‘my_wsrep_cluster’
150728 10:34:17 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
150728 10:34:17 [Note] WSREP: Waiting for SST to complete.
150728 10:34:17 [Note] WSREP: Flow-control interval: [8, 16]
150728 10:34:17 [Note] WSREP: Received NON-PRIMARY.
150728 10:34:17 [Note] WSREP: New cluster view: global state: 9ba720b3-8d53-11e1-0800-52407463bac7:304911180, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1
150728 10:34:17 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
150728 10:34:18 [Warning] WSREP: last inactive check more than PT1.5S ago, skipping check