Node Shutdown after start

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

Hello!

After fail start node i see log-file:


2014-06-02 18:59:06 7162 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
2014-06-02 18:59:06 7162 [Note] WSREP: wsrep_load(): Galera 3.5(r178) by Codership Oy <info&#64;codership.com> loaded successfully.
2014-06-02 18:59:06 7162 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
2014-06-02 18:59:06 7162 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2014-06-02 18:59:06 7162 [Note] WSREP: Passing config to GCS: base_host = 10.10.48.8; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://10.10.48.8; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_t
2014-06-02 18:59:06 7162 [Note] WSREP: Service thread queue flushed.
2014-06-02 18:59:06 7162 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2014-06-02 18:59:06 7162 [Note] WSREP: wsrep_sst_grab()
2014-06-02 18:59:06 7162 [Note] WSREP: Start replication
2014-06-02 18:59:06 7162 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2014-06-02 18:59:06 7162 [Note] WSREP: protonet asio version 0
2014-06-02 18:59:06 7162 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
2014-06-02 18:59:06 7162 [Note] WSREP: backend: asio
2014-06-02 18:59:06 7162 [Note] WSREP: GMCast version 0
2014-06-02 18:59:06 7162 [Note] WSREP: (d3a72715-ea6e-11e3-9862-2a026dbf4089, 'tcp://10.10.48.8:4567') listening at tcp://10.10.48.8:4567
2014-06-02 18:59:06 7162 [Note] WSREP: (d3a72715-ea6e-11e3-9862-2a026dbf4089, 'tcp://10.10.48.8:4567') multicast: , ttl: 1
2014-06-02 18:59:06 7162 [Note] WSREP: EVS version 0
2014-06-02 18:59:06 7162 [Note] WSREP: PC version 0
2014-06-02 18:59:06 7162 [Note] WSREP: gcomm: connecting to group 'rxmydb', peer '10.10.48.8:,10.10.48.10:'
2014-06-02 18:59:07 7162 [Note] WSREP: declaring ed87b5f4-ea6c-11e3-b5f2-6782c2dc0b6f stable
2014-06-02 18:59:07 7162 [Note] WSREP: Node ed87b5f4-ea6c-11e3-b5f2-6782c2dc0b6f state prim
2014-06-02 18:59:07 7162 [Note] WSREP: view(view_id(PRIM,d3a72715-ea6e-11e3-9862-2a026dbf4089,6) memb {
d3a72715-ea6e-11e3-9862-2a026dbf4089,0
ed87b5f4-ea6c-11e3-b5f2-6782c2dc0b6f,0
} joined {
} left {
} partitioned {
})
2014-06-02 18:59:07 7162 [Note] WSREP: gcomm: connected
2014-06-02 18:59:07 7162 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2014-06-02 18:59:07 7162 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2014-06-02 18:59:07 7162 [Note] WSREP: Opened channel 'rxmydb'
2014-06-02 18:59:07 7162 [Note] WSREP: Waiting for SST to complete.
2014-06-02 18:59:07 7162 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2014-06-02 18:59:07 7162 [Note] WSREP: STATE_EXCHANGE: sent state UUID: d3f3d7f1-ea6e-11e3-82ae-f38d73c4898d
2014-06-02 18:59:07 7162 [Note] WSREP: STATE EXCHANGE: sent state msg: d3f3d7f1-ea6e-11e3-82ae-f38d73c4898d
2014-06-02 18:59:07 7162 [Note] WSREP: STATE EXCHANGE: got state msg: d3f3d7f1-ea6e-11e3-82ae-f38d73c4898d from 0 (mysql)
2014-06-02 18:59:07 7162 [Note] WSREP: STATE EXCHANGE: got state msg: d3f3d7f1-ea6e-11e3-82ae-f38d73c4898d from 1 (db)
2014-06-02 18:59:07 7162 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 5,
members = 1/2 (joined/total),
act_id = 5587,
last_appl. = -1,
protocols = 0/5/2 (gcs/repl/appl),
group UUID = 07f3e316-ea5c-11e3-a67b-7f95f5026990
2014-06-02 18:59:07 7162 [Note] WSREP: Flow-control interval: [23, 23]
2014-06-02 18:59:07 7162 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 5587)
2014-06-02 18:59:07 7162 [Note] WSREP: State transfer required:
Group state: 07f3e316-ea5c-11e3-a67b-7f95f5026990:5587
Local state: 00000000-0000-0000-0000-000000000000:-1
2014-06-02 18:59:07 7162 [Note] WSREP: New cluster view: global state: 07f3e316-ea5c-11e3-a67b-7f95f5026990:5587, view# 6: Primary, number of nodes: 2, my index: 0, protocol version 2
2014-06-02 18:59:07 7162 [Warning] WSREP: Gap in state sequence. Need state transfer.
2014-06-02 18:59:09 7162 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address '10.10.48.8' --auth 'replicate:password' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '7162' '' '
WSREP_SST: [INFO] Streaming with xbstream (20140602 18:59:09.569)
WSREP_SST: [INFO] Using socat as streamer (20140602 18:59:09.572)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[&#64;]} ) (20140602 18:59:09.592)
2014-06-02 18:59:09 7162 [Note] WSREP: Prepared SST request: xtrabackup|10.10.48.8:4444/xtrabackup_sst
2014-06-02 18:59:09 7162 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-06-02 18:59:09 7162 [Note] WSREP: REPL Protocols: 5 (3, 1)
2014-06-02 18:59:09 7162 [Note] WSREP: Service thread queue flushed.
2014-06-02 18:59:09 7162 [Note] WSREP: Assign initial position for certification: 5587, protocol version: 3
2014-06-02 18:59:09 7162 [Note] WSREP: Service thread queue flushed.
2014-06-02 18:59:09 7162 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (07f3e316-ea5c-11e3-a67b-7f95f5026990): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():447. IST will be unavailable.
2014-06-02 18:59:09 7162 [Note] WSREP: Member 0.0 (mysql) requested state transfer from '*any*'. Selected 1.0 (db)(SYNCED) as donor.
2014-06-02 18:59:09 7162 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5587)
2014-06-02 18:59:09 7162 [Note] WSREP: Requesting state transfer: success, donor: 1
WSREP_SST: [ERROR] xtrabackup process ended without creating '/var/lib/mysql//xtrabackup_galera_info' (20140602 18:59:10.064)
2014-06-02 18:59:10 7162 [Warning] WSREP: 1.0 (db): State transfer to 0.0 (mysql) failed: -22 (Invalid argument)
2014-06-02 18:59:10 7162 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.
2014-06-02 18:59:10 7162 [Note] WSREP: gcomm: terminating thread
2014-06-02 18:59:10 7162 [Note] WSREP: gcomm: joining thread
2014-06-02 18:59:10 7162 [Note] WSREP: gcomm: closing backend
WSREP_SST: [INFO] Contents of datadir (20140602 18:59:10.067)
WSREP_SST: [INFO] -rw-rw---- 1 mysql mysql 8820 Jun 2 17:52 /var/lib/mysql//mysql/columns_priv.frm
-rw-rw---- 1 mysql mysql 0 Jun 2 17:52 /var/lib/mysql//mysql/columns_priv.MYD
-rw-rw---- 1 mysql mysql 4096 Jun 2 17:52 /var/lib/mysql//mysql/columns_priv.MYI
***
more tables.... (cut)
***
(20140602 18:59:10.161)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20140602 18:59:10.173)
WSREP_SST: [INFO] Removing the sst_in_progress file (20140602 18:59:10.176)
2014-06-02 18:59:10 7162 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address '10.10.48.8' --auth 'replicate:pass' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '7162' '' : 32 (Broken pipe)
2014-06-02 18:59:10 7162 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2014-06-02 18:59:10 7162 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2014-06-02 18:59:10 7162 [ERROR] Aborting

2014-06-02 18:59:11 7162 [Note] WSREP: view(view_id(NON_PRIM,d3a72715-ea6e-11e3-9862-2a026dbf4089,6) memb {
d3a72715-ea6e-11e3-9862-2a026dbf4089,0
} joined {
} left {
} partitioned {
ed87b5f4-ea6c-11e3-b5f2-6782c2dc0b6f,0
})
2014-06-02 18:59:11 7162 [Note] WSREP: view((empty))
2014-06-02 18:59:11 7162 [Note] WSREP: gcomm: closed
2014-06-02 18:59:11 7162 [Note] WSREP: /usr/sbin/mysqld: Terminated.
140602 18:59:11 mysqld_safe mysqld from pid file /tmp/mysql.pid ended

How start node ‘mysql’ without errors?

Node db running at CentOS 6 (selinux disabled, iptables disabled) and percona DB (Percona XtraDB Cluster (GPL), Release rel35.0, Revision 756, WSREP version 25.10, wsrep_25.10.r3985)
Node mysql running at Ubuntu 12.04 and (Percona XtraDB 5.6.15-rel63)

2

Server up only on settings wsrep_sst_method is rsync, but no xtrabackup. Any ideas?

3

Node (db) down after many instert data to node (mysql)


140603 10:45:20 [Note] WSREP: Signalling provider to continue.
140603 10:45:20 [Note] WSREP: SST received: 07f3e316-ea5c-11e3-a67b-7f95f5026990:5613
140603 10:45:20 [Note] WSREP: 1.0 (mysql): State transfer from 0.0 (mysql) complete.
140603 10:45:20 [Note] WSREP: Shifting JOINER -> JOINED (TO: 5613)
140603 10:45:20 [Note] WSREP: Member 1.0 (mysql) synced with group.
140603 10:45:20 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 5613)
140603 10:45:20 [Note] WSREP: Synchronized with group, ready for connections
140603 10:45:20 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140603 10:45:20 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.37-35.0-55-log' socket: '/tmp/mysql.sock' port: 3306 Percona XtraDB Cluster (GPL), Release rel35.0, Revision 756, WSREP version 25.10, wsrep_25.10.r3985





140603 12:35:40 [ERROR] Error in Log_event::read_log_event(): 'Sanity check failed', data_len: 50, event_type: 30
140603 12:35:40 [ERROR] WSREP: applier could not read binlog event, seqno: 5614, len: 0
140603 12:35:40 [Warning] WSREP: Failed to apply app buffer: seqno: 5614, status: 1
at galera/src/trx_handle.cpp:apply():340
Retrying 2th time
140603 12:35:40 [ERROR] Error in Log_event::read_log_event(): 'Sanity check failed', data_len: 50, event_type: 30
140603 12:35:40 [ERROR] WSREP: applier could not read binlog event, seqno: 5614, len: 0
140603 12:35:40 [Warning] WSREP: Failed to apply app buffer: seqno: 5614, status: 1
at galera/src/trx_handle.cpp:apply():340
Retrying 3th time
140603 12:35:40 [ERROR] Error in Log_event::read_log_event(): 'Sanity check failed', data_len: 50, event_type: 30
140603 12:35:40 [ERROR] WSREP: applier could not read binlog event, seqno: 5614, len: 0
140603 12:35:40 [Warning] WSREP: Failed to apply app buffer: seqno: 5614, status: 1
at galera/src/trx_handle.cpp:apply():340
Retrying 4th time
140603 12:35:40 [ERROR] Error in Log_event::read_log_event(): 'Sanity check failed', data_len: 50, event_type: 30
140603 12:35:40 [ERROR] WSREP: applier could not read binlog event, seqno: 5614, len: 0
140603 12:35:40 [ERROR] WSREP: Failed to apply trx: source: f4882da2-eaed-11e3-9489-b66dd25d467d version: 3 local: 0 state: APPLYING flags: 1 conn_id: 4434 trx_id: 10039 seqnos (l: 5, g: 5614, s: 5613, d: 5613, ts: 63733643875911)
140603 12:35:40 [ERROR] WSREP: Failed to apply trx 5614 4 times
140603 12:35:40 [ERROR] WSREP: Node consistency compromized, aborting...
140603 12:35:40 [Note] WSREP: Closing send monitor...
140603 12:35:40 [Note] WSREP: Closed send monitor.
140603 12:35:40 [Note] WSREP: gcomm: terminating thread
140603 12:35:40 [Note] WSREP: gcomm: joining thread
140603 12:35:40 [Note] WSREP: gcomm: closing backend
140603 12:35:41 [Note] WSREP: view(view_id(NON_PRIM,f4882da2-eaed-11e3-9489-b66dd25d467d,26) memb {
f6af48e1-eaf2-11e3-9723-db01c76922f6,0
} joined {
} left {
} partitioned {
f4882da2-eaed-11e3-9489-b66dd25d467d,0
})
140603 12:35:41 [Note] WSREP: view((empty))
140603 12:35:41 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140603 12:35:41 [Note] WSREP: gcomm: closed
140603 12:35:41 [Note] WSREP: Flow-control interval: [16, 16]
140603 12:35:41 [Note] WSREP: Received NON-PRIMARY.
140603 12:35:41 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 5614)
140603 12:35:41 [Note] WSREP: Received self-leave message.
140603 12:35:41 [Note] WSREP: Flow-control interval: [0, 0]
140603 12:35:41 [Note] WSREP: Received SELF-LEAVE. Closing connection.
140603 12:35:41 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 5614)
140603 12:35:41 [Note] WSREP: RECV thread exiting 0: Success
140603 12:35:41 [Note] WSREP: recv_thread() joined.
140603 12:35:41 [Note] WSREP: Closing replication queue.
140603 12:35:41 [Note] WSREP: Closing slave action queue.
140603 12:35:41 [Note] WSREP: /usr/sbin/mysqld: Terminated.
140603 12:35:41 mysqld_safe Number of processes running now: 0
140603 12:35:41 mysqld_safe WSREP: not restarting wsrep node automatically
140603 12:35:41 mysqld_safe mysqld from pid file /tmp/mysql.pid ended
4

How run Cluster with xtrabackup?

5

Can you paste your my.cnf file? Did you also try xtrabackup-v2?
[url]Index of wsrep system variables

6

Option xtrabackup-v2 work good, but not successfully join node3 (node1 ex db and node2 ex mysql working):


140610 11:49:21 [Note] WSREP: gcomm: connected
140610 11:49:21 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
140610 11:49:21 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
140610 11:49:21 [Note] WSREP: Opened channel 'mydb'
140610 11:49:21 [Note] WSREP: Waiting for SST to complete.
140610 11:49:21 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3
140610 11:49:21 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 1da6f842-f07c-11e3-b4e6-170c73cd6cd1
140610 11:49:21 [Note] WSREP: STATE EXCHANGE: sent state msg: 1da6f842-f07c-11e3-b4e6-170c73cd6cd1
140610 11:49:21 [Note] WSREP: STATE EXCHANGE: got state msg: 1da6f842-f07c-11e3-b4e6-170c73cd6cd1 from 0 (mydb03)
140610 11:49:21 [Note] WSREP: STATE EXCHANGE: got state msg: 1da6f842-f07c-11e3-b4e6-170c73cd6cd1 from 1 (db)
140610 11:49:21 [Note] WSREP: STATE EXCHANGE: got state msg: 1da6f842-f07c-11e3-b4e6-170c73cd6cd1 from 2 (mysql)
140610 11:49:21 [Note] WSREP: Quorum results:
version = 3,
component = PRIMARY,
conf_id = 57,
members = 2/3 (joined/total),
act_id = 850,
last_appl. = -1,
protocols = 0/5/2 (gcs/repl/appl),
group UUID = 4c570ccd-ebfe-11e3-916a-3223af6b893e
140610 11:49:21 [Note] WSREP: Flow-control interval: [28, 28]
140610 11:49:21 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 850)
140610 11:49:21 [Note] WSREP: State transfer required:
Group state: 4c570ccd-ebfe-11e3-916a-3223af6b893e:850
Local state: 00000000-0000-0000-0000-000000000000:-1
140610 11:49:21 [Note] WSREP: New cluster view: global state: 4c570ccd-ebfe-11e3-916a-3223af6b893e:850, view# 58: Primary, number of nodes: 3, my index: 0, protocol version 2
140610 11:49:21 [Warning] WSREP: Gap in state sequence. Need state transfer.
140610 11:49:23 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.10.48.21' --auth 'replicator:Shoh4ool' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '26671''
WSREP_SST: [INFO] Streaming with xbstream (20140610 11:49:23.589)
WSREP_SST: [INFO] Using socat as streamer (20140610 11:49:23.590)
WSREP_SST: [INFO] Evaluating timeout 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | pv -f -i 10 -N joiner 2>>/tmp/sstl.progress | xbstream -x; RC=( ${PIPESTATUS[&#64;]} ) (20140610 11:49:23.728)
140610 11:49:23 [Note] WSREP: Prepared SST request: xtrabackup-v2|10.10.48.21:4444/xtrabackup_sst
140610 11:49:23 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140610 11:49:23 [Note] WSREP: REPL Protocols: 5 (3, 1)
140610 11:49:23 [Note] WSREP: Service thread queue flushed.
140610 11:49:23 [Note] WSREP: Assign initial position for certification: 850, protocol version: 3
140610 11:49:23 [Note] WSREP: Service thread queue flushed.
140610 11:49:23 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (4c570ccd-ebfe-11e3-916a-3223af6b893e): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():447. IST will be unavailable.
140610 11:49:23 [Note] WSREP: Member 0.0 (mydb03) requested state transfer from '*any*'. Selected 1.0 (db)(SYNCED) as donor.
140610 11:49:23 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 850)
140610 11:49:23 [Note] WSREP: Requesting state transfer: success, donor: 1
2014/06/10 11:49:24 socat[26946] E write(1, 0xd963e0, 133): Broken pipe
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 127 0 (20140610 11:49:24.305)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20140610 11:49:24.306)
WSREP_SST: [INFO] Removing the sst_in_progress file (20140610 11:49:24.307)
140610 11:49:24 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.10.48.21' --auth 'xxx:yyyyy' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '26671': 32 (Broken pipe)
140610 11:49:24 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
140610 11:49:24 [ERROR] WSREP: SST failed: 32 (Broken pipe)
140610 11:49:24 [ERROR] Aborting

140610 11:49:26 [Note] WSREP: Closing send monitor...
140610 11:49:26 [Note] WSREP: Closed send monitor.
140610 11:49:26 [Note] WSREP: gcomm: terminating thread
140610 11:49:26 [Note] WSREP: gcomm: joining thread
140610 11:49:26 [Note] WSREP: gcomm: closing backend
140610 11:49:26 [Note] WSREP: view(view_id(NON_PRIM,1d5a4878-f07c-11e3-a03d-caab901a1248,58) memb {
1d5a4878-f07c-11e3-a03d-caab901a1248,0
} joined {
} left {
} partitioned {
4aff8818-f079-11e3-b2be-2b3e4a2741ed,0
f8408c3e-f078-11e3-ab86-da1737463817,0
})
140610 11:49:26 [Note] WSREP: view((empty))
140610 11:49:26 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140610 11:49:26 [Note] WSREP: gcomm: closed
140610 11:49:26 [Note] WSREP: Flow-control interval: [16, 16]
140610 11:49:26 [Note] WSREP: Received NON-PRIMARY.
140610 11:49:26 [Note] WSREP: Shifting JOINER -> OPEN (TO: 850)
140610 11:49:26 [Note] WSREP: Received self-leave message.
140610 11:49:26 [Note] WSREP: Flow-control interval: [0, 0]
140610 11:49:26 [Note] WSREP: Received SELF-LEAVE. Closing connection.
140610 11:49:26 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 850)
140610 11:49:26 [Note] WSREP: RECV thread exiting 0: Success
140610 11:49:26 [Note] WSREP: recv_thread() joined.
140610 11:49:26 [Note] WSREP: Closing replication queue.
140610 11:49:26 [Note] WSREP: Closing slave action queue.
140610 11:49:26 [Note] WSREP: Service disconnected.
140610 11:49:26 [Note] WSREP: rollbacker thread exiting
140610 11:49:27 [Note] WSREP: Some threads may fail to exit.
140610 11:49:27 [Note] /usr/sbin/mysqld: Shutdown complete

My.cnf:

[sst]
streamfmt=xbstream
#incremental=1
#rebuild=1
progress=/tmp/sstl.progress
#nsferfmt=socat

[mysql]
port = 3306
# socket = /tmp/mysql.sock

[mysqld]
bind-address = 0.0.0.0
user = mysql
default-storage-engine = InnoDB
# socket = /tmp/mysql.sock
# pid-file = /tmp/mysql.pid
character-set-server=utf8
collation-server=utf8_general_ci
init-connect = 'SET NAMES utf8'

wsrep_cluster_address=gcomm://ALL_NODES_IPS

key-buffer-size = 128Mb
myisam-recover = FORCE,BACKUP

# SAFETY #
max-allowed-packet = 16M
max-connect-errors = 1000000
skip-name-resolve
sql-mode = STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY
sysdate-is-now = 1
innodb = FORCE
innodb-strict-mode = 1
#innodb_force_recovery = 1
log-slave-updates
# DATA STORAGE #
datadir = /var/lib/mysql/

# BINARY LOGGING #
log-bin = /var/lib/mysql/mysql-bin
expire-logs-days = 60
sync-binlog = 1
server-id = 2
max_binlog_size = 1G
binlog_format = row
wsrep_provider = /usr/lib64/libgalera_smm.so
wsrep_slave_threads = 8
wsrep_cluster_name = mydb
wsrep_provider_options = "gmcast.listen_addr=tcp://mydb03;"

wsrep_sst_method = xtrabackup-v2

wsrep_sst_auth = xxxx:yyyy
wsrep_node_name = mydb03

# CACHES AND LIMITS #
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0
max-connections = 5000000
thread-cache-size = 1000
open-files-limit = 65535
table-definition-cache = 4096
table-open-cache = 128

# INNODB #
innodb-flush-method = O_DIRECT
innodb-log-files-in-group = 2
innodb-log-file-size = 256M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 5000M
innodb_locks_unsafe_for_binlog = 1
innodb_autoinc_lock_mode = 2
innodb_locks_unsafe_for_binlog = 1
# LOGGING #
log-error = /var/log/mysql-error.log
log-queries-not-using-indexes = 0
slow-query-log = 0
slow-query-log-file = /var/log/mysql-slow.log
general_log_file = /var/log/mysql.log
general_log = 0


[mysqld_safe]
wsrep_cluster_address=gcomm://ALL_NODES_IPS
7

any ideas?

8

Please help to fix this problem with perconaDB

9

PLEASE HELP!

10

140610 11:49:24 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
140610 11:49:24 [ERROR] WSREP: SST failed: 32 (Broken pipe)

Seems that SST failed on donor (db). When a SST is executed on a server the log of that xtrabackup process goes to files with the name innobackup.backup.log and/innobackup.prepare.log. They are on the data dir. If you share them (with error log too) we can check it to see what could be the problem. Maybe the answer is there :slight_smile:

11

i found file /tmp/sstl.progress with contents:


/usr//bin/wsrep_sst_xtrabackup-v2: line 106: pv: command not found

PS: Prompts to me that percona DB is not yet stable enough and ready to run on the stable projects. Because more time to start cluster and so more errors + no support.

12

Percona XtraDB Cluster is stable, being used in really large projects and you have all the support you need in this link:

[url]http://www.percona.com/products/mysql-support[/url]

This is a community forum :wink:

I still would need innobackup.backup.log and/innobackup.prepare.log files, but the error you pasted indicates that you need to install pv tool (you can use yum or apt). pv is used to throttle the backup, as you can read in our documentation:

[url]Percona XtraBackup

So, install the tool on all the nodes and try again please.

13

No user replicator Shoh4ool created by 10.10.48.21
#mysql
create user ‘replicator’@‘10.10.48.21’ identified by ‘Shoh4ool’;
grant all privileges on . to ‘replicator’@‘10.10.48.21’ with grant option;
grant reload, process on . ‘replicator’@‘10.10.48.21’;

and by all Nodes, for all Nodes IPs

[url]http://dev.mysql.com/doc/mysql-security-excerpt/5.5/en/adding-users.html[/url]