2014-10-03 02:45:28 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT8.31637S), skipping check 2014-10-03 02:45:28 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:28 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:29 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.91.2:4567 tcp://10.10.91.3:4567 2014-10-03 02:45:29 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') reconnecting to 9ee1b0f1 (tcp://10.10.91.3:4567), attempt 0 2014-10-03 02:45:30 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') reconnecting to ae3cf18c (tcp://10.10.91.2:4567), attempt 0 2014-10-03 02:45:30 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:30 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:30 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:30 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:32 1043 [Note] WSREP: evs::proto(e3517d46, OPERATIONAL, view_id(REG,9ee1b0f1,3)) suspecting node: 9ee1b0f1 2014-10-03 02:45:32 1043 [Note] WSREP: evs::proto(e3517d46, OPERATIONAL, view_id(REG,9ee1b0f1,3)) suspected node without join message, declaring inactive 2014-10-03 02:45:33 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') turning message relay requesting off 2014-10-03 02:45:33 1043 [Note] WSREP: evs::proto(e3517d46, GATHER, view_id(REG,9ee1b0f1,3)) suspecting node: ae3cf18c 2014-10-03 02:45:33 1043 [Note] WSREP: evs::proto(e3517d46, GATHER, view_id(REG,9ee1b0f1,3)) suspected node without join message, declaring inactive 2014-10-03 02:45:34 1043 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2014-10-03 02:45:34 1043 [Note] WSREP: Flow-control interval: [16, 16] 2014-10-03 02:45:34 1043 [Note] WSREP: Received NON-PRIMARY. 2014-10-03 02:45:34 1043 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 8457497) 2014-10-03 02:45:34 1043 [Warning] WSREP: Last Applied Action message in non-primary configuration from member 0 2014-10-03 02:45:34 1043 [Note] WSREP: New cluster view: global state: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8457497, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 2014-10-03 02:45:37 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:37 1043 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2014-10-03 02:45:37 1043 [Note] WSREP: Flow-control interval: [16, 16] 2014-10-03 02:45:37 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.91.2:4567 tcp://10.10.91.3:4567 2014-10-03 02:45:37 1043 [Note] WSREP: Received NON-PRIMARY. 2014-10-03 02:45:38 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT4.03575S), skipping check 2014-10-03 02:45:38 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') reconnecting to ae3cf18c (tcp://10.10.91.2:4567), attempt 0 2014-10-03 02:45:38 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') reconnecting to 9ee1b0f1 (tcp://10.10.91.3:4567), attempt 0 2014-10-03 02:45:38 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:38 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:38 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:38 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:45:39 1043 [Note] WSREP: declaring 9ee1b0f1 at tcp://10.10.91.3:4567 stable 2014-10-03 02:45:39 1043 [Note] WSREP: declaring ae3cf18c at tcp://10.10.91.2:4567 stable 2014-10-03 02:45:39 1043 [Note] WSREP: Node 9ee1b0f1 state prim 2014-10-03 02:45:39 1043 [Note] WSREP: save pc into disk 2014-10-03 02:45:42 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') turning message relay requesting off 2014-10-03 02:45:42 1043 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3 2014-10-03 02:45:42 1043 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2014-10-03 02:45:42 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.551S), skipping check 2014-10-03 02:45:42 1043 [Note] WSREP: STATE EXCHANGE: sent state msg: 37174a3c-4a8e-11e4-9600-3a98085430f2 2014-10-03 02:45:42 1043 [Note] WSREP: STATE EXCHANGE: got state msg: 37174a3c-4a8e-11e4-9600-3a98085430f2 from 0 (neon) 2014-10-03 02:45:42 1043 [Note] WSREP: STATE EXCHANGE: got state msg: 37174a3c-4a8e-11e4-9600-3a98085430f2 from 1 (blackbird) 2014-10-03 02:45:42 1043 [Note] WSREP: STATE EXCHANGE: got state msg: 37174a3c-4a8e-11e4-9600-3a98085430f2 from 2 (natrium) 2014-10-03 02:45:42 1043 [Note] WSREP: Quorum results: version = 3, component = PRIMARY, conf_id = 4, members = 2/3 (joined/total), act_id = 8458323, last_appl. = 8457479, protocols = 0/6/3 (gcs/repl/appl), group UUID = ae3d96be-4a1d-11e4-8abd-f7453e074c6e 2014-10-03 02:45:42 1043 [Note] WSREP: Flow-control interval: [28, 28] 2014-10-03 02:45:42 1043 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 8458323) 2014-10-03 02:45:44 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.26452S), skipping check 2014-10-03 02:45:47 1043 [Note] WSREP: New cluster view: global state: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8457497, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 2014-10-03 02:45:47 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.39619S), skipping check 2014-10-03 02:45:47 1043 [Note] WSREP: State transfer required: Group state: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8458323 Local state: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8457497 2014-10-03 02:45:47 1043 [Note] WSREP: New cluster view: global state: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8458323, view# 5: Primary, number of nodes: 3, my index: 2, protocol version 3 2014-10-03 02:45:47 1043 [Warning] WSREP: Gap in state sequence. Need state transfer. 2014-10-03 02:45:47 1043 [Note] WSREP: You have configured 'xtrabackup-v2' state snapshot transfer method which cannot be performed on a running server. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server. 2014-10-03 02:45:47 1043 [Note] WSREP: REPL Protocols: 6 (3, 2) 2014-10-03 02:45:47 1043 [Note] WSREP: Service thread queue flushed. 2014-10-03 02:45:47 1043 [Note] WSREP: Assign initial position for certification: 8458323, protocol version: 3 2014-10-03 02:45:47 1043 [Note] WSREP: Service thread queue flushed. 2014-10-03 02:45:47 1043 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.10.91.4:4568 2014-10-03 02:45:47 1043 [Note] WSREP: Member 2.0 (natrium) requested state transfer from '*any*'. Selected 0.0 (neon)(SYNCED) as donor. 2014-10-03 02:45:47 1043 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 8458330) 2014-10-03 02:45:47 1043 [Note] WSREP: Requesting state transfer: success, donor: 0 2014-10-03 02:45:47 1043 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): discarded 7871526516 bytes 2014-10-03 02:45:47 1043 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): found 1/8 locked buffers 2014-10-03 02:45:48 1043 [Note] WSREP: Receiving IST: 826 writesets, seqnos 8457497-8458323 2014-10-03 02:45:49 1043 [Note] WSREP: IST received: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8458323 2014-10-03 02:45:49 1043 [Note] WSREP: 2.0 (natrium): State transfer from 0.0 (neon) complete. 2014-10-03 02:45:49 1043 [Note] WSREP: Shifting JOINER -> JOINED (TO: 8458418) 2014-10-03 02:45:49 1043 [Note] WSREP: 0.0 (neon): State transfer to 2.0 (natrium) complete. 2014-10-03 02:45:49 1043 [Note] WSREP: Member 0.0 (neon) synced with group. 2014-10-03 02:45:49 1043 [Note] WSREP: Member 2.0 (natrium) synced with group. 2014-10-03 02:45:49 1043 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 8458418) 2014-10-03 02:45:49 1043 [Note] WSREP: Synchronized with group, ready for connections 2014-10-03 02:45:59 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.86856S), skipping check 2014-10-03 02:46:03 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.51348S), skipping check 2014-10-03 02:46:08 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50307S), skipping check 2014-10-03 02:46:08 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:46:09 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.91.2:4567 tcp://10.10.91.3:4567 2014-10-03 02:46:09 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:46:09 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:46:09 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:46:09 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:46:12 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') turning message relay requesting off 2014-10-03 02:46:19 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.90905S), skipping check 2014-10-03 02:46:19 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:46:19 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.91.2:4567 tcp://10.10.91.3:4567 2014-10-03 02:46:24 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') reconnecting to ae3cf18c (tcp://10.10.91.2:4567), attempt 0 2014-10-03 02:46:24 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') reconnecting to 9ee1b0f1 (tcp://10.10.91.3:4567), attempt 0 2014-10-03 02:46:24 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT4.73499S), skipping check 2014-10-03 02:46:24 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:46:24 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:46:24 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:46:24 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:46:24 1043 [Note] WSREP: evs::proto(e3517d46, OPERATIONAL, view_id(REG,9ee1b0f1,5)) suspecting node: 9ee1b0f1 2014-10-03 02:46:24 1043 [Note] WSREP: evs::proto(e3517d46, OPERATIONAL, view_id(REG,9ee1b0f1,5)) suspected node without join message, declaring inactive 2014-10-03 02:46:24 1043 [Note] WSREP: evs::proto(e3517d46, OPERATIONAL, view_id(REG,9ee1b0f1,5)) suspecting node: ae3cf18c 2014-10-03 02:46:24 1043 [Note] WSREP: evs::proto(e3517d46, OPERATIONAL, view_id(REG,9ee1b0f1,5)) suspected node without join message, declaring inactive 2014-10-03 02:46:25 1043 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2014-10-03 02:46:25 1043 [Note] WSREP: Flow-control interval: [16, 16] 2014-10-03 02:46:25 1043 [Note] WSREP: Received NON-PRIMARY. 2014-10-03 02:46:25 1043 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 8458704) 2014-10-03 02:46:25 1043 [Warning] WSREP: Last Applied Action message in non-primary configuration from member 0 2014-10-03 02:46:25 1043 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2014-10-03 02:46:25 1043 [Note] WSREP: Flow-control interval: [16, 16] 2014-10-03 02:46:25 1043 [Note] WSREP: New cluster view: global state: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8458704, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 2014-10-03 02:46:25 1043 [Note] WSREP: Received NON-PRIMARY. 2014-10-03 02:46:25 1043 [Note] WSREP: declaring 9ee1b0f1 at tcp://10.10.91.3:4567 stable 2014-10-03 02:46:25 1043 [Note] WSREP: declaring ae3cf18c at tcp://10.10.91.2:4567 stable 2014-10-03 02:46:25 1043 [Note] WSREP: Node 9ee1b0f1 state prim 2014-10-03 02:46:25 1043 [Note] WSREP: save pc into disk 2014-10-03 02:46:25 1043 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3 2014-10-03 02:46:25 1043 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2014-10-03 02:46:25 1043 [Note] WSREP: STATE EXCHANGE: sent state msg: 5293efb3-4a8e-11e4-89b4-266792049c13 2014-10-03 02:46:25 1043 [Note] WSREP: STATE EXCHANGE: got state msg: 5293efb3-4a8e-11e4-89b4-266792049c13 from 0 (neon) 2014-10-03 02:46:25 1043 [Note] WSREP: STATE EXCHANGE: got state msg: 5293efb3-4a8e-11e4-89b4-266792049c13 from 1 (blackbird) 2014-10-03 02:46:25 1043 [Note] WSREP: STATE EXCHANGE: got state msg: 5293efb3-4a8e-11e4-89b4-266792049c13 from 2 (natrium) 2014-10-03 02:46:25 1043 [Note] WSREP: Quorum results: version = 3, component = PRIMARY, conf_id = 6, members = 2/3 (joined/total), act_id = 8458821, last_appl. = 8458702, protocols = 0/6/3 (gcs/repl/appl), group UUID = ae3d96be-4a1d-11e4-8abd-f7453e074c6e 2014-10-03 02:46:25 1043 [Note] WSREP: Flow-control interval: [28, 28] 2014-10-03 02:46:25 1043 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 8458821) 2014-10-03 02:46:25 1043 [Note] WSREP: New cluster view: global state: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8458704, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 2014-10-03 02:46:25 1043 [Note] WSREP: State transfer required: Group state: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8458821 Local state: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8458704 2014-10-03 02:46:25 1043 [Note] WSREP: New cluster view: global state: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8458821, view# 7: Primary, number of nodes: 3, my index: 2, protocol version 3 2014-10-03 02:46:25 1043 [Warning] WSREP: Gap in state sequence. Need state transfer. 2014-10-03 02:46:25 1043 [Note] WSREP: You have configured 'xtrabackup-v2' state snapshot transfer method which cannot be performed on a running server. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server. 2014-10-03 02:46:25 1043 [Note] WSREP: REPL Protocols: 6 (3, 2) 2014-10-03 02:46:25 1043 [Note] WSREP: Service thread queue flushed. 2014-10-03 02:46:25 1043 [Note] WSREP: Assign initial position for certification: 8458821, protocol version: 3 2014-10-03 02:46:25 1043 [Note] WSREP: Service thread queue flushed. 2014-10-03 02:46:25 1043 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.10.91.4:4568 2014-10-03 02:46:25 1043 [Note] WSREP: Member 2.0 (natrium) requested state transfer from '*any*'. Selected 0.0 (neon)(SYNCED) as donor. 2014-10-03 02:46:25 1043 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 8458823) 2014-10-03 02:46:25 1043 [Note] WSREP: Requesting state transfer: success, donor: 0 2014-10-03 02:46:25 1043 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): discarded 3840890 bytes 2014-10-03 02:46:25 1043 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): found 1/3 locked buffers 2014-10-03 02:46:25 1043 [Note] WSREP: Receiving IST: 117 writesets, seqnos 8458704-8458821 2014-10-03 02:46:26 1043 [Note] WSREP: 0.0 (neon): State transfer to 2.0 (natrium) complete. 2014-10-03 02:46:26 1043 [Note] WSREP: Member 0.0 (neon) synced with group. 2014-10-03 02:46:26 1043 [Note] WSREP: IST received: ae3d96be-4a1d-11e4-8abd-f7453e074c6e:8458821 2014-10-03 02:46:26 1043 [Note] WSREP: 2.0 (natrium): State transfer from 0.0 (neon) complete. 2014-10-03 02:46:26 1043 [Note] WSREP: Shifting JOINER -> JOINED (TO: 8458823) 2014-10-03 02:46:26 1043 [Note] WSREP: Member 2.0 (natrium) synced with group. 2014-10-03 02:46:26 1043 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 8458823) 2014-10-03 02:46:26 1043 [Note] WSREP: Synchronized with group, ready for connections 2014-10-03 02:46:27 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') turning message relay requesting off 2014-10-03 02:47:04 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.89547S), skipping check 2014-10-03 02:47:12 1043 [Warning] WSREP: last inactive check more than PT1.5S ago (PT7.66844S), skipping check 2014-10-03 02:47:12 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') address 'tcp://10.10.91.4:4567' pointing to uuid e3517d46 is blacklisted, skipping 2014-10-03 02:47:13 1043 [Note] WSREP: (e3517d46, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.10.91.2:4567 tcp://10.10.91.3:4567 ... and it is repeated.