Cluster failed, can you provide any insight?

VictorT · November 8, 2013, 1:54pm

We had an influx of connections/queries , ~2000 in a matter of 5 seconds, which resulted in one of the servers dropping out of the cluster 10.19.0.44.

We are currently running version: 5.5.31-23.7.5.438.rhel6.x86_64
I know there is a new version out but after reading over the change logs I only saw a couple changes in 5.5.34-23.7.6 that could be semi related.

Attempting to check the status of mysql resulted in the following error on all nodes still online resulted with the following message:

mysql> show global status;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

I have the wsrep_cluster_address set as follows:
wsrep_cluster_address=gcomm://10.19.0.40,10.19.0.41,10.19.0.42,10.19.0.43,10.19. 0.44,10.19.0.45

One possibly unrelated configuration question I have is, should I not include the servers own IP in the wsrep_cluster_address variable, I looked all over but could not find any documentation to allow me to come to a conclusion either way. During testing, I found that it would detect that it was itself, and then would skip that node, however is this bad form, should you include the the servers own IP in the cluster address or not?

At 15:51:45 the issue began, at 16:05:37 I reinitialized the cluster using db6. I stopped mysql and restarted with it using the following command:
/etc/init.d/mysql start --wsrep_cluster_address=gcomm://

At this point db6 was the primary, db4 was offline, db1,db2,db5 all joined db6

I had to stop db3 and restart db3 and db4 which both performed an SST from db6, and everything was back up and running normally

What I am looking to figure out is what cause the servers to no longer work and result in a Lock wait timeout exceeded when checking the status.

There are no log entries up to the point where the influx of connections occurred
The logs presented are from the server 10.19.0.42

VictorT · November 8, 2013, 1:56pm


131107 15:51:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:51:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:51:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.43:4567 tcp://10.19.0.44:4567 tcp://10.19.0.45:4567
131107 15:51:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:51:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:51:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567), attempt 0
131107 15:51:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 0
131107 15:51:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 644c6f70-4716-11e3-9900-c2dfa5a8512f (tcp://10.19.0.45:4567), attempt 0
131107 15:51:47 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.85633S), skipping check
131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:51:48 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:51:49 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:51:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:51:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:51:50 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:51:51 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:51:51 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:51:52 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:51:52 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:51:52 [Note] WSREP: evs:roto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,263)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:51:52 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 15:51:52 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 15:51:52 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
131107 15:51:53 [Note] WSREP: Node 27013c03-3740-11e3-aec6-4a0d195491f5 state prim
131107 15:51:53 [Note] WSREP: view(view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,264) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
644c6f70-4716-11e3-9900-c2dfa5a8512f,
a21cd90d-373e-11e3-b146-560f037992a6,
c089c14e-3738-11e3-877c-8aa6e988bded,
} joined {
} left {
} partitioned {
b5387adf-4564-11e3-9ffe-f38d17a91a12,
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
})
131107 15:51:53 [Note] WSREP: forgetting b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567)
131107 15:51:53 [Note] WSREP: deleting entry tcp://10.19.0.44:4567
131107 15:51:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:51:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:51:53 [Note] WSREP: forgetting b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567)
131107 15:51:53 [Note] WSREP: deleting entry tcp://10.19.0.43:4567
131107 15:51:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

VictorT · November 8, 2013, 1:56pm



131107 15:51:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:51:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
131107 15:51:53 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 4
131107 15:51:53 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
131107 15:51:53 [Note] WSREP: STATE EXCHANGE: sent state msg: 6ed09247-47ee-11e3-b5aa-6b2a9aeb8771
131107 15:51:53 [Note] WSREP: STATE EXCHANGE: got state msg: 6ed09247-47ee-11e3-b5aa-6b2a9aeb8771 from 0 db1
131107 15:51:53 [Note] WSREP: STATE EXCHANGE: got state msg: 6ed09247-47ee-11e3-b5aa-6b2a9aeb8771 from 3 db4
131107 15:51:53 [Note] WSREP: STATE EXCHANGE: got state msg: 6ed09247-47ee-11e3-b5aa-6b2a9aeb8771 from 1 db5
131107 15:51:53 [Note] WSREP: STATE EXCHANGE: got state msg: 6ed09247-47ee-11e3-b5aa-6b2a9aeb8771 from 2 db2
131107 15:51:53 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 232,
members = 4/4 (joined/total),
act_id = 8175717,
last_appl. = 8175401,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 05be2ccc-012d-11e3-8cb2-c69c9281c29b
131107 15:51:53 [Note] WSREP: Flow-control interval: [32, 32]
131107 15:51:53 [Warning] WSREP: gcs_caused() returned -1 (Operation not permitted)

...

131107 15:51:53 [Warning] WSREP: gcs_caused() returned -1 (Operation not permitted)
131107 15:51:53 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175717, view# 233: Primary, number of nodes: 4, my index: 2, protocol version 2
131107 15:51:53 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 15:51:53 [Note] WSREP: Assign initial position for certification: 8175717, protocol version: 2
131107 15:51:59 [Note] WSREP: cleaning up b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567)
131107 15:51:59 [Note] WSREP: cleaning up b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567)
131107 15:51:53 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175717, view# 233: Primary, number of nodes: 4, my index: 2, protocol version 2
131107 15:51:53 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 15:51:53 [Note] WSREP: Assign initial position for certification: 8175717, protocol version: 2
131107 15:51:59 [Note] WSREP: cleaning up b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567)
131107 15:51:59 [Note] WSREP: cleaning up b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567)
131107 15:52:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:52:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:52:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.45:4567
131107 15:52:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 644c6f70-4716-11e3-9900-c2dfa5a8512f (tcp://10.19.0.45:4567), attempt 0
131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
131107 15:52:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.44:4567
131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.43:4567
131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:52:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:52:27 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 15:52:27 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 15:52:27 [Note] WSREP: declaring b5387adf-4564-11e3-9ffe-f38d17a91a12 stable
131107 15:52:27 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
131107 15:52:28 [Note] WSREP: Node 27013c03-3740-11e3-aec6-4a0d195491f5 state prim
131107 15:52:29 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 15:52:29 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 15:52:29 [Note] WSREP: declaring b5387adf-4564-11e3-9ffe-f38d17a91a12 stable
131107 15:52:29 [Note] WSREP: declaring b7de24de-4713-11e3-8a4f-cb158b3ea3e2 stable
131107 15:52:29 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
131107 15:52:30 [Note] WSREP: Node 27013c03-3740-11e3-aec6-4a0d195491f5 state prim
131107 15:52:30 [Note] WSREP: view(view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
644c6f70-4716-11e3-9900-c2dfa5a8512f,
a21cd90d-373e-11e3-b146-560f037992a6,
b5387adf-4564-11e3-9ffe-f38d17a91a12,
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
c089c14e-3738-11e3-877c-8aa6e988bded,
} joined {
} left {
} partitioned {
})
131107 15:52:30 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 6
131107 15:52:30 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
131107 15:52:30 [Note] WSREP: STATE EXCHANGE: sent state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d
131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 0 (db1)
131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 1 (db5)
131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 2 (db2)
131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 5 (db6)
131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 3 (db4)
131107 15:52:30 [Note] WSREP: STATE EXCHANGE: got state msg: 84bea604-47ee-11e3-bae7-7b0ffba1ce0d from 4 (db3)
131107 15:52:30 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 233,
members = 4/6 (joined/total),
act_id = 8175755,
last_appl. = 8175401,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 05be2ccc-012d-11e3-8cb2-c69c9281c29b

VictorT · November 8, 2013, 1:57pm



131107 15:52:30 [Note] WSREP: Flow-control interval: [39, 39]
131107 15:52:30 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175755, view# 234: Primary, number of nodes: 6, my index: 2, protocol version 2
131107 15:52:30 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 15:52:30 [Note] WSREP: Assign initial position for certification: 8175755, protocol version: 2
131107 15:52:33 [Note] WSREP: Node 3 (db4) requested state transfer from 'db6'. Selected 5 (db6)(SYNCED) as donor.
131107 15:52:35 [Note] WSREP: 5 (db6): State transfer to 3 (db4) complete.
131107 15:52:35 [Note] WSREP: Member 5 (db6) synced with group.
131107 15:52:35 [Note] WSREP: 3 (db4): State transfer from 5 (db6) complete.
131107 15:52:35 [Note] WSREP: Member 3 (db4) synced with group.
131107 15:53:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.43:4567
131107 15:53:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567), attempt 0
131107 15:53:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 0
131107 15:53:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:04 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:53:04 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:53:05 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:53:05 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:53:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 644c6f70-4716-11e3-9900-c2dfa5a8512f (tcp://10.19.0.45:4567), attempt 0
131107 15:53:05 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:53:05 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:53:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:53:06 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:53:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:53:07 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:53:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b5387adf-4564-11e3-9ffe-f38d17a91a12
131107 15:53:08 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,266)) suspecting node: b7de24de-4713-11e3-8a4f-cb158b3ea3e2
131107 15:53:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:09 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 15:53:09 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 15:53:09 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
131107 15:53:10 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

VictorT · November 8, 2013, 1:58pm


131107 15:53:13 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:13 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 0
131107 15:53:13 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:13 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:14 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 644c6f70-4716-11e3-9900-c2dfa5a8512f (tcp://10.19.0.45:4567), attempt 0
131107 15:53:14 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:14 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:15 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,267)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:53:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:15 [Note] WSREP: evs::proto(a21cd90d-373e-11e3-b146-560f037992a6, GATHER, view_id(REG,27013c03-3740-11e3-aec6-4a0d195491f5,267)) suspecting node: 644c6f70-4716-11e3-9900-c2dfa5a8512f
131107 15:53:16 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:16 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:16 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,267) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
a21cd90d-373e-11e3-b146-560f037992a6,
c089c14e-3738-11e3-877c-8aa6e988bded,
} joined {
} left {
} partitioned {
644c6f70-4716-11e3-9900-c2dfa5a8512f,
b5387adf-4564-11e3-9ffe-f38d17a91a12,
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
})
131107 15:53:16 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 3
131107 15:53:16 [Warning] WSREP: Dropping input, message pcmsg{ type=STATE, seq=0, flags= 0, node_map { 27013c03-3740-11e3-aec6-4a0d195491f5,prim=1,un=0,last_seq=5,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
644c6f70-4716-11e3-9900-c2dfa5a8512f,prim=1,un=0,last_seq=8,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
a21cd90d-373e-11e3-b146-560f037992a6,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
b5387adf-4564-11e3-9ffe-f38d17a91a12,prim=1,un=0,last_seq=4,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,prim=1,un=0,last_seq=1,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
c089c14e-3738-11e3-877c-8aa6e988bded,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
}} in state TRANS
131107 15:53:16 [Warning] WSREP: Dropping input, message pcmsg{ type=STATE, seq=0, flags= 0, node_map { 27013c03-3740-11e3-aec6-4a0d195491f5,prim=1,un=0,last_seq=5,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
644c6f70-4716-11e3-9900-c2dfa5a8512f,prim=1,un=0,last_seq=8,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
a21cd90d-373e-11e3-b146-560f037992a6,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
b5387adf-4564-11e3-9ffe-f38d17a91a12,prim=1,un=0,last_seq=4,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,prim=1,un=0,last_seq=1,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
c089c14e-3738-11e3-877c-8aa6e988bded,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
}} in state TRANS
131107 15:53:16 [Warning] WSREP: Dropping input, message pcmsg{ type=STATE, seq=0, flags= 0, node_map { 27013c03-3740-11e3-aec6-4a0d195491f5,prim=1,un=0,last_seq=5,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
644c6f70-4716-11e3-9900-c2dfa5a8512f,prim=1,un=0,last_seq=8,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
a21cd90d-373e-11e3-b146-560f037992a6,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
b5387adf-4564-11e3-9ffe-f38d17a91a12,prim=1,un=0,last_seq=4,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,prim=1,un=0,last_seq=1,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
c089c14e-3738-11e3-877c-8aa6e988bded,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
}} in state TRANS
131107 15:53:16 [Warning] WSREP: Dropping input, message pcmsg{ type=STATE, seq=0, flags= 0, node_map { 27013c03-3740-11e3-aec6-4a0d195491f5,prim=1,un=0,last_seq=5,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
644c6f70-4716-11e3-9900-c2dfa5a8512f,prim=1,un=0,last_seq=8,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
a21cd90d-373e-11e3-b146-560f037992a6,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
b5387adf-4564-11e3-9ffe-f38d17a91a12,prim=1,un=0,last_seq=4,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,prim=1,un=0,last_seq=1,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541805,weight=1
c089c14e-3738-11e3-877c-8aa6e988bded,prim=1,un=0,last_seq=3,last_prim=view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,266),to_seq=8541829,weight=1
}} in state TRANS
131107 15:53:16 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 15:53:16 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
131107 15:53:16 [Note] WSREP: Flow-control interval: [28, 28]
131107 15:53:16 [Note] WSREP: Received NON-PRIMARY.
131107 15:53:16 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 8175767)
131107 15:53:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:53:16 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 3, my index: 1, protocol version 2
131107 15:53:16 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 15:53:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)

...

131107 15:53:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:53:16 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,270) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
a21cd90d-373e-11e3-b146-560f037992a6,
c089c14e-3738-11e3-877c-8aa6e988bded,
} joined {
} left {
} partitioned {
644c6f70-4716-11e3-9900-c2dfa5a8512f,
b5387adf-4564-11e3-9ffe-f38d17a91a12,
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
})

VictorT · November 8, 2013, 1:58pm


131107 15:53:16 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 3
131107 15:53:16 [Note] WSREP: Flow-control interval: [28, 28]
131107 15:53:16 [Note] WSREP: Received NON-PRIMARY.
131107 15:53:16 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 3, my index: 1, protocol version 2
131107 15:53:16 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 15:53:16 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 15:53:16 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 15:53:16 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
131107 15:53:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:53:18 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,271) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
644c6f70-4716-11e3-9900-c2dfa5a8512f,
a21cd90d-373e-11e3-b146-560f037992a6,
c089c14e-3738-11e3-877c-8aa6e988bded,
} joined {
} left {
} partitioned {
b5387adf-4564-11e3-9ffe-f38d17a91a12,
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
})
131107 15:53:18 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 2, memb_num = 4
131107 15:53:18 [Note] WSREP: Flow-control interval: [32, 32]
131107 15:53:18 [Note] WSREP: Received NON-PRIMARY.
131107 15:53:18 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 4, my index: 2, protocol version 2
131107 15:53:18 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 15:53:18 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
131107 15:53:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:22 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:22 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.44:4567
131107 15:53:22 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 0
131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
131107 15:53:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.44:4567
131107 15:53:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:27 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 15:53:27 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 15:53:27 [Note] WSREP: declaring b7de24de-4713-11e3-8a4f-cb158b3ea3e2 stable
131107 15:53:27 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
131107 15:53:27 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,272) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
644c6f70-4716-11e3-9900-c2dfa5a8512f,
a21cd90d-373e-11e3-b146-560f037992a6,
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
c089c14e-3738-11e3-877c-8aa6e988bded,
} joined {
} left {
} partitioned {
b5387adf-4564-11e3-9ffe-f38d17a91a12,
})
131107 15:53:27 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 2, memb_num = 5
131107 15:53:27 [Note] WSREP: Flow-control interval: [36, 36]
131107 15:53:27 [Note] WSREP: Received NON-PRIMARY.
131107 15:53:27 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 5, my index: 2, protocol version 2
131107 15:53:27 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 15:53:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 0
131107 15:53:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:28 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:53:29 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:41 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:53:41 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:41 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:41 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:53:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:51 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:53:51 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:51 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:52 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:53:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:53:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:53:56 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:53:57 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

VictorT · November 8, 2013, 1:59pm


131107 15:54:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:54:03 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:54:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:54:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:54:06 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:54:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:54:11 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:54:11 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:54:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 30
131107 15:54:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:54:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:54:13 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:54:14 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:54:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:54:21 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:54:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:54:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:54:25 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:54:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:54:35 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:54:36 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:54:36 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:54:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:54:47 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:54:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:54:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:54:57 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:54:57 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 60
131107 15:54:57 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:54:57 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:54:58 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:54:59 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:55:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:02 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:55:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:09 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:09 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:55:11 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:12 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:55:17 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:17 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:18 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:55:20 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:20 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:55:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:27 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:55:30 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:32 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:55:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:32 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:33 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

VictorT · November 8, 2013, 1:59pm


131107 15:55:41 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:42 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 90

...

131107 15:55:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:42 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:44 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:55:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:55:53 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:55:54 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:04 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:14 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:15 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:17 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:18 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:18 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:23 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:24 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:26 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:26 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 120
131107 15:56:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:35 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:36 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:36 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:36 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:37 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:38 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:41 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:41 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:48 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:49 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:51 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:51 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:54 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:56:56 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:56:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:56:59 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:57:00 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:57:00 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:57:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:57:06 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:57:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:57:11 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:57:11 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:57:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 150
131107 15:57:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:57:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:57:22 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:57:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

VictorT · November 8, 2013, 2:00pm


131107 15:57:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:57:33 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:57:33 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:57:41 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:57:41 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:57:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:57:44 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:57:44 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:57:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:57:45 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:57:46 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:57:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:57:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:57:55 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:57:55 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:57:55 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:57:56 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:57:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to b5387adf-4564-11e3-9ffe-f38d17a91a12 (tcp://10.19.0.44:4567), attempt 180
131107 15:57:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:58:01 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:58:01 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:58:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:58:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:58:05 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:58:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:58:11 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:58:11 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:58:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:58:15 [Note] WSREP: remote endpoint tcp://10.19.0.44:4567 changed identity b5387adf-4564-11e3-9ffe-f38d17a91a12 -> 526e2b4e-47ef-11e3-8c28-d21f3034d100
131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:58:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:58:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:58:16 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 15:58:16 [Note] WSREP: declaring 526e2b4e-47ef-11e3-8c28-d21f3034d100 stable
131107 15:58:16 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 15:58:16 [Note] WSREP: declaring b7de24de-4713-11e3-8a4f-cb158b3ea3e2 stable
131107 15:58:16 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
131107 15:58:16 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,273) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
526e2b4e-47ef-11e3-8c28-d21f3034d100,
644c6f70-4716-11e3-9900-c2dfa5a8512f,
a21cd90d-373e-11e3-b146-560f037992a6,
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
c089c14e-3738-11e3-877c-8aa6e988bded,
} joined {
} left {
} partitioned {
b5387adf-4564-11e3-9ffe-f38d17a91a12,
})
131107 15:58:16 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 3, memb_num = 6
131107 15:58:16 [Note] WSREP: Flow-control interval: [39, 39]
131107 15:58:16 [Note] WSREP: Received NON-PRIMARY.
131107 15:58:16 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 6, my index: 3, protocol version 2
131107 15:58:16 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 15:58:17 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:58:26 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:58:28 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:58:38 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:58:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:58:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:58:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.44:4567
131107 15:58:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

VictorT · November 8, 2013, 2:01pm


131107 15:58:46 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:58:46 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 15:58:46 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 15:58:46 [Note] WSREP: declaring b7de24de-4713-11e3-8a4f-cb158b3ea3e2 stable
131107 15:58:46 [Note] WSREP: declaring c089c14e-3738-11e3-877c-8aa6e988bded stable
131107 15:58:46 [Note] WSREP: view(view_id(NON_PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,274) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
644c6f70-4716-11e3-9900-c2dfa5a8512f,
a21cd90d-373e-11e3-b146-560f037992a6,
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
c089c14e-3738-11e3-877c-8aa6e988bded,
} joined {
} left {
} partitioned {
526e2b4e-47ef-11e3-8c28-d21f3034d100,
b5387adf-4564-11e3-9ffe-f38d17a91a12,
})
131107 15:58:46 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 2, memb_num = 5
131107 15:58:46 [Note] WSREP: Flow-control interval: [36, 36]
131107 15:58:46 [Note] WSREP: Received NON-PRIMARY.
131107 15:58:46 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# -1: non-Primary, number of nodes: 5, my index: 2, protocol version 2
131107 15:58:46 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 15:58:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567), attempt 0
131107 15:58:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:58:48 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:58:50 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:58:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:00 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:00 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:02 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:02 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:03 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:07 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:11 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:11 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:12 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:17 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:17 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:18 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:22 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:22 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:30 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567), attempt 30
131107 15:59:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:32 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:32 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:33 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:33 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:36 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:37 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:38 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:44 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:44 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:47 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:47 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:48 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 15:59:54 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 15:59:55 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 15:59:56 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:00:05 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:00:05 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 16:00:06 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:00:15 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:00:16 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)

VictorT · November 8, 2013, 2:01pm



131107 16:00:17 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') reconnecting to 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567), attempt 60
131107 16:00:17 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:00:21 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:00:23 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 16:00:23 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:00:27 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:00:28 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 16:00:28 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 16:00:29 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:00:36 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:00:38 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 16:00:38 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:00:38 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:00:38 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 16:00:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:00:42 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:00:43 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 16:00:44 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:00:48 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:00:49 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 16:00:50 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:00:51 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:00:53 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)
131107 16:00:53 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:00:57 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:00:58 [Warning] WSREP: gcs_caused() returned -107 (Transport endpoint is not connected)

131107 16:05:37 [Note] WSREP: remote endpoint tcp://10.19.0.40:4567 changed identity c089c14e-3738-11e3-877c-8aa6e988bded -> 59fea0f7-47f0-11e3-b16f-2b76a5181294
131107 16:05:37 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:05:38 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:05:39 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 16:05:39 [Note] WSREP: declaring 59fea0f7-47f0-11e3-b16f-2b76a5181294 stable
131107 16:05:39 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 16:05:39 [Note] WSREP: declaring b7de24de-4713-11e3-8a4f-cb158b3ea3e2 stable
131107 16:05:39 [Note] WSREP: Node 59fea0f7-47f0-11e3-b16f-2b76a5181294 state prim
131107 16:05:39 [Note] WSREP: view(view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,277) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
59fea0f7-47f0-11e3-b16f-2b76a5181294,
644c6f70-4716-11e3-9900-c2dfa5a8512f,
a21cd90d-373e-11e3-b146-560f037992a6,
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
} joined {
} left {
} partitioned {
526e2b4e-47ef-11e3-8c28-d21f3034d100,
b5387adf-4564-11e3-9ffe-f38d17a91a12,
c089c14e-3738-11e3-877c-8aa6e988bded,
})
131107 16:05:39 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 3, memb_num = 5
131107 16:05:39 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
131107 16:05:39 [Note] WSREP: forgetting 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567)
131107 16:05:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:05:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:05:39 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
131107 16:05:39 [Note] WSREP: STATE EXCHANGE: sent state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6
131107 16:05:39 [Note] WSREP: STATE EXCHANGE: got state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6 from 0 (db1)
131107 16:05:39 [Note] WSREP: STATE EXCHANGE: got state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6 from 1 (db6)
131107 16:05:39 [Note] WSREP: STATE EXCHANGE: got state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6 from 2 (db5)
131107 16:05:39 [Note] WSREP: STATE EXCHANGE: got state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6 from 3 (db2)
131107 16:05:39 [Note] WSREP: STATE EXCHANGE: got state msg: 5aba8e58-47f0-11e3-9d12-96d0db6484d6 from 4 (db3)
131107 16:05:39 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 1,
members = 4/5 (joined/total),
act_id = 8175767,
last_appl. = 8175401,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 05be2ccc-012d-11e3-8cb2-c69c9281c29b
131107 16:05:39 [Note] WSREP: Flow-control interval: [36, 36]
131107 16:05:39 [Note] WSREP: Restored state OPEN -> SYNCED (8175767)
131107 16:05:39 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# 2: Primary, number of nodes: 5, my index: 3, protocol version 2
131107 16:05:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 16:05:39 [Note] WSREP: Assign initial position for certification: 8175767, protocol version: 2
131107 16:05:39 [Note] WSREP: Synchronized with group, ready for connections
131107 16:05:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 16:05:39 [Note] WSREP: Node 4 (db3) requested state transfer from 'db6'. Selected 1 (db6)(SYNCED) as donor.
131107 16:05:40 [Warning] WSREP: 1 (db6): State transfer to 4 (db3) failed: -125 (Operation canceled)
131107 16:05:40 [Note] WSREP: Member 1 (db6) synced with group.
131107 16:05:40 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 16:05:40 [Note] WSREP: declaring 59fea0f7-47f0-11e3-b16f-2b76a5181294 stable
131107 16:05:40 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 16:05:40 [Note] WSREP: Node 27013c03-3740-11e3-aec6-4a0d195491f5 state prim
131107 16:05:40 [Note] WSREP: view(view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,278) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
59fea0f7-47f0-11e3-b16f-2b76a5181294,
644c6f70-4716-11e3-9900-c2dfa5a8512f,
a21cd90d-373e-11e3-b146-560f037992a6,
} joined {
} left {
} partitioned {
b7de24de-4713-11e3-8a4f-cb158b3ea3e2,
})
131107 16:05:40 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 3, memb_num = 4
131107 16:05:40 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
131107 16:05:40 [Note] WSREP: forgetting 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567)
131107 16:05:40 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

VictorT · November 8, 2013, 2:02pm


131107 16:05:40 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:05:40 [Note] WSREP: forgetting b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567)
131107 16:05:40 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:05:40 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:05:40 [Note] WSREP: STATE EXCHANGE: sent state msg: 5ba0232f-47f0-11e3-b84d-d7533c1bf359
131107 16:05:40 [Note] WSREP: STATE EXCHANGE: got state msg: 5ba0232f-47f0-11e3-b84d-d7533c1bf359 from 0 (db1)
131107 16:05:40 [Note] WSREP: STATE EXCHANGE: got state msg: 5ba0232f-47f0-11e3-b84d-d7533c1bf359 from 1 (db6)
131107 16:05:40 [Note] WSREP: STATE EXCHANGE: got state msg: 5ba0232f-47f0-11e3-b84d-d7533c1bf359 from 2 (db5)
131107 16:05:40 [Note] WSREP: STATE EXCHANGE: got state msg: 5ba0232f-47f0-11e3-b84d-d7533c1bf359 from 3 (db2)
131107 16:05:40 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 2,
members = 4/4 (joined/total),
act_id = 8175767,
last_appl. = 0,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 05be2ccc-012d-11e3-8cb2-c69c9281c29b
131107 16:05:40 [Note] WSREP: Flow-control interval: [32, 32]
131107 16:05:40 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8175767, view# 3: Primary, number of nodes: 4, my index: 3, protocol version 2
131107 16:05:40 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 16:05:40 [Note] WSREP: Assign initial position for certification: 8175767, protocol version: 2
131107 16:05:45 [Note] WSREP: cleaning up b7de24de-4713-11e3-8a4f-cb158b3ea3e2 (tcp://10.19.0.43:4567)
131107 16:05:45 [Note] WSREP: cleaning up 526e2b4e-47ef-11e3-8c28-d21f3034d100 (tcp://10.19.0.44:4567)
131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.19.0.44:4567
131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping
131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') turning message relay requesting off
131107 16:17:08 [Note] WSREP: (a21cd90d-373e-11e3-b146-560f037992a6, 'tcp://0.0.0.0:4567') address 'tcp://10.19.0.42:4567' pointing to uuid a21cd90d-373e-11e3-b146-560f037992a6 is blacklisted, skipping

...

131107 16:17:08 [Note] WSREP: declaring 27013c03-3740-11e3-aec6-4a0d195491f5 stable
131107 16:17:08 [Note] WSREP: declaring 59fea0f7-47f0-11e3-b16f-2b76a5181294 stable
131107 16:17:08 [Note] WSREP: declaring 644c6f70-4716-11e3-9900-c2dfa5a8512f stable
131107 16:17:08 [Note] WSREP: declaring f5b78e53-47f1-11e3-b8c6-6f4c302349ca stable
131107 16:17:08 [Note] WSREP: Node 27013c03-3740-11e3-aec6-4a0d195491f5 state prim
131107 16:17:08 [Note] WSREP: view(view_id(PRIM,27013c03-3740-11e3-aec6-4a0d195491f5,279) memb {
27013c03-3740-11e3-aec6-4a0d195491f5,
59fea0f7-47f0-11e3-b16f-2b76a5181294,
644c6f70-4716-11e3-9900-c2dfa5a8512f,
a21cd90d-373e-11e3-b146-560f037992a6,
f5b78e53-47f1-11e3-b8c6-6f4c302349ca,
} joined {
} left {
} partitioned {
})
131107 16:17:08 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 3, memb_num = 5
131107 16:17:08 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
131107 16:17:08 [Note] WSREP: STATE EXCHANGE: sent state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d
131107 16:17:08 [Note] WSREP: STATE EXCHANGE: got state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d from 0 (db1)
131107 16:17:08 [Note] WSREP: STATE EXCHANGE: got state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d from 1 (db6)
131107 16:17:08 [Note] WSREP: STATE EXCHANGE: got state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d from 2 (db5)
131107 16:17:08 [Note] WSREP: STATE EXCHANGE: got state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d from 3 (db2)
131107 16:17:09 [Note] WSREP: STATE EXCHANGE: got state msg: f5bac6ca-47f1-11e3-8589-6b6d1f9da25d from 4 (db4)
131107 16:17:09 [Note] WSREP: Quorum results:
version = 2,
component = PRIMARY,
conf_id = 3,
members = 4/5 (joined/total),
act_id = 8176745,
last_appl. = 0,
protocols = 0/4/2 (gcs/repl/appl),
group UUID = 05be2ccc-012d-11e3-8cb2-c69c9281c29b
131107 16:17:09 [Note] WSREP: Flow-control interval: [36, 36]
131107 16:17:09 [Note] WSREP: New cluster view: global state: 05be2ccc-012d-11e3-8cb2-c69c9281c29b:8176745, view# 4: Primary, number of nodes: 5, my index: 3, protocol version 2
131107 16:17:09 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
131107 16:17:09 [Note] WSREP: Assign initial position for certification: 8176745, protocol version: 2
131107 16:17:11 [Note] WSREP: Node 4 (db4) requested state transfer from 'db6'. Selected 1 (db6)(SYNCED) as donor.
131107 16:19:06 [Note] WSREP: 1 (db6): State transfer to 4 (db4) complete.
131107 16:19:06 [Note] WSREP: 4 (db6): State transfer from 1 (db6) complete.
131107 16:19:06 [Note] WSREP: Member 1 (db6) synced with group.
131107 16:19:20 [Note] WSREP: Member 4 (db4) synced with group.

Topic		Replies	Views
Percona XtraDB Cluster - Lock wait timeout exceeded Percona XtraDB Cluster 5.x	3	2466	September 11, 2019
Database not responding after adding a server to the cluster Percona XtraDB Cluster 8.x mysql , percona	11	866	November 16, 2022
wsrep_cluster_size=1 & wsrep_ready=ON. What is wrong? Percona XtraDB Cluster 5.x	0	2034	October 4, 2012
Mysql Service Stopped Due to timeout with cluster address Percona XtraDB Cluster 5.x	0	843	May 25, 2018
percona xtradb 5.6.37 cluster crash Percona XtraDB Cluster 5.x	1	734	December 15, 2017

Cluster failed, can you provide any insight?

Related topics