I have 3-node cluster. At some point it went bad, so after that it wast started again, with one node being started with ‘bootstrap-pxc’ option, and two others just syncing to it.
Now I have two nodes crashing all the time, while the node which was started first is always ok.
On the crashing nodes, in the log I see the following:
2014-12-04 14:58:05 16834 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 204996287)
2014-12-04 14:58:05 16834 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 204996287
2014-12-04 14:58:05 16834 [Warning] WSREP: failed to replay trx: source: 8c76e5c3-7b99-11e4-b260-4be15515cbc1 version: 3 local: 1 state: REPLAYING flags: 1 conn_id: 43055 trx_id: 1783377670 seqnos (l: 80620, g: 204996287, s: 204996283, d: 204996171, ts: 21382320446607328)
2014-12-04 14:58:05 16834 [Warning] WSREP: Failed to apply trx 204996287 4 times
2014-12-04 14:58:05 16834 [ERROR] WSREP: trx_replay failed for: 6, query: void
2014-12-04 14:58:05 16834 [ERROR] Aborting
This error almost always shows ‘Delete_rows apply warning’, however the general-log show no delete queries at all.
The average life of crashing node is about 15-20 minutes, however when I run it inside the gdb trying to catch the problem, it works for days without any problems, which makes the situation even more strange.