Percona Cluster won't start after VM Crashed

Hi everyone and thanks by advance for your sharp advices.

We’ve got a setup of 3 Percona Nodes installed on Ubuntu 10.04-LTS.

Two of the 3 nodes had crashed and now this is impossible to start again the service.

if i type sudo service mysql start, i received ERROR 1047 (08S01) at line 1: Unknown command

While inspecting the log i found this :
140514 15:54:12 [ERROR] Slave SQL: Could not execute Write_rows event on table SWIAT_LIVE_V3.CAR_WS_RESULT; Cannot add or update a child row: a foreign key constraint fails (SWIAT_LIVE_V3.CAR_WS_RESULT, CONSTRAINT CAR_WS_RESULT_FK_1 FOREIGN KEY (PROJECT_ID) REFERENCES CAR_DETAILS (CAR_DETAILS)), Error_code: 1452; handler error HA_ERR_NO_REFERENCED_ROW; the event’s master log FIRST, end_log_pos 190, Error_code: 1452
140514 15:54:12 [Warning] WSREP: RBR event 2 Write_rows apply warning: 151, 8414353
140514 15:54:12 [ERROR] WSREP: receiving IST failed, node restart required: Failed to apply app buffer: seqno: 8414353, status: WSREP_FATAL

sounds a issue with the replication… but sincerly i don’t know how to handle this as long i’m not a DBA.

Any advices are welcome.

First of all, it should go under “Percona XtraDB Cluster” category.

ERROR 1047 (08S01) at line 1: Unknown command
Above error occurred because In this state, the working node will stop to serve any client requests as a protection for a possible split brain situation. Any SQL statement will result with:
mysql> select 1;
ERROR 1047 (08S01): Unknown command

In order to restore the cluster, execute below command on the working node and it will establish this node to form the primary component again and restarting previously crashed nodes will join the cluster hopefully.
mysql> SET GLOBAL wsrep_provider_options=‘pc.bootstrap=true’;

Hope that helps.