Recently we installed 3 node Percona Xtra DB Custer 5.7 on CentOS 7. It was working good for some time but after 10 days , mysql on 1st and 2nd nodes was crashed down due to filling up of “/” partition because of binary logs. And the 3rd node was still up and running but it could not serve read/write for the clients.
After cleaning/making space in the file system, i tried to start the mysql on 1st node as well as 2nd node but it failed to start on both the nodes. Later by going through some of the forums, i realized that the 3rd node which was up and running was not bootstrapped. So i bootstrapped the 3rd node and tried to start the mysql on 1st and 2nd nodes. This time it was successful and all the 3 nodes were in sync.
This arises a lot of questions w.r.t high availability… I have the following questions in my mind:
- When the 1st and 2nd nodes were down, why the surviving 3rd node could not serve reads and writes?
- Even after making space on the 1st and 2nd nodes, when i tried to start mysql , why it couldn’t start? If bootstrap was the reason then why it didn’t automatically shift to surviving node?
I hope there might be a configuration issues which needs to be changed…
Any help here would be highly appreciated…