I have 3 nodes running on Google Cloud.
The server verions is 5.6 the Galera version 3.8 running on a Debian 7.
Today I had a second occurency of a catastrophic problem that causes the nodes crashing. First one node crashes after some time a second node crashes and them all system is down.
I’d take a look at the error log and I found these messages right before the first node to crash:
2015-05-26 05:41:11 18566 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.58995S), skipping check
2015-05-26 05:41:13 18566 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.81387S), skipping check
2015-05-26 05:41:15 18566 [Note] WSREP: (21052d7d, ‘tcp://0.0.0.0:4567’) turning message relay requesting on, nonlive peers: tcp://10.240.31.226:4567
2015-05-26 05:41:16 18566 [Note] WSREP: (21052d7d, ‘tcp://0.0.0.0:4567’) reconnecting to 2cf899b9 (tcp://10.240.31.226:4567), attempt 0
2015-05-26 05:41:19 18566 [Warning] WSREP: last inactive check more than PT1.5S ago (PT6.08689S), skipping check
2015-05-26 05:41:22 18566 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.68869S), skipping check
2015-05-26 05:41:25 18566 [Note] WSREP: (21052d7d, ‘tcp://0.0.0.0:4567’) reconnecting to 39e0cac1 (tcp://10.240.135.197:4567), attempt 0
Killed
150526 05:41:28 mysqld_safe Number of processes running now: 0
150526 05:41:28 mysqld_safe WSREP: not restarting wsrep node automatically
150526 05:41:28 mysqld_safe mysqld from pid file /var/lib/mysql/mysql.pid ended
This log is from the bootstrap node, it was the first one to crash.
Can anyone can tell me what is happening?
Thanks