Hello,
Some days ago we had shutdown all nodes (3 nodes cluster) and we couldn’t find a reason. Only what we found in logs was:
1st node
2017-03-13T15:58:05.464969Z 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT1.74085S), skipping check
2017-03-13T15:58:07.639025Z 0 [Warning] WSREP: discarding established (time wait) 2f0310b0 (tcp://192.168.150.177:4567)
2017-03-13T15:58:12.791141Z 224 [ERROR] WSREP: Certification failed for TO isolated action: source: 43f9e143-e948-11e6-b96a-7308d80d4481 version: 3 local: 0 state: CERTIFYING flags: 65 conn_id: 44314804 trx_id: -1 seqnos (l: 2349880499, g: 8244067270, s: 8244066132, d: -1, ts: 6410594034574940)
2nd node
2017-03-13T15:58:07.228471Z 0 [Warning] WSREP: Action message in non-primary configuration from member 1
2017-03-13T15:58:07.232030Z 0 [Warning] WSREP: Action message in non-primary configuration from member 1
2017-03-13T15:58:07.232051Z 0 [Warning] WSREP: Action message in non-primary configuration from member 1
2017-03-13T15:58:07.232067Z 0 [Warning] WSREP: Action message in non-primary configuration from member 1
2017-03-13T15:58:07.232082Z 0 [Warning] WSREP: Action message in non-primary configuration from member 1
2017-03-13T15:58:07.232098Z 0 [Warning] WSREP: Action message in non-primary configuration from member 1
3rd node:
2017-03-13T15:58:05.480916Z 0 [Warning] WSREP: last inactive check more than PT1.5S ago (PT4.92684S), skipping check
2017-03-13T15:58:07.887999Z 34610542 [Warning] WSREP: Send action {(nil), 686, TORDERED} returned -107 (Transport endpoint is not connected)
2017-03-13T15:58:07.889008Z 34610541 [Warning] WSREP: Send action {(nil), 686, TORDERED} returned -107 (Transport endpoint is not connected)
2017-03-13T15:58:07.889060Z 34610542 [Warning] WSREP: Send action {(nil), 686, TORDERED} returned -107 (Transport endpoint is not connected)
2017-03-13T15:58:07.889588Z 34610541 [Warning] WSREP: Send action {(nil), 686, TORDERED} returned -107 (Transport endpoint is not connected)
2017-03-13T15:58:07.889739Z 34610542 [Warning] WSREP: Send action {(nil), 686, TORDERED} returned -107 (Transport endpoint is not connected)
2017-03-13T15:58:07.890079Z 34610541 [Warning] WSREP: Send action {(nil), 686, TORDERED} returned -107 (Transport endpoint is not connected)
What could be a reason those fault ?