# Disconnected Node and KeepAlives

**URL:** <https://forums.percona.com/t/disconnected-node-and-keepalives/14175>\
**Category:** Percona XtraDB Cluster\
**Created:** [January 31, 2022, 3:30pm UTC](https://forums.percona.com/t/disconnected-node-and-keepalives/14175 "2022-01-31T15:30:07Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![meyerder](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/meyerder/32/16994_2.png) [@meyerder](https://forums.percona.com/u/meyerder)\
**Post date:** [January 31, 2022, 3:30pm UTC](https://forums.percona.com/t/disconnected-node-and-keepalives/14175/1 "2022-01-31T15:30:07Z")

</div>

To All:

I know a couple of bugs Exist in PXC 8. Some result in race conditions that will freeze the cluster for 600 seconds until it crashes… As such a co-worker and I worked out the following Two Items for our setup which is located behind AVI/Netscalers (Not using HAProxy) I know this is a bit extreme in reducing the End-user impact but I could not figure out any other means of doing it

\*\*My setup all reads/writes go to one system. The others act as a backup in the event that the primary has an issue.

Script 1. - Runs on the AVI/Netscaler to check Each server’s condition. Will remove a node if it becomes a Donor or otherwise.

[keepalive\_procedure.txt](https://forums.percona.com/uploads/short-url/cdSeQk5L8fDqGdwhJmBNFzE1bs5.txt) (1.2 KB)

Script#2  
High Level - Does Everything the KeepAlive check does and also processes an insert. If the Insert takes \> 20 seconds it issues a pkill on the system if it is the primary system (this accounts for the Stalled condition and reduces the time due to the race condition from 600 seconds to less than a minute.)

[local\_script.txt](https://forums.percona.com/uploads/short-url/pmCiykNUDqcsNfDRFBzQQk1rkLb.txt) (13.2 KB)

I have one condition that I have had happen several times that I have yet to figure out how to address. For some reason once in a bit I have an insert procedure that happens that initiates a quarm vote when it hits a duplicate key (1062 Error) The Primary system goes into Disconnected and the VIP/GSLB moves traffic to another system. The issue is any persistent connections are still attached to the system that is now Disconnected and receive wsrep errors until the connections are terminated. I do not know yet how to deal with that condition.

Thank you

---

<div class="post-metadata">

**Author:** ![Michael\_Coburn](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/michael_coburn/32/18_2.png) [@Michael\_Coburn](https://forums.percona.com/u/Michael_Coburn)\
**Post date:** [January 31, 2022, 7:01pm UTC](https://forums.percona.com/t/disconnected-node-and-keepalives/14175/2 "2022-01-31T19:01:15Z")

</div>

Hi @meyerder

Could you share the error logs from the node being evicted along with a node that remained in the cluster?

This is not expected behaviour on a DUPLICATE KEY violation - these should be resolved within InnoDB and never raise to the wsrep galera level. Unless you actually do have data differences on your nodes, in which case you’d want to validate consistency by either forcing SST from a single node, or executing [pt-table-checksum](https://www.percona.com/doc/percona-toolkit/LATEST/pt-table-checksum.html) in order to identify data differences.
