# Database cluster hang problem **URL:** \ **Category:** Percona XtraDB Cluster\ **Tags:** mysql, percona\ **Created:** [February 20, 2023, 1:21pm UTC](https://forums.percona.com/t/database-cluster-hang-problem/20194 "2023-02-20T13:21:52Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![bthnklc](https://avatars.discourse-cdn.com/v4/letter/b/e0b2c6/32.png) [@bthnklc](https://forums.percona.com/u/bthnklc)\ **Post date:** [February 20, 2023, 1:21pm UTC](https://forums.percona.com/t/database-cluster-hang-problem/20194/1 "2023-02-20T13:21:52Z")
Hello, We have a 5 node percona xtradb cluster structure and all nodes work as master-master. We are using MySQL version 8.0.29. 3 nodes are located on dc1. 2 nodes are located on dc2. I added a slave db node to one of the nodes located on dc2 with a 3 hour delay replication. After about half an hour, all the master servers were down. note: I installed the slave db database server I added as percona server. In the attachment, I have added the log of the node - slave db’ log and the log of the master server to which the slave db is connected. I would like you to share your knowledge and experience about the operability of this structure and the reason for its closure. [dc2\_error\_20\_158\_slave.log](https://forums.percona.com/uploads/short-url/3PhV41rKLviMzjBPAq2TAgTCKkJ.log) (7.1 KB) [error\_19\_156\_other.log](https://forums.percona.com/uploads/short-url/iUXBogCENz3nHM5kVDgVLM4kkFa.log) (16.0 KB) [dc1\_error\_19\_156\_main.log](https://forums.percona.com/uploads/short-url/wNrl8MA6Wz8QuRGMkXSKn2Zl1yj.log) (370.5 KB) [dc2\_error\_20\_157.log](https://forums.percona.com/uploads/short-url/4qANBLzDCQP68s3EI0vGipljK7e.log) (41.8 KB) ---
**Author:** ![Pep\_Pla](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/pep_pla/32/9735_2.png) [@Pep\_Pla](https://forums.percona.com/u/Pep_Pla)\ **Post date:** [February 20, 2023, 9:41pm UTC](https://forums.percona.com/t/database-cluster-hang-problem/20194/2 "2023-02-20T21:41:16Z")
Hi @bthnklc I think there are several things in your configuration that can lead to problems: - PXC 5 node cluster split in two DCs (If, for whatever reason, communication is broken dc2 will go down) - PXC with multiple writing nodes and foreign keys. It looks like something happened to your cluster, and then there was a cascade of events that brought the whole cluster down. I recommend that you try to simplify the architecture to find the root cause of the problem: - Do not write into multiple nodes. - Remove triggers (or use them only for integrity validation, not to propagate changes). Pep ---
**Author:** ![bthnklc](https://avatars.discourse-cdn.com/v4/letter/b/e0b2c6/32.png) [@bthnklc](https://forums.percona.com/u/bthnklc)\ **Post date:** [March 2, 2023, 12:15pm UTC](https://forums.percona.com/t/database-cluster-hang-problem/20194/3 "2023-03-02T12:15:10Z")
There is no problem in the connection between DC1 and DC2. It never happened. We use 10gig connection. At the existing 5 knotted structure, applications are adjusted to only one server on the DNS. There is a question here. If any writing process is performed on one of the other nodes in the structure I specified, does the cluster work? ---
**Author:** ![Pep\_Pla](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/pep_pla/32/9735_2.png) [@Pep\_Pla](https://forums.percona.com/u/Pep_Pla)\ **Post date:** [March 2, 2023, 9:02pm UTC](https://forums.percona.com/t/database-cluster-hang-problem/20194/4 "2023-03-02T21:02:37Z")
```auto 2023-02-20T08:00:29.003926Z 0 [Note] [MY-000000] [Galera] (92d437c2-a685, 'ssl://0.0.0.0:4567') connection to peer f31f99f1-92e7 with addr ssl://172.19.0.157:4567 timed out, no messages seen in PT3S, ``` Looks like there were some timeouts. This is why I was talking about possible network issues. The cluster should work if any node gets writes, but this can cause concurrency issues. ---
**Author:** ![bthnklc](https://avatars.discourse-cdn.com/v4/letter/b/e0b2c6/32.png) [@bthnklc](https://forums.percona.com/u/bthnklc)\ **Post date:** [March 3, 2023, 6:48am UTC](https://forums.percona.com/t/database-cluster-hang-problem/20194/5 "2023-03-03T06:48:08Z")
Thank you for the information. Regardless of this issue, I have one more question. I start bootstrap from a server in a 3-node cluster structure and start the other nodes in order. When restarts are required, I can restart other nodes one by one with systemctl restart, but the node I started with bootstrap does not restart when shut down. I would like your help on how to proceed here. ---
**Author:** ![Pep\_Pla](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/pep_pla/32/9735_2.png) [@Pep\_Pla](https://forums.percona.com/u/Pep_Pla)\ **Post date:** [March 14, 2023, 2:26pm UTC](https://forums.percona.com/t/database-cluster-hang-problem/20194/6 "2023-03-14T14:26:35Z")
Do you mean the database does not restart when you restart the node? Or if you issue a `systemct restart mysql` does nothing? In the second case, try the following: ```auto systemctl stop mysql@bootstrap # only if the other nodes are running! systemctl start mysql ``` Then check if `systemctl restart mysql` works.