# Nodes got aborted one by one when creating/droping index **URL:** \ **Category:** Percona XtraDB Cluster 8.x\ **Created:** [July 11, 2025, 2:13pm UTC](https://forums.percona.com/t/nodes-got-aborted-one-by-one-when-creating-droping-index/38654 "2025-07-11T14:13:39Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![hung\_le](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/hung_le/32/12170_2.png) [@hung\_le](https://forums.percona.com/u/hung_le)\ **Post date:** [July 11, 2025, 2:13pm UTC](https://forums.percona.com/t/nodes-got-aborted-one-by-one-when-creating-droping-index/38654/1 "2025-07-11T14:13:39Z")
Hi all, I’m having a 3 nodes Cluster. > mysql\> SHOW VARIABLES LIKE ‘version%’; > ±------------------------±--------------------------------------------------------------------------------------+ > | Variable\_name | Value | > ±------------------------±--------------------------------------------------------------------------------------+ > | version | 8.0.37-29.1 | > | version\_comment | Percona XtraDB Cluster (GPL), Release rel29, Revision d29a325, WSREP version 26.1.4.3 | > | version\_compile\_machine | x86\_64 | > | version\_compile\_os | Linux | > | version\_compile\_zlib | 1.2.13 | > | version\_suffix | .1 | > ±------------------------±--------------------------------------------------------------------------------------+ > 6 rows in set (0.01 sec) On top of those 3 nodes, I have a proxysql: > ±-------------±-----------±-----±----------±--------±-------±------------±----------------±--------------------±--------±---------------±--------+ > | hostgroup\_id | hostname | port | gtid\_port | status | weight | compression | max\_connections | max\_replication\_lag | use\_ssl | max\_latency\_ms | comment | > ±-------------±-----------±-----±----------±--------±-------±------------±----------------±--------------------±--------±---------------±--------+ > | 10 | prod-db-01 | 3306 | 0 | SHUNNED | 10000 | 0 | 1000 | 0 | 0 | 0 | | > | 10 | prod-db-02 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | | > | 10 | prod-db-03 | 3306 | 0 | SHUNNED | 9999 | 0 | 1000 | 0 | 0 | 0 | | > | 20 | prod-db-01 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | | > | 20 | prod-db-03 | 3306 | 0 | ONLINE | 9999 | 0 | 1000 | 0 | 0 | 0 | | > | 30 | prod-db-01 | 3306 | 0 | ONLINE | 10000 | 0 | 1000 | 0 | 0 | 0 | | > | 30 | prod-db-03 | 3306 | 0 | ONLINE | 9999 | 0 | 1000 | 0 | 0 | 0 | | > ±-------------±-----------±-----±----------±--------±-------±------------±----------------±--------------------±--------±---------------±--------+ I applied write/read split: > \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* 1. row \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* > rule\_id: 100 > active: 1 > username: NULL > schemaname: NULL > flagIN: 0 > client\_addr: NULL > proxy\_addr: NULL > proxy\_port: NULL > digest: NULL > match\_digest: NULL > match\_pattern: ^SELECT .\* FOR UPDATE > negate\_match\_pattern: 0 > re\_modifiers: CASELESS > flagOUT: NULL > replace\_pattern: NULL > destination\_hostgroup: 10 > cache\_ttl: NULL > cache\_empty\_result: NULL > cache\_timeout: NULL > reconnect: NULL > timeout: NULL > retries: NULL > delay: NULL > next\_query\_flagIN: NULL > mirror\_flagOUT: NULL > mirror\_hostgroup: NULL > error\_msg: NULL > OK\_msg: NULL > sticky\_conn: NULL > multiplex: NULL > gtid\_from\_hostgroup: NULL > log: NULL > apply: 1 > attributes: > comment: NULL > \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* 2. row \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* > rule\_id: 200 > active: 1 > username: NULL > schemaname: NULL > flagIN: 0 > client\_addr: NULL > proxy\_addr: NULL > proxy\_port: NULL > digest: NULL > match\_digest: NULL > match\_pattern: ^SELECT .\* > negate\_match\_pattern: 0 > re\_modifiers: CASELESS > flagOUT: NULL > replace\_pattern: NULL > destination\_hostgroup: 30 > cache\_ttl: NULL > cache\_empty\_result: NULL > cache\_timeout: NULL > reconnect: NULL > timeout: NULL > retries: NULL > delay: NULL > next\_query\_flagIN: NULL > mirror\_flagOUT: NULL > mirror\_hostgroup: NULL > error\_msg: NULL > OK\_msg: NULL > sticky\_conn: NULL > multiplex: NULL > gtid\_from\_hostgroup: NULL > log: NULL > apply: 1 > attributes: > comment: NULL > \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* 3. row \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* > rule\_id: 300 > active: 1 > username: NULL > schemaname: NULL > flagIN: 0 > client\_addr: NULL > proxy\_addr: NULL > proxy\_port: NULL > digest: NULL > match\_digest: NULL > match\_pattern: .\* > negate\_match\_pattern: 0 > re\_modifiers: CASELESS > flagOUT: NULL > replace\_pattern: NULL > destination\_hostgroup: 10 > cache\_ttl: NULL > cache\_empty\_result: NULL > cache\_timeout: NULL > reconnect: NULL > timeout: NULL > retries: NULL > delay: NULL > next\_query\_flagIN: NULL > mirror\_flagOUT: NULL > mirror\_hostgroup: NULL > error\_msg: NULL > OK\_msg: NULL > sticky\_conn: NULL > multiplex: NULL > gtid\_from\_hostgroup: NULL > log: NULL > apply: 1 > attributes: > comment: NULL > 3 rows in set (0.00 sec) Before anything happened we ran a script (ssh directly to Node02 to run) but got error because of missing the index. > ERROR 1822 (HY000) at line 94: Failed to add the foreign key constraint. Missing index for constraint ‘fk\_ms\_shortname’ in the referenced table ‘catalogitems’ **Then : The problem is that we tried to create and drop indexes via DBeaver Tool (while there are user still using the system → query, update to the DB), we got the Cluster crashed (Cluster crashed around 2025-07-10T12:45).** These are log files relatively to Node01, Node02 and Node03. [Node01 log](https://drive.google.com/file/d/1gaJdo6XAVWY1i6qqAtN57SmRdg0uU0lN/view?usp=drive_link) [Node02 log](https://drive.google.com/file/d/1kkeykTnmhJZVz1R-39QRIyLcV2z09K0x/view?usp=sharing) [Node03 log](https://drive.google.com/file/d/1UG8GNlweZdzaVJWo6SIRfF4xfeFynsUv/view?usp=sharing) To bring back the cluster again, I have to restart Node01 and Node03 (these are aborted), then the SST flow started. Please help me to identify what is the root cause. I tried to reproduce the issue by running queries while creating/dropping indexes but cannot have the same issue (Nodes got aborted). Thanks. ---
**Author:** ![jrivera](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jrivera/32/13_2.png) [@jrivera](https://forums.percona.com/u/jrivera)\ **Post date:** [July 12, 2025, 12:26am UTC](https://forums.percona.com/t/nodes-got-aborted-one-by-one-when-creating-droping-index/38654/2 "2025-07-12T00:26:26Z")
You could be affected by this bug: > **[Jira](https://perconadev.atlassian.net/browse/PXC-4348)** I would suggest upgrading to the fixed version (8.0.36-28) or better yet the latest release. For DDL operations we highly recommend using pt-online-schema-change using the TOI method. Alternatively, check this blog post for the RSU method: > **[How to Perform Compatible Schema Changes in Percona XtraDB Cluster (Advanced...](https://www.percona.com/blog/how-to-perform-compatible-schema-changes-in-percona-xtradb-cluster-advanced-alternative/)** > > Making schema changes in Percona XtraDB Cluster are a problem, and challenge high availability. Przemyslaw Malkowski reviews galera schema change options. ---
**Author:** ![meyerder](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/meyerder/32/16994_2.png) [@meyerder](https://forums.percona.com/u/meyerder)\ **Post date:** [July 16, 2025, 6:39pm UTC](https://forums.percona.com/t/nodes-got-aborted-one-by-one-when-creating-droping-index/38654/3 "2025-07-16T18:39:15Z")
Also noticed it looks like the fk is dropped but you are createing a full text index so you may be doing several changes at one time including dropping tables so you may have triggers or other items that are going on at the same time..