# Galera/PXC Cluster Permanent Stall: Failed DDL Voting Storm + Pending TOI Deadlock **URL:** \ **Category:** Percona XtraDB Cluster 8.x\ **Tags:** mysql, percona\ **Created:** [April 24, 2026, 2:50pm UTC](https://forums.percona.com/t/galera-pxc-cluster-permanent-stall-failed-ddl-voting-storm-pending-toi-deadlock/40709 "2026-04-24T14:50:21Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![reddy\_nishanth](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/reddy_nishanth/32/21004_2.png) [@reddy\_nishanth](https://forums.percona.com/u/reddy_nishanth)\ **Post date:** [April 24, 2026, 2:50pm UTC](https://forums.percona.com/t/galera-pxc-cluster-permanent-stall-failed-ddl-voting-storm-pending-toi-deadlock/40709/1 "2026-04-24T14:50:21Z")
## Description: A burst of ~50 failed `ALTER TABLE … DROP PARTITION` statements (Error 1507 - “Error in list of partitions to DROP”) issued sequentially across ~50 databases by a single application thread caused all three nodes of a PXC 8.0.42-33.1 cluster to enter consecutive Galera voting rounds. Combined with concurrent client DML load, this overwhelmed the receiver appliers. A subsequent legitimate TOI DDL (`ALTER TABLE … REORGANIZE PARTITION`) on the writer then got stuck indefinitely in wsrep: preparing for TO isolation, creating a flow-control deadlock that did not self-recover. The cluster remained in this stalled state for 26+ hours until manual recovery via pod deletion. During the stall, all clients write blocked, MySQL on the writer became completely unresponsive (could not even answer SELECT 1), and the receivers were frozen at the same seqno (wsrep\_last\_committed) for the entire duration. The same failed DDL pattern on standalone MySQL is harmless — the application catches Error 1507 and proceeds. On Galera, each failed DDL triggered cluster-wide voting before the application’s error handler received the error, paying the cluster cost regardless of how the application handled it. ## Steps to Reproduce: The bug pattern (Error 1507 + voting on all nodes) reproduces in 1 second on a 3-node cluster: 1. Create `N` databases, each with a partitioned table containing a catchall partition: ```auto CREATE DATABASE repro_db_001; USE repro_db_001; CREATE TABLE t ( id BIGINT NOT NULL AUTO_INCREMENT, ts DATETIME NOT NULL, PRIMARY KEY (id, ts) ) PARTITION BY RANGE (TO_DAYS(ts)) ( PARTITION p_min VALUES LESS THAN (TO_DAYS(‘2025-01-01’)), PARTITION p_2025_01 VALUES LESS THAN (TO_DAYS(‘2025-02-01’)), PARTITION p_2025_02 VALUES LESS THAN (TO_DAYS(‘2025-03-01’)), PARTITION old_max VALUES LESS THAN MAXVALUE ); – Repeat for repro_db_002 … repro_db_050 ``` 1. From a single MySQL session against the writer, issue the buggy pattern across all N databases sequentially: ```auto USE repro_db_001; ALTER TABLE t REORGANIZE PARTITION old_max INTO ( PARTITION p_2025_03 VALUES LESS THAN (TO_DAYS(‘2025-04-01’)), PARTITION new_max VALUES LESS THAN MAXVALUE ); ALTER TABLE t DROP PARTITION old_max; – FAILS with Error 1507 (old_max already gone) USE repro_db_002; ALTER TABLE t REORGANIZE PARTITION old_max INTO (…); ALTER TABLE t DROP PARTITION old_max; – FAILS – … repeat for all 50 databases ``` 1. Observe the mysqld error log on any node — each failed DROP triggers a 3-node voting round. Conditions required to escalate from voting storm into full deadlock (we observed all three in production but could only reliably reproduce conditions 1 and 2 in a controlled test): - Condition 1: The failed-DDL pattern above (~50 in rapid succession). - Condition 2: Tables with substantial data (so each REORGANIZE on the receiver is non-trivial work). Production tables had GBs of data; empty test tables let receivers keep up trivially. - Condition 3: Concurrent client DML load competing for receiver applier resources. With all three present, the deadlock is reliably triggered by the next legitimate TOI DDL issued after the storm. ## Version: - Server: `8.0.42-33.1` Percona XtraDB Cluster (GPL), Release rel33, Revision 6673f8e - WSREP: `26.1.4.3` - Operator: Percona Operator for PXC (Kubernetes deployment) - OS: Linux x86\_64 Relevant Galera provider options: `repl.commit_order = 3 (strict commit ordering)` `cert.optimistic_pa = no (conservative parallel applying)` `gcs.fc_limit = 100` `gcs.fc_factor = 1.0` `gcache.size = 4G` `gcache.recover = yes` Server config: `innodb_buffer_pool_size = 8G` `max_connections = 1000` `wsrep_slave_threads = 8` `wsrep_sst_method = xtrabackup-v2` ## Logs: Failed DDL voting events (50 in 6.8 seconds) `2026-04-23T10:31:08.108514Z 15 [Warning] [MY-000000] [WSREP] Event 1 Query apply failed: 1, seqno` `1453156` `2026-04-23T10:31:08.262998Z 11 [Warning] [MY-000000] [WSREP] Event 1 Query apply failed: 1, seqno` `1453159` `2026-04-23T10:31:08.377533Z 15 [Warning] [MY-000000] [WSREP] Event 1 Query apply failed: 1, seqno` `1453162` `… (continuing every ~130ms)` `2026-04-23T10:31:14.890197Z 2 [Warning] [MY-000000] [WSREP] Event 1 Query apply failed: 1, seqno` `1453343` Single voting round (representative example) `2026-04-23T10:31:13.110814Z 15 [Warning] [MY-000000] [WSREP] Event 1 Query apply failed: 1, seqno` `1453302` `2026-04-23T10:31:13.112502Z 0 [Note] [MY-000000] [Galera] Member 0(node-2) initiates vote on` `:1453302,94ebcd795f69ea70: Error in list of partitions to DROP, Error_code: 1507;` `2026-04-23T10:31:13.112627Z 0 [Note] [MY-000000] [Galera] Recomputed vote based on error codes: 1507.` `New vote 885fc42092da4791 will be used for further steps. Old Vote: 94ebcd795f69ea70` `2026-04-23T10:31:13.112650Z 0 [Note] [MY-000000] [Galera] Member 2(node-1) initiates vote on` `:1453302,94ebcd795f69ea70: Error in list of partitions to DROP, Error_code: 1507;` `2026-04-23T10:31:13.112745Z 0 [Note] [MY-000000] [Galera] Votes over :1453302:` `885fc42092da4791: 2/3` `Winner: 885fc42092da4791` Pending TOI DDL stuck for 27,517 seconds (from INFORMATION\_SCHEMA.PROCESSLIST) ```auto Id Time State Info 2264721 27517s wsrep: preparing for TO isolation ALTER TABLE other_table REORGANIZE PARTITION om_max INTO (…) 2465436 24350s wsrep: preparing for TO isolation TRUNCATE TABLE foo /* … / 2465437 24350s wsrep: preparing for TO isolation TRUNCATE TABLE foo / … / 2465438 24350s wsrep: preparing for TO isolation TRUNCATE TABLE foo / … */ … (348 client connections in ‘wsrep: replicating and certifying write set’ state) ``` ```auto SHOW STATUS LIKE ‘wsrep_%’ during the stall ┌────────────────────────────┬────────────────┬──────────────────┬──────────────────┐ │ Variable │ writer (pxc-0) │ receiver (pxc-1) │ receiver (pxc-2) │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_cluster_status │ Primary │ Primary │ Primary │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_cluster_size │ 3 │ 3 │ 3 │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_local_state_comment │ Synced │ Synced │ Synced │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_last_committed │ 1453533 │ 1453350 │ 1453350 │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_replicated │ 1,453,505 │ 0 │ 0 │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_received │ 27,302 │ 1,480,747 │ 1,480,741 │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_flow_control_paused │ 0.33 → 0.62 │ 0.33 → 0.62 │ 0.33 → 0.62 │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_flow_control_sent │ 0 │ 57 │ 31 │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_flow_control_recv │ 88 │ 88 │ 88 │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_local_recv_queue_avg │ 0.003 │ 0.26 │ 0.81 │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_apply_window │ 2.01 │ 1.58 │ 1.65 │ ├────────────────────────────┼────────────────┼──────────────────┼──────────────────┤ │ wsrep_cert_deps_distance │ 1.99 │ 1.99 │ 1.99 │ └────────────────────────────┴────────────────┴──────────────────┴──────────────────┘ ``` `wsrep_flow_control_paused = 0.62` after ~26 hours indicates the cluster has been 100% paused since the stall began (cumulative average since metric reset). The receivers were stuck only 7 seqnos past the last failed DROP (seqno 1453343 → frozen at 1453350) for the entire 26-hour duration. ## Expected Result: Cluster shouldn’t go to Stall state permanently, it can probably be stalled for sometime may be, but should heal in sometime ## Actual Result: - Each failed DDL triggered a 3-node voting round consuming a global seqno and ~100-150ms of cluster wall time - 50 voting rounds in 6.8 seconds overwhelmed receiver applier throughput - A subsequent legitimate TOI DDL got stuck in wsrep: preparing for TO isolation indefinitely (27,517+ seconds observed) - All client writes on the writer accumulated in wsrep: replicating and certifying write set state behind the pending TOI (repl.commit\_order = 3 enforces commit ordering) - Receivers sent flow control to writer; writer could not release pending TOI until receivers caught up; receivers had no new writesets to process because writer was blocked → permanent deadlock - After ~26 hours, MySQL on the writer became completely unresponsive (could not answer SELECT 1) - No self-recovery occurred. Manual recovery required deleting all 3 PXC pods to force a bootstrap from the most-advanced node’s PVC ## Additional Information: Why this matters The same code path on standalone MySQL is completely benign: the application catches Error 1507 and continues, with zero impact on other connections or future operations. The pattern of “issue a DDL, catch the error if it fails, move on” is common in MySQL-targeted application code. On Galera/PXC, the same code path can stall the entire cluster — but only when the failed DDL pattern coincides with sufficient receiver load (table data + concurrent DMLs). This makes the bug latent: it can lurk in production for months and only surface when conditions converge. In our case, the application code had been running this pattern across our cluster for 3+ months before the deadlock manifested. Application-side workaround we deployed Check partition existence before issuing the DROP DDL, so the failed DDL never enters Galera replication: # Before `db.execute(“ALTER TABLE t DROP PARTITION old_max”) # may fail with 1507, app catches it` # After ```auto if partition_exists(db, “t”, “old_max”): db.execute(“ALTER TABLE t DROP PARTITION old_max”) ``` This eliminates the voting events entirely. We also audited all “catch DDL error and continue” patterns in our codebase, since these are silent on standalone MySQL but trigger cluster-wide consensus on Galera. Recovery procedure that worked 1. Delete all 3 PXC pods (operator-managed cluster) 2. Operator ran `wsrep_recover` on each PVC to determine local seqno via InnoDB redo log replay 3. Operator bootstrapped from the most-advanced node (the pre-stall writer, seqno 1453533 — vs receivers at 1453350) 4. Other two nodes joined via SST (xtrabackup-v2, ~25 GiB each at ~180 MiB/s) 5. Total recovery time: ~20 minutes, no committed data lost (committed-but-not-applied writesets on the writer were preserved because we bootstrapped from its PVC) Questions 1. Is there anything wrong we are doing, or something we can do so that we avoid such issus in the future? 2. Are there existing bug reports tracking this scenario? Happy to share additional logs (full mysqld.log, gcache state, anything else useful). ---
**Author:** ![reddy\_nishanth](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/reddy_nishanth/32/21004_2.png) [@reddy\_nishanth](https://forums.percona.com/u/reddy_nishanth)\ **Post date:** [May 6, 2026, 6:34pm UTC](https://forums.percona.com/t/galera-pxc-cluster-permanent-stall-failed-ddl-voting-storm-pending-toi-deadlock/40709/2 "2026-05-06T18:34:29Z")
@Ege_Gunes Can you please take a look at this? ---
**Author:** ![Ege\_Gunes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ege_gunes/32/4595_2.png) [@Ege\_Gunes](https://forums.percona.com/u/Ege_Gunes)\ **Post date:** [May 15, 2026, 6:47am UTC](https://forums.percona.com/t/galera-pxc-cluster-permanent-stall-failed-ddl-voting-storm-pending-toi-deadlock/40709/3 "2026-05-15T06:47:40Z")
@reddy_nishanth I’ll take a look next week ---
**Author:** ![Ege\_Gunes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ege_gunes/32/4595_2.png) [@Ege\_Gunes](https://forums.percona.com/u/Ege_Gunes)\ **Post date:** [May 21, 2026, 7:00am UTC](https://forums.percona.com/t/galera-pxc-cluster-permanent-stall-failed-ddl-voting-storm-pending-toi-deadlock/40709/4 "2026-05-21T07:00:23Z")
@reddy_nishanth as far as I see operator is handled its side okay. This is more like a PXC question so I’ll defer to experts. @matthewb could you please take a look? ---
**Author:** ![TableWorker](https://avatars.discourse-cdn.com/v4/letter/t/53a042/32.png) [@TableWorker](https://forums.percona.com/u/TableWorker)\ **Post date:** [May 21, 2026, 6:55pm UTC](https://forums.percona.com/t/galera-pxc-cluster-permanent-stall-failed-ddl-voting-storm-pending-toi-deadlock/40709/5 "2026-05-21T18:55:07Z")
I think we just hit the same bug today. I need to do further debug with the information from the first post and try to replicate the issue, but just reading it seems like it exactly follows our scenario. We don’t have partitioned table, but the code was running ALTER … ADD INDEX queries and were getting error 1072 (Key column ‘\__non\_\_existent\_\_column\_\_name_\_’ doesn’t exist in table). After some time, the vote didn’t finish and nodes went to deadlock. We also saw “preparing for TO isolation” state for the client running the query and the rest of the clients in either “wsrep: replicating and certifying write set” or “wsrep: write-set replicated and certified”. Will do more test and report back. ---
**Author:** ![TableWorker](https://avatars.discourse-cdn.com/v4/letter/t/53a042/32.png) [@TableWorker](https://forums.percona.com/u/TableWorker)\ **Post date:** [July 7, 2026, 6:07am UTC](https://forums.percona.com/t/galera-pxc-cluster-permanent-stall-failed-ddl-voting-storm-pending-toi-deadlock/40709/6 "2026-07-07T06:07:24Z")
Is anybody looking into this by any chance or is this something that won’t be fixed?