Since upgrading our Percona XtraDB Cluster to 8.0.46-38, one node remains stuck in Donor/Desynced after a mydumper backup.
This backup setup had been working for years before the upgrade.
Our original backup script was:
mysql --defaults-extra-file="${CREDFILE}" \
--execute "SET GLOBAL wsrep_desync = ON"
mydumper \
--defaults-file="${CREDFILE}" \
--use-savepoints \
-t 2 \
-c ZSTD \
-o "${DESTDIR}/0"
mysql --defaults-extra-file="${CREDFILE}" \
--execute "SET GLOBAL wsrep_desync = OFF"
After the backup, wsrep_desync is correctly set to OFF, but the node does not return to Synced.
The status then looks like this:
wsrep_desync OFF
wsrep_desync_count 1
wsrep_local_state 2
wsrep_local_state_comment Donor/Desynced
wsrep_local_recv_queue 0
wsrep_connected ON
wsrep_ready ON
Running SET GLOBAL wsrep_desync = OFF again only gives: wsrep_desync is already OFF. The node only returns to Synced after restarting MySQL.
We tested a few different scenarios.
First, we tested wsrep_desync without mydumper:
SET GLOBAL wsrep_desync = ON;
-- wait about 10 minutes
SET GLOBAL wsrep_desync = OFF;
That works normally. wsrep_desync_count returns to 0 and the node becomes Synced again. With the original backup script, the values change like this:
Before backup:
wsrep_desync=0
wsrep_desync_count=0
Synced
During mydumper:
wsrep_desync=1
wsrep_desync_count=2
Donor/Desynced
After SET GLOBAL wsrep_desync=OFF:
wsrep_desync=0
wsrep_desync_count=1
Donor/Desynced
We then ran mydumper without setting wsrep_desync ourselves. The result was:
Before mydumper:
wsrep_desync=0
wsrep_desync_count=0
Synced
After mydumper started:
wsrep_desync=0
wsrep_desync_count=1
Donor/Desynced
The counter stayed at 1 after mydumper completed.
So mydumper alone is enough to trigger the problem. The explicit wsrep_desync commands are not the cause. We first reproduced this with: mydumper v0.16.1-3. We then upgraded to the latest mydumper version, but the result was the same.
The problem does not occur when using:
--sync-thread-lock-mode=SAFE_NO_LOCK
--no-backup-locks
It also does not occur with:
--sync-thread-lock-mode=GTID
--no-backup-locks
With both settings, the backup completes and the node returns to:
wsrep_desync=OFF
wsrep_desync_count=0
wsrep_local_state_comment=Synced
At this point, it looks like the issue is in the interaction between PXC 8.0.46-38 and the default lock path selected by mydumper with sync-thread-lock-mode=FTWRL.
My assumption is that mydumper’s use of FTWRL causes PXC to create an internal desync reference. After the lock is released, that reference is not removed, leaving the node stuck with:
wsrep_desync=OFF
wsrep_desync_count=1
Donor/Desynced
The issue appeared immediately after upgrading to PXC 8.0.46-38. Since that release includes changes to FTWRL handling on backup nodes, including PXC-4799, it seems possible that the change is related.
Has anyone else seen this with PXC 8.0.46-38?