Encrypted Backup and Restore is failing for PXC 8.0.32-24.2

MySQL & MariaDB Percona XtraDB Cluster 8.x
1

I’ve been trying Backup and Restore for encrypted InnoDB tables using auto generated transition keys. I’m using hashicorp vault as a keying store.

I got it working fine for the following setup
PERCONA_OPERATOR_VERSION = ‘1.19.0’
PERCONA_CLUSTER_VERSION = ‘8.0.44-35.1’
XTRABACKUP_VERSION = ‘8.0.35-34.1’

But when I try the same with cluster version 8.0.32-24.2 and keeping everything else same, then while restoring I get the following error
Error reading xtrabackup_keys: failed to decrypt key and iv for tablespace 83886080. Wrong transition key?

xtrabackup --use-memory=100MB --prepare --transition-key=EGSEyNUf2uTywGndZlLH4um3xN9zdKjk --rollback-prepared-trx --xtrabackup-plugin-dir=/usr/lib64/xtrabackup/plugin --target-dir=/datadir/pxc_sst_zIuY
2026-02-17T13:48:13.149567-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --innodb_checksum_algorithm=crc32 --innodb_log_checksums=1 --innodb_data_file_path=ibdata1:12M:autoextend --innodb_log_file_size=50331648 --innodb_page_size=16384 --innodb_undo_directory=./ --innodb_undo_tablespaces=2 --server-id=27962852 --innodb_log_checksums=ON --innodb_redo_log_encrypt=1 --innodb_undo_log_encrypt=1
2026-02-17T13:48:13.149630-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --use-memory=100MB --prepare=1 --transition-key=* --transition-key=* --rollback-prepared-trx=1 --xtrabackup-plugin-dir=/usr/lib64/xtrabackup/plugin --target-dir=/datadir/pxc_sst_zIuY
xtrabackup version 8.0.35-34 based on MySQL server 8.0.35 Linux (x86_64) (revision id: c8a25ff9)
2026-02-17T13:48:13.149653-00:00 0 [Note] [MY-011825] [Xtrabackup] cd to /datadir/pxc_sst_zIuY/
2026-02-17T13:48:13.149699-00:00 0 [Note] [MY-011825] [Xtrabackup] This target seems to be not prepared yet.
2026-02-17T13:48:13.157656-00:00 0 [Note] [MY-011825] [Xtrabackup] xtrabackup_logfile detected: size=8388608, start_lsn=(32621804)
2026-02-17T13:48:13.158488-00:00 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration for recovery:
2026-02-17T13:48:13.158504-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_home_dir = .
2026-02-17T13:48:13.158512-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_file_path = ibdata1:12M:autoextend
2026-02-17T13:48:13.158538-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_group_home_dir = .
2026-02-17T13:48:13.158544-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_files_in_group = 1
2026-02-17T13:48:13.158549-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_file_size = 8388608
2026-02-17T13:48:13.158619-00:00 0 [Note] [MY-011825] [Xtrabackup] Loading xtrabackup_keys
2026-02-17T13:48:13.173211-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Error reading xtrabackup_keys: failed to decrypt key and iv for tablespace 83886080. Wrong transition key?
2026-02-17T13:48:13.173253-00:00 0 [ERROR] [MY-011825] [Xtrabackup] failed to load tablespace keys

I went through the code and the percona-xtradb-cluster/scripts/wsrep_sst_xtrabackup-v2.sh at 8.0 · percona/percona-xtradb-cluster · GitHub and couldn’t find any significant difference between the 2 versions, could you help me find the issue with 8.0.32-24.2 version.
Thanks!

2

Can you give some step-by-step commands that you are using so I can attempt to reproduce in my cluster?

3

Hi @Yash_Daga,

I dug into the XtraBackup source and reproduced the exact error. The root cause is a binary format incompatibility in the xtrabackup_keys file between PXB 8.0.32 and PXB 8.0.35.

In PXB 8.0.32, a sizeof(char*) bug made the transition key name field 77 bytes instead of 74 in the xtrabackup_keys file. This was silently fixed in PXB 8.0.35-30 (PXB-3139), but both versions write the same KEYSV02 header, so the reader has no way to detect which layout was used.

In the pxc-operator’s default backup flow, the PXC 8.0.32 donor uses its bundled PXB 8.0.32-26 (77-byte field), but the restore job uses spec.backup.image which the version service auto-assigns to PXB 8.0.35-34.1 for PXC 8.0 clusters (expects 74). The 3-byte misalignment shifts all record reads, so space_id 5 (0x00000005) gets read as 83886080 (0x05000000), and the CRC32 check fails.

I confirmed this by backing up PXC 8.0.32 with PXB 8.0.32-26 and preparing with PXB 8.0.35-34, which reproduced the exact error:

Error reading xtrabackup_keys: failed to decrypt key and iv
for tablespace 83886080. Wrong transition key?

Workaround: pin spec.backup.image to match the donor’s PXB version. In your deploy/cr.yaml:

spec:
  upgradeOptions:
    apply: Disabled          # prevent version service from overriding
  backup:
    image: percona/percona-xtrabackup:8.0.32-26   # match PXC 8.0.32 bundled PXB

If you later upgrade PXC, update spec.backup.image to match the new bundled PXB version, otherwise backups will break again for the same reason.

Alternatively, upgrade PXC to 8.0.35+. PXC 8.0.35 and later bundle PXB with the corrected 74-byte field size, matching the operator’s default backup image. This is the cleaner long-term fix.

References:

4

Ohh this makes sense, I didn’t realise that the the cluster uses it’s bundled PXB image.

Thanks a lot for your investigation!

1 Like