Cannot specify custom --transition-key for XtraBackup in Percona PXC Cluster

Yash_Daga · January 29, 2026, 12:12pm

Hi, I’m using Percona Operator 1.19.0 with PXC 8.0 and trying to use --transition-key for InnoDB encrypted backups. but while trying to restore using the same transition key I’m getting “Wrong transition key?“ error.

Backup CR config
spec:
pxcCluster: mysqlcluster
storageName: minio-s3
containerOptions:
args:
xtrabackup:
- “–transition-key=MySecretKey123”

From the backup pod logs, I see that it runs SST on donor node via garbd and not run xtrabackup and it seems that it doesn’t pass the transition-key arg.

I wanted to ask if i’m doing something incorrectly, how can I use the transition key to take backup of encrypted tables?

Thanks!
@matthewb

matthewb · January 29, 2026, 8:41pm

Native encryption: Built-in support for encrypted backups with proper key management. This functionality is not yet available in version 1.19.0 but will be added in future releases.

Two things: 1) you must enable the XtrabackupSidecar feature gate in the operator, otherwise the backup method remains the default SST. 2) Wait until the feature for encryption is added to the operator.

Yash_Daga · January 30, 2026, 5:41am

Hi @matthewb thanks a lot for your response.

I wanted to ask is there a recommended way (or even a hacky way) to backup and restore encrypted tables currently?

My setup:
Percona Operator 1.19.0 with PXC 8.0 on multi node cluster.
I’m using component_keyring_file for encryption and I’m mounting the file as an extra PVC which is on rwx file system shared by all pxc pods.

I initially tried without using a transition key and let the backup encrypted by one of the existing pod’s master key. While restoring I added the keys to the new cluster’s keyring-file but that restoration failed with
2026-01-28T05:07:06.494451-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Error reading xtrabackup_keys: failed to derive encryption key
2026-01-28T05:07:06.494502-00:00 0 [ERROR] [MY-011825] [Xtrabackup] failed to load tablespace keys

I’m guessing because the restore job pods didn’t have the keyring file mounted, but I was not sure how can I mount it and also ensure that component_keyring_file config is also present there.

Yash_Daga · January 30, 2026, 9:06am

Is it possible to add extraPVC feature to the Restore pod, similar to the pxc pods, so we can use it to mount the keyring file containing old cluster’s keys and then use the --component-keyring-config xtrabackup arg to mention the keyring config? Do you see any issue with this approach or is there any simpler way?

I can help by raising a PR for this.

matthewb · January 30, 2026, 3:22pm

HI @Yash_Daga,
In reading the code for the operator, xtrabackup is already executed using --transition-key, however, this only works when using component-keyring-vault and is not supported with keyring-file.

Thus, if you want to use encrypted tables, and be able to backup/restore those encrypted tables, then you need to migrate to using Vault.

Yash_Daga · February 2, 2026, 3:25am

Thanks for the response, @matthewb.

I see that the generated transistion key is stored, in the sst_info file inside the backup itself.

I tried restoring using that, and the operator was able to decrypt the xtrabackup keys file, but failed at the –generate-new-master key step since the restore pod didn’t have any keyring file mounted. So if we could just mount the extraPVC(which stores our keyring file) of the cluster pod to the restore pod, we might be able to restore without using vault.

Or we could also maybe allow an argument to stop the operator from trying to generate a new master key?

matthewb · February 2, 2026, 5:46pm

github.com/percona/percona-xtradb-cluster-operator

build/backup/recovery-cloud.sh

204c90437


      
          		# If backup-my.cnf does not contain plugin_load, then --prepare will fail if you pass the --keyring-vault-config option.
          		if [[ -n "$(parse_ini 'plugin_load' "${tmp}/backup-my.cnf")" ]] && [[ -z $transition_key ]]; then
          			PXB_VAULT_PREPARE_ARGS="--keyring-vault-config=${VAULT_CONFIG_FILE}"
          		fi
          	fi
          fi
          
          
          
          if [ -f "${tmp}/xtrabackup_keys" ]; then
          	master_key_options="--generate-new-master-key"
          fi
          
          # Extract --defaults-file from XB_EXTRA_ARGS if present and place it as the first argument
          # This fixes the issue where --defaults-file must be the first argument for xtrabackup and innobackupex
          DEFAULTS_FILE=""
          REMAINING_XB_ARGS=""
          if [[ "$XB_EXTRA_ARGS" =~ --defaults-file=([^[:space:]]+) ]]; then
          	defaults_file_path="${BASH_REMATCH[1]}"
          	# If the path is relative (doesn't start with /), prepend $tmp directory
          	if [[ "$defaults_file_path" != /* ]]; then

I don’t see an option to override that, or an option for mounting additional PVC into the restore pod.

Yash_Daga · February 3, 2026, 3:20am

Thanks for your help @matthewb
Would it be fine if I raised a PR for that?

Also I was trying to perform backup+restore in setup with a vault I was still getting wrong transition key error.

260202 14:53:45 xbcloud: Download completed.

set +o xtrace
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100   217  100   217    0     0  21700      0 --:–:-- --:–:-- --:–:-- 21700
transition-key exists

xtrabackup  --use-memory=100MB --prepare 	 --rollback-prepared-trx  --xtrabackup-plugin-dir=/usr/lib64/xtrabackup/plugin 	--target-dir=/datadir/pxc_sst_Q6K2
2026-02-02T14:53:45.577442-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --innodb_checksum_algorithm=crc32 --innodb_log_checksums=1 --innodb_data_file_path=ibdata1:12M:autoextend --innodb_log_file_size=50331648 --innodb_page_size=16384 --innodb_undo_directory=./ --innodb_undo_tablespaces=2 --server-id=32648292 --innodb_log_checksums=ON --innodb_redo_log_encrypt=1 --innodb_undo_log_encrypt=1
2026-02-02T14:53:45.577534-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --use-memory=100MB --prepare=1 --transition-key=* --rollback-prepared-trx=1 --xtrabackup-plugin-dir=/usr/lib64/xtrabackup/plugin --target-dir=/datadir/pxc_sst_Q6K2
xtrabackup version 8.0.35-34 based on MySQL server 8.0.35 Linux (x86_64) (revision id: c8a25ff9)
2026-02-02T14:53:45.577567-00:00 0 [Note] [MY-011825] [Xtrabackup] cd to /datadir/pxc_sst_Q6K2/
2026-02-02T14:53:45.577620-00:00 0 [Note] [MY-011825] [Xtrabackup] This target seems to be not prepared yet.
2026-02-02T14:53:45.623422-00:00 0 [Note] [MY-011825] [Xtrabackup] xtrabackup_logfile detected: size=8388608, start_lsn=(32596787)
2026-02-02T14:53:45.624170-00:00 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration for recovery:
2026-02-02T14:53:45.624188-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_home_dir = .
2026-02-02T14:53:45.624193-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_file_path = ibdata1:12M:autoextend
2026-02-02T14:53:45.624226-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_group_home_dir = .
2026-02-02T14:53:45.624230-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_files_in_group = 1
2026-02-02T14:53:45.624238-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_file_size = 8388608
2026-02-02T14:53:45.624443-00:00 0 [Note] [MY-011825] [Xtrabackup] Loading xtrabackup_keys
2026-02-02T14:53:45.645169-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Error reading xtrabackup_keys: failed to decrypt key and iv for tablespace 83886080. Wrong transition key?
2026-02-02T14:53:45.645243-00:00 0 [ERROR] [MY-011825] [Xtrabackup] failed to load tablespace keys

Even though the transition was stored inside the vault. It would be a great help if you could take a look. Thanks.

matthewb · February 4, 2026, 1:48am

For the feature of being able to mount additional PVC during restore, that does seem reasonable to me.

I don’t think the transition key is stored in vault; that’s in the sst_info. It’s the InnoDB master key which goes into vault, and a new one is generated on restore. After you configured Vault, did you reconfigure the cluster so that PXC started using that key as well?

Yash_Daga · February 5, 2026, 5:17am

From the code it seems that if vault was configured in the cluster where we take backup the transistion key is deleted from the sst_info file and stored into the vault.

github.com/percona/percona-xtradb-cluster-operator

build/backup/lib/pxc/vault.sh

221aaba40


      
          		return 0
          	fi
          
          	export VAULT_TOKEN=$(get_vault_option "token")
          	export VAULT_ADDR=$(get_vault_option "vault_url")
          	local vault_root=$(get_vault_option "secret_mount_point")/backup
          	local ca_path=$(get_vault_option "vault_ca")
          
          	local gtid=$(parse_ini "galera-gtid" "${sst_info}")
          
          	curl ${ca_path:+--cacert $ca_path} \
          		-X PUT \
          		-H "X-Vault-Request: true" \
          		-H "X-Vault-Token: ${VAULT_TOKEN}" \
          		-H "Content-Type: application/json" \
          		-d "{\"transition_key\":\"${transition_key}\"}" \
          		"${VAULT_ADDR}/v1/${vault_root}/${gtid}"
          
          	set -o xtrace
          	sed -i '/transition-key/d' "$sst_info" >/dev/null
          }

I also verified this in the backup I took.

Yes, i redeployed the cluster after configuring vault.

@matthewb @Ege_Gunes

Ege_Gunes · February 5, 2026, 5:58am

@Yash_Daga We recently created K8SPXC-1798 to mount extraPVCs into restore container. If you are willing to work on it, your PR is welcome.

Yash_Daga · February 5, 2026, 6:04am

Hi sure if we end up using keyring component on a extra pvc, I can help by raising a PR.

But I was also trying to perform backup and restore of encrypted tables using vault, which is supported by the operator on pxc 8.0 but I keep getting the error I mentioned above, could you help me in debugging that, thanks!

Ege_Gunes · February 5, 2026, 6:13am

OK, few questions:

Are you using XtrabackupSidecar?
Are you setting a custom transition key in Vault before restore?

Yash_Daga · February 5, 2026, 6:20am

No, I’m using the default method used by PerconaXtraDBClusterBackup(SST).
No, I’m not passing any transistion key the operator is storing the transistion inside the vault during the backup and while restoring on the same cluster it successfully reads it from the same vault(verified it by using custom image with logging).

@Ege_Gunes

Yash_Daga · February 8, 2026, 12:40pm

Updating the following versions fixed the issue (Wrong transistion key error while trying to restore):

operator 1.17.0 → 1.19.0
cluster 8.0.32-24.2 → 8.0.44-35.1
xtrabackup 1.17.0-pxc8.0-backup-pxb8.0.35 → 8.0.35-34.1

Do you know what could be the issue earlier?
@matthewb @Ege_Gunes

Topic		Replies	Views
Problem when trying creating backup in encrypted database Percona XtraBackup percona	7	1595	September 10, 2021
Encrypted Backup and Restore is failing for PXC 8.0.32-24.2 Percona XtraDB Cluster 8.x percona	3	27	February 22, 2026
How to mount keyring using extraPVC in percona 1.19.0 Percona Operator for MySQL	5	342	February 5, 2026
"Observed a panic in reconciler" on the backup restore request [perconaxtradbclusterrestore.pxc.percona.com] Percona Operator for MySQL mysql , percona , kubernetes	7	599	January 13, 2025
Percona XtraDB Cluster on Kubernetes mysql operator (Point-In-Time Recovery) not happening Percona Operator for MySQL community , troubleshooting , mysql , percona	13	2831	April 18, 2022

Cannot specify custom --transition-key for XtraBackup in Percona PXC Cluster

Related topics