Restore of Xtrabackup from standalone database to PXC on Kubernetes fails

MySQL & MariaDB Percona Operator for MySQL
1

Description:

We are in the process of moving from a Proxmox cluster to Kubernetes and need to transfer data from a Percona MySQL Server 8.0 to the Percona Operator for MySQL XtraDBCluster. I’m following this documentation.

We successfully created an xtrabackup and saved it to S3 storage using the following command

xtrabackup --defaults-extra-file=/etc/mysql/conf.d/admin.conf --backup --stream=xbstream --extra-lsndir=/tmp --target-dir=/tmp | xbcloud put --storage=s3 --s3-endpoint='[[EDITED]]' --s3-access-key=[[EDITED]] --s3-secret-key=[[EDITED]] --s3-bucket='[[EDITED]]' --md5 --parallel=10 `date +%Y%m%d-%H%M`-full_backup

but the restore job fails. Looks like a file ‘.sst_info’ is missing. The restore job is very straightforward:

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBClusterRestore
metadata:
  name: restore-stage
spec:
  pxcCluster: percona-cluster-pxc-d
  backupSource:
    destination: s3://ts-protected-dev/20250821-1539-full_backup
    s3:
      bucket: ts-protected-dev
      credentialsSecret: [[EDITED]]
      endpointUrl: "[[EDITED]]"
      region: "eu-central-1"

The job pods start successfully and have access to the S3 bucket

+ aws s3 ls ts-protected-dev/20250821-1539-full_backup
                           PRE 20250821-1539-full_backup/
2025-08-21 13:44:12     284139 20250821-1539-full_backup.md5

But as one can see in the file listings an .md5 file is available, but not .sst_info file. This results in the following error:

+ xbcloud get --parallel=2 --curl-retriable-errors=7 s3://ts-protected-dev/20250821-1539-full_backup.sst_info
250821 14:23:44 xbcloud: Successfully connected.
250821 14:23:44 xbcloud: Download failed. Cannot list 20250821-1539-full_backup.sst_info.
++ parse_ini mysql-version /datadir/pxc_sst_2sNi/sst_info
++ local key=mysql-version
++ local file_path=/datadir/pxc_sst_2sNi/sst_info
++ awk -F '=[ ]*' '/mysql-version[ ]*=/ {print $2}' /datadir/pxc_sst_2sNi/sst_info
awk: fatal: cannot open file `/datadir/pxc_sst_2sNi/sst_info' for reading: No such file or directory
+ MYSQL_VERSION=

After this error the pod exits and a new pod retries the restore, but all of them fail with the same problem.

What is the expected behaviour? Should the process continue even if no .sst_info file is available or do we have to modify the backup command so a .sst_file will be created?

Steps to Reproduce:

Tried several times, always the same result

Version:

Percona MySQL Server 8.0.42-33-1.focal / Xtrabackup 8.0.35-31 on Proxmox virtual machine

Logs:

see description

2

Hi @letsmakesense , this file is needed to detect the MySQL version of the backup. Please modify the backup command. Thanks.

3

Hi @Slava_Sarzhan thanks for your reply! It looks like the ‘–galera-info’ options creates this file. But this option does not work on a standalone Percona MySQL server. Would it be possible to prepare the backup, save it on a volume used by the PVC of the first node of a new Percona Operator XtraDB Cluster and bootstrap the node with this prepared volume?

4

@letsmakesense I have created a test branch Allow restore without sst_info by hors · Pull Request #2182 · percona/percona-xtradb-cluster-operator · GitHub and an image perconalab/percona-xtradb-cluster-operator:test-without_sst_info for you. It is based on the main branch, so please do not use it for production. Please test it and inform me about the results.

5

Wow great. Thanks for the image. I will try it out the next days. Meanwhile I found a solution by creating a Kubernetes job which downloads the backup from S3, prepares it and copies the files to the PVC of the first node.

apiVersion: batch/v1
kind: Job
metadata:
  name: pxc-prepare-job
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: restore
          image: percona/percona-xtrabackup:8.0
          securityContext:
            runAsUser: 0
          command: ["/bin/sh", "-c"]
          args:
            - |
              set -e
              rm -rf /var/lib/mysql/*
              chown -R 1001:1001 /var/lib/mysql
              echo "=== Downloading backup from S3 ==="
              xbcloud get --storage=s3 \
                --s3-bucket=$BUCKET \
                --s3-endpoint='$ENDPOINT' \
                --s3-access-key=$AWS_ACCESS_KEY_ID \
                --s3-secret-key=$AWS_SECRET_ACCESS_KEY \
                --parallel=10 \
                $BACKUP | xbstream -x -C /var/lib/mysql
              
              echo "=== Preparing backup ==="
              xtrabackup --prepare --target-dir=/var/lib/mysql

              echo "=== Fixing ownership ==="
              chown -R 1001:1001 /var/lib/mysql
              echo "=== Restore complete ==="
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
          env:
			- name: ENDPOINT
			  value: [[YOUR_ENDPOINT]]
			- name: BUCKET
			  value: [[YOUR_BUCKET]]
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: [[YOUR_S3_SECRET]]
                  key: AWS_ACCESS_KEY_ID
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: [[YOUR_S3_SECRET]]
                  key: AWS_SECRET_ACCESS_KEY
            - name: AWS_DEFAULT_REGION
              value: eu-central-1
            - name: BACKUP
              value: "[[YOUR_BACKUP_PREFIX]]"
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: datadir-[[YOUR_CLUSTERNAME]]-pxc-0
1 Like
6

Hey @Slava_Sarzhan
Just posting to say I ran into the exact same issue trying to restore a backup on the latest version of the operator - worth noting that these restores worked just fine on 1.17.1 but updating to 1.18.0 has broken them. Tried to change the image in my image to perconalab/percona-xtradb-cluster-operator:test-without_sst_info but the image is missing aws so it fails before it gets to the step needed

+ LIB_PATH=/opt/percona/backup/lib/pxc
+ . /opt/percona/backup/lib/pxc/check-version.sh
+ . /opt/percona/backup/lib/pxc/vault.sh
++ set -o errexit
++ keyring_vault=/etc/mysql/vault-keyring-secret/keyring_vault.conf
+ . /opt/percona/backup/lib/pxc/aws.sh
++ set -o errexit
++ export AWS_SHARED_CREDENTIALS_FILE=/tmp/aws-credfile
++ AWS_SHARED_CREDENTIALS_FILE=/tmp/aws-credfile
++ export AWS_REGION=us-east-1
++ AWS_REGION=us-east-1
++ export AWS_ENDPOINT_URL=https://s3.us-east-1.amazonaws.com
++ AWS_ENDPOINT_URL=https://s3.us-east-1.amazonaws.com
++ '[' -n true ']'
++ [[ true == \f\a\l\s\e ]]
+ XBCLOUD_ARGS='--curl-retriable-errors=7 '
+ '[' -n true ']'
+ [[ true == \f\a\l\s\e ]]
+ '[' -n <backup> ']'
/opt/percona/backup/lib/pxc/aws.sh: line 29: aws: command not found

So I’m looking into making a copy of that image but with aws installed, but if you release another testing image let me know.