Encrypted backup configuration in PXC Cluster Resource

MySQL & MariaDB Percona Operator for MySQL
1

Hi, I am attempting to confiugure encrypted backups for a Percona cluster using the PXC operator.

After scouring the documentation I came up with an approach that performs backups to S3, but I am not sure if encryption is actually enabled.

Can anybody confirm if this approach should work?

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  name: cluster
  namespace: percona
  finalizers:
  - delete-pxc-pods-in-order
spec:
  # ...
  backup:
    image: percona/percona-xtradb-cluster-operator:1.14.0-pxc8.0.36-backup-pxb8.0.35
    backoffLimit: 6
    storages:
      backup:
        type: s3
        containerOptions:
          env:
          - name: ENCRYPT_KEY
            valueFrom:
              secretKeyRef:
                name: backup-env
                key: ENCRYPTION_KEY
          args:
            xtrabackup:
            - "--encrypt=AES256"
            - "--encrypt-key=$ENCRYPT_KEY"

Is the configured env variable actutally passed to the backup tool or will it use the literal $ENCRYPT_KEY as key for encryption?

2

This does not seem to work. I also tried to configure encryption on xbstream and the log output of the backup job shows that the passed parameter is apparently quoted using ', so the shell will not perform any replacements.

xbstream -x -C /tmp '--encrypt-key=$ENCRYPT_KEY'
3

@jens_hausherr right now encrypted backups are not supported in the Operator.

Your thought process is correct, but there are other variables in the air that makes it really a complex process within the currect backup architecture.

As a workaround we recommend encryption on the bucket level.

4

Thank you for the quick response, @Sergey_Pronin

I hoped for it to work, but at least I do not have to search any longer .

Are there plans to implement encrypted backups in the future?

5

There are, we have it in our roadmap.
But to implement it within current architecture is a huge effort, so it is hard for me to share any ETA.

@jens_hausherr will you be open to collaborate on it? We are fully open source, so if you have any ideas or want to contribute code - let’s do this.

6

I am happy to look into it. Might not make it to the top of my stack immediately, though.

Do you have a few pointers what the known obstacles are to integrating encrypted backups.

My first naïve understanding of the code is that backups are basically k8s jobs that run a shell script in the backup image, so passing the key should be straightforward.

I admit that I have not looked into restoring yet, so this still might complicate things.

My preferred solution would also require more work, because passing keys as environment is not really safe.

I would rather prefer mounting the secret as volume and using the key file approach, but this requires changing the CRD/Operator as currently no extra volumes can be specified in the PXC/Backup/Restore CRs IIRC.

Please let me know if there are any constraints that have to be taken into account. Few things are more frustrating than pouring time into code only to learn in the PR discussion that the solution is not acceptable :slight_smile:

7

Mounting the secret is easy.
The bigger problem is that in the Operator we use garbd (Galera arbitrator) to take backups and SST (State Snapshot Transfer). So the reality is that it is not really a pure xtrabackup. If it was, it would be just adding a secret key and a flag.