Encryption at rest with Percona MongoDB backups

Hi there,

I am running a 3 member replica set of Percona MongoDB server, deployed by the Percona Kubernetes Operator. I have encryption at rest enabled. I have verified in the MongoDB logs that it is enabled, by checking for the line percona_encryption_extension_init as per WiredTiger Encryption at Rest with Percona Server for MongoDB - Percona Database Performance Blog.

In addition, I am using Percona backup manager, to store backups inside remote s3 storage. If I completely delete my Kubernetes cluster, and create a new cluster, which generates for itself a new mongodb encryption key, I am able to restore my previous backup from my remote storage.

I was expecting this action to not work, due to the encryption keys not matching.

My question - are the data files stored by Percona backup manager in S3 unencrypted? Or am I misunderstanding what is happening here?

I would expect this action to only work if my new cluster had the same encryption key.

Thank you

1 Like

Hi Kylem.

The Percona Kubernetes Operator uses Percona Backup for MongoDB (PBM) as the backup mechanism. The files created like PBM are like those created by mongodump in that they are BSON files exported from each replica set. Compression and an archive file format are also involved, but not encryption.

Data-at-rest encryption in Percona Server for MongoDB (PSMDB) and hence of the K8s operator for it is of the WiredTiger files in the mongod’s data directory. It doesn’t apply to the export of collection documents and oplog as BSON.

If K8s operator made backups by copying the underlying WT files then it would have gone the way you expected, i.e. fail to restart due to not having the original key to decrypt upon opening.

As long as the K8s operator for PSMDB exports already-decrypted BSON records it will be this way. There is the potential that K8s operator be changed in future development to use PSMDB’s hot backup instead of PBM. Cluster-consistent hot backups are still a new feature in PSMDB that I’m awaiting community feedback on though. (Experimental Feature: $backupCursorExtend in Percona Server for MongoDB - Percona Database Performance Blog)

This is not to say you can’t have encrypted backup files - S3 encryption can be used. But tha is about the options of the remote store, not PBM itself (and hence the K8s operator, or current generation at least). See “serverSideEncryption” in Initial setup.

1 Like

Hi Akira,

Thank you for the explanation - I understand it now.

I will look into a solution at the remote storage end.

Thank you very much

1 Like

Cool :+1:

Happy trucking Kyle.

1 Like