In addition, I am using Percona backup manager, to store backups inside remote s3 storage. If I completely delete my Kubernetes cluster, and create a new cluster, which generates for itself a new mongodb encryption key, I am able to restore my previous backup from my remote storage.
I was expecting this action to not work, due to the encryption keys not matching.
My question - are the data files stored by Percona backup manager in S3 unencrypted? Or am I misunderstanding what is happening here?
I would expect this action to only work if my new cluster had the same encryption key.
The Percona Kubernetes Operator uses Percona Backup for MongoDB (PBM) as the backup mechanism. The files created like PBM are like those created by mongodump in that they are BSON files exported from each replica set. Compression and an archive file format are also involved, but not encryption.
Data-at-rest encryption in Percona Server for MongoDB (PSMDB) and hence of the K8s operator for it is of the WiredTiger files in the mongod’s data directory. It doesn’t apply to the export of collection documents and oplog as BSON.
If K8s operator made backups by copying the underlying WT files then it would have gone the way you expected, i.e. fail to restart due to not having the original key to decrypt upon opening.
As long as the K8s operator for PSMDB exports already-decrypted BSON records it will be this way. There is the potential that K8s operator be changed in future development to use PSMDB’s hot backup instead of PBM. Cluster-consistent hot backups are still a new feature in PSMDB that I’m awaiting community feedback on though. (Experimental Feature: $backupCursorExtend in Percona Server for MongoDB - Percona Database Performance Blog)
This is not to say you can’t have encrypted backup files - S3 encryption can be used. But tha is about the options of the remote store, not PBM itself (and hence the K8s operator, or current generation at least). See “serverSideEncryption” in Set up and configure - Percona Backup for MongoDB.
