Percona K8s Operator disaster recovery

I’ve been struggling with disaster recovery of PerconaMongo operator running on EKS (v1.14). My operator creates data volumes using PVC that are EBS backed. I need a plan for disaster recovery. For instance, if the whole EKS gets corrupted or I need to create a spare cluster of mongo (or different deployment) with the same data. The backup capability that the operator provides is for “point in time” recovery using oplog. My goal is to have a copy of mongo cluster with the same data in it.

I’ve tried aws-ebs CSI driver which has the capability to create VPC using a snapshot but this feature is in alpha for 1.14 and as I’m running EKS I have no access to control-plane and cannot enable it.

I’ve been thinking about mounting the PVC that creates an operator and a snapshot volume into another container and manually moving data from backup volume but this sound awfully slow and I’m not sure it might even work.

Does anybody have a solution for complete disaster recovery using snapshots?

Hi Oleksii

could you try this document [url]Percona Operator for MongoDB

the following scenario should work fine:

  1. start “cluster-first” and put data into it
  2. make backup according to doc
  3. create “cluster-second”
  4. list backups, restore the backup from step 2) into “cluster-second”

Hi Mykola,

My concern for operator backup is that it is an incremental backup. I should have mentioned that the solution I try to find is having backups for the previous ~30 days and the ability to restore the database from any of them. Using the PBM does not really fit this. While PBM creates incremental backups I cannot clean up the very first one as the latest depends on it. I cannot also create an on-demand full backup. The way for creating the full backup is to delete all the previous backups. And still deleting should only be done use PBM interface as removing the files directly from s3 will not update the JSON file located in the backup-coordinator pod.

Maybe there are some tricks in PBM that can fit all my requirements?