Creating a cluster from S3 backup path

Percona Everest
1

Hi,

I’ve been evaluating Everest and testing it’s backup and restore functionality for MongoDB see how it behaves during disaster recovery. The backup and restore works without any issues as long as the database is up and running.

But when db is accidentally deleted, there is no way to restore the database from the backup in s3, (It’s fairly easy to accidentally delete the db since there are no non root users or deletion protection that can be enabled on db)

In order to test restoring from s3 when server is accidentally deleted, I deleted a db with backups and created a new db with the same name as deleted db and realised that backups from deleted db are visible in new db, but when I tried to restore the backup it’s stuck in a perpetual state of restoring with operator logs below

2024-02-25T08:52:25.358Z	INFO	Waiting for restore metadata	{"controller": "psmdbrestore-controller", "object": {"name":"restore-i8h","namespace":"percona"}, "namespace": "percona", "name": "restore-i8h", "reconcileID": "181fbeb8-6730-4210-acba-64a87926fb74", "pbmName": "2024-02-25T07:44:50.291387568Z", "restore": "restore-i8h", "backup": "backup-t5p"}
2024-02-25T08:52:28.239Z	INFO	Waiting for restore to complete	{"controller": "psmdbbackup-controller", "object": {"name":"cron-mongodb-4jy-20240225080000-f7nbc","namespace":"percona"}, "namespace": "percona", "name": "cron-mongodb-4jy-20240225080000-f7nbc", "reconcileID": "16de2ae5-3fb7-4507-96fe-343b1382a0cd", "restore": "restore-i8h", "status": "requested"}
2024-02-25T08:52:30.383Z	INFO	Waiting for restore metadata	{"controller": "psmdbrestore-controller", "object": {"name":"restore-i8h","namespace":"percona"}, "namespace": "percona", "name": "restore-i8h", "reconcileID": "8e41a7e5-bbad-46a1-84b1-e080e6f14b7a", "pbmName": "2024-02-25T07:44:50.291387568Z", "restore": "restore-i8h", "backup": "backup-t5p"}
2024-02-25T08:52:33.253Z	INFO	Waiting for restore to complete	{"controller": "psmdbbackup-controller", "object": {"name":"cron-mongodb-4jy-20240225080000-f7nbc","namespace":"percona"}, "namespace": "percona", "name": "cron-mongodb-4jy-20240225080000-f7nbc", "reconcileID": "bb0e9e97-9fbf-4355-b022-59144f62f4f8", "restore": "restore-i8h", "status": "requested"}

Everest provides a way to create a new db from a specific backup when the source db is up and running so I think a way to restore the db from s3 path is feasible and important for disaster recovery.

I hope adding this functionality is considered in future versions.

Everest version: 0.8.0
MongoDB version: 6.0.9-7

Best regards
Harsha

1 Like
2

Hi, @harsha

Welcome to the forum, and thanks for trying Everest.

Your input or observation sounds very logical and helpful. I have passed it on to the team.

We should probably add a new section in Everest where all backups can be tracked and managed. This will ensure that you are not linked to databases that may actually be deleted or lost in the list.

Also, you may have many backup settings, and going manually through all the databases to check the settings can be time-consuming.

Maybe the team will suggest some other solution, or you have suggestions for implementation.

Thank you very much.

3

We should probably add a new section in Everest where all backups can be tracked and managed. This will ensure that you are not linked to databases that may actually be deleted or lost in the list.

Also, you may have many backup settings, and going manually through all the databases to check the settings can be time-consuming.

That is one way to handle backups but as you said it’d be time consuming to filter through all the dbs and it’s versions. For the time being, I was thinking about adding an extra step called restore(or source) while creating a new db, after backup storage-location has been selected. In this step there are three ways to select an existing backup

In the first one, there are two fields to select, one to select the existing db(from a drop-down) and second field to select the backup you want to restore of the db selected. This is not much different to current flow of creating db from backup i.e, selecting a db, going to it’s backups and clicking create new db on backup. This is not related to disaster recovery but helps in creating of new db with existing backups and cluster in one creation flow.

For the second one, it’s just a path to s3 folder in storage-location selected in previous steps. This way there is no need to list all the backups of every cluster in one place, the admin can enter the backup file path manually and Everest would load data from that path.

For the third one, it’s very similar to first one but instead of selecting the db from existing db, the user has to enter the deleted db name(existing db name work too if they are in same storage-location), then Everest can fetch the backups from storage-location (selected in previous steps) corresponding to the cluster, if it can find any, show them to the user for selection and then proceed to next step.

For second and third method the backups should reside in the same storage location as the db for it to get the backups, unless an option to select storage location is provided.

This is one way i think restore can be implemented while creating a new db.

1 Like
4

Thanks for your suggestions; this seems like a good tool, including for easy migration to Percona Everest or for launching pre-prepared databases on demand.

We discussed it today at the daily meeting.

We have a similar task in Roadmap, and the team will design a solution, according to the workflow. How it will be implemented is not yet known.

5

Hello @harsha,
Thank you for the detailed suggestions, they were really great.

Like @daniil.bazhenov mentioned we already have an item in our roadmap to add the ability to restore from an “external” backup. There’s still no ETA for this feature.

Nonetheless, I wanted to share with you that we already implemented one improvement related to this.
We added a confirmation prompt that asks users to type the DB cluster name before proceeding with the deletion. This should help to avoid accidental deletion of DBs.
This improvement will be part of the upcoming v0.9.0 release, which is planned to come out next week.

When we have a better understanding of when the restore from “external” backup feature will be ready, we’ll let you know.

6

I noticed that the v0.9.0 release was created about an hour ago (:tada:). At the top of the release notes, there is an obvious callout to uninstall previous versions of Everest before installing v0.9.0. This is great.

However, as part of the uninstallation process, we’re unable to leave running database clusters in place (right?). It makes sense, particularly if CRDs need to be updated. However, this does bring up the OP in a different way. It would be wonderful to be able to perform a manual backup of each cluster, remove each clusters, uninstall Everest, install the latest version of Everest, then recreate the original clusters directly from the most recent backups.

In my case, I was already running v0.9.0-rc2, so I just risked running the everestctl upgrade command without nuking my clusters :slight_smile: Obviously this is the more ideal option, but we can’t expect this to be possible in major version upgrades.

1 Like
7

Hi Josh, you’re right, you can’t leave database clusters running during an uninstall of everest.
The highest priority feature we are developing right now is precisely to allow users to do an everest upgrade without affecting running databases. If everything goes according to plan, when 0.10.0 comes out at the end of April, you’ll be able to seamlessly upgrade from 0.9.x.

2 Likes
8

Fantastic! I look forward to testing this feature out :slight_smile:

1 Like