Description:
We tried to restore from pitr backup in s3, however, the restore results in the following error “there is no oplogs that can cover the date/time or no oplogs at all”
Steps to Reproduce:
- I’m trying to restore the db to cron-percona-mongo-ps-20230919010000-jgwx4
kg psmdb-backup | tail -n 2
cron-percona-mongo-ps-20230919010000-jgwx4 percona-mongo-psmdb-d s3-us-west 2023-09-19T01:00:21Z logical ready 36h 36h
cron-percona-mongo-ps-20230920010000-pzz9w percona-mongo-psmdb-d s3-us-west 2023-09-20T01:00:21Z logical ready 12h 12h
These are the list of file in the s3
[2023-09-18 21:00:56 EDT] 428B STANDARD cfg/admin.pbmAgents.gz
[2023-09-18 21:00:56 EDT] 17KiB STANDARD cfg/admin.pbmBackups.gz
[2023-09-18 21:00:56 EDT] 5.3KiB STANDARD cfg/admin.pbmCmd.gz
[2023-09-18 21:00:58 EDT] 553B STANDARD cfg/admin.pbmConfig.gz
[2023-09-18 21:00:58 EDT] 267B STANDARD cfg/admin.pbmLock.gz
[2023-09-18 21:00:59 EDT] 3.7MiB STANDARD cfg/admin.pbmLog.gz
[2023-09-18 21:00:56 EDT] 14KiB STANDARD cfg/admin.pbmOpLog.gz
[2023-09-18 21:00:58 EDT] 2.6MiB STANDARD cfg/admin.pbmPITRChunks.gz
[2023-09-18 21:00:58 EDT] 259B STANDARD cfg/admin.pbmRRoles.gz
[2023-09-18 21:00:56 EDT] 2.2KiB STANDARD cfg/admin.pbmRUsers.gz
[2023-09-18 21:00:56 EDT] 259B STANDARD cfg/admin.system.roles.gz
[2023-09-18 21:00:55 EDT] 2.2KiB STANDARD cfg/admin.system.users.gz
[2023-09-18 21:00:56 EDT] 141B STANDARD cfg/admin.system.version.gz
[2023-09-18 21:00:56 EDT] 49KiB STANDARD cfg/config.actionlog.gz
[2023-09-18 21:00:56 EDT] 77KiB STANDARD cfg/config.changelog.gz
[2023-09-18 21:00:56 EDT] 17KiB STANDARD cfg/config.chunks.gz
[2023-09-18 21:00:56 EDT] 590B STANDARD cfg/config.collections.gz
[2023-09-18 21:00:58 EDT] 198B STANDARD cfg/config.databases.gz
[2023-09-18 21:00:58 EDT] 82B STANDARD cfg/config.lockpings.gz
[2023-09-18 21:00:56 EDT] 519B STANDARD cfg/config.locks.gz
[2023-09-18 21:00:57 EDT] 206B STANDARD cfg/config.settings.gz
[2023-09-18 21:00:58 EDT] 184B STANDARD cfg/config.shards.gz
[2023-09-18 21:00:58 EDT] 97B STANDARD cfg/config.version.gz
[2023-09-18 21:01:20 EDT] 16KiB STANDARD cfg/local.oplog.rs.bson.gz
[2023-09-18 21:00:59 EDT] 14KiB STANDARD cfg/metadata.json
[2023-09-18 21:00:56 EDT] 1.2KiB STANDARD rs0/AIModeration.ModerationActions.gz
[2023-09-18 21:00:56 EDT] 488B STANDARD rs0/AIModeration.ModerationLogs.gz
[2023-09-18 21:00:56 EDT] 141B STANDARD rs0/AIModeration.ServiceSettings.gz
[2023-09-18 21:00:56 EDT] 243B STANDARD rs0/AIModeration._migrations.gz
[2023-09-18 21:00:56 EDT] 1.5KiB STANDARD rs0/Hive.ClassThresholds.gz
[2023-09-18 21:00:56 EDT] 259B STANDARD rs0/admin.pbmRRoles.gz
[2023-09-18 21:00:56 EDT] 1.4KiB STANDARD rs0/admin.pbmRUsers.gz
[2023-09-18 21:00:56 EDT] 259B STANDARD rs0/admin.system.roles.gz
[2023-09-18 21:00:56 EDT] 1.4KiB STANDARD rs0/admin.system.users.gz
[2023-09-18 21:00:56 EDT] 454B STANDARD rs0/admin.system.version.gz
[2023-09-18 21:00:56 EDT] 94B STANDARD rs0/config.vectorClock.gz
[2023-09-18 21:01:25 EDT] 2.8KiB STANDARD rs0/local.oplog.rs.bson.gz
[2023-09-18 21:00:56 EDT] 14KiB STANDARD rs0/metadata.json
[2023-09-18 21:00:56 EDT] 1.2KiB STANDARD rs1/AIModeration.ModerationActions.gz
[2023-09-18 21:00:59 EDT] 143B STANDARD rs1/AIModeration._migrations.gz
[2023-09-18 21:00:56 EDT] 1.8KiB STANDARD rs1/Hive.ClassThresholds.gz
[2023-09-18 21:00:56 EDT] 143B STANDARD rs1/Hive._migrations.gz
[2023-09-18 21:00:56 EDT] 259B STANDARD rs1/admin.pbmRRoles.gz
[2023-09-18 21:00:56 EDT] 1.5KiB STANDARD rs1/admin.pbmRUsers.gz
[2023-09-18 21:00:56 EDT] 259B STANDARD rs1/admin.system.roles.gz
[2023-09-18 21:00:56 EDT] 1.4KiB STANDARD rs1/admin.system.users.gz
[2023-09-18 21:00:56 EDT] 361B STANDARD rs1/admin.system.version.gz
[2023-09-18 21:01:29 EDT] 2.5KiB STANDARD rs1/local.oplog.rs.bson.gz
[2023-09-18 21:00:56 EDT] 8.7KiB STANDARD rs1/metadata.json
- Create the PerconaServerMongoDBRestore
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
name: restore1
spec:
clusterName: percona-mongo-psmdb-d
backupName: cron-percona-mongo-ps-20230919010000-jgwx4
pitr:
type: date
date: 2023-09-19 01:00:21
kg perconaservermongodbrestore.psmdb.percona.com/restore1
NAME CLUSTER STATUS AGE
restore1 percona-mongo-psmdb-d waiting 13s
- Check the status of PerconaServerMongoDBRestore
Name: restore1
Namespace: data
Labels: <none>
Annotations: <none>
API Version: psmdb.percona.com/v1
Kind: PerconaServerMongoDBRestore
Metadata:
Creation Timestamp: 2023-09-20T13:40:19Z
Generation: 1
Resource Version: 577343681
UID: f772db92-f3c6-407b-9f7d-b8095beeebbe
Spec:
Backup Name: cron-percona-mongo-ps-20230919010000-jgwx4
Cluster Name: percona-mongo-psmdb-d
Pitr:
Date: 2023-09-19 01:00:21
Type: date
Status:
Error: there is no oplogs that can cover the date/time or no oplogs at all
State: error
Events: <none>
Version:
kd psmdb/percona-mongo-psmdb-d | grep -i image
Image: percona/percona-backup-mongodb:2.0.5
Image: percona/percona-server-mongodb:6.0.5-4
Image Pull Policy: Always
Image: percona/pmm-client:2.35.0
Mongo Image: percona/percona-server-mongodb:6.0.5-4
Logs:
2023-09-20T13:43:15.086Z ERROR failed to make restore {"controller": "perconaservermongodbrestore-controller", "object": {"name":"restore1","namespace":"data"}, "namespace": "data", "name": "restore1", "reconcileID": "e2333b41-5cc0-494e-b2d6-67cfa2754f3e", "restore": "restore1", "backup": "cron-percona-mongo-ps-20230919010000-jgwx4", "error": "there is no oplogs that can cover the date/time or no oplogs at all", "errorVerbose": "there is no oplogs that can cover the date/time or no oplogs at all\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.init\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:343\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6329\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6306\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6306\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6306\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"}
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile.func1
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/perconaservermongodbrestore_controller.go:123
github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/perconaservermongodbrestore_controller.go:158
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-09-20T13:43:15.086Z INFO Restore state changed {"controller": "perconaservermongodbrestore-controller", "object": {"name":"restore1","namespace":"data"}, "namespace": "data", "name": "restore1", "reconcileID": "e2333b41-5cc0-494e-b2d6-67cfa2754f3e", "previous": "waiting", "current": "error"}
2023-09-20T13:43:15.109Z ERROR Reconciler error {"controller": "perconaservermongodbrestore-controller", "object": {"name":"restore1","namespace":"data"}, "namespace": "data", "name": "restore1", "reconcileID": "e2333b41-5cc0-494e-b2d6-67cfa2754f3e", "error": "reconcile logical restore: there is no oplogs that can cover the date/time or no oplogs at all", "errorVerbose": "there is no oplogs that can cover the date/time or no oplogs at all\ngithub.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.init\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:343\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6329\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6306\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6306\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6306\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594\nreconcile logical restore\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/perconaservermongodbrestore_controller.go:158\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"}
Expected Result:
The backup restored without any error
Actual Result:
Fails with the following error “there is no oplogs that can cover the date/time or no oplogs at all”
Additional Information:
Backup config from psmdb
Backup:
Enabled: true
Image: percona/percona-backup-mongodb:2.0.5
Pitr:
Enabled: true
Service Account Name: percona-server-mongodb-operator
Storages:
s3-us-west:
s3:
Bucket: dev1-xxxxxxxxxxx
Credentials Secret: percona-xxxxx
Endpoint URL: https://s3.us-west-000.backblazeb2.com
Insecure Skip TLS Verify: false
Region: us-west-000
Type: s3
Tasks:
Enabled: true
Keep: 30
Name: 1am-nightly-backup
Schedule: 0 1 * * *
Storage Name: s3-us-west