@Sergey_Pronin Thank you for your reply!! 
I re tried pitr restore. The result is below.
just get psmdb-restore result is below. - requested
kubectl get psmdb-restore
NAME CLUSTER STATUS AGE
restore-pitr-source mongodb-test-psmdb-db requested 2m37s
kubectl get psmdb-restore -o yaml result is below. (no error showing)
kubectl get psmdb-restore restore-pitr-source -oyaml
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
creationTimestamp: "2024-01-17T02:18:55Z"
generation: 1
name: restore-pitr-source
namespace: mongodb-test
resourceVersion: "100038724"
uid: f370ff7a-c0f7-4bf6-874e-0b07a5d79262
spec:
backupSource:
destination: s3://percona-mongodb-backup/prod/2024-01-17T01:19:19Z
clusterName: mongodb-test-psmdb-db
pitr:
date: "2024-01-17 02:09:14"
type: date
storageName: s3-percona-mongodb-backup
status:
pbmName: "2024-01-17T02:19:14.497636748Z"
state: requested
pbm status command result in backup-agent container is below. - I stopped pitr after pitr backup created
kubectl exec -it mongodb-test-psmdb-db-rs0-0 -c backup-agent -- pbm status
Cluster:
========
cfg:
- cfg/mongodb-test-psmdb-db-cfg-0.mongodb-test-psmdb-db-cfg.mongodb-test.svc.cluster.local:27017 [P]: pbm-agent v2.3.1 OK
rs0:
- rs0/mongodb-test-psmdb-db-rs0-0.mongodb-test-psmdb-db-rs0.mongodb-test.svc.cluster.local:27017 [P]: pbm-agent v2.3.1 OK
PITR incremental backup:
========================
Status [OFF]
Currently running:
==================
(none)
Backups:
========
S3 ap-northeast-2 s3://percona-mongodb-backup/prod
Snapshots:
2024-01-17T01:19:19Z 1.42MB <physical> [restore_to_time: 2024-01-17T01:19:21Z]
PITR chunks [7.25MB]:
2024-01-17T01:19:22Z - 2024-01-17T02:09:14Z
2024-01-17T01:18:15Z - 2024-01-17T01:19:21Z (no base snapshot)
After delete cluster (including pvc), recreate cluster. And I tried to pitr like below.
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
name: restore-pitr-source
spec:
clusterName: mongodb-test-psmdb-db
pitr:
type: date
date: 2024-01-17 02:09:14
storageName: s3-percona-mongodb-backup
backupSource:
destination: s3://percona-mongodb-backup/prod/2024-01-17T01:19:19Z
operator logs - (INFO Waiting for restore metadata logs continue…)
2024-01-17T02:19:13.000+0000 D [resync] got backups list: 9
2024-01-17T02:19:13.000+0000 D [resync] bcp: 2024-01-09T09:30:07Z.pbm.json
2024-01-17T02:19:13.000+0000 D [resync] bcp: 2024-01-11T02:27:13Z.pbm.json
2024-01-17T02:19:13.000+0000 D [resync] bcp: 2024-01-11T02:32:50Z.pbm.json
2024-01-17T02:19:13.000+0000 D [resync] bcp: 2024-01-11T02:48:31Z.pbm.json
2024-01-17T02:19:13.000+0000 D [resync] bcp: 2024-01-11T03:01:15Z.pbm.json
2024-01-17T02:19:13.000+0000 D [resync] bcp: 2024-01-11T03:10:06Z.pbm.json
2024-01-17T02:19:13.000+0000 D [resync] bcp: 2024-01-11T03:24:09Z.pbm.json
2024-01-17T02:19:13.000+0000 D [resync] bcp: 2024-01-11T04:50:19Z.pbm.json
2024-01-17T02:19:13.000+0000 D [resync] bcp: 2024-01-17T01:19:19Z.pbm.json
2024-01-17T02:19:14.502Z INFO Restore state changed {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "38f935e8-c450-437f-98e9-ef8dcca7bad2", "previous": "", "current": "requested"}
2024-01-17T02:19:14.589Z INFO Waiting for restore metadata {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "1d468ec5-e586-4e9d-abab-cfb9a3238823", "pbmName": "2024-01-17T02:19:14.497636748Z", "restore": "restore-pitr-source", "backup": ""}
2024-01-17T02:19:19.588Z INFO Waiting for restore metadata {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "d97ac96d-4f65-43f5-bcb5-c1a817f19389", "pbmName": "2024-01-17T02:19:14.497636748Z", "restore": "restore-pitr-source", "backup": ""}
2024-01-17T02:19:24.671Z INFO Waiting for restore metadata {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "ec974261-2a09-4b0c-b573-44ea4a39a009", "pbmName": "2024-01-17T02:19:14.497636748Z", "restore": "restore-pitr-source", "backup": ""}
2024-01-17T02:19:29.755Z INFO Waiting for restore metadata {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "8ff50ad6-4046-4131-83e6-b5232948dcd8", "pbmName": "2024-01-17T02:19:14.497636748Z", "restore": "restore-pitr-source", "backup": ""}
2024-01-17T02:19:34.846Z INFO Waiting for restore metadata {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "34810c1a-4711-4772-9656-d196e7ef6e20", "pbmName": "2024-01-17T02:19:14.497636748Z", "restore": "restore-pitr-source", "backup": ""}
2024-01-17T02:19:39.929Z INFO Waiting for restore metadata {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "21b61a56-d6fc-4bdb-96b0-404e0910ca0a", "pbmName": "2024-01-17T02:19:14.497636748Z", "restore": "restore-pitr-source", "backup": ""}
2024-01-17T02:19:45.009Z INFO Waiting for restore metadata {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "68ce5ee9-fce0-4910-b055-e3bd4677d90d", "pbmName": "2024-01-17T02:19:14.497636748Z", "restore": "restore-pitr-source", "backup": ""}
2024-01-17T02:19:50.109Z INFO Waiting for restore metadata {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "81081c6c-0247-4909-a658-f42f450885d7", "pbmName": "2024-01-17T02:19:14.497636748Z", "restore": "restore-pitr-source", "backup": ""}
2024-01-17T02:19:55.191Z INFO Waiting for restore metadata {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "4afda217-2e3f-4255-ac0b-2f98929bf8be", "pbmName": "2024-01-17T02:19:14.497636748Z", "restore": "restore-pitr-source", "backup": ""}
2024-01-17T02:20:00.280Z INFO Waiting for restore metadata {"controller": "psmdbrestore-controller", "object": {"name":"restore-pitr-source","namespace":"mongodb-test"}, "namespace": "mongodb-test", "name": "restore-pitr-source", "reconcileID": "c95bf534-be42-470b-889e-a58649da9349", "pbmName": "2024-01-17T02:19:14.497636748Z", "restore": "restore-pitr-source", "backup": ""}
log of backup-agent in config-server pod
Version: 2.3.1
Platform: linux/amd64
GitCommit: 8c4265cfb2d9a7581b782a829246d8fcb6c7d655
GitBranch: release-2.3.1
BuildTime: 2023-11-29_13:31_UTC
GoVersion: go1.19
2024-01-17T02:14:38.000+0000 I starting PITR routine
2024-01-17T02:14:38.000+0000 I node: cfg/mongodb-test-psmdb-db-cfg-0.mongodb-test-psmdb-db-cfg.mongodb-test.svc.cluster.local:27017
2024-01-17T02:14:38.000+0000 I listening for the commands
2024-01-17T02:14:43.000+0000 W [agentCheckup] get current storage status: query mongo: mongo: no documents in result
2024-01-17T02:19:14.000+0000 I got command restore [name: 2024-01-17T02:19:14.497636748Z, snapshot: 2024-01-17T01:19:19Z point-in-time: <1705457354,0>] <ts: 1705457954>
2024-01-17T02:19:14.000+0000 I got epoch {1705457935 1}
2024-01-17T02:19:14.000+0000 I [restore/2024-01-17T02:19:14.497636748Z] to time: 2024-01-17T02:09:14Z
2024-01-17T02:19:14.000+0000 I [restore/2024-01-17T02:19:14.497636748Z] backup: 2024-01-17T01:19:19Z
2024-01-17T02:19:14.000+0000 I [restore/2024-01-17T02:19:14.497636748Z] recovery started
2024-01-17T02:19:15.000+0000 D [restore/2024-01-17T02:19:14.497636748Z] port: 27865
2024-01-17T02:19:15.000+0000 E [restore/2024-01-17T02:19:14.497636748Z] restore: check mongod binary: run: exec: "mongod": executable file not found in $PATH. stderr:
2024-01-17T02:19:15.000+0000 D [restore/2024-01-17T02:19:14.497636748Z] hearbeats stopped
log of backup-agent in replicaset pod
Version: 2.3.1
Platform: linux/amd64
GitCommit: 8c4265cfb2d9a7581b782a829246d8fcb6c7d655
GitBranch: release-2.3.1
BuildTime: 2023-11-29_13:31_UTC
GoVersion: go1.19
2024-01-17T02:14:37.000+0000 I node: rs0/mongodb-test-psmdb-db-rs0-0.mongodb-test-psmdb-db-rs0.mongodb-test.svc.cluster.local:27017
2024-01-17T02:14:37.000+0000 I listening for the commands
2024-01-17T02:14:42.000+0000 W [agentCheckup] get current storage status: query mongo: mongo: no documents in result
2024-01-17T02:19:14.000+0000 I got command restore [name: 2024-01-17T02:19:14.497636748Z, snapshot: 2024-01-17T01:19:19Z point-in-time: <1705457354,0>] <ts: 1705457954>
2024-01-17T02:19:14.000+0000 I got epoch {1705457935 1}
2024-01-17T02:19:14.000+0000 I [restore/2024-01-17T02:19:14.497636748Z] to time: 2024-01-17T02:09:14Z
2024-01-17T02:19:14.000+0000 I [restore/2024-01-17T02:19:14.497636748Z] backup: 2024-01-17T01:19:19Z
2024-01-17T02:19:14.000+0000 I [restore/2024-01-17T02:19:14.497636748Z] recovery started
2024-01-17T02:19:14.000+0000 D [restore/2024-01-17T02:19:14.497636748Z] port: 27738
2024-01-17T02:19:14.000+0000 E [restore/2024-01-17T02:19:14.497636748Z] restore: check mongod binary: run: exec: "mongod": executable file not found in $PATH. stderr:
2024-01-17T02:19:14.000+0000 D [restore/2024-01-17T02:19:14.497636748Z] hearbeats stopped
After terminated mongos pod, cfg and rs pod is still Running but any actions not occurs.
kubectl get pod
NAME READY STATUS RESTARTS AGE
mongodb-test-operator-psmdb-operator-7cf5dcd97c-67tsm 1/1 Running 0 7d16h
mongodb-test-psmdb-db-cfg-0 2/2 Running 0 21m
mongodb-test-psmdb-db-rs0-0 2/2 Running 0 21m