Description:
We’re unable to restore a physical backup taken from a non-kubernetes percona mongodb replicaset to a PerconaServerMongodb mongodb replicaset in Kubernetes.
Steps to Reproduce:
To reproduce this situation:
- Take a backup of a non-kubernetes percona mongodb replicaset using this command executed via a suitable user
db.adminCommand(
{
createBackup: 1,
s3: {
bucket: "MONGODB_BACKUP_S3_BUCKET_NAME",
path: "FOLDER_NAME/TIMESTAMP",
accessKeyId: "MONGODB_BACKUP_S3_ACCESS_KEY_ID",
secretAccessKey: "MONGODB_BACKUP_S3_SECRET_ACCESS_KEY",
region: "MONGODB_BACKUP_S3_REGION"
}
}
);
- Create a
PerconaServerMongoDBresource that spins up a three-node RS like so and wait for it to start up.
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
annotations:
meta.helm.sh/release-name: percona-mongodb-deployments
meta.helm.sh/release-namespace: percona-mongodb-deployments
labels:
app.kubernetes.io/managed-by: Helm
name: mongodb-dev-rs
namespace: percona-mongodb-deployments
spec:
allowUnsafeConfigurations: true
clusterServiceDNSMode: external
crVersion: 1.15.0
image: percona/percona-server-mongodb:5.0.20-17
replsets:
- name: rs0
resources:
requests:
cpu: 500m
memory: 768Mi
size: 3
volumeSpec:
persistentVolumeClaim:
resources:
requests:
storage: 10Gi
secrets:
users: mongodb-dev-rs-mongodb-init-users
- Create a
PerconaServerMongoDBRestoreresource like so:
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
annotations:
meta.helm.sh/release-name: percona-mongodb-deployments
meta.helm.sh/release-namespace: percona-mongodb-deployments
labels:
app.kubernetes.io/managed-by: Helm
name: mongodb-dev-rs-restore
namespace: percona-mongodb-deployments
spec:
backupSource:
destination: s3://<bucket>/<directory>
s3:
bucket: <bucket>
credentialsSecret: mongodb-dev-rs-s3-credentials
endpointUrl: https://s3.<region>.amazonaws.com/
prefix: <prefix>
region: <region>
type: physical
clusterName: mongodb-dev-rs
Version:
- We’re relying on the
1.15.0version of thepsmdb-operator. - The operator was installed via the
percona/psmdb-operatorhelm chart available athttps://percona.github.io/percona-helm-charts/
Logs:
The following log is seen in the operator’s pods:
2023-11-21T17:24:35.098Z ERROR failed to make restore {"controller": "psmdbrestore-controller", "object": {"name":"mongodb-dev-rs-restore","namespace":"percona-mongodb-deployments"}, "namespace": "percona-mongodb-deployments", "name": "mongodb-dev-rs-restore", "reconcileID": "22375786-249f-4ace-a794-57200414e0dc", "restore": "mongodb-dev-rs-restore", "backup": "", "error": "disable PiTR stderr: stdout: : Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"f485b36e82a5b3df5d6f0894892c66ff4fe14a8ed6d34ea843bcf0e21cf5b613\": OCI runtime exec failed: exec failed: unable to start container process: exec: \"/opt/percona/pbm\": stat /opt/percona/pbm: no such file or directory: unknown", "errorVerbose": "Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"f485b36e82a5b3df5d6f0894892c66ff4fe14a8ed6d34ea843bcf0e21cf5b613\": OCI runtime exec failed: exec failed: unable to start container process: exec: \"/opt/percona/pbm\": stat /opt/percona/pbm: no such file or directory: unknown\ndisable PiTR stderr: stdout: \ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).disablePITR\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:906\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).reconcilePhysicalRestore\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/physical.go:78\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore.(*ReconcilePerconaServerMongoDBRestore).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbrestore/perconaservermongodbrestore_controller.go:170\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.1/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"}
The same logs in plain text:
disable PiTR stderr: stdout: : Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "f485b36e82a5b3df5d6f0894892c66ff4fe14a8ed6d34ea843bcf0e21cf5b613": OCI runtime exec failed: exec failed: unable to start container process: exec: "/opt/percona/pbm": stat /opt/percona/pbm: no such file or directory: unknown
Expected Result:
The operator restores the specified backup to the mongodb cluster cluster.
Actual Result:
-
The operator doesn’t restore the specified backup to the mongodb replicaset cluster.
-
The status on the operator looks like so:
status:
error: 'disable PiTR stderr: stdout: : Internal error occurred: error executing
command in container: failed to exec in container: failed to start exec "f485b36e82a5b3df5d6f0894892c66ff4fe14a8ed6d34ea843bcf0e21cf5b613":
OCI runtime exec failed: exec failed: unable to start container process: exec:
"/opt/percona/pbm": stat /opt/percona/pbm: no such file or directory: unknown'
state: error