High memory consumption during restore

MongoDB Percona Operator for MongoDB
,
1

Description:

I am experiencing an issue when performing a restore of an 80GB backup from S3 using the Percona MongoDB Operator (version 1.13.0) on Kubernetes. During the restore process, the node running the database pod (backup container of the leading rs0 pod) managed by the operator ends up consuming a significant amount of memory, potentially causing performance degradation or node failure.

Steps to Reproduce:

  1. Set up a Kubernetes cluster and install the Percona MongoDB Operator version 1.13.0.
  2. Ensure that the S3 bucket containing the 80GB backup is properly configured and accessible from the Kubernetes cluster.
  3. Trigger the restore process of the backup from the S3 bucket using the Percona MongoDB Operator.
  4. Monitor both the pod’s and the node’s memory usage during the restore process to track the increase in memory consumption.

Version:

Percona Operator for MongoDB → 1.13.0
Backup image tag → 1.8.1

Logs:

I don’t have concrete logs beyond monitoring an increase of about 10GB in memory usage.

Expected Result:

I expected the restore process to complete successfully without causing excessive memory consumption on the node where the MongoDB pod is running. Ideal behavior would include efficient memory management during the restoration of the backup.

Actual Result:

During the restore, the memory usage of the node significantly increased, potentially causing performance issues or crashes on the node hosting the restore process.

Additional Information:

I modified the cacheSizeRatio to 0.1 under storage → wiredTiger in the cr.yaml, but it didn’t work.

2

I’ve created the account specifically for you since we encountered the same issue a few days ago. Our backup size was 150GB, and we tried numerous solutions. Finally, we resolved the issue by using mongorestore with –noIndexRestore. Try this approach—it should work for you.

We also had another issue where our Kubernetes environment was crashing during the restore process, but that’s been resolved as well.

1 Like