Pbm-agent uses all memory and crashes during a resync

I run the following command and pbm-agent uses all memory on my server then crashes because of OOM
pbm config --force-resync --mongodb-uri=$PBM_MONGODB_URI

It shows it reading the json file, uses all memory (over 64GB) then gets stopped
This is the last entry in journal before crash
Apr 13 15:27:31 CARSB61MON001 pbm-agent[14261]: 2023-04-13T15:27:31.000-0500 D [resync] bcp: 2023-03-16T03:01:01Z.pbm.json

Has anyone seen this and know of a fix?

I have added memory and gone to 8 cpu and 128gb ram same result

I have matched both mongodb versions and pbm versions also tried with newer versions

pbm ver 2.0.2 mongodb 4.4.12-1

Hi @gregl

Several factors can trigger suboptimal memory usage. If memory pressure grows, OOM Killer tends to kill the process demanding more memory. Things to check:

I guess that a backup/restore operation uses a lot of filesystem page caching, and the OOM Killer just kills the process that is more memory demanding. I would try:

  • Reduce swappiness.
  • Increase vfs_cache_pressure
  • Disable OOM Killer
  • Disable THP

I hope this helps.

Pep

1 Like

This is not MongoDB doing this but pbm-agent that was consuming memory until killed by OOM. I watched was was using memory and it was pbm-agent growing to consume all memory in the span of 2 minutes. I added over 128GB of RAM so I don’t think this is a memory optimization issue. I even tried limiting memory used by pbm-agent per documentation in the config. It didn’t help. After doing some research I saw that I used version 1.6.1 in my repo originally. This problem stopped happening once I got the older version (1.6.1) installed on the system I was restoring to. The really odd part is if I copied a backup that was done with a newer version (2.0.2) I still had to restore it with the older version of pbm-agent. Apparently there is some serious differences between file formats versions that the newer version does not like.

My comments aside, Thank for the well thought response.