I run the following command and pbm-agent uses all memory on my server then crashes because of OOM
pbm config --force-resync --mongodb-uri=$PBM_MONGODB_URI
It shows it reading the json file, uses all memory (over 64GB) then gets stopped
This is the last entry in journal before crash
Apr 13 15:27:31 CARSB61MON001 pbm-agent: 2023-04-13T15:27:31.000-0500 D [resync] bcp: 2023-03-16T03:01:01Z.pbm.json
Has anyone seen this and know of a fix?
I have added memory and gone to 8 cpu and 128gb ram same result
I have matched both mongodb versions and pbm versions also tried with newer versions
pbm ver 2.0.2 mongodb 4.4.12-1
Several factors can trigger suboptimal memory usage. If memory pressure grows, OOM Killer tends to kill the process demanding more memory. Things to check:
I guess that a backup/restore operation uses a lot of filesystem page caching, and the OOM Killer just kills the process that is more memory demanding. I would try:
- Reduce swappiness.
- Increase vfs_cache_pressure
- Disable OOM Killer
- Disable THP
I hope this helps.
This is not MongoDB doing this but pbm-agent that was consuming memory until killed by OOM. I watched was was using memory and it was pbm-agent growing to consume all memory in the span of 2 minutes. I added over 128GB of RAM so I don’t think this is a memory optimization issue. I even tried limiting memory used by pbm-agent per documentation in the config. It didn’t help. After doing some research I saw that I used version 1.6.1 in my repo originally. This problem stopped happening once I got the older version (1.6.1) installed on the system I was restoring to. The really odd part is if I copied a backup that was done with a newer version (2.0.2) I still had to restore it with the older version of pbm-agent. Apparently there is some serious differences between file formats versions that the newer version does not like.
My comments aside, Thank for the well thought response.