I run the following command and pbm-agent uses all memory on my server then crashes because of OOM
pbm config --force-resync --mongodb-uri=$PBM_MONGODB_URI
It shows it reading the json file, uses all memory (over 64GB) then gets stopped
This is the last entry in journal before crash
Apr 13 15:27:31 CARSB61MON001 pbm-agent[14261]: 2023-04-13T15:27:31.000-0500 D [resync] bcp: 2023-03-16T03:01:01Z.pbm.json
Has anyone seen this and know of a fix?
I have added memory and gone to 8 cpu and 128gb ram same result
I have matched both mongodb versions and pbm versions also tried with newer versions
Several factors can trigger suboptimal memory usage. If memory pressure grows, OOM Killer tends to kill the process demanding more memory. Things to check:
I guess that a backup/restore operation uses a lot of filesystem page caching, and the OOM Killer just kills the process that is more memory demanding. I would try:
This is not MongoDB doing this but pbm-agent that was consuming memory until killed by OOM. I watched was was using memory and it was pbm-agent growing to consume all memory in the span of 2 minutes. I added over 128GB of RAM so I don’t think this is a memory optimization issue. I even tried limiting memory used by pbm-agent per documentation in the config. It didn’t help. After doing some research I saw that I used version 1.6.1 in my repo originally. This problem stopped happening once I got the older version (1.6.1) installed on the system I was restoring to. The really odd part is if I copied a backup that was done with a newer version (2.0.2) I still had to restore it with the older version of pbm-agent. Apparently there is some serious differences between file formats versions that the newer version does not like.