The mongodb operator error is just responding to the fact that mongodb has crashed and can’t make a connection.
It turns out this issue was related to a new change introduced which sets default resources limits and requests on the mongodb server itself. Which caused an out of memory error.
State: Running
Started: Thu, 18 Jul 2024 13:06:16 +0100
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Thu, 18 Jul 2024 13:02:03 +0100
Finished: Thu, 18 Jul 2024 13:05:50 +0100
Ready: True
Restart Count: 9
Limits:
cpu: 300m
memory: 500M
Requests:
cpu: 300m
memory: 500M
Not sure when this change was introduced but it meant that after i upgraded from 1.14.0 to 1.16.1 the mongodb instance had the above default resource limits and requests assigned to them.
After setting the appropriate resource limits and requests, our monogdb instance seems to now be stabilised.