Debugging the whole setup on GKE, it turns out that mongod in a Kubernetes container sees the max RAM of the node instead of the limits.memory set for the pod.
So in our case mongod sees 49GB (from the node) and I guess does not respect the limit of 8GB set in limits.memory.
Basically this means the pod gets OOM killed sooner or later.
Thanks - we are now tracking the memory consumption and have a alert, when a certain memory used limit is reached. May be some time in the future there is some kind of auto-scaling based on memory surge in the operator …
May be as an input for others - to avoid the Memory Usage Peak, we load the databases on demand into MongoDB. This way we can have a lot fewer db’s active and in case a pod gets rescheduled, it need most likely much less memory.