node1 - primary
node 2 - secondary
node 3 - secondary
Node 1 and node 2 are in same datacenter and always in sync but node 3 is in different datacenter and we see it is always lagging for long hours from primary.
We have also increased the oplog to a quite large about 4TB and in our case lot of data gets loaded and modified every minute.
The memory and load average is also not occupying much and we have baremetals configured and even when we check with network team they say there is no issue between both the datacenters and after 35 hours of lag it is going to recovery state.
We have done initial sync twice → from startup2 it changes to secondary but still i see it is lagging for 2 hours and it continues to lag, I have also tried restarting the node but still lagging is increasing.
Please can anyone help here what else I need to do to make this sync again to primary as this is a production env?
Thanks in advance.