We are using Percona MongoDB v3.6.11-3.1, running a replicaSet of 5 nodes. The nodes 1-2-3 are on the same datacenter1 same VLAN and the backends are pointing to those 3 servers, reading from secondaries. The node4 is on the datacenter1 but on another VLAN, and it has been setup to do not vote and it doesn’t get backends traffic. The node 5 is on datacenter2, on a VLAN that can reach datacenter1 but only node 4, meaning Node 5 can’t reach Node 1-2-3. No vote and no backends traffic for Node 5.
We are experiencing an issue, apparently we are writing to the primary and very soon trying to read some data, but it seems the data is not being propagated to the node 2 & node 3 (secondaries) yet. Then, I am looking to implement the Write Concern "2 or perhaps majority”. We use Percona Monitoring and Management v1, and I am seeing 2 seconds of replication lag for node 2 and node 3, and sometimes the PMM shows 3-4 secs of replication lag.
We have a database of 50GB of data, and the main collection has way to many indexes over 50 indexes and we are close to 90MM records/documents. We mainly use it for a Learning Records Store “LRS”, where the official “Learning Locker software” has official 30 indexes. With our +50 indexes plus the data, our databases on disk is about 101GB of disk size. We run our backends/databases on VMs - Debian 8, using 4 Hosts (ESXi 6.7) some Dell servers, we use vCenter, vSAN, vMotion, etc.
It seems to me 2 seconds of replicate lag is a lot. Should I add the Write Concern “majority” on the Mongod.conf or on the connection string on the backends?
Thanks in advance.