MongoDB replica set member replication lag issue

Percona Monitoring and Management (PMM) Database Monitoring and Management
1

I have added mongo as a service through pmm UI, when i am checking replication lag for any replica set if one node is going into lag all nodes on replication lag monitoring dashboard showing in lag with same amout.how this happening please help.

2

@manishkr47

Did you also verified if there is no such lag observed among other nodes which is showing false lag ? You can use the below command in order to confirm the replication lag in secondary nodes.

mongo> rs.printSecondaryReplicationInfo()

Can you also please share the below command output to verify the oplog status ?

mongo> rs.printReplicationInfo()

Would it be possible if you can share with us the [Replication Lag] snapshot from MongoDB/MongoDB ReplSet Summary where you seeing the lag occurrences ?

Exactly, what version of PMM Server/MongoDB you are using ?

3

Hi Anil,

I am sharing the required screenshot below also i am reproducing the issue by apply db.fsysncLock() on a single node but again same result i am getting.

PMM Server version:- 2.41.2
MongoDB version:- 4.0, 4.2,4,4 and 5.0

rs.printSecondaryReplicationInfo()
rs.printSecondaryReplicationInfo()1485×303 101 KB

4

Please find another attachment.

rs.printReplicationInfo()
rs.printReplicationInfo()1450×136 58.9 KB

5

@manishkr47

I tried to test this with using db.fsysncLock() on any one of the secondaries however other node seems fine on few conducted tests and no lag is observed there.

Let me share the details.

MongoDB server version: 5.0.26

db.fsysncLock()

rs0:PRIMARY> for (var i = 1; i <= 30000; i++) db.users.insert( { user_id : "user"+i,created_at :new Date() } )

rs0:SECONDARY> rs.printSecondaryReplicationInfo()
source: mongo_sec:27017
	syncedTo: Sat Apr 27 2024 15:39:34 GMT+0000 (UTC)
	15 secs (0 hrs) behind the primary 
source: mongo_sec2:27017
	syncedTo: Sat Apr 27 2024 15:39:49 GMT+0000 (UTC)
	0 secs (0 hrs) behind the primary

source: mongo_sec:27017
	syncedTo: Sat Apr 27 2024 15:39:34 GMT+0000 (UTC)
	29 secs (0.01 hrs) behind the primary 
source: mongo_sec2:27017
	syncedTo: Sat Apr 27 2024 15:40:03 GMT+0000 (UTC)
	0 secs (0 hrs) behind the primary

db.fsyncUnlock()

source: mongo_sec:27017
	syncedTo: Sat Apr 27 2024 15:40:29 GMT+0000 (UTC)
	0 secs (0 hrs) behind the primary 
source: mongo_sec2:27017
	syncedTo: Sat Apr 27 2024 15:40:29 GMT+0000 (UTC)
	0 secs (0 hrs) behind the primary

Did you also test exactly like this and then observed the lag on almost all secondary nodes ? Is that reproducible ? Is there any sequence/order in which a particular node receives the lag followed by other nodes then ?

Can you please confirm the below output for a review ?

rs.conf()
rs.status()

Is the delay output same on all nodes ?

mongo> rs.printSecondaryReplicationInfo()

Still, there are lot of things which can cause the genuine increase in replication lag . Please read the below manual for the same.