Hello all,
I don’t know maybe my problem is normal behavior of MongoDB but I need to find a solution for that. So, I am using 3 node replicaset with 1 primary, 1 secondary and 1 arbiter. My servers are powerfull and can handle 3K ops . When secondary is down for a reason, MongoDB slows down significant like from 3000 ops to 50 ops. As I said that may be normal behavior.
I need to solve this slowness when a node is down.
- I can not remove down node and re-add it when become online because data is big, i can not wait to re-sync.
- when I lost a node, I restore it from backup and open it. It catches from primary in a minute.
MongoDB version is 4.2.4. do you have any suggestion about my problem? Thank you!
2 Likes
Maybe it’s a write-concern issue. PSA (Primary, Secondary and Arbiter) replicasets since v3.6 or v4.0 have been finicky about it. “w:majority” is default and writes cannot be made to a majority when a node is going and all you have is a ‘PDA’ instead of PSA.
A simple differential diagnostic test to perform to see if this hypothesis is hitting the mark or not is to change the write concern to just w:1.
2 Likes
Hi @Akira_Kurogane , I found what couses slowness when a node is down. It is because “Flow Control”, by default is “enableFlowControl” is “true”. When I disabled it, my slowness problem is solved. Right now, I am searching why Flow Control is needed. Thank you!
2 Likes
Hi Ghan.
Ah, I should have thought of that. Sorry, I was focused on some other issues with PSAs, caused by majority read concern, that have been reported several times recently.
As I like to say, “Flow Control” is a marketing name. What is really is a primary throttle mechanism.
For a lot of detail, including why a primary should be throttled, please see slide #28 onwards in this 2019 presentation:
If your goal is to allow running with only one data-bearing node out of two up at a time you will need to disable flow control in v4.2+. But you already worked that out so kudos.
1 Like
Thanks @Akira_Kurogane , my goal is ; when a node down at PSA replicas, I dont want any slowness while using MongoDB. I know it is because of Flow Control but I am not knowing why I need Flow control if I am using w:1 as write concern ? I tried to change its configuration (flowControlTargetLagSeconds, flowControlWarnThresholdSeconds ) but nothing changed, only Flow control activation time increased/decreased.
At above SS, I think yellow ones retired. If I can change red ones, I may continue to use Flow control, otherwise I will disable it incase its consequences.
1 Like
Hi.
Regardless of whether apps are happy with w:1 write consistency a MongoDB replica set has a duty to make sure replication doesn’t get too far behind. If it does then nice elections become impossible; and sudden failover would have a large amount of rolled back ops.
Having said that I expect a dead secondary to be something that flow control will ignore. I should retract this statement I made last time:
If your goal is to allow running with only one data-bearing node out of two up at a time you will need to disable flow control in v4.2+.
On more thought slowness isn’t a side effect I expect when the secondary is down; I only expect it to occur when the secondary is slow and lagging > 5(?) secs (by default).
1 Like
