We have nearly 500 MongoDB replica sets (normal + sharded clusters) span across tow data centers. Each cluster is having minimum 3 members on an average. so we have 500 x 3 = 1500 members (physical servers + virtual servers combination based app preferences).
Due to enterprise standards, every host shoudl undergo monthly patching. which is created a big challenge to DBAs to plan the database server high availability during the month Linux patching.
The challenge we are facing currently is to ensure the database server availability through manual checks to ensure one server from each cluster go for maintenance where the patching activity start 2 week of the month and ends last week of the month.
Wondering how the big companies managing the linux patchign by achieving the high availability by keepign majority of the voting members in the cluster during the patching?
This is good topic to open a discussion about. MongoDBâs replica sets means that regular maintenance doesnât have to cause any downtime for the database clients, but (important) you must not take down the majority of nodes in any given replica set at a given time.
Rule of âMinority maximum deathâ?
Before stopping and restarting any mongod node, or the host it is on, the administration script/program needs to determine what the other nodes in the same replica set are and confirm the majority (i.e. two, for the typical three-node replica set) are healthy before stopping that node. âHealthyâ = either PRIMARY, or SECONDARY with no noticeable replication lag.
Parallelization for the win
You can do replica sets in parallel so it doesnât take longer in theory if you have one replica set or five hundred. Of course in practice youâll probably stagger launches of the update procedure so there isnât too much to be watched at any given moment.
The tricky parts
The tricky part is maintaining âMinority maximum deathâ as the script/program runs
I think the root issue behind this is devops tools are built without a sense of distributed data. Theyâre made to run per server using only what state they can sense local to the server.
Parallel launch in the same replica set can easily lead to race conditions. For example a local script running on hosts A, B and C of the same replicaset might first run a check âAre the other hosts healthy PRIMARY or SECONDARY state now?â. If theyâre done in parallel at the starting moment the answer will be true. Then A, B and C all restart in another moment, taking down the entire replicaset simultaneously.
Safety mechanisms have to be programmed. In short I think a semaphore of some kind should be put into the replica setâs own data so external agents can see which node is schedule to be restarted at any given moment. But at this point we should start determining some concrete points about your case.
What is being patched? The entire host server, or just the mongod binaries? (The former is what I assume.)
What is the sys admin method? Bash scripts and passwordless SSH? A devops tool like Puppet, Ansible, etc? Program using API of a computing service like AWS or whole-real-bare-metal system like Ubuntuâs MAAS?
If I understand correctly, parallelization is nothing but rolling method upgrades to ensure we do not have majority deaths.
Even for rolling fashion OS upgrades, we are following the below steps,
Linux team (sysadmins) releases the list of servers that are planned for patching in the first week of every month
DBA admins process the list and ensure any 2 of the server identifies and preapre the master list in nas location
Upgrade process (puppet process by sysadmin) refers master list and upgrades the database server one at a time as mentioned in the master list
But still this process involves atleast 3 days of one single DBA resource effort to verify and prepare the master list
Wondering how the large organizations manages the upgrades process on linux server that hosts mondodb instances monthly or quarterly or whatever defined time.
As you highlighted, safety mechanism had to be programmed.
What is being patched? The entire host server, or just the mongod binaries? (The former is what I assume.)
What is the sys admin method? Bash scripts and passwordless SSH? A devops tool like Puppet, Ansible, etc? Program using API of a computing service like AWS or whole-real-bare-metal system like Ubuntuâs MAAS?
OS (RHEL) is being patched every monthly
sysadmin method is mostly through puppet using flyway tools
OS (RHEL) is being patched every monthly
sysadmin method is mostly through puppet using flyway tools
Got it.
I had the experience of programming a solution using Ansible ~3 years ago whilst I was a contractor DBA for a while. Ansible was a fixed requirement because it had already been used for the non-distributed databases in that department. I found I could use Ansibleâs group hierarchy well for organizing environments, clusters, and replica sets. Furthermore the number of nodes at that site wasnât big - so I could review the list of servers manually within several page scrolls. At least for a single environment (prod/qa/dev). So I could for example have a subgroup for every replica set
But then there was no way (at least with Ansible at the time) to get it to do work serially within each group.
If I recall correctly I executed the same shell script in parallel. The script was very careful to wait and not proceed if more than one replica set member was down; when it did do a restart it exited. I canât recall what I used to choose the next member to restart - but the one with the maximum uptime at a given time seems like a suitable property.
If puppet (or modern Ansible) has a way to instruct work be performed serially within a subgroup, or follow a âdisruption budgetâ of 1 that would make it simpler.
Also if it is a sharded cluster it should be the config server replica set first, then shards in parallel after.
In your case youâre not just doing a âsystemctl restart mongodâ, youâre doing a âsystemctl stop mongod; [systemctl] rebootâ + âsystemctl start mongodâ after the server is communicating again so thatâs a bit different.