# Percona MongoDB Operator Multi-Cluster Switchover/Failover **URL:** \ **Category:** Percona Operator for MongoDB\ **Tags:** mongodb\ **Created:** [March 18, 2025, 12:17pm UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258 "2025-03-18T12:17:51Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Laurentiu\_Soica](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/laurentiu_soica/32/20523_2.png) [@Laurentiu\_Soica](https://forums.percona.com/u/Laurentiu_Soica)\ **Post date:** [March 18, 2025, 12:17pm UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/1 "2025-03-18T12:17:51Z")
## Description: I went through the tutorial to setup Percona MongoDB Multi-Cluster with the K8S Operator. [About Multi-cluster and multi-region deployments - Percona Operator for MongoDB](https://docs.percona.com/percona-operator-for-mongodb/replication.html) Everything went smooth. I want now to simulate a switchover/failover but I was not able to find instructions. What I tried: Start with: - 2 K8S clusters, A and B - Percona MongoDB operator deployed in each - Deploy a RS with 3 nodes in B, unmanaged, expose it with type NodePort. Sharding disabled. - Deploy a RS with 3 nodes in A, managed, expose it with type NodePort. Sharding disabled.Use B nodes as externalNodes (using host IPs and NodePorts ports) - make sure the credentials and certificates are the same - The setup is successful, I have a 6 node ReplicaSet and both PSMDB CRs are “ready” As a switchover process: - Change PSMDB CR in A from managed to unmanaged - Connect to MongoDB and remove A’s nodes from the ReplicaSet. Now the replicaSet have 3 nodes, the nodes from B - Change PSMDB CR in B from unmanaged to managed - The CR in B moves to error state with error: message: ‘Error: failed to update config members: delete: write mongo config: replSetReconfig: (NodeNotFound) No host described in new configuration with {version: 131364, term: 2} for replica set rs0 maps to this node’ ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [March 19, 2025, 11:38am UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/2 "2025-03-19T11:38:44Z")
Hi, please take a look at [Disaster Recovery for MongoDB on Kubernetes](https://www.percona.com/blog/disaster-recovery-for-mongodb-on-kubernetes) ---
**Author:** ![Laurentiu\_Soica](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/laurentiu_soica/32/20523_2.png) [@Laurentiu\_Soica](https://forums.percona.com/u/Laurentiu_Soica)\ **Post date:** [March 19, 2025, 1:19pm UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/3 "2025-03-19T13:19:27Z")
Thanks for sharing the link. The failover (disaster recovery) process worked as described. Now, in case of a switchover, where both the clusters are healthy but I just want to switch the PRIMARY/ACTIVE from A to B, I did not manage to make it work. What I did: - Start with the same setup as described in the blog post - Changed the CR in A from managed to unmanaged - Leave the ReplicaSet config as it was (6 nodes, all healthy) - Changed the CR in B from unmanaged to managed - The CR in B moves to error state: ```auto message: 'Error: failed to update config members: fix member hostname: write mongo config: replSetReconfig: (NewReplicaSetConfigurationIncompatible) Found two member configurations with same host field, members.1.host == members.4.host == :' mongoImage: percona/percona-server-mongodb:7.0.15-9-multi ``` LE: Removing the tags from all Mongo nodes and moving the primary to B instead of A resulted in a successful switchover. However, the process involved quite a few manual steps, and I’m unsure whether it is entirely deterministic. ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [March 20, 2025, 10:54am UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/4 "2025-03-20T10:54:22Z")
What you did looks good to me. Can you share the output of rs.conf() ? also what is the value of `clusterServiceDNSMode` ? ---
**Author:** ![Laurentiu\_Soica](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/laurentiu_soica/32/20523_2.png) [@Laurentiu\_Soica](https://forums.percona.com/u/Laurentiu_Soica)\ **Post date:** [March 20, 2025, 12:15pm UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/5 "2025-03-20T12:15:27Z")
```auto clusterServiceDNSMode: "External" ``` rs.conf() before switchover: ```auto { _id: 'rs0', version: 78580, term: 3, members: [ { _id: 0, host: '1.2.3.4:30864', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2, tags: { serviceName: 'mongodb', nodeName: 'mw-3', podName: 'mongodb-rs0-0' }, secondaryDelaySecs: Long('0'), votes: 1 }, { _id: 1, host: '1.2.3.5:31383', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2, tags: { serviceName: 'mongodb', nodeName: 'mw-1', podName: 'mongodb-rs0-1' }, secondaryDelaySecs: Long('0'), votes: 1 }, { _id: 2, host: '1.2.3.6:32351', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2, tags: { serviceName: 'mongodb', nodeName: 'mw-2', podName: 'mongodb-rs0-2' }, secondaryDelaySecs: Long('0'), votes: 1 }, { _id: 3, host: '1.2.3.7:30308', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1, tags: { external: 'true' }, secondaryDelaySecs: Long('0'), votes: 1 }, { _id: 4, host: '1.2.3.8:32118', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1, tags: { external: 'true' }, secondaryDelaySecs: Long('0'), votes: 1 }, { _id: 5, host: '1.2.3.9:30498', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0, tags: { external: 'true' }, secondaryDelaySecs: Long('0'), votes: 0 } ], protocolVersion: Long('1'), writeConcernMajorityJournalDefault: true, settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('67dbf1520289a4b6b57b904b') } } ``` cleanup tags and switch primary: ```auto cfg = rs.config() cfg.members[0].tags = {} cfg.members[1].tags = {} cfg.members[2].tags = {} cfg.members[3].tags = {} cfg.members[4].tags = {} cfg.members[5].tags = {} cfg.members[3].priority = 10 rs.reconfig(cfg, {force: true}) ``` rs.conf() after switchover ```auto { _id: 'rs0', version: 148264, term: 4, members: [ { _id: 0, host: '1.2.3.4:30864', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1, tags: { external: 'true' }, secondaryDelaySecs: Long('0'), votes: 1 }, { _id: 1, host: '1.2.3.5:31383', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1, tags: { external: 'true' }, secondaryDelaySecs: Long('0'), votes: 1 }, { _id: 2, host: '1.2.3.6:32351', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0, tags: { external: 'true' }, secondaryDelaySecs: Long('0'), votes: 0 }, { _id: 3, host: '1.2.3.7:30308', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2, tags: { podName: 'mongodb-rs0-0', serviceName: 'mongodb', nodeName: 'mw-3' }, secondaryDelaySecs: Long('0'), votes: 1 }, { _id: 4, host: '1.2.3.8:32118', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2, tags: { podName: 'mongodb-rs0-1', serviceName: 'mongodb', nodeName: 'mw-2' }, secondaryDelaySecs: Long('0'), votes: 1 }, { _id: 5, host: '1.2.3.9:30498', arbiterOnly: false, buildIndexes: true, hidden: false, priority: 2, tags: { nodeName: 'mw-1', podName: 'mongodb-rs0-2', serviceName: 'mongodb' }, secondaryDelaySecs: Long('0'), votes: 1 } ], protocolVersion: Long('1'), writeConcernMajorityJournalDefault: true, settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('67dbf1520289a4b6b57b904b') } } ``` ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [March 20, 2025, 12:25pm UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/6 "2025-03-20T12:25:00Z")
Tag cleanup should be performed by operator when you change the site B to managed. I believe there might be an issue with the cr.yaml on site B. Anyway, we will update the documentation with clear steps for switchover. Feel free to subscribe to [Jira](https://perconadev.atlassian.net/browse/K8SPSMDB-1315) for updates ---
**Author:** ![Laurentiu\_Soica](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/laurentiu_soica/32/20523_2.png) [@Laurentiu\_Soica](https://forums.percona.com/u/Laurentiu_Soica)\ **Post date:** [March 20, 2025, 12:58pm UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/7 "2025-03-20T12:58:08Z")
In case of a failover, when I manually remove the A’s nodes from rs.conf(), like described in the blog post, I can confirm the operator refreshes the tags. In case of a switchover, where I want to keep all the nodes but switch the “Active” cluster, so the active operator and the primary node, the operator gets confused by the existing tags and fails as described above. Anyway, indeed it could be a config error on my end so a reproducible procedure will certainly help me identify any misconfigurations. ---
**Author:** ![Laurentiu\_Soica](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/laurentiu_soica/32/20523_2.png) [@Laurentiu\_Soica](https://forums.percona.com/u/Laurentiu_Soica)\ **Post date:** [August 29, 2025, 12:13pm UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/8 "2025-08-29T12:13:27Z")
> [@Ivan\_Groenewold](#): > > Tag cleanup should be performed by operator when you change the site B to managed. I believe there might be an issue with the cr.yaml on site B. Anyway, we will update the documentation with clear steps for switchover. Feel free to subscribe to [Jira](https://perconadev.atlassian.net/browse/K8SPSMDB-1315) for updates Hi @Ivan_Groenewold Is this still on your radar? I still can’t perform the switchover if I do not manually cleanup the tags. ---
**Author:** ![Ege\_Gunes](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ege_gunes/32/4595_2.png) [@Ege\_Gunes](https://forums.percona.com/u/Ege_Gunes)\ **Post date:** [September 1, 2025, 7:02am UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/9 "2025-09-01T07:02:08Z")
@Laurentiu_Soica I will check this sometime this week. ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [September 9, 2025, 4:49pm UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/10 "2025-09-09T16:49:19Z")
I think the issue with the tags might be due to your manually removing nodes from the replica set. Can you try instead letting the operator handle changes? basically add nodes from site B as external on site A, and add nodes from site A as external on site B. At switchover time, you set the cluster on A to “unmanaged”, and then on B to “managed”. The operator should take care of promoting a node on B site as primary. ---
**Author:** ![Laurentiu\_Soica](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/laurentiu_soica/32/20523_2.png) [@Laurentiu\_Soica](https://forums.percona.com/u/Laurentiu_Soica)\ **Post date:** [September 16, 2025, 6:24am UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/11 "2025-09-16T06:24:02Z")
I think that’s the process I follow (I do node removal on failover only, not switchover). In case of a switchover, this is the process: - Start with cluster A with unmanaged: false, updateStrategy: SmartUpdate, 3 local nodes, 3 externalNodes and cluster B with unmanaged: true, updateStrategy: OnDelete, 3 local nodes, 3 externalNodes - Demote cluster A: unmanaged: true, updateStrategy: OnDelete, 3 local nodes, 3 externalNodes - Promote cluster B: unmanaged: false, updateStrategy: SmartUpdate, 3 local nodes, 3 externalNodes At this point, cluster B CR reports: > message: ‘Error: failed to update config members: fix member hostname: write mongo > config: replSetReconfig: (NewReplicaSetConfigurationIncompatible) Found two member > configurations with same host field, members.1.host == members.4.host == IP:PORT’ > mongoImage: percona/percona-server-mongodb:8.0.8-3 > mongoVersion: 8.0.8-3 > observedGeneration: 4 > ready: 3 > replsets: > rs0: > initialized: true > ready: 3 > size: 3 > status: ready > size: 3 > state: error ---
**Author:** ![Ivan\_Groenewold](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/ivan_groenewold/32/6299_2.png) [@Ivan\_Groenewold](https://forums.percona.com/u/Ivan_Groenewold)\ **Post date:** [September 23, 2025, 11:39am UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/12 "2025-09-23T11:39:36Z")
I couldn’t reproduce the same behavior. In my case the promotion of cluster B works fine. Would you mind opening a bug at [https://perconadev.atlassian.net/](https://perconadev.atlassian.net/) and providing the full cr.yaml for both clusters? that way the dev team can take a look. ---
**Author:** ![Laurentiu\_Soica](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/laurentiu_soica/32/20523_2.png) [@Laurentiu\_Soica](https://forums.percona.com/u/Laurentiu_Soica)\ **Post date:** [September 23, 2025, 3:37pm UTC](https://forums.percona.com/t/percona-mongodb-operator-multi-cluster-switchover-failover/37258/13 "2025-09-23T15:37:15Z")
I tried to add the details here [Jira](https://perconadev.atlassian.net/browse/K8SPSMDB-1478) Wasn’t able to edit description or add files so I’ve add the details as comments.