Description:
Hello,
We currently have two geo-redundant OpenShift 4.18 clusters running Percona MongoDB deployments.
These MongoDB clusters are configured in a primary/replica topology, where cluster A serves as the primary site and cluster B serves as the replica site.
The MongoDB pods are exposed through LoadBalancer Services using static IP addresses assigned by the MetalLB Operator. These IP addresses are advertised through L2Advertisement over a bridge network interface.
This configuration has been successfully tested in other lab environments and operates as expected. However, in the current environment, we are experiencing intermittent replication failures and frequent MongoDB pod restarts.
Steps to Reproduce:
Setup two sharded clusters.
Copy the secrets (cluster-users,ssl, ssl-internal) from the main cluster to the other before releasing the replica one.
Switch the services from ClusterIP to LoadBalancer on the cr.yaml.
Create on Metallb the ipadresspool with static IPs which are advertised via L2advertisement for the services exposed.
Paste the cfs/rs0 IPs as external nodes to the main cr.yaml e.g
externalNodes:
- host: 10.201.10.65
port: 27017
priority: 1
votes: 1 - host: 10.201.10.67
port: 27017
priority: 1
votes: 1 - host: 10.201.10.69
port: 27017
priority: 0
votes: 0
Version:
1.19.1, 1.20.1
Logs:
with rs.status() i can see
replica set IDs do not match, ours: 6a563e8e6fcf6a59e5a1878e; remote node’s: 6a5643dc07ad62b04b1f56eb
also:
{“t”:{“$date”:“2026-07-15T08:36:57.263+00:00”},“s”:“F”,“c”:“CONTROL”,“id”:6384300,“ctx”:“ShardRegistry-0”,“msg”:“Writing fatal message”,“attr”:{“message”:“terminate() called. An exception is active; attempting to gather more information\n”}}
{“t”:{“$date”:“2026-07-15T08:36:57.263+00:00”},“s”:“F”,“c”:“CONTROL”,“id”:6384300,“ctx”:“ShardRegistry-0”,“msg”:“Writing fatal message”,“attr”:{“message”:“DBException::toString(): Location6493100: Time monotonicity violation: lookup time { topologyTime: Timestamp(1784037060, 2), rsmIncrement: 1, forceReloadIncrement: 1 } which is less than the earliest expected timeInStore { topologyTime: Timestamp(1784038411, 4), rsmIncrement: 0, forceReloadIncrement: 1 }.\nActual exception type: mongo::error_details::throwExceptionForStatus(mongo::Status const&)::NonspecificAssertionException\n\n”}}
Expected Result:
Mongos pods should be connected and running.
Actual Result:
Mongos pods are in CrashLoopBackOff
Additional Information:
[Include any additional information that could be helpful to diagnose the issue, such as browser or device information]