# Cross-site two way (Or more than two) replication **URL:** \ **Category:** Percona Operator for MySQL\ **Tags:** percona\ **Created:** [April 27, 2024, 7:17am UTC](https://forums.percona.com/t/cross-site-two-way-or-more-than-two-replication/29896 "2024-04-27T07:17:46Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jithin\_devaws](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jithin_devaws/32/16176_2.png) [@jithin\_devaws](https://forums.percona.com/u/jithin_devaws)\ **Post date:** [April 27, 2024, 7:17am UTC](https://forums.percona.com/t/cross-site-two-way-or-more-than-two-replication/29896/1 "2024-04-27T07:17:46Z")
## Description: I want to test out percona operator for my use-case. I’m looking for a multi master (within the same cluster) + two way (Or more than two) cross-site replication. ie, All pods in asia, us & eu regions should accept read/write and then replicate to the other two sites. Is this possible ? I have checked the doc [Multi-cluster and multi-region deployment - Percona Operator for MySQL](https://docs.percona.com/percona-operator-for-mysql/pxc/replication.html) and it specifies only one way replication from one cluster to another. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [April 27, 2024, 12:24pm UTC](https://forums.percona.com/t/cross-site-two-way-or-more-than-two-replication/29896/2 "2024-04-27T12:24:47Z")
> [@jithin\_devaws](#): > > ie, All pods in asia, us & eu regions should accept read/write and then replicate to the other two sites. Hey @jithin_devaws, This is a bad design decision, in general. Your application will experience extremely high lag/latency because every write must be synchronously replicated to other nodes. Typically applications have a single point for writes, and replicas for reading locally. ---
**Author:** ![jithin\_devaws](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jithin_devaws/32/16176_2.png) [@jithin\_devaws](https://forums.percona.com/u/jithin_devaws)\ **Post date:** [April 27, 2024, 5:25pm UTC](https://forums.percona.com/t/cross-site-two-way-or-more-than-two-replication/29896/3 "2024-04-27T17:25:53Z")
> [@matthewb](#): > > This is a bad design decision, in general. Your application will experience extremely high lag/latency because every write must be synchronously replicated to other nodes. Typically applications have a single point for writes, and replicas for reading locally. Apps (dc1) will face latency **only** if it did not receive an ACK for the write operation from the local DC (dc1). And if I’m correct master (dc1) =\> slave (dc2) replication can be done in asynchronous way, so that the apps will get immediate ACK from local DC (dc1) even if the replication breaks/delayed with slave. So I don’t think that will be a problem. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [April 27, 2024, 7:03pm UTC](https://forums.percona.com/t/cross-site-two-way-or-more-than-two-replication/29896/4 "2024-04-27T19:03:24Z")
Maybe I misunderstood what you said, but above you said you wanted PXC pods (ie: nodes) in each DC: asia, us, and eu. That’s 3 nodes in geographically dispersed regions. Any writes to either of these 3 nodes will require _synchronous_ replication. This means that the ACK in US will not be received by the app in US until _after_ it is replicated to EU and ASIA. Perhaps a diagram of what you are envisioning would be helpful? I like [https://excalidraw.com/](https://excalidraw.com/) for quick, easy diagrams. ---
**Author:** ![jithin\_devaws](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jithin_devaws/32/16176_2.png) [@jithin\_devaws](https://forums.percona.com/u/jithin_devaws)\ **Post date:** [April 28, 2024, 8:34am UTC](https://forums.percona.com/t/cross-site-two-way-or-more-than-two-replication/29896/5 "2024-04-28T08:34:07Z")
@matthewb Thank you for the quick diagram tool 🙂 Here is the high level view of the db setup I’m trying to achieve. > **[Excalidraw — Collaborative whiteboarding made easy](https://excalidraw.com/#json=XmnhBsTdFb4Ft-ruV2np3,SrYV0guddi9kevbQb4dGgw)** > > Excalidraw is a virtual collaborative whiteboard tool that lets you easily sketch diagrams that have a hand-drawn feel to them. Let’s replace the US and EU region based DCs with DCs within the same region because ours are private DCs located 300-400 miles from one another. They are; asia-dc1 (k8s) - db-cluster-1 (3 master replicas) asia-dc2 (k8s) - db-cluster-2 (3 master replicas) asia-dc3 (k8s) - db-cluster-3 (3 master replicas) In every k8s cluster, we have the stateless apps deployed which connects to the DB endpoint _db-cluster-ip-svc_ locally where this SVC endpoints load balance write/read request to any of the healthy master nodes. Eg, in dc-1, the apps will connect to any of asia-dc1-master-1, asia-dc1-master-2, asia-dc1-master-3 via the svc. This health is decided by the k8s probes. Similarly, apps hosted in respective clusters will be connecting only to the local db end point _db-cluster-ip-svc_ . I believe this sync within the local dc is _synchronous_ which might not cause any serious latency if I’m correct. Now for the cross-site replication, we can expose these master pods via svc called _replication-svc-nodeport/lb_. As you can see from the diagram, there are two dotted lines originated from a single cluster to other two clusters. Eg; from **db-cluster-1** two dotted blue lines connecting to _replication-svc_ of **db-cluster-2** and **db-cluster-3** for replicating data from those two clusters. Similarly every other dc is connected to other two DCs. If I understood this correctly, these cross site replication is something we can control as **asynchronous** , so that it won’t interrupt any application writes to the local dc and then replicate to other DC’s in an **asynchronous** way. I may be wrong here. But if you feel like this is not the right way to implement HA for database within multiple DCs, you could share any useful docs/diagram. ---
**Author:** ![matthewb](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/matthewb/32/34_2.png) [@matthewb](https://forums.percona.com/u/matthewb)\ **Post date:** [April 29, 2024, 3:20pm UTC](https://forums.percona.com/t/cross-site-two-way-or-more-than-two-replication/29896/6 "2024-04-29T15:20:48Z")
Hello @jithin_devaws, Thanks for the drawing. What you have detailed out still amounts to ‘circular replication’, which is a huge “don’t do that”. You can find many blogs about how MySQL circular replication will cause more harm than good. Nobody at Percona would ever recommend a circular-replication topology. Instead, to achieve the multi-DC HA, you would have 1 PXC node in each DC (3 nodes total), all connected as the same cluster. The apps in each DC would connect to a local load balancer (ie: proxysql) and that proxysql would send all writes to DC1. Reads would go to the local PXC node. If DC1 (or just the pxc1 pod in DC1) goes down/offline, the proxysql pods in DC1, DC2, and DC3 detect this. A new leader is elected from the surviving 2 PXC nodes, and writes are now sent to this node. When K8S in DC1 restores pxc1-pod, it will join the cluster and sync any missing data before accepting connections. > **[Excalidraw — Collaborative whiteboarding made easy](https://excalidraw.com/#json=63_poSw5E1-sQMk4aCHSX,CmtZjl6GkTnZJ136Ov1cjw)** > > Excalidraw is a virtual collaborative whiteboard tool that lets you easily sketch diagrams that have a hand-drawn feel to them. Even if you did not have K8S, this would still be the recommendation for DC-HA with PXC. ---
**Author:** ![jithin\_devaws](https://sea1.discourse-cdn.com/flex019/user_avatar/forums.percona.com/jithin_devaws/32/16176_2.png) [@jithin\_devaws](https://forums.percona.com/u/jithin_devaws)\ **Post date:** [May 2, 2024, 9:07am UTC](https://forums.percona.com/t/cross-site-two-way-or-more-than-two-replication/29896/7 "2024-05-02T09:07:46Z")
Based on the design you shared, in a normal scenario with 3 DCs the apps hosted in the DC3 will have to connect to DC1 node over proxsql for all write operations (Considering DC1 node is the master). In my opinion, this design will add latency on every single write operation from DC2 and DC3 due to the geo separation. Anywya, we have the 3 way circular replication implemented already with mariadb galera but that too helm based approach but no operators. This is leading to many operational challenges especially when it comes to crash recovery/unintentional node failures in k8s.