Hi all, Good day!
We had implemented 3 PXC clusters, running each on a different OCP cluster. Like the diagram attached.
What are the best practices in the following scenarios?
- We need to take the OCP Cluster hosting PXC1 offline for maintenance but do not want to disrupt database services.
Is there an automatic way to switch over pxc, I mean, to change from Source to Replica and from Replica to Source? Or what is your recommended approach for this needs.
Current pxc and ocp version included in the diagram.
Just to clarify, do you have 3-node clusters in each OCP cluster (total 9 servers)? Or just 1 PXC, with 3-nodes (total 3 servers), each node in a different OCP?
Your diagram seems to show the former, 3 separate clusters, using async replication to keep 2 clusters in sync with the primary cluster. If this is the case, I would do the following:
- Stop replication from PXC1->PXC3
- Establish replication from PXC2->PXC3
- Shut down PXC1 as needed based on your plan
- Move applications to PXC2
The tricky part comes with PXC1 is back online. You will need to configure replication from PXC2->PXC1 in order to sync all the changes that occurred on PXC2 while PXC1 was offline
Your supposition was right, We have 3-node clusters in each OCP cluster (total 9 servers).
Analyzing your suggested action path… and I apologize in advance if I am doing so basic questions, This is our very first implementation of the Percona Operator for MySQL, hopefully not the last one.
When you mention: stopping the replication is just as simple as removing the replicationChannels section from the cr.yaml file, or involve a pause: true of the PXC cluster?
Your step number two, involves first Stop replication from PXC2, and after that configure as a isSource: true for PXC2 and finally configure replication from PXC2 to PXC3.
The step number three, involve a pause: true of the PXC1 cluster, right? for a graceful poweroff of the running instances.
I think we understood the logic of your suggested steps, I have a final doubt, all those steps can be safely without any db service disruption? Is there a plan to do the mentioned process automatically? a sort of Orchestrator: MySQL Replication Topology Manager, for Percona Operator for MySQL.
I must apologize, I am not familiar enough with K8S and our operator to give guidance on those aspects within the config files. The steps I gave were a high-level process.
You have a highly complex and unusual setup. There isn’t anything I know of that can do this automatically. If you were using a standard MySQL replication, then yes, Orchestrator would be able to handle this via some drag-n-drop. But since you are using PXC in a multi-cluster setup, Orchestrator can’t help here.