Recommended actions for PXC in case maintenance windows

Hi all, Good day!

We had implemented 3 PXC clusters, running each on a different OCP cluster. Like the diagram attached.

What are the best practices in the following scenarios?

  • We need to take the OCP Cluster hosting PXC1 offline for maintenance but do not want to disrupt database services.

Is there an automatic way to switch over pxc, I mean, to change from Source to Replica and from Replica to Source? Or what is your recommended approach for this needs.

Current pxc and ocp version included in the diagram.

Hello @Francisco_Ruben_Jime,
Just to clarify, do you have 3-node clusters in each OCP cluster (total 9 servers)? Or just 1 PXC, with 3-nodes (total 3 servers), each node in a different OCP?

Your diagram seems to show the former, 3 separate clusters, using async replication to keep 2 clusters in sync with the primary cluster. If this is the case, I would do the following:

  • Stop replication from PXC1->PXC3
  • Establish replication from PXC2->PXC3
  • Shut down PXC1 as needed based on your plan
  • Move applications to PXC2

The tricky part comes with PXC1 is back online. You will need to configure replication from PXC2->PXC1 in order to sync all the changes that occurred on PXC2 while PXC1 was offline

1 Like

Your supposition was right, We have 3-node clusters in each OCP cluster (total 9 servers).

Analyzing your suggested action path… and I apologize in advance if I am doing so basic questions, This is our very first implementation of the Percona Operator for MySQL, hopefully not the last one.

When you mention: stopping the replication is just as simple as removing the replicationChannels section from the cr.yaml file, or involve a pause: true of the PXC cluster?

Your step number two, involves first Stop replication from PXC2, and after that configure as a isSource: true for PXC2 and finally configure replication from PXC2 to PXC3.

The step number three, involve a pause: true of the PXC1 cluster, right? for a graceful poweroff of the running instances.

I think we understood the logic of your suggested steps, I have a final doubt, all those steps can be safely without any db service disruption? Is there a plan to do the mentioned process automatically? a sort of Orchestrator: MySQL Replication Topology Manager, for Percona Operator for MySQL.

I must apologize, I am not familiar enough with K8S and our operator to give guidance on those aspects within the config files. The steps I gave were a high-level process.

You have a highly complex and unusual setup. There isn’t anything I know of that can do this automatically. If you were using a standard MySQL replication, then yes, Orchestrator would be able to handle this via some drag-n-drop. But since you are using PXC in a multi-cluster setup, Orchestrator can’t help here.