Best way to upgrade a 5.7 cluster?

MySQL & MariaDB Percona XtraDB Cluster 8.x
1

Hello,

According to this link Percona Release Lifecycle Overview I have to upgrade a 3 nodes percona cluster 5.7 to 8.0 before October 2023.

I read this link Upgrade Percona XtraDB Cluster - Percona XtraDB Cluster but it still confused.

My cluster is behind a haproxy LB, with master / backup (no write over all the nodes).

What is the best way to go ?

  • adding a new node, in 8.0 version to the 5.7 cluster, promote it as master then upgrading other node “in place” ?
  • directly upgrading inplace 5.7 to 8.0 ?

Thanks

Yathus

2

Hi @Yathus welcome back to the Percona forums!

You have three main choices to your upgrade, and the scenario to choose depends on the amount of downtime you can take (several hours to zero seconds):

  1. High downtime - stop all 5.7 nodes. replace PXC packages with 8.0. Start one node PXC 8 in bootstrap mode. Join the other two nodes.
  2. Several seconds downtime: Configure a new cluster running PXC 8. Replicate data from PXC 5.7 → PXC 8 using asynchronous replication. Stop writes on PXC 5.7, point application to read/write from PXC 8.
  3. Zero downtime: stop a member of the PXC 5.7 cluster. replace PXC packages with 8.0. Start this PXC 8.0 node and let it join the PXC 5.7 cluster. Then upgrade the remaining PXC 5.7 → PXC 8.0

Your decision will be influenced by the amount of testing on PXC 8 you’d like to do, as well as your budget (whether you can afford to run multiple clusters at the same time), and finally your ability to take downtime (or not). My preference is usually #2 as it provides the most control over the upgrade process, and also a very clear method for rolling back to PXC 5.7 if any issues arise on PXC 8.0 that will take longer than the maintenance window to resolve.

3

Hello Michael,

Thanks for the three choices, option 1 is not possible.

We’ll try the option 3 in a labs environnement, so we could test “Mysql 8.0 compatibility” with our applications.

Choice 2 is a good idea if we take advantage of this migration to also change the hardware of the 3 nodes.

4

Hi Yathur,

Thanks for reaching out.
Yes it is true that PS MySQL is having its EOL by October 2023 and it would be a wise idea to upgrade to 8.0

May I know what exact confusion you have from this link, so that I can guide you correctly. We are actively working to improve our document.

I understand that you have 3 node Percona Xtradb cluster with writes on only one node and before this layer, you have deployed HAproxy for load balancing and now you are planning to upgrade your all nodes to 8.0 version and would like to understand which approach is the best to adopt. Please correct me if I am wrong.

With my experience, I would recommend you to adopt the in-place which should have minimal downtime or negligible. I would recommend you to test it in testing environment first before moving it to the production and get familiar with 8 since MySQL has changed alot after 5.7 in terms of structure and feature. Check here. Below are the steps you can take.

  1. Stop read only node and replace the binary with 8.
  2. Restart it and let it re-join the cluster.
  3. Once everything is fine, do it same with other read only node.
  4. Switch your writes to upgraded one and move old master out of the cluster.
  5. Upgrade it and bring it back in the cluster.

Things to be cautious about :

  1. Your application should be capable to handle two nodes i.e. the lesser number of nodes and make sure no split brain situation will occur.
  2. While upgrading please switch off pxc_encrypt_cluster_traffic to OFF as it is ON by default in 8 and can create issues when data will flow from 5.7 to 8
  3. Keep pxc_strict_mode to PERMISSIVE as it is ENFORCING by default in 8 and if there is any incompatibility, it wont start the PXC. You can fix any incompatibility if any and then roll it back to ENFORCING.

Hope this helps

5

Hello Ankit,

With your answer, it is much more understandable !