Please tell me an upgrade howto / path for table encrypted PXC v. 5.7.25

MySQL & MariaDB Percona XtraDB Cluster 5.x
1

Hello,

I am running a version PXC cluster on Ubuntu 18.04 LTS servers since a year+. I’ve regularly updated it so that at the moment it shows the following version manifest:

Server version: 5.7.25-28-57-log Percona XtraDB Cluster (GPL), Release rel28, Revision a2ef85f, WSREP version 31.35, wsrep_31.35

Galera version 3.35(rddf9876)

Relevant configuration settings:


innodb_file_per_table=ON
early-plugin-load="keyring_vault=keyring_vault.so"
keyring_vault_config="/var/lib/mysql-keyring/keyring_vault.conf"
keyring-vault-timeout=30
innodb_encrypt_tables=ON
innodb_temp_tablespace_encrypt=ON
binlog_checksum=CRC32
master_verify_checksum=ON

As you can see from the above config snippet, it’s running the keyring vault plugin, connected to an HashiCorp Vault server and it’s setup for tables based encryption.

Tables have been encrypted with:

ALTER TABLE `table_name` ENCRYPTION='Y'

Everything works perfectly.

However, I cannot upgrade this setup any more.

I can shut down the cluster and upgrade the servers one by one, if needed. I have no “100% uptime” requirement.

Every time I perform a:

apt update
apt dist-upgrade

the existing PXC gets upgraded and… stops starting any more.
The log states that encrypted tables are not supported.

The plugin and vault server communication work, I have double checked them by hand on the command line.
Plus, when I restore the server(s) image backup, everything works again.

I have read somewhere on Percona’s website that newer PXC versions don’t support table based encryption.

Is it true?
If so, how do I upgrade PXC to a current version?
I don’t need to switch to version 8, just to upgrade to the most recent 5.XX version.

Best regards,
Dario Fumagalli

2

Hello,

please excuse me if I look ungrateful but…

… is it normal that 3 weeks after a documented question about a server blocking issue there’s no answer, not even a “we don’t know / send more configuration files / logs” reply?

How am I supposed to keep suggesting Percona products if there’s zero support - but even zero reply at all - for blocking issues?

3

Hi dfumagalli (Dario)

I apologize that you are disappointed, and we value your support for Percona products. Percona offers a paid support service where there are Service Level Agreements and guaranteed response times. We don’t guarantee responses for our free Forum, which is not intended to be a technical support service, as we also invite community members to contribute their advice and experiences.

Nevertheless, reading this, I appreciate that in this case you have an issue that needs resolution that we may have overlooked. In such cases we do normally try to respond as we realize that they present important issues to the user.

I know that the team generally want to see the mysql.conf files for each node, so please could you upload those. If there are any error logs associated with this experience, please also provide that. Meanwhile I will see if I can get any more immediate advice for you regarding this situation. Thanks.

4

I also have a few more questions for you:
[LIST]
[]You’re on almost the latest version of PXC (5.7.25 and the latest is 5.7.26), so did the problem happen only with your upgrade from 5.7.25 to 5.7.26 or did you have problems with previous upgrades?
[]Do you have any references/links abut the newer PXC versions don’t support table based encryption, as we really need to check out those resources to make sure there’s not an ambiguity that needs to be corrected?
[*]We really need the logs that record the problem(s) that occurred with the upgrades, otherwise we’re too much in the dark as to what’s the underlying issue. It could be that the encrypted table statement is masking something else.
[/LIST] If you could possibly get these answers to us, it would be great. Thanks.

5

Hello,

As you say, it’s been really surprising to see a “catastrophically failed” upgrade on such a minor update.

In the end I discovered (it’s not documented anywhere!!!) that the updated engine does not accept the following configuration file switches any longer:

innodb_encrypt_tables=ON
innodb_temp_tablespace_encrypt=ON

Setting both to OFF or removing them, immediately let the engine start.
I’d expected some more informative error about this, like “blah has been deprecated, remove it”.

Because in the end it was really just the switches. Just by setting them to off, the engine could start again and correctly encrypt / decrypt the data as it’s doing since years.

Now, how or why it’s still encrypting with the relevant options set to OFF is a mystery but, for now, I am happy it just works.

Best regards,
Dario Fumagalli

6

After you set innodb_encrypt_tables=OFF ,you could start your PXC and encrypted your data. I think that 's because you set encrypt option in your table.
“ALTER TABLE table_name ENCRYPTION=‘Y’”