Vault encryption at rest binlog master key issue

MySQL & MariaDB Percona XtraDB Cluster 8.x
1

Hi, after some cluster error and vault being unavailable for some time, pxc cluster is not able to start. I see this error, but keys are not changed, they were generated by pxc first time I enabled enc at rest

 [ERROR] [MY-013362] [Repl] Unable to recover binary log master key, the combination of new_master_key_seqno=0, master_key_seqno=3 and old_master_key_seqno=39 are wrong.\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log"}
2024-02-13T11:45:33+01:00 {"log":"2024-02-13T10:45:31.371827Z 0 [ERROR] [MY-013288] [Server] Failed to initialize binlog encryption, please check if keyring is loaded.\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log"}
2024-02-13T11:45:33+01:00 {"log":"2024-02-13T10:45:31.372599Z 0 [ERROR] [MY-010119] [Server] Aborting\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log"}

Is it possible to recover from this? Thank you for any help

2

Do you have the keyring plugin set for early_plugin_load in my.cnf? Also, if you were using Hashicorp Vault to store encryption keys, and that Vault is no longer available, then, no, you usually can’t recover. However, this error message looks related to binlog encryption, so you can disable that in my.cnf and see if that allows mysql to start.

1 Like
3

No, all keys and the same vault is available. It was a temporary downtime.
I saw more in the logs, that I don’t have vault_ca, so now I changed URL to HTTP and now I’m stuck with this:

[Warning] [MY-011197] [Server] Plugin keyring_vault reported: 'Probing pxc-secret-stage for being a mount point unsuccessful - skipped.'\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log"

Secret path is kv1 and it is available.
I’m not sure why this issue arrised, because everything was working this morning with same settings. I used https vault URL (on VPN) with let’s encrypt SSL and haven’t used vault_ca TLS.
I can only think of this issue happening because the pxc cluster was completely down today and I configured encryption some months ago on some previous pxc version?

4

I also don’t have the keyring plugin set for early_plugin_load in my.cnf, nor binlog encryption in there, I enabled encryption afterwards using phpmyadmin

5

Also pxc version is percona-xtradb-cluster:8.0.29-21.1

I also tried accessing vault api from the pxc node using curl and that works too. Vault token is ok.
The path in vault is /pxc-secret-stage and policy is the same as when it was working, so I changed nothing except vault URL so I can use HTTP

@matthewb I also see this in the logs

The InnoDB Encryption Master Key has been rotated in 2 tablespaces.\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log

[InnoDB] Encryption key is loaded for undo tablespace 'innodb_undo_002'.\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log"}

could it mean that keys are in fact loaded, but only binlog is causing pxc not to start?

How can I disable binlog at this point if possible?

6

I was able to recover cluster by disabling binlog_encryption=OFF in my.cnf

But would be nice to know why this happened so we can prevent it in the future.

Thanks