Keyring_vault Encryption

MySQL & MariaDB Percona XtraDB Cluster 8.x
1

I’ve been putting together a proof of concept of an architecture to make sure everything works.

OS: Centos7.8
PXC: 8.0.19 - 3 node cluster setup
Hashicorp Vault: 1.4.2 - 3 node cluster setup

I’m trying to get encryption at rest working using the keyring_vault plugin. I have my configuration below:

early-plugin-load             = "keyring_vault=keyring_vault.so"
loose-keyring_value_config    = "/etc/xtradb/keyring_vault.conf"

However, the cluster fails to start up with the following error:

2020-10-19T04:22:44.478556Z 0 [ERROR] [MY-011370] [Server] Plugin keyring_vault reported: 'File '' not found (OS errno 2 - No such file or directory)'
2020-10-19T04:22:44.479119Z 0 [System] [MY-011197] [Server] Plugin keyring_vault reported: 'Could not open file with credentials.'
2020-10-19T04:22:44.479628Z 0 [System] [MY-011197] [Server] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.'
2020-10-19T04:22:44.480595Z 0 [ERROR] [MY-010202] [Server] Plugin 'keyring_vault' init function returned error.
2020-10-19T04:22:44.482688Z 0 [ERROR] [MY-010167] [Server] Failed to initialize early plugins.
2020-10-19T04:22:44.485491Z 0 [ERROR] [MY-010119] [Server] Aborting

I’m not sure what file is missing that the plugin can’t locate.
my keyring_vault.conf file looks like this:

vault_url          = https://vault.uri:8200
secret_mount_point = secrets/pxc
token              = s.sdl;fjslfjsdfods
vault_ca           = /etc/pki/tls/certs/keyring_vault.crt

Getting rid of the plugin declaration and PXC loads back up just fine

2

I wonder if the mysql system user can read or access the keyring_vault.conf file in /etc/xtradb directory. Maybe change /etc/xtradb to be owned by mysql user and group?

chown -R mysql:mysql /etc/xtradb

3

Thanks for the suggestion, I did check permissions and ownership is correct and owned by xtradb user.

4

I figured out the problem by stumbling across this article:

https://forums.percona.com/discussion/52994/solved-but-painfully-keyring-vault-conf-file-is-not-a-conf-file

  • The key_vault.conf file doesn’t seem to like comments. I had some comments in the file and once I removed them, it started up ok.

    I can validate the configuration is read and loaded

  • The keyring vault plugin is loaded

But now I am on the new hurdle, the encryption key isn’t being created within my vault instance. I don’t know when this is supposed to happen - whether on startup or when I encrypt a table or tablespace.

5
  • Ok I figured out the issue. Seems that the keyring_vault plugin only works with Hashicorp vault KV v1 secrets engine. Once I recreated using v1, the master keys got generated once I ran the rotate master key alter statement. I still have a lot of questions though with regards to long term operations and the intricacies:

    It seems that the master key is unique to each server instance, If you generate a master key on each node in the cluster into the same vault instance, it generates 3 different keys. Which one will be used for encryption? Since it’s multi-master, I’m assuming each master would use a different master key? but since the tablespace headers are replicated, wouldn’t I encounter failures? Should you only generate a master key on a single node within the cluster?

  • I'm using asynchronous replication to a secondary DR cluster that has a second vault instance and I'm using consul to replicate to it. How does encryption work in such a setup? As long as the master keys are replicated to the secondary instance, it should be ok?