Percona keyring_vault plugin: Providing vault_ca

MySQL & MariaDB Percona Server for MySQL 8.0
1

We are using Percona for MySQL 8.0 with TDE enabled, using HashiCorp Vault as the Key Management System through the Percona keyring_vault plugin. The documentation specifies that the vault_ca can be provided if the machine does not trust the Vault’s CA certificate (Using the Keyring Plugin — Percona Server 8.0 Documentation).

Currently, we have set up Percona and HashiCorp Vault set up on two different servers. We’re using Let’s Encrypt to generate a signed certificate and private key, a different ones for each server. Intuitively, it doesn’t seem right to copy and paste the TLS cert file for the Vault server into the keyring_vault.conf file on the Percona server.

The following output is produced by Percona at start up when the vault_ca is not specified:

[ERROR] [MY-011197] [Server] Plugin keyring_vault reported: 'vault_ca is not specified but vault_url is https://. Please make sure that Vault's CA certificate is trusted by the machine from which you intend to connect to Vault.'

Is a TLS multi domain SAN or Wildcard the only way to enable TLS between Percona and Vault? The biggest concern is the single point of failure if the TLS cert or private key is compromised.

1 Like
2

Luke,

First, I think that the paragraph about “vault_ca” in the official documentation is a bit inaccurate.
It should state that if you are using “https” then you must specify “vault_ca”.
“vault_ca” is not needed only when you are using “http” (which you should do only for TESTING purposes like running Vault server in the “-dev” mode).

Second, you do not need to get real signed certificates for the Vault server (signed by Let’s Encrypt or any other certification authority). Vault servers are not usually made publicly available and therefore do not require real certificates. You can achieve the same level of security in the connection between Vault Server and Percona Server using self-signed CA certificate.
Here is how this can be set up.

On the Vault Server machine:

  1. You generate a key pair (private and public keys) for the self-signed CA certificate (ca_key_priv.pem and ca_key_pub.pem).
  2. You generate a self-signed CA certificate ca.crt - it will include CA public key .
  3. You generate a key pair for Vault Server server_priv.pem and server_pub.pem.
  4. You generate a certificate for the Vault Server (server.crt) and sign it with the self-signed CA certificate ca.crt (using ca_key_priv.pem).
  5. You configure Vault Server to use server.crt and server_priv.pem for HTTPS.
  6. You copy ca_key_priv.pem to a secure storage and remove it from the Vault Sever machine (you may only need this key when you decide to generate a new certificate signed with your self-signed CA for another Vault Server).

On the Percona Server machine:

  1. You copy ca.crt from the Vault Server machine to the Percona Server machine.
  2. You configure keyring_vault plugin in Percona Server to use this ca.crt as “vault_ca” configuration parameter.

Third, if you really need to use Let’s encrypt, you will need to ask them to generate a certificate for you which can be used as a CA under which you will be able to sign your server certificates yourself. However, I am not sure if Let’s Encrypt provides such functionality to end-users. I only remember that certificates with signing capabilities from giants like Digicert and IdenTrust were up to 10 times more expensive.

Just to sum up and make the procedure for you clearer, a brief description of how “vault_ca” parameter is used by the “keyring_vault” plugin. When “keyring_vault” establishes a secure HTTPS connection to the Vault Server it receives the Vault Server certificate that it tries to validate. And this is where it needs “vault_ca”. Basically, the plugin extracts the info about the CA that signed the received certificate and compares this with “vault_ca”. If those are identical, the validation succeeds. So, this is basically why certificates generated by Let’s Encrypt cannot be used as “vault_ca” because nothing is signed with them.

2 Likes
3

Thx @Yura_Sorokin . So we should be creating the HashiCorp High Availability cluster on a private network at our data centre, managing the DNS internally. We will need to look into the HA IP handling for failover, I suppose.

1 Like
4

We are using Percona* for MySQL 8.0 with TDE enabled, using HashiCorp Vault as the Key Management System through 192.168.l.254 the Percona keyring_vault … keyring_vault provides* an interface for the database with a HashiCorp Vault server to store key and secure encryption keys. Note. The keyring_file plugin should …

2 Likes