Best Practice for Automatic Vault Token Generation and Percona keyring_vault plugin

According to HashiCorp Vault best practices, tokens generated should be set to expire after a period of time, requiring the token to be renewed or a new token to be generated all together.

The use of Vault Agent and AppRole authentication seems to be a good approach to allow new tokens to be generated after they expire. However, since the keyring_vault plugin and its configuration file containing the token is only read when Percona restarts, it implies a period of downtime when the token needs to be updated in Percona.

For some context, we currently have a Master and Slave Percona server using replication in addition to Orchestrator and ProxySQL for failover. Percona with the keyring_vault plugin interfaces with HashiCorp vault secrets engine to store the encryption key used for Percona TDE.

Question: What is standard practice for automatically updating tokens while minimizing Percona downtime? We’re just not sure whether we should generate a token that does not expire after a long period of time (ie. 10 years), implement automatic token generation and find someway to restart Percona without downtime, or if downtime is just inevitable if we want automatic token generation.

1 Like

After doing some research I have found a way that might work for automatic vault token rotation without the need to restart Percona and potential down time.

There are a few important facts that this solution is based on:

  1. If a Vault token is revoked while the Percona server with TDE is running, the data is still decrypted and accessible. This is because the only time Percona needs to get the master decryption key stored in Vault is when the server is first started to decrypt the data. For the rest of the the session, the data is decrypted.

  2. Running the commands below causes Percona to read the keyring_vault.conf file again and implements any changes in the file, such as the token being changed. With reference to the previous point, the period between uninstalling and reinstalling the plugin will not result in downtime because a client will still have complete access to the data.

mysql> UNINSTALL PLUGIN keyring_vault;
mysql> INSTALL PLUGIN keyring_vault SONAME 'keyring_vault.so';

Implementation:

  1. Activate AppRole authentication on HashiCorp Vault Server
  2. Create a new role that is linked to the policy needed to access Vault KV secrets engine and generate the Role ID and Secret ID
  3. Install Vault Agent and configure Auto-Auth with the Role ID and and Secret ID
  4. Create a cron job that will automatically tell Vault Agent to generate a new Vault token and set the token value in keyring_vault.conf to the new token
  5. In MySQL, create a new event that automatically uninstalls and reinstalls the keyring_vault plugin and enable Event Scheduler
1 Like

@LukeYangJMA, I am new to the vault-percona combination, and I am doing the same task. Thanks a lot for the solution. I have a question though. Based on the observation in the solution, the token is only needed when mysql starts up, I am wondering the necessity of running uninstall/install keyring_vault plugin to get the server reread the keyring-vault config file.

It appears to me that I only need keep the token in the keyring-vault config file updated for mysql’s next startup (i.e. performing steps 1 - 4 is sufficient?).

Lastly, I am wondering if an agent.hcl sample file could be posted here. Do I need to learn consul template for this? Thanks in advance.

1 Like