Hello again,
while I am in “issues reporting Zen mode”, I’ll add this.
It’s not a real Percona Server error, but if a system administrator mistakenly forgets a slash at the end of the secret mount point, a disaster happens and [COLOR=#B22222]all encrypted data on the whole server is immediately lost.
I’d classify it as “small usability issue with large consequences”. Since it’s very easy to add one or so line of code to check for a trailing slash, I suggest you do it and save admins some very bad days.
Let me show by example. Here are two anonymised plugin configurations. Spot the difference!
vault_url = https://kv01.domain.tld:8200
secret_mount_point = secret/mysql/customer01/lxsrv01/
token = 0188a1a9-<censored>
vault_ca = /usr/share/ca-certificates/mycerts/priv-cert-rootCA.crt
vault_url = https://kv01.domain.tld:8200
secret_mount_point = secret/mysql/customer01/lxsrv01
token = 0188a1a9-<censored>
vault_ca = /usr/share/ca-certificates/mycerts/priv-cert-rootCA.crt
The first configuration will destroy all data on next server reboot. The last works fine.
The difference and what it does
The difference is tiny and happened to me for real: I pasted the secret_mount_point (by the way, Linux shell tends to add a traling slash to paths) and it ends with a “/”.
Then I happily encrypted tables. No error reported or logged by the key vault whatsoever.
Then one day I rebooted the server and the key vault plugin put errors on the log, stating it could not retrieve the keys from the HashiCorp Key Vault server.
Whatever you do, you’ll never recover the data, because the tables were encrypted, the keys were generated but never stored on the key vault server.
By studying HashiCorp Key Vault logs, at next server / Percona daemon reboot, you can see the key vault plug in opening a connection and reading exactly that path. Sadly, due to the trailing slash, the Key Vault server never wrote the keys. Even querying the same path, it’ll return an empty keys list and thus the “link” between Percona server and Key Vault server is severed.
Now, I don’t know whether it should be the Key Vault server or the Percona plugin to show “hey dude, you passed a trailing slash, don’t do it!” but as of now, it silently fails and then hell happens.